Brave NUI World: Designing Natural User Interfaces for Touch and Gesture [Paperback]

The authors both have significant experience in the design of novel user interfaces and collaborated at Microsoft in the design of the revolutionary tabletop interface known as Microsoft Surface. This book takes a wide view of the literature of user interface development (but significantly omitting from the bibliography the other major book on "Designing Gestural Interfaces" by Dan Saffer). The approach of this volume is scientific relying on GOMS-like state machine paradigms for describing touch and gestural interaction and giving a formal approach that the authors employed in designing Microsoft Surface for design iterations entitled "RITE". While the authors and others in the field have a wide body of experience in the design of novel user interfaces they emphasize the formal scientific techniques that they did employ successfully in the Surface system. Formal methods using a "Must", "Should", and "Could" prescriptive approach are described throughout the volume.

Because this book is well researched, illustrated and based largely on successfully employed experience it well deserves an excellent rating on Amazon and addition to the practitioner's library. It provides much of the scientific basis that "Should" be employed in the design of "Gestural" and "Tactile" Interfaces. I would have welcomed an approach involving more sketching and rapid prototyping like "Sketching User Experiences" by Bill Buxton also of Microsoft and available in the same publishers series. For a book on novel gestural interfaces that takes this approach I would also recommend Dan Saffer's "Designing Gestural Interfaces", I also am somewhat surprised that the authors coming from the Microsoft Research staff didn't describe the design approach used in the XBOX 360 Kinect System.

On the whole it offers a valuable and scientific approach to the design of Natural User Interfaces.

--Ira Laefsky, MSE/MBA HCI Researcher and Consultant
formerly on the Senior Consulting Staff of Arthur D. Little, Inc. and Digital Equipment Corporation

Advances in human-computer interaction technologies have progressively shrunk the gulf separating user and computing device, making it easier for a user to accomplish a task using means that feel more and more natural to the user. Early in the history of computing, for example, communicating with a computer was cumbersome and consisted of plugging wires in patch panels of circuits. The arrival of programming languages and the ability to communicate with computers through the use of symbolic commands empowered users to think at a more natural level, and further progress was achieved with the advent of graphical user interfaces where users can point and click on labeled screen objects such as menu items and buttons, instead of taxing their memory to recall the exact syntaxes for various commands.

But even today's WIMP-based GUIs (WIMP stands for Windows, Icons, Menus, and Pointers) may be sub-optimal for performing certain tasks. If you've watched the movie Minority Report or CNN's Election Reporting, where screenfuls of information can slide in and out of view with a flick of a finger, or images enlarged or reduced with just the simultaneous movements of two fingers, or multiple people changing the contents in different parts of a screen simultaneously using gestures only, you'll know what kind of tasks I'm talking about. With more and more devices having capabilities to recognize touch and gestures, the next evolutionary step in software application development would surely involve the seamless integration of these new input modalities to the design of natural user interfaces (NUIs). This book will help you understand what you need to know in order to get started with such an endeavor.

The book begins with a discussion of what the authors mean by a natural user interface, qualities to look for in such interfaces, and computing niches where touch- and gesture-responsive NUIs will have an important role to play.

The authors emphasize * repeatedly * in the book that when attempting to integrate new modalities into user interfaces, one should avoid the temptation to simply copy old paradigms. For example, some of the earlier graphical menuing systems that attempted to mimic the way command-centric applications worked by requiring users to first select a desired operation before they could select the object to be acted upon failed miserably because that interaction style was not the optimal way for interacting with a GUI. Equating touches with mouse clicks would similarly not work because there are important differences between those two kinds of inputs.

The authors provide ample discussions of similarities and differences among touch, gesture, mouse click, and mouse movement, and give plenty of guidelines on how to handle touch and gesture inputs, provide effective feedback to users so they'll know whether their inputs are getting received and interpreted correctly by the system, and if not, potentially why, and how to compose interaction patterns that are easy to learn. The book then concludes with some suggestions on how to test the learnability of those proposed interaction patterns.

Recommended readings are provided at the end of each chapter.

Overall, I thought the book is well written but a bit dry. The information provided is practical and valuable.

If you're an experienced designer of user interfaces, you're either really excited about the new possibilities opened up by gesture and motion detection controllers or you're convinced that you already know everything you need to know to create touch and gesture based interfaces. This book is useful for both types of designers.

Lots of games are already based on gestures that mimic real life closely, mainly for sports and combat. But what about regular control - finding, sorting, selecting, enlarging,shrinking, moving through groups of item and so forth - the bread and butter of most UIs? Do they require new primitives? How precisely can users gesture? How many gestures can users differentiate? Because touch and gesture are directly physical, do users need more help with them? How does that physicality transform the user experience? Do users already have useful intuitions to exploit?

This book answers those questions by presenting principles, discussing issues, offering examples, and linking to lots more reading. If you've done lots of UI design for anything beyond standard workpiece software, you probably know many of the answers to these questions - and you'll find lots of what you know reflected in sections called "Lessons from the Past". The book goes beyond those lessons and serves as an excellent guide and review for UI designers faced with this new world.

There are 30 scenarios through which the authors, combining research and practical experience, present the issues and offer guidance. Practice in this area has not yet ossified into received wisdom, and there isn't much out there covering the same ground.

So save yourself some time and thought, and pick up this book to help you understand the issues. Expect clear but not sparkling prose.