James Patten, Ph.D.

We stand at the dawn of a huge shift in the field of interaction design, one where the dynamism of the digital world is embodied in the physical, where our everyday objects have even richer interactivity than today’s smartphones, and where the built environment and interactive media become one and the same.

The work of Patten Studio is defining what this transformation means and how it can enhance the human experience.

Tomorrow’s interaction designers will help create the furniture we use, the spaces we live and work in, and the objects we surround ourselves with. As the line between interface and object blurs, we need a new set of design principles focused on enhancing the human experience, not dictating it.

A New Set of Design Principles

As designers, we should develop solutions through a humanist lens in order to create richer, real-world experiences that empower those who use them. Our approach must ensure that the interactivity we bring into the physical world connects us, rather than isolates us, from one another; and enhances our quality of life, rather than detracts from it. As creators, we need to empower people to be the authors of their own experiences.

The following design principles, developed through our practice in both academic and commercial contexts, guide us when architecting experience and designing for interactivity beyond the screen.

Design for Experience

Our main aim is to create interactions with technology that create an experience, triggering emotional responses, be they a sense of wonderment, surprise, or otherwise. This goal requires a fundamentally different approach than one might use when designing an app or a website. In those contexts, consistency is a guiding principle that enables ease of use, but applying task-oriented interaction ideas to interactive objects and spaces strips away a fundamental aspect of the human condition – the wonder in discovery.

Combine Physicality + Interactivity

We need to think beyond the traditional graphical user interface, where most of the user’s interaction is confined to a mouse or touch screen, and instead incorporate interactivity directly into physical objects and spaces.

By using, for example, Thumbles, the tiny graspable robots that combine the versatility of a graphical interface with the tactile advantages of physical controls, we can make abstract information tangible and open up an entirely new array of combinations that enhance materiality, take advantage of form and texture, and engage the body — ultimately creating an experience far more engaging than just looking at pixels on a screen.

Create Legible Interactions

Many graphical interfaces are designed to be used by only one person at a time. As interactivity moves beyond screens, it is increasingly important to design systems for use by multiple people. This might mean face-to-face collaboration or play, or it might mean one person is interacting while others watch. In either case, it is important that whomever is watching the interaction take place is able to grasp and follow what is going on. We call this interaction legibility.

Legible interactions, particularly in the context of a public space, simultaneously engage onlookers while showing them how to step in themselves. Key considerations for interaction legibility include:

  • I/O Coincidence —  The input and output should happen in the same physical location. For example, a touch screen has I/O coincidence because the display and the touch sensor are located right on top of one another. In contrast, a computer mouse does not, because input happens on a tabletop and output happens on screen. I/O coincidence helps a bystander focus their attention in one place and still follow along.
  • Visibility — Interactions are more legible when onlookers can see the same things as those who are actively engaged.
  • Immediate Feedback — When a system responds immediately to people’s actions, it helps onlookers discover and understand the cause-and-effect relationships surrounding those actions.
  • Body Engagement — When the movement of a person’s body drives an interaction, the interaction is more legible if it is based on larger body movements.
  • Tangibility — By mapping actions and data to physical objects that people can touch, move and manipulate, designers can tap into pre-existing knowledge and social constructs about how we use tools and share, making the interaction easy to follow and participate in.

Many objects that predate computers embody many of the above mentioned characteristics. A violin is a good example: It responds to the movement of a body (a bow pulling on a string), emitting sound, immediately, from the same object that the player interacts with. The legibility of a violin player’s interaction with their instrument is a big part of why a musical performance is fun to watch. One gets to see, and begin to understand, the process through which the music is made.

Empower People to Explore

Give people the pleasure of authoring their own experience. Rather than embody a series of step-by-step instructions, interactive environments should sense and respond in a way that encourages the user’s exploration through visual, auditory, or tactile cues.

In the SenseScape installation, for instance, visitors get to engage with an aquatic-like environment by using their body to create movements that the screen will in turn react to. A raised arm, say, causes fish to scatter away, while strumming harp strings (another component of the installation) results in both dramatic on-screen ripples and otherworldly seascape sounds — lending legibility to an interaction whose cause and effect can be clearly seen and heard. This sparks curiosity and initiates a process through which one can discover a variety of different styles of interaction. While instructions can be appropriate when using a task-specific tool, being told what to do without understanding why, makes you a tool instead of a user of one.

Enable Improvisation

When we design to enable improvisation, we are allowing for the possibility that interactions with our creations may extend beyond those originally intended. Take, for example, the record turntable: While it was not conceived as a tool for musical improvisation, experimental musicians in the 1930s and, later, hip hop DJs, used it to create new forms of music. Systems that enable this sort of improvisation share a few key properties:

  • Simplicity of Interaction — A typical turntable supports a few simple interactions, such as play, stop, switching songs and switching records. This makes it easy to mentally unpack any interaction with the object into its component parts.
  • Immediate Feedback — The sound from a turntable immediately changes in response to the motion of the record, enabling a direct understanding of the cause-and-effect relationships surrounding its operation.
  • Ease of Exploration — Because a record is visible and touchable on a turntable while it is playing, one can easily reach in, move the record, and see what happens. There are no mechanisms designed to prevent you from doing something the product’s designer did not envision, thereby allowing for spontaneity.

Interactions we did not originally envision took place in the ‘Create a Chemical Reaction’ exhibit that we developed for the Museum of Science and Industry to allow visitors to explore the elements in the Periodic Table. While we had intended for the exhibit’s electronically tagged pucks to be used in contact with its tabletop surface, we noticed people picking the pucks up off of the surface as a way of sharing the tools. Family members would pass pucks from hand to hand, away from the tabletop surface, letting each member contribute an atom to help build the collectively desired molecule.

Connect People

In the end, we are focused on allowing people to reconnect with each other. Technology has developed a reputation for joining us together only in a virtual sense, while making us feel increasingly distant from each other. But it doesn’t have to work this way. Technology, particularly used in the context of play, can add to our face-to-face interactions, rather than detract from them. It can give us an opportunity to let our guard down, feed off of one another’s actions, and allow for an experience we might not otherwise have.

What Next?

As interactive technology permeates more and more of the spaces around us, interdisciplinary collaboration will be a key factor in the quality of experience that this technology produces. Architects, interaction designers, electrical engineers, product designers, software developers and more will have to work together more closely. At the same time, each of us in our respective disciplines should question the way we’ve been looking at things for a while, just to see if there might be different ways of living and working. If technological constraints of the past are holding us back, we should not be averse to finding and forging a new way forward.

Most importantly, we need to collectively reconsider our expectations of what our relationship to technology should be, aiming for a constructive physical resonance that ensures our daily experience with it remains of the utmost quality. We cannot simply reproduce the screen-based apps of today if we want to revolutionize the world for tomorrow.