Principles of Interaction

As seen above, different kinds of interactive media offer different media inherent conditions and depths of interaction. Usually the principle of interaction is determined by the content that in turn should determine the technology of choice. For users of “WIMPs and GUIs”* this may sound as an unusual statement as users benefit from the intended familiarities between applications, yet visitors of interactive environments may not (yet) expect something as a canonical approach.

In the example of the authors work “Interface to go” (to be found under "Projects") we have seen four different variations of interacting with typography on the floor. The interaction paradigm might be described as “natural physical interaction”, while the interaction principle has been the same in all four examples: direct physical engagement.

Some environments appear to be truly interactive yet could be described as only semi-interactive, meaning interactive within a very limited or defined range of possibilities.
“Shovel yourself free” is such an example for a very low grade of interactivity – nevertheless interactivity is necessary to bring this piece of art into existence. The piece of art is a linear narrative that doesn’t react at all to the visitors’ intervention. The visitors’ interactions are limited and consist of a natural interface, direct physical action at the same spot the effect takes place is required, which is shovelling sand. Yet, only through this intervention the piece attains the content it holds and comes to its surprising, ephemeral and poetic existence. This first level of interactivity might be called “semi-interactive”.
As Rokeby points out non interactive environments have a paradoxical nature: “The non interactive system can be seen as stubborn in refusing to reflect the presence and the actions of the spectator, or, it can be seen as giving the spectator complete freedom of reflection and interpretation by not intervening in the process.” So, a chair might be used to stand on or even used as firewood whereas as computer mediated interaction offers only a very limited range compared to “real” things.

Realising the different concepts of "alternative input devices" and "computer mediated interaction" as used by Nardi , was an important insight for the author.

*WIMP: windows, icons, mouse & pull-down menus, GUI: graphical user interface

Intro: Natural, Gestural & Body interaction

The canonical interaction principles of WIMP interfaces have had an influence upon many gesture-controlled environments as well, determining in their command structure and discreet interaction. Most interactive systems that use gestural input usually require the participant to learn a certain repertoire of gestures to interact with the system. Otherwise interaction will occur on a frustrating trial and error basis. These semantic gestures, be they tracked with a data glove in a virtual environment or via shadow tracking or otherwise are a vocabulary that is not acquired by exploring but by studying and training from the manual or with the help of an expert. Also these gestures might seem natural to the observer, they have not naturally developed and do not necessarily originate from our rich repertoire of gestural or non-verbal communication although they appear to be. Few professional systems seem to pay attention to this matter, imprinting and conditioning the human “wetware” into fitness for communication with the machine.

Without going too much into detail I want to shortly describe the basic categories of gestures and body interaction and their particular relevance to my work.

Gestures, as they appear in everyday life occur in three different categories , again with their own sub-categories (depending on the source).

- Semiotic gestures: which are used to communicate useful information or supporting speech
- Ergotic gestures: used to manipulate the physical world or to create artefacts
- Epistemic gestures : those, to learn from the environment through tactile or haptic exploration

Myron Krugers “Videoplace” makes use of gestures in a natural and intuitive way, that is seamlessly integrated into the work and its participants appear to make use of it intuitively.

In my exploration of a number of existing pieces i have created seven useful categories of principles of interaction:
1. Direct Natural (Physical) Interaction
2. Direct Instrumental (Physical) Interaction
3. Unconscious (Physical) Interaction
4. Indirect (Physical) Interaction or Displaced Response
5. Indirect Instrumental (Physical) Interaction
6. Cognitive, Instrumental or Functional Interaction
7. Poetic, alive or otherwise stimulating

Direct Natural (Physical) Interaction

“Direct” in this context implies being engaged at the very same location the result will “take place”. (This also includes unencumbered interaction. The researcher considers true physical action of the body as a necessary quality - to which keyboard or mouse input not account.) This is natural in real life but not necessary or sometimes even desired in human computer interaction. So, it is possible to move the mouse about the screen, but not desirable as it makes it impossible to see the results at the same time. In “direct natural physical interaction” contiguity is important otherwise computer interaction becomes irritating and ambiguous. Interaction with unknown systems, to which most interactive environments may count, as their interaction principles result from their content, can’t rely on “expected” behaviours, therefore some paradigms from the real world have been adapted. Which enable visitors to apply existing knowledge.

See other Examples in Appendix: A-Volve, Stream of Consciousness, Shovel yourself out, interface to go, Hidden Worlds, Beware Satellite, T’Garden, Legible City, Shapetape, Very Nervous System, McLuhan Machine

Time shifted response

Acting now – with an effect later. This may happen intended or a an effect of latency from the system. Very ambiguous, as used in T’garden, playing with the “rule” of contiguity. Under these circumstances it is difficult for a first time visitor to grasp the rules of the interaction and may lead to a frustrating experience. But maybe “Playing is the process of discovering how the model works” .

Similar effects occur with Perry Hobermans’ “Bar Code Motel”. Actions with objects on the screen via a haptic bar code interface. Some functions effect the own virtual representation other virus like other participants avatars or the appearance of the whole virtual environment. They have a cascading hierarchy of effects resulting in significant delays of actions. The work is utterly confusing for a novice participant or when played in a group and its success may come due to its nature of being a fast, colourful and noisy spectacle.

Direct Instrumental (Physical) Interaction

“Direct” in this context implies again being engaged at the very same location the result will “take place”, but there is an instrumental interface involved, mediating between user and machine. A “code” of gestures and behaviours has to be learned to successfully interact. The actions are often limited within a necessary repertoire and generally task-related and remind more of work, reaching a goal or playing an instrument.

See CD “categories: Flight simulator, Artwick (1975-), “Troikaranch”, Troikaranch (1993), “Shapetape”, Alias Wavefront (1999)

Unconscious (Physical) Interaction

In this category interactions between system and spectator are the result of unconscious bodily reactions. These may include the electric resistance of the skin, blood pressure the heartbeat, or in these examples frequency of breath and brainwaves.
As instruments of control these are far from being intuitive and difficult or stressful to master within a reasonable amount of time. Therefore they don’t belong into this collection of interaction principles and are listed only for reasons of completeness.

Osmose, Ephémère, Charlotte Davies
”Osmose has provoked 70 mainly scientific articles in the last four years, a remarkable amount of attention by any standard. Osmose is a visually powerful real-time simulation of a dozen nature and text zones. Participants navigate through the data space, seen on a HMD, using a chest-hugging rubber vest that contains devices sensitive to the body's breathing - a quasi natural interface. Just as when diving, the lungs fill and you rise. The impression for the viewer is deeply meditative, an effect which makes clear the psychological and manipulative power of Virtual art.”

(See also CD: categories_collection.pdf > Terrain 02, Ulrike Gabriel)

Indirect (Physical) Interaction or Displaced Response

Indirect Physical Action implies that the place of action is not identical with the place of presentation. Although the use is often quite intuitive this seperation makes the pieces abstract and forces participants to split their attention between coordinated action on the input side and the output. This requires moments of orientation and is not the an completely intuitive and elegant solution – although sometimes necessary. It reminds of the mouse movement on a GUI. An observer who is not operating the mouse and just following cursor movement orients himself much faster than the actual operator.

Examples: Bits'n People, Tower of the Winds, Your in Control, Feldheim, Trigger Happy, Beyond Pages.
Movies on CD: Loser Raum (loose space), Bodymovies, Frequent Traveller, Your in Control, Trigger Happy

“Trigger Happy”, audiorom (1997)
A sensor based installation. Operated by placing hands in the light projections - the gestures of the audience members' trigger images and sounds in order to create together a live music performance. The installation uses light sensors and projects images onto a table surface.

Susanne Schuricht, freequent traveller
The motion of a hammock is animating text on a projected screen. Very flow, extraordinary hypnotic and meditative; it seems so for two reasons: getting a pleasant, engaging feedback without any effort but swinging in the hammock. Being immersed in the stimulating synchronised loop of ones own physical motion and perceived visual dynamics.
“freequent traveller” is “semi interactive” as well. The animated text is prewritten, not dynamically generated nor pulled from the web neither pushed from a content-provider. Its narrative is an absolute condition, linearity as well as the sequential order of the text isn’t changed. Yet the form of its animated visuals dynamics, a progressive spline-curve rendered by complex algorithms, depends on the dynamics of the hammocks movement. Nevertheless the piece is truly interactive to a certain extent. Evident in both cases of semi-interaction is, that the piece itself doesn’t change its general appearance, isn’t generative.

The Interaction paradigm could be described as “natural physical interaction”.
The interaction principles and qualities of use are its simple and straightforward way of interaction resulting in low action depth. The basic interaction principle is indirect physical interaction with a displaced, yet live response. It is designed to be used by one or two users who have a personal experience. Its more immersive than observant qualities result form the synchronized motion of the animated letters. Although it is content based and intended to be cognitive stimulative it appears to be rather experiential through its form.
“freequent traveller” is passive, relaxed and contemplative. Through its synchronized visual activity it appears experiential, explorative, sensuous and playful. It is contiguous, perceived motion is synchronized with visuals. Its use is intuitive, and overall feel “pull”. Interactions are continuous and not in discreet steps; it is unencumbered, poetic, involving, engaging (hot), has cognitive (visual/texty) as well as sensuous (feely) qualities and incorporates degrees of self recognition / perception because of its synchronization. The content consists of statements from artists from all over the world about mobility, home and identity and raise global awareness to a certain extent. Gestalt effects are achieved by the high feedback through synchronization of perceived body movement and dynamic visuals. freequent traveller applies to multiple senses as orientation and visual. Its nature is ephemeral.

Indirect Instrumental (Physical) Interaction

The very well documented and explored world of Smartboards™ and “Shared Workspaces”; Bill Buxton, Data Gloves & Gesture Recognition and shared workspaces. Although often appearing to by “natural” or intuitive an abstract “language” or “code” of interaction has to be learned to successfully interact with the system. Courses and “introduction sessions” might by offered by the industries.

Cognitive, Instrumental or Functional Interaction

The media has a predominant tool character and covers most of the applications we use in everyday computer based interaction. It might be screen based, but not necessarily It might be easy to use for first time as well was for experienced participants. The actions are cognitive and conscious and overall less experiential, the motions instrumental and functional as operating a tool. In this context mouse and keyboard input is not considered as “physical motion”. In case the interface is well conceived they might appear to be intuitive e.g. the game “Myst”. Most popular applications would be placed here – if they where interactive media at public places. Examples: Automatic Teller Machine (ATM) by Luther Simjian (1971). “Radiomap” as environment by the Author (1997), “Blinkenlights / Arcade” by the Berlin based Chaos Computer Club (CCC) (2001), “Bar Code Hotel” by Perry Hoberman (1994) where the interaction consists of scanning bar codes.

Poetic, alive or otherwise stimulating

As the categories above aren't a holistic approach and miss all of the magic that some environments do incorporate.There seems to be a certain quality within works that is hard to point out. It is a result of a variety of factors particularly its liveliness, charm, poetic quality or its stimulation of insight by its content. All these properties create a meta-pattern of a felicitous work. Perhaps may refer to it as what Christopher Alexander describes as “The Quality without a Name” .
Examples: “Global Village Square”, McLuhan Program for Culture and Technology (1998), “Feldheim 3”, AEC artist group (1995), “Global Windchimes Project” (2000), “Shishi Odoshi”, vernacular, “While you where”, Sensorium (2000). Examples on CD: “Tornado”, Ned Kahn (1990), “Night and Day”, Sensorium (1998) (Masaki Fujihata), “Windlight”, Nishimura (2001)