Rapid advances in digital technologies for interaction and visualisation offer the potential to bring the active, exploratory, manipulative and expressive ways in which we work with real materials, using our hands and tools, into the digital realm. Haptic (touch and force-feedback) devices can provide a sensation of touch and physical properties when interacting with virtual models. Physical devices can be linked to virtual models to allow direct manipulation, and gesture interfaces enable more naturalistic interaction. A variety of techniques exist which allow a user to 'see' a virtual model in three dimensions, and methods of displaying virtual 3D models in true physical space are increasingly viable. New modelling systems are being developed which allow designers to use their existing skills while working in the virtual environment. The potential of such technologies to allow a less constrained, more naturalistic interaction with virtual models has increased the drive towards computer support for the whole design process, in particular for conceptual design.

 

This guide provides a brief introduction to some of the technologies and principles involved, followed by selected examples to illustrate ways in which these technologies are being, or could be, integrated into the working processes of artists and designers.

Haptic interaction

 

Haptic devices allow users to experience a sensation of touch and physical properties when they interact with virtual materials. They exert force in response to a user's action, at the point of action. They enable active 'two-way' interaction with virtual objects, where action and perception are brought together. The devices described below can be used not only to interact intuitively with virtual models, but also to interact with such models in 3D space, allowing hand and eye to work together on the model.

 

The PHANTOM^TM from SensAble Technologies is a desktop haptic feedback device which provides single point, 3D force-feedback to the user via a stylus (or thimble) attached to a moveable arm. The position of the stylus point/fingertip is tracked, and resistive force is applied to it when the device comes into 'contact' with the virtual model, providing accurate, ground-referenced force feedback. The physical working space is determined by the extent of the arm, and a number of models are available to suit different user requirements.

 

The illustration shows the PHANTOM Desktop being used to work with a three-dimensional model of a molecule.

 

Further information can be obtained from http://www.sensable.com/haptics/products/phantom.html

 

Immersion 3D Interaction (formerly Virtual Technologies, Inc.) produce a family of products based around their CyberGlove®, a tethered, instrumented glove that can sense the position and movement of the fingers and wrist. With the appropriate software, it can be used to interact with systems using hand gestures, and when combined with a tracking device to determine the hand's position in space, it can be used to manipulate virtual objects. The CyberTouch^TM option provides a sense of tactile feedback through the addition of vibrotactile stimulators to the palm and fingers of the CyberGlove. These produce a buzzing vibration when the wearer comes into 'contact' with the virtual object. While not true tactile feedback, it can give the perception of touching an object.

 

The CyberGrasp ^TM is a full hand force-feedback exoskeletal device, which is worn over the CyberGlove. When the wearer makes contact with a virtual object, resistive force is exerted on the fingers through a series of 'tendons' controlled by actuators, allowing them to 'feel' the object. This force is hand-referenced: it can prevent the user from crushing a virtual object in their hand, but it cannot prevent them pushing through a wall, or allow them to feel weight, for example. This can be achieved through the CyberForce®, a moveable, force-feedback 'arm' on a fixed base which, when used with the GyberGrasp, provides the hard and arm with force-feedback relative to the ground.

 

 

Further information can be obtained from http://www.immersion.com/products/3d/interaction/overview.shtml

 

Another method of achieving direct manipulation of virtual objects is to couple them with physical devices or objects. Although such devices, or 'props', do not give haptic feedback to the user, they enable tangible interaction, often with both hands, taking advantage of our existing skills and experience in manipulating objects. A well-designed prop has a physical form which gives cues to the way it works, making it more intuitive and easier to learn than traditional techniques for manipulating virtual objects.

 

The CubicMouse^TM, initially developed at GMD (the German National Research Centre for Information Technology) and now supplied by Fakespace Systems, is a hand-held cube with three rods running through its centre, one along each of the x, y, and z axes (Kruijff 2000). The cube is mapped to the position and orientation of the virtual environment, and the rods to the co-ordinate system of an object within that environment. Rotating a rod rotates the object around its corresponding axis, while pulling or pushing a rod through the cube will move the object along that plane within the environment.

 

Further information can be obtained from http://www.fakespacesystems.com/worktools.htm

Integrating advanced technologies for interaction and visualisation

 

Integrating advanced technologies for interaction and visualisation combines the benefits of more natural ways of working, with moving the three-dimensional virtual model into the user's physical workspace, allowing hand and eye to work together with the model. Many of the examples in this guide use this approach.

 

A variety of techniques exist which allow a user to 'see' a virtual model in three dimensions. These range from fully-immersive stereoscopic systems, where images for each eye are displayed on goggles worn by the user, 'immersing' them in the virtual environment, to volumetric and holographic systems, where the model is displayed in true 3D space, allowing the user to work in the physical environment. Currently, some of the most common systems are based on semi-immersive stereoscopic displays. In semi-immersive systems, a stereo pair of images is projected onto a display, and viewed through glasses which restrict each eye to receiving a single image, producing the illusion of seeing a three-dimensional image. Unlike immersive systems, objects in the physical environment can still be seen, allowing the hand to interact with the virtual model, for example. Such system range from the size of small rooms, where walls, floor and ceiling can all be used as display surfaces, to small, desktop displays.

 

Commercially-available systems based on integrated visualisation and interaction are now reaching the desktop: Reachin Technologies' Reachin Display combines a stereoscopic display, a haptic device, and a positioning device, allowing eye and hand to be co-located with the three-dimensional virtual model. Different configurations are available to suit a variety of applications.

 

Building applications which combine haptics and advanced visualisation presents a major challenge. To create an environment that appears convincing to the user the haptics, graphics (and often audio) hardware and software must work together seamlessly, virtual objects must behave credibly, and interaction must take place in real time. It is a complex process requiring expert programming to integrate the many components, simulate properties and behaviours, and satisfy the real-time constraints.

 

 

Reachin Technologies address this challenge with Reachin API, their application programming interface. It handles these complex ‘low-level’ activities (including the ability to quickly integrate new devices), freeing developers to focus on the application. Virtual objects, along with their properties, behaviours and relationships, are defined and managed through a modular interface. This enables ‘multi-sensory’ applications to be developed more quickly, more reliably, and at lower cost, making them more accessible to a wide variety of application areas.

Further information can be obtained from http://www.reachin.se/

Software modelling to support interaction and visualisation

 

Haptic devices and 3D displays are of little value without software to model the 'physical' properties of the virtual material and its response to interaction, both haptically and visually: when you press a springy material, for example, you expect to feel it 'give', and see it deform. This is a growing research area where much development is required.

 

 

One example of such work is by Doug James at the University of British Columbia, who is researching techniques in Linear Elastic Modelling which allow people to interact in real time with simulations of elastic or 'springy' materials. The virtual material responds haptically and visually to the user's touch, providing an engaging experience. The example shown illustrates a PHANTOM being used to interact with elastic models.

 

For further information, and to view additional movies of James' work, see http://www.cs.ubc.ca/~djames/deformable/index.html

Advanced digital technologies in art and design

 

The examples below illustrate a range of approaches being taken to harness the potential benefits that can arise from combining the capabilities of computer systems with the traditional skills and working methods of artists and designers.

 

The selected examples are by no means exhaustive, however they illustrate a variety of ways in which the types of hardware and software described above are being used. They include new ways of creating virtual models as an alternative to the precise, geometric techniques currently provided. Expressive, intuitive, playful and quick methods are sought, particularly for the early stages of design. While not all the examples involve true haptic manipulation, all place strong emphasis on using the hands, and direct modelling.

 

MIT's Spatial Imaging Group combined computer-generated holographic video and a haptic device to explore naturalistic, real time interaction with a fully three-dimensional image (Plesniak and Pappu 1998). A PHANTOM with a stylus interface was used with holographic video of a cylinder in a 'lathe' scenario. The user has the sensation of feeling the cylinder spinning beneath their touch, and when they apply sufficient force, the cylinder surface deforms in response.

 

Further information can be obtained from http://www.media.mit.edu/groups/spi/HHlathe.htm

 

 

Research at the Digital Design Studio (DDS), Glasgow School of Art, focuses on a human centred approach to advanced digital 3D modelling, visualisation, interaction and virtual prototyping. The DDS, in conjunction with DERA (the UK Defence Evaluation and Research Agency) and the Ford Motor Company, are developing a 3D workstation which integrates an advanced 3D display with gestural, haptic and audio technologies (Anderson and Slinger 2000). A Fakespace Immersive Workbench is being used as a testbed to develop new 3D interfaces, and for integrating haptic and 3D audio hardware and software. An Evaluation Tool for the Automotive Industry is being developed as a demonstration application.

 

Researchers at Alias|Wavefront and the University of Toronto are exploring new interaction techniques around ShapeTape^TM, a sensored strip that can measure its own bend and twist (Balakrishnan et al 1999). Their prototype system uses ShapeTape to control NURBS* curves in Maya, Alias|Wavefront's 3D modelling and animation software. The user can directly manipulate virtual curves and surfaces with both hands, rather than using geometric techniques. This system explores more intuitive ways of creating and manipulating 'geometry' in a more 'traditional' modelling environment.

 

A movie demonstrating this system can be downloaded from http://www.measurand.com/videos/SHAPETAPEVideo1min.mpeg

* NURBS: Non-Uniform Rational B-Spline. A type of curve where control points are manipulated to define the degree of curvature.

 

Global Haptics are developing the geOrb^TM, a spherical device with sensors distributed over the surface, which is held in both hands. Pressing on any part of the surface deforms the virtual model mapped to the device in the direction of the pressure. Switches on the surface allow the model to be deformed inwards to or outwards from the centre of the orb, and the model to be rotated. Switching modes allows the device to be used to navigate through virtual environments.

 

The example shown is of a prototype which has now been superseded, but it demonstrates the interaction between the geOrb and the virtual model. A CD including more recent videos and a limited software demo can be obtained from Global Haptics.

 

For the most recent developments and further information see http://www.globalhaptics.com/

 

 

SensAble Technologies' FreeForm^TM modelling system provides a 'clay sculpting'-based technique for creating 3D digital models, based around their PHANTOM haptic device. Users work directly with the "digital clay" using the PHANTOM stylus as a modelling tool. The hardness and surface smoothness of the 'clay' can be varied, and different modelling 'tools' selected. Unlike real clay, you can also work from the inside out!

 

To view the many other features offered by FreeForm, and to obtain further information see http://www.sensable.com/freeform/freeform.html

 

Surface Drawing is a system being developed at California Institute of Technology and Bell Labs to allow artists and others to create organic and expressive 3D shapes in an intuitive and immediate manner (Schkolne et al 2001). Using their hand, users sweep out 3D marks or 'strokes' which appear to 'float' in space above the semi-immersive bench-type display. This system extend the principles of drawing to 3D space, using repeated strokes to build up surfaces. A set of physical 'tangible tools' allows the user to manipulate the 3D drawing: a pair of tongs is used to move the drawing in space (two pairs can scale the drawing up or down), an eraser tool allows small portions of the drawing to be removed, and a 'magnet' tool enables small deformations and smoothing of surfaces.

 

The 3D shape shown, "Spun Tubule" (artist: tabalip) illustrates the fluid and expressive ways of working this system enables.

 

For more information and further examples of work produced using this system, see http://www.cs.caltech.edu/~ss/sdraw/

References

 

Anderson, P. & Slinger, C. (2000) Virtual Replaces Physical - Key Areas of Research Within Replacement Reality. In Virtual Design and Manufacture- Institute of Mechanical Engineers Seminar Publication, Bury St.Edmunds: Professional Engineering Publishing, pp35-48.

 

Balakrishnan, R., Fitzmaurice, G., Kurtenbach, G. & Singh, K. (1999) Exploring Interactive Curve and Surface Manipulation Using a Bend and Twist Sensitive Input Strip. In Proceedings of the Symposium on Interactive 3D Graphics, New York: Association for Computing Machinery Press, pp. 111-118.

 

Kruijff, E. (2000) Exploring complex data visualisations using the Cubic Mouse. In 3D User Interface Design: fundamental techniques, theory and practice (Course 36 Notes), SIGGRAPH 2000 Conference, Association for Computing Machinery, New Orleans, 23-28 July.

 

Plesniak, W. & Pappu, R. (1998) Coincident Display Using Haptics and Holographic Video. In Conference Proceedings on Human Factors in Computing Systems (CHI 1998), New York: Association for Computing Machinery Press and Addison-Wesley Publishing Co., pp304-311.

 

Schkolne, S., Pruett, M. & Schröder, P. (2001) Surface Drawing: Creating Organic 3D Shapes with the Hand and Tangible Tools. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2001), New York: Association for Computing Machinery Press, pp261-268.