Printer Friendly

A Scoping Review on Tangible and Spatial Awareness Interaction Technique in Mobile Augmented Reality-Authoring Tool in Kitchen.

1. Introduction

Augmented Reality (AR) receives a lot of attention globally. There are a lot of efforts in research and ongoing development in this area as it is the newest technology. Augmented Reality can provide attractive and interactive experience in kitchen design. Interior design has gained tons of attention among the users in order to make a convenient and comfortable kitchen environment. Based on the previous finding, AR offers many advantages in digital architectural design and construction fields [1]. It is a new design approach for interior design, where AR environment is able to display virtual furniture and it can be modified in real-time where the user can directly see the virtual outcome of the design.

Some of the kitchen design companies are using physical board to show rough design by arranging the physical blocks into the empty kitchen layout as shown in Figure 1. However, with the limitation of human imagination, it is challenging to visualize how the cabinet will look like or align in the real kitchen space [2]. Mobile Augmented Reality (MAR) application allows the virtual furniture to overlay into real scene. Thus, the user can see the outcome of the kitchen design whether the design outcome could fit with the actual kitchen environment. The interaction with the augmented reality environment plays an important role in this authoring type application. It could affect the performance and user preference such as to discourage the user if the interaction is too complicated. Thus, this paper is focusing on the interaction technique design in kitchen design. There are two types of potential interaction techniques, which are suitable for the kitchen design: tangible and spatial awareness interaction technique. This work reviews the existing studies on the tangible and spatial awareness technique in order to identify and design a new interaction technique for the kitchen design in MAR environment.

This paper consists of six parts: the first part is the introduction of the study. The second part discusses the documents selection process. The third part is the reviews of the studies related to tangible and spatial awareness interaction technique to explore their possibilities in kitchen design MAR authoring tool. Next, the kitchen design process is defined based on the interview section with the Malaysian kitchen design company. The fifth part of the paper discusses the proposed tangible and spatial awareness interaction techniques in mobile augmented reality setting. The conclusion and the future works are in the final part of the paper.

2. Documents Selection

This study focused on two types of interaction techniques, which are tangible interaction technique and spatial awareness interaction technique. Tangible interaction technique is an approach of designing the AR interfaces that focus on interactions using related physical object to interact with the virtual object [4, 5]. Tangible interaction technique registers the virtual object to a physical object, and a user interacts with the virtual objects by manipulating the corresponding physical objects [1]. In the designing session, it allows the user to interact with the virtual furniture in order to move, rotate, and copy them as in the real environment using physical object. With the latest development, RGB and depth camera embedded in the smart phone have enabled the spatial awareness interaction technique in a mobile setting [6]. It allows the user to measure the depth of the object by using depth sensor. Spatial awareness interaction technique involves motion tracking, depth sensing, and area learning where the environment properties could be recognized by the AR system. Both types of interaction techniques require further exploration on their effectiveness in helping design process between the kitchen designer and customer.

2.1. Documents (Tangible Interaction Technique). Based on Figure 2, the initial result of query search is 1768 from SCOPUS database. Then, the result was filtered by the document type, field, language, and year. The filtration ended up with 1528 documents. After title and abstract filtering, 1528 documents have been reduced to 145. Then, full text reading process excluded 133 documents, resulting in 12 documents as the final document to be analysed.

2.2. Documents (Spatial Interaction Technique). Based on Figure 3, the initial query shows 3565 raw results in SCOPUS. The field was then specified to computer science leaving 771 results. 717 articles and conference papers were then chosen to be included in this filtering. The documents from year 2000 until 2017 were chosen, reducing the documents into 650 documents. After scanning the title and abstract, only 116 documents were applicable. At the end, 8 documents were chosen after finishing the full text reading.

3. Result and Statistical Information of Documents

In summary, 20 articles have been selected from the SCOPUS database and reviewed in which 60% are in tangible interaction technique and 40% are related to spatial awareness interaction technique. All the papers were published in proceedings and journals from 2002 until 2017. Study on spatial awareness interaction technique is a new topic and the reviewed papers are from 2013 to 2017. 50% of the reviewed papers for tangible interaction technique are proceeding papers and another 50% are from journal articles. Meanwhile, all the reviewed papers for spatial awareness interaction technique are proceeding papers. The selection process will be discussed in Section 2. There are 6 scenarios that used tangible interaction technique in their study as shown in Table 1.

Based on the review there are four types of AR, which are projection based, see-through based, PC based, and mobile based which used tangible and spatial awareness interaction techniques. The statistics of the types of AR used in the reviewed papers are shown in Table 2.

3.1. Studies Related to Tangible Interaction Technique. Table 3 shows a review on existing tangible interaction technique. There are 12 researches reviewed; most of the researches (50%) used card with marker as interaction medium to interact with virtual object. 18% used cube to manipulate the virtual object. Another 32% used specific physical object such as LP records, paddle, pen, and cup as medium of interaction. For the physical setup statistic, 58% of the researchers used HMD as a device to display the virtual object. 17% of the studies used projector and 17% used PC to display the virtual environment. Only one study used mobile phone as a display device.

A collaborative AR system with tangible interaction that is designed to support a scenario where a group of experts meet to discuss the design of a product has been presented in [7]. The tangible interaction includes a physical round plate for the object rotation manipulation, seeing the internal view of an object using clip plane method controlled by a normal marker to cut through the virtual object, and other manipulation functions. The result showed the interaction technique runs stably and is easy to use. However, based on the user feedback, several aspects of the system could be improved such as reducing the time consumed for tracking and improve quality of image and HMD. Another four applications have been developed using the design principles of tangible interaction technique [8]. All the tangible interaction approaches are using different image patterns on physical card for representing different virtual object such as Shared Space for gaming purpose, ARgroove focus on interactive music, rapid prototyping for aircraft instrument panels in Tiles application, and VOMAR for furniture relocation application. Their finding concludes that the tangible interaction allows seamless spatial interaction with virtual objects in the physical workspace [8].

Physical cube is one of the popular mediums of interaction in tangible interaction. For example, a game that has been developed by [9] uses two cubes as a medium of interaction with the virtual object and supports two players to play in turn. This game uses Singapore's map as the game board. Players were able to view the 3D map of Singapore by using HMD. Cube number one is used as dice and cube number two is used as a controller. When the player rolls cube one, the metaphor will move based on the number. Cube two consists of three functions which are to view photos and videos, as a shopping cart and as a navigation tool. Player can rotate cube two to change the displayed photo and the video. When the player stacks two cubes together, it acts as shopping cart function. It allows the player to add virtual object into an empty virtual cart. Cube two can be turned to left or right and tilted upwards and downwards as a navigation tool to tour around in the virtual world. Another research that used cube as medium interaction is Magic Story Cube [10]. Magic Story Cube is a foldable cube for storytelling with 3D graphics, 3D sound, music, and speech. Users can explore the story content by unfolding the cube and view the content by using HMD. A digilog book that supports novel visual, audio, and haptic authoring functions has been introduced in [11]. This system uses pen-type manipulation prop as medium interaction by placing the pen-type manipulation prop on the marker in digilog book to copy the virtual object. Meanwhile, pressing and holding the button on the pen-type manipulation prop will edit the shape of the trajectory object.

A system that uses card marker as a medium interaction has been introduced in [12]. This system allows the user to create virtual AR content by selection menu technique. Then, the user can manipulate the virtual object by using the card marker. For example, when the user puts two markers side by side, a new virtual content will be shown. Another research on tangible augmented reality is called AR-SEE [13]. Passive Solar Energy Education has presented AR-SEE as an augmented reality application for mobile phones. It is a combination of mobile phone-based AR with the physical model of a house compartment such as roof styles, windows, and other building materials. Those design changes will then affect the internal temperature inside the house and the temperature displayed through particle visualization. Then, the users can realize how passive solar energy affects the temperature inside the house and the energy usage efficiency. The users can interact with the AR-SEE via phone by looking through it and pointing at the big green marker. Then, three buttons are allocated on the phone with one of them for the material selection for the roof of the house, another button for the material for the base of the house, and the last button for X-ray vision function which makes the house partially transparent to allow the user to see the visualization inside the house. Other than that, it uses virtual human with different actions to represent temperature condition. For example, the virtual human is sitting if the temperature is less than 81 degrees Fahrenheit and falling forward if temperature rises above 95 degrees Fahrenheit. Their finding shows that the virtual human visualization enables significantly lower temperature estimation error than the particle visualization.

The previous researches which are almost similar to this study are VOMAR [8] and TAR for interior design [1]. A study on tangible AR for interior design has been conducted in [1]. This study introduced an augmented reality system for designing or educating in interior design projects by overlaying virtual furniture in a physical environment on a regular PC system. The system uses marker card as a medium to interact with the virtual furniture. The colour and material of the virtual furniture can be adjusted by adding or hiding the card marker. Another similar study was a cup-shaped handheld compact that has been labelled with marker as the medium interaction between user and virtual furniture [14]. This system includes six main functions, which are picking, allocating, moving, rotating, deleting, and changing mode. The user can pick the virtual object by covering the virtual object with the cup. To allocate the virtual object, user can place the cup that contains the virtual object on the table. User can move the virtual object by sliding the cup. Furthermore, rotating the cup will rotate the virtual object and shaking the cup will delete the object. In interior design scenario, basic manipulation ability of the interaction technique is picking, allocating, moving, rotating, and deleting the virtual furniture. All the virtual furniture manipulations in the previous studies were limited where user cannot change the feature of the virtual furniture.

3.2. Studies Related to Spatial Awareness Interaction Technique. Depth sensing and motion tracking are useful features in spatial awareness interaction. It allows user to move in 3D space (x, y, and z position) and to track its orientation in yaw, pitch, and roll [23]. This 6 degrees of freedom of sensing capabilities allows the phone to respond to the physical movements of the user. Table 4 shows a review on existing spatial awareness interaction technique. There are 8 researches that have been reviewed.

Projection-based augmented reality is a normal setup for the spatial augmented reality. General technique of dynamically augmenting physical models was introduced back then. iSarProjection is using a handheld projector attached with an RGBD camera to project the texture on a physical model [15]. The physical model is mapped with corresponding textures, and it develops a real-time dynamic calibration, registration, tracking, and rendering. However, this study only focuses on the visualization in the spatial augmented reality environment.

A hybrid space combining spatial augmented reality and virtual reality has been proposed and has obtained a positive feedback from interviewees [16]. The hybrid spatial augmented reality (SAR)/Virtual Reality (VR) hybrid space is a projection-based AR that enables multiusers manipulation with an augmented object, and immersive experiences. Virtual objects can be manipulated using the controller's grab button. To start drawing, the user can directly point at the target location and press the controller's trigger. Garden by Sing [17] is a study which used the game scenario to explore the mixed reality (MR) experience that combines both VR with AR. It allows users to transform their physical environment into a virtual garden they can play in. The exploration of mixed reality with motion tracking and depth sensing is going very well here. An immersive experience allows players to move through their surroundings untethered while exploring and interacting with the virtual world by using Durovis Dive headset where users can see the world in VR via tablet mounted on it. It consists of a simple hand and leg interaction where user draws a virtual object via voxel that is created by their hand and leg. Another projection-based AR is the Tabletop SAR [18] where it uses physical object such as cube and cone to interact with the 3D data. It is a combination of spatial augmented reality with the tangible interaction technique. The physical object in their study comes with a push button for selection and scroll wheel for 2D zooming. Users manipulate the physical object and push the button to lock the zooming position in the 3D data visualization area and then increase or decrease the zoom level using the scroll wheel.

Besides the projection-based AR, researchers have also studied wearable augmented reality and spatial awareness mobile device such as Google Project Tango. Nassani et al. [19] invented a wearable system that allows people to place and interact with 3D virtual tags placed around them and it is named Tag It!. A head-worn wearable computer (Google Glass) and a chest-worn depth sensor (Tango device) were used as wearable technology. The users can create virtual tags with text content generated by voice input and place it on where they are looking at by wearing the system. Tango is utilized to extend Google Glass' sensing ability by sharing the location and pose of the user as well as the tagged target position in the real world. The Google Glass then overlays virtual tags based on the spatial information received from the Tango, and the background of the glass display is set to black to act as an optical see-through display. Via the Google Glass touchpad, the user can initiate the wireless connection by using a three-finger touch gesture on it. Besides the Tag It! invention that benefits the user, the social spatial mash-up (SSM) [20] for information sharing in public space also improves the interaction among human with overlaying the virtual information surrounding them. The proposed SSM is based on RGB-D SLAM to create a 3D feature map and allows users to locate information and contents in 3D space. Information placing is through tapping button in the Tango. The virtual information is presented in an image (a small logo) form, which is located on the face of the 3D object after social spatial mash up. The virtual information is well registered even with the movement of the Tango device.

For the interior design area such as in the study of Tang et al. [21], Kinect was used to detect the spatial information of the physical environment where the furniture reseller can create virtual 3D furniture. The virtual furniture is rendered in the real scene of the home space via AR technology. The study performed an evaluation on the system usability, flexibility, and accuracy where overall result is positive, and the system is helpful, usable, efficient, and effective for home furnishing and furniture purchase. It is related to this study; however, it does not have the mobile ability where user can move and see the furniture in different perspective. Blanks (white cabinet) is a metaphor of the cabinets that are placed in an environment of the kitchen with actual shape and size [22]. It supports the early design process of kitchen. It will ease the discussion session between the designer and the customer. The designer can move the cabinet blanks to the desired location. Then, the designer and the customer can decide on the cupboards, the drawers, and other components. The system automatically arranges the components to fit the available space. The component can be manipulated by using the manipulation tools, which are two identical cards to resize the component and to use SAR Swatches for choosing the colour and loading and saving the kitchen designs. It also allows user to compare different designs simultaneously by quickly flicking through several different designs in a more natural way than using a menu. Final design phase allows the designer and the customer to set the material of the cabinet by tapping the cabinet surface with the Stylus.

In conclusion, the previous studies have shown the possibility of the spatial awareness interaction between user and the systems such as the SAR/VR Hybrid Space, Tabletop SAR, Tag It!, Social Spatial Mashup, and Garden.

4. Current Kitchen Design Process

We have conducted interview session with the kitchen design company in Malaysia. Figure 4 presents the current kitchen design procedures, which start with the face-to-face meeting between the customer and the designer. At the beginning of the process, the customer will need the measurement of the kitchen space and might come with the initial plan of their desired kitchen design to discuss with the designer. Some of them come with an example of their wanted design from the magazine or catalogue. However, the designer will have difficulty to follow the design from the magazine due to lack of measurement information in the magazine. Next, the designer will ask if there is any appliance such as fridge or microwave to be placed in the kitchen. Then, the designer will start to sketch the design after the customer has made the selection of cabinet material, colour, and accessories. Common furniture pieces that the customer requires are sink, stove, kitchen ventilation, and cabinet. The type of furniture, measurement, material, and accessories will have an impact on the price of the kitchen cabinet. After the customer makes the payment, the designer will produce a full 3D design in the design software. The designer and the customer will continue the discussion to finalize the design and then the designer will go to the customer's home to confirm the actual measurement. After all the procedures have been done, the designer will proceed with the production of kitchen cabinet.

The weaknesses of the current procedure include difficulty of understanding the kitchen design imagination of both customer and designer. The designer cannot understand the customer needs through verbal communication. This problem causes time-consuming delay in the designing process. During the interview section, we have explained the MAR approach to solve the problem to the kitchen designer. The designers feel positive to use the MAR in smartphone platform instead of tablet, as smartphone is a common device that they use daily. The designers suggested that this application should allow the user to design their kitchen at their home and obtain the actual measurement beforehand so that the designer does not have to go to the customer's home and confirm the measurement. Then, the customer can email or print the output design to show to the designer for further discussion. Based on the designer's opinion, this application is useful and will ease the designers' work in the designing process and overcome different imagination between customer and designer. Besides, it will also help to reduce the designing time.

5. Proposed Interaction Techniques

Based on the findings of the scoping review, this study has proposed two types of interaction techniques: tangible interaction technique and spatial awareness interaction technique for kitchen design. Tangible interaction technique is used mainly in the situation where the customer and designer are together in a discussion session. Normally, the discussion session is conducted in the kitchen design company. Tangible interaction technique requires physical board as interaction space and cube as medium interaction. The tangible interaction technique will use cube as interaction medium to represent kitchen furniture and kitchen features as shown in Table 5. For example, the cubes will represent cabinets, sinks, stoves, and kitchen ventilation labelled with related markers to be arranged on the physical board as shown in Figure 5(a). For sink, stove, and kitchen ventilation, the colour features of this furniture are included in the physical cube where the user can choose the colour of the virtual object by rotating the cube that is labelled with its colour. For the cabinet, the cube is labelled with the surface of cabinet. For example, the user can choose the surface of the cabinet by manipulating the cabinet cubes that are labelled with different type of cabinet's surface. Figure 5(b) illustrates the virtual cabinet shown after the user places the cabinet cubes on the physical board. For the cabinet accessories and the cabinet colour features, the proposed technique uses the cards as interaction medium. The user can choose the colour and accessories of the cabinet by placing the colour card and the accessories card on the designated physical board.

Spatial awareness interaction techniques do not require physical board because users can design their kitchens directly in their home kitchen using mobile phones. Spatial awareness techniques enable the users to measure available space to add new furniture. Besides, the users can view 1: 1 scale of the virtual furniture to see the suitability of the virtual furniture with the available space. The user interface of this technique contains four menu buttons representing cabinets, stoves, kitchen ventilation, and sinks as shown in Figure 6(a). When user clicks on the cabinet menu button, it will display a list of cabinets for the user to choose. A popup box will be displayed for the user to select the surface type, colour, and accessories of the cabinet as shown in Figure 6(b). If the user clicks on the sink menu button, list of the sink will be displayed. After the user selected the sink they want, a popup will be displayed for the user to select the colour of the sink. A similar process will be used for stove and kitchen ventilation menu button. Spatial awareness interaction techniques can detect walls and floors. Therefore, after the selection has been made the virtual furniture will snap on and not penetrate the physical wall.

There are few limitations in spatial awareness interaction technique; namely, this technique cannot be used to fully furnish the kitchen. This is because the spatial awareness interaction technique can detect the existing furniture and act as the boundary or wall of the kitchen space. Thus, the virtual furniture that customer choose will appear in front of the physical furniture. Other than that, customers lack the knowledge in kitchen design; thus, some of the existing water piping or the electric socket area might make some kitchen furniture unable to be located at the location they desired. However, the customer does not know those limitations, and this makes the kitchen design not achievable. This problem can be tackled by having further discussion session about the design between the customer and designer. The discussion session will be easier because they already have the design based on the customer's idea. This will minimize the gap of the different imagination issue.

6. Conclusion

In kitchen design, the traditional approach has a few limitations such as mistake in measurement and different imagination between customer and designer. The user has different final kitchen output, which is usually different from the sketch design from the designer. Hence, this traditional method is inconvenient and not efficient. As a solution, the virtual furniture can be overlaid into real scene by using MAR application. The user can see the overlaid virtual kitchen outcome that could fit with the actual kitchen environment.

This paper has presented a scoping review on the tangible and spatial awareness interaction technique and proposed the tangible and spatial awareness interaction technique for the kitchen design authoring tool. The proposed interaction techniques will be developed and refined by conducting a heuristic evaluation with the AR expert. After the refining has been made, both techniques will be improved and evaluated with the kitchen designer and the customer through usability testing.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.


This study is supported the Universiti Kebangsaan Malaysia under research grant scheme of GGPM-2015-023 and GUP2017-050.


[1] V. T. Phan, P Seung, and Y. Choo, "A Study on Tangible AR for interior design," International Journal of CAD/CAM, vol. 10, no. 1, pp. 21-28, 2010.

[2] A. Adiyodi, M. Agrahara, P. Gamnani, and K. Pampattiwar, "Interior design using augmented reality," International Journal of Innovative Research in Science, Engineering and Technology, pp. 6104-6110, 2016.

[3] kubiq, "What Is Ezikit,",, 2016.

[4] R. Jafri, A. M. Aljuhani, and S. A. Ali, "A Tangible Interface-based Application for Teaching Tactual Shape Perception and Spatial Awareness Sub-Concepts to Visually Impaired Children," Procedia Manufacturing, vol. 3, pp. 5562-5569, 2015.

[5] N. Tuli, Interaction Techniques: helping user manipulate AR virtual content Interaction Techniques: helping user manipulate AR virtual content, 2016.

[6] O. Bimber and R. Raskar, Spatial Augmented Reality Merging Real and Virtual Worlds, vol. 6, 2005.

[7] H. T. Regenbrecht, M. T. Wagner, and G. Baratoff, "Magic-meeting: A collaborative tangible augmented reality system," Virtual Reality, vol. 6, no. 3, pp. 151-166, 2002.

[8] M. Billinghurst and I. Poupyrev, "Tangible augmented reality," in Proceedings of the ACM SIGGRAPH ASIA 2008 Courses, pp. 1-10, 2008.

[9] Z. Zhou, A. D. Cheok, T. Chan, and Y. Li, "Jumanji Singapore: an interactive 3D board game turning hollywood fantasy into reality categories and subject descriptors," Advances in Computer Entertainment Technology, pp. 362-363, 2004.

[10] Z. Zhou, A. D. Cheok, and J. Pan, "3D story cube: An interactive tangible user interface for storytelling with 3D graphics and audio," Personal and Ubiquitous Computing, vol. 8, no. 5, pp. 374-376, 2004.

[11] T. Ha, W. Woo, Y. Lee et al., "ARtalet: Tangible user interface based immersive augmented reality authoring tool for digilog book," in Proceedings of the 8th International Symposium on Ubiquitous Virtual Reality, ISUVR2010, pp. 40-43, kor, July 2010.

[12] Y. Yang, J. Shim, S. Chae, and T. Han, "Mobile augmented reality authoring tool," in Proceedings of the 2016 IEEE Tenth International Conference on Semantic Computing (ICSC), pp. 358-361, 2016.

[13] V. Ferrer, A. Perdomo, H. R. Ali, C. Fies, and J. Quarles, "Virtual humans for temperature visualization in a tangible augmented reality educational game," in Proceedings of the 2017 IEEE Virtual Reality Workshop on K-12 Embodied Learning through Virtual and Augmented Reality, KELVAR 2017, usa.

[14] M. Billinghurst, H. Kato, and S. Myojin, "Advanced interaction techniques for augmented reality applications," in Proceedings of the 3rd Int. Conf. Virtual Mix. Reality, VMR 2009. Held as Part HCI Int, vol. 5622, pp. 13-22.

[15] M. Tan, W. Xu, and D. Weng, "ISarProjection: A kinect-fusion based handheld dynamic spatial augmented reality system," in Proceedings of the 13th International Conference on Computer-Aided Design and Computer Graphics, CAD/Graphics 2013, pp. 425-426, chn, November 2013.

[16] J. S. Roo and M. Hachet, "Towards a hybrid space combining Spatial Augmented Reality and virtual reality," in Proceedings of the 2017 IEEE Symp. 3D User Interfaces, 3DUI2017, pp. 195-198, 2017.

[17] K. H. Sing, "Garden: a mixed reality experience combining virtual reality and 3D reconstruction," in Proceeding of the 2016 CHI Conf. Ext. Abstr. Hum. Factors Comput. Syst.--CHI EA '16, pp. 180-183, 2016.

[18] B. H. Thomas, M. Marner, R. T. Smith et al., "Spatial augmented reality--A tool for 3D data visualization," in Proceedings of the IEEE VIS International Workshop on 3DVis, 3DVis 2014, pp. 4550, France.

[19] A. Nassani, H. Bai, G. Lee, and M. Billinghurst, "Tag it!: AR annotation using wearable sensors," in Proceedings of the SIGGRAPH Asia Mobile Graphics and Interactive Applications, pp. 1-4, 2015.

[20] C. Shin, Y. Kim, J. Hong, S. Hong, and H. Kang, "Social Spatial Mashup for Place and Object," in Proceedings of the the 2016 Symposium, pp. 185-185, Tokyo, Japan, October 2016.

[21] J. K. Tang, W. Lau, K. Chan, and K. To, "AR interior designer: Automatic furniture arrangement using spatial and functional relationships," in Proceedings of the 2014 International Conference on Virtual Systems & Multimedia (VSMM), pp. 345-352, Hong Kong, China, December 2014.

[22] M. R. Marner and B. H. Thomas, "Poster: Spatial Augmented Reality user interface techniques for room size modeling tasks," in Proceedings of the 8th IEEE International Symposium on 3D User Interfaces, 3DUI2013, pp. 155-156, usa, March 2013.

[23] J. Lee, " 4-1:," SID Symposium Digest of Technical Papers, vol. 48, no. 1, pp. 17-18, 2017.

Siti Soleha Muhammad Nizam (iD), Meng Chun Lam (iD), Haslina Arshad (iD), and Nur Asylah Suwadi (iD)

Mixed Reality and Pervasive Computing Lab, Centre of Artificial Intelligence Technology, Faculty of Information Science and Technology, Universiti Kebangsaan Malaysia, 43600 Bangi, Selangor, Malaysia

Correspondence should be addressed to Meng Chun Lam;

Received 30 November 2017; Revised 16 March 2018; Accepted 12 April 2018; Published 21 May 2018

Academic Editor: Tuomas Makila

Caption: Figure 1: Physical kitchen miniature modelling kit [3].

Caption: Figure 2: Document selection for tangible interaction technique.

Caption: Figure 3: Document selection for spatial interaction technique.

Caption: Figure 4: Current design procedures.

Caption: Figure 5:(a) User interface of tangible interaction technique. (b) Virtual cabinet is displayed based on the selection.

Caption: Figure 6: (a) User interface of spatial awareness interaction technique. (b) Display of popup box.
Table 1: Scenarios.

Interaction         Scenario        Percentage

                 Product design         8%
                 Rapid Prototype        8%
Tangible              Game              18%
                      Music             8%
                 Interior design        25%
                    Education           33%

                  Architecture          37%
Spatial        Information Sharing      37%
awareness             Game              13%
                     Medical            13%

Table 2: Type of AR.

                  Type of interaction

Type of AR    Tangible   Spatial awareness

Projection      17%             50%
See-through     58%            12.5%
PC              17%            12.5%
Mobile           8%             25%

Table 3: Scoping review on existing tangible interaction technique.

Name of          technique:
invention          medium     Interaction technique

Magic            Tangible:    MagicMeeting is a
Meeting [7]    cake platter,  collaborative AR system
                 clipping     designed to support
                plane, card   a scenario where a
                              group of experts meet
                              to discuss the design
                              of a product.
                              (i) Rotate cake platter:
                              rotate view
                              (ii) Transfer the virtual
                              model: put PDA near
                              cake platter:
                              (iii) Operate virtual model:
                              use clipping plane
                              (iv) Change light source:
                              control light prop device
                              (v) Change virtual model
                              colour: place the marker
                              near the virtual model

Shared           Tangible:    Shared space is an AR
Space [8]           card      collaborative card game.
                              (i) Place the card side
                              by side. If cards contain
                              correct matches, an
                              animation is triggered

ARGroove [8]     Tangible:    Manipulate the LP records
                 LP records   to composite music.
                              (i) Up: pitch
                              (ii) Down: amplitude
                              (iii) Rotate: filter
                              cut-off frequency
                              (iv) Tilt: delay mix

Tiles [8]        Tangible:    Tiles is an AR authoring
                    card      interface that supported
                              rapid prototyping for
                              aircraft instrument panels.
                              (i) Data tiles: show virtual
                              object from menu book
                              (ii) Operator tiles: delete,
                              copy, help

VOMAR [8]        Tangible:    An AR multimodal interface
                   Paddle     for designing house via
                              combination of speech and
                              paddle gestures interaction.
                              (i) Place paddle on the
                              marker of menu book to
                              pick the object.
                              (ii) Tilt the paddle to
                              place the object.
                              (iii) Push to move the
                              (iv) Shake to delete the
                              object from paddle.
                              (v) Hit to delete object
                              from virtual room.

Jumanji          Tangible:    Jumanji Singapore is a game
Singapore        Two Cubes    that allows users to have a
[9]                           magical tour of Singapore
                              by simple manipulations
                              of two physical cubes.
                              (i) Dice: Rolls the dice,
                              the metaphor will move
                              based on the number.
                              (ii) Control cube:
                              (1) Rotate to view different
                              photo and video.
                              (2) Stack the cubes will
                              function as shopping cart.
                              (3) Navigation tool: Turn
                              left, turn right, tilt
                              upwards and tilt downwards.

Magic            Tangible:    A foldable cube for
Story             Foldable    storytelling with 3D
Cube [10]           Cube      graphics, 3D sound,
                              music and speech.
                              (i) Unfold the cube will
                              function as turning pages
                              of traditional book.

ARTalet [11]     Tangible:    A digilog book that supports
                  Pen-type    novel visual, audio and haptic
               Haptic Device  authoring functions.
                              (i) Place the pen-type on
                              the marker in digilog to
                              copy the virtual object.
                              (ii) Pressing and holding the
                              button on the pen to edit
                              shape of the trajectory

MAR              Tangible:    Mobile AR authoring tool
authoring           Card      system that enables user to
tool [12]                     create AR content.
                              (i) Place the card side by
                              side to augment the new
                              virtual object to be

AR-SEE:          Tangible:    AR-SEE system that allows
Augmented          Marker     user to learn about science
Reality             Card      of architectural design by
for Passive                   interacting with a tangible
Solar                         physical 3D model house.
Energy                        (i) Marker: Manipulate the
Education                     type of roof and window
[13]                          for the house.

TAR for          Tangible:    An augmented reality
interior          3 Cards     system for designing/
design [1]         Marker     educating/presenting
                              interior design projects.
                              (i) Add or hide marker to
                              adjust colour and material
                              of the furniture

MagicCup         Tangible:    The MagicCup is a cup-shaped
AR [14]          Cup-shaped   handheld compact AR input
                 Handheld     device with a tracker that
                  Compact     detects six dimensional
                              positions and pose
                              (i) Pick: Cover a virtual
                              object with the cup.
                              (ii) Allocate: Place the cup
                              with the virtual object.
                              (iii) Move: Slide the cup.
                              (iv) Rotate: Rotate the cup.
                              (v) Delete: Shake the cup.
                              (vi) Change Mode: Shake the
                              empty cup

Name of
invention         Scenario     Type of AR   Physical setup

Magic             Product      Projection   (i) Meeting table
Meeting [7]        design         based     (ii) HMD
                                            (iii) Projection screen
                                            (iv) Cake platter
                                            (v) PDA
                                            (vi) Clipping plane
                                            (vii) Three card colour
                                            (red, green, yellow)
                                            (viii) Light prop

Shared              Game       See-through  i) HMD + camera
Space [8]                         based     ii) Card

ARGroove [8]   Entertainment   Projection   (i) Projector
                                  based     (ii) LP records
                                            (ii) Projection

Tiles [8]          Rapid       See-through  (i)Whiteboard
                prototyping       based     (ii) Book
                                            (iii) Magnetic tiles
                                            (iv) HMD

VOMAR [8]        Interior      See-through  (i) Menu Book
                   Design         Based     (ii) Cardboard Paddle
                                            (iii) Large Piece
                                            of Paper
                                            (iv) HMD

Jumanji             Game       See-through  (i) HMD
Singapore                         Based     (ii) Game Board
[9]                                         (iii) Two Cube as
                                            Dice and Controller

Magic            Education     See-through  (i) HMD + Camera
Story                             Based     (ii) Foldable Cube
Cube [10]

ARTalet [11]     Education     See-through  (i) Input Images
                                  Based     from a Camera
                                            (ii) CV Based
                                            Tracking Book
                                            (iii) Menu Prop
                                            (iv) Pen-type
                                            Manipulation Prop
                                            (v) HMD

MAR              Education       Mobile     (i) Phone
authoring                         Based     (ii) Card Marker
tool [12]

AR-SEE:          Education       Mobile     (i) Phone
Augmented                         Based     (ii) Markers
for Passive

TAR for          Interior       PC Based    (i) PC +Webcam
interior           Design                   (ii) Logitech
design [1]                                  Quickcam Vision
                                            (iii) Markers

MagicCup         Interior      See-through  (i) Menu Book
AR [14]            Design         Based     (ii) Cup-shaped
                                            Handheld Compact
                                            (iii) HMD

Table 4: Scoping review on existing spatial awareness
interaction technique.

Name of            Interaction     Interaction
invention           technique:     technique

iSar                  Spatial      (i) Involves only
Projection [15]                    projection

SAR/VR                Spatial      (i)Wand controller:
HYBRID                             points at a certain
SPACE [16]                         location and presses
                                   the controller's
                                   trigger to start
                                   the drawing

Garden [17]           Spatial      (i) Durovis Dive headset:
                                   user can see the world
                                   in VR via tablet that has
                                   been mounted on it
                                   (ii) Hands and Legs:
                                   create voxels to fill
                                   in the wireframe and
                                   draw objects

Tabletop           Spatial and     Tangible
SAR [18]             tangible:     (i) Rectangular object:
                   rectangular     specification of zoom
                  object, scroll   region in the virtual
                       wheel       volume with desired
                                   location and orientation
                                   (ii) Push button: locks
                                   zoom position in place
                                   (iii) Scroll wheel:
                                   increases or decreases
                                   the zoomed volume.
                                   (i) Handheld input device
                                   (push button): selection
                                   (ii) Handheld input device
                                   (scroll wheel): 2D zooming
                                   (iii) 6DOF pointing device:
                                   drills down on 3D projected
                                   onto the physical objects
                                   (iv) 4K monitor: displays
                                   detailed views of 3D

Tag It! [19]          Spatial      (i) Tango: extends Google
                                   Glass' sensing ability by
                                   sharing the location and
                                   pose of the user as well
                                   as tag the virtual
                                   information in real world
                                   (ii) Glass: overlays virtual
                                   tags based on the spatial
                                   information received from
                                   the Tango
                                   (iii) Glass touchpad: user
                                   can initiate the wireless
                                   connection by using a
                                   three-finger touch
                                   gesture on it

Social spatial        Spatial      (i) Tango: registers the
mashup                             virtual information on
(SSM) [20]                         the face of the 3D object
                                   after social spatial
                                   mashup via tapping the
                                   virtual button on the

AR interior           Spatial      A depth camera is used to
designer [21]                      measure the size of available
                                   space in terms of supporting
                                   surfaces for new furniture
                                   (i) User uses mouse to
                                   interact and manipulate
                                   virtual furniture

PimpMy                Spatial      A system that supports the
Kitchen [22]                       early design process of
                                   kitchens and helps to ease
                                   designer and client in
                                   meeting session
                                   (i) Blank (white cabinet):
                                   moveable cabinet placed
                                   in an environment of the
                                   shape and size of the
                                   target kitchen
                                   (ii) Identical card:
                                   resizing projected components.
                                   Initially, the components
                                   automatically fit the
                                   available space
                                   (iii) SAR Swatches: choose
                                   colour, loading different
                                   design and save design
                                   (iv) Stylus: taping the
                                   cabinet surface to change

Name of
invention          Scenario     Type of AR  Physical setup

iSar                Medical    Projection   (i) Handheld projector
Projection [15]                   based     (ii) RGBD camera
                                            (iii) Physical model

SAR/VR           Architecture  Projection   (i)Wand controller
HYBRID                            based     (ii) Projector
SPACE [16]

Garden [17]          Game        Mobile     (i) Tango tablet
                                  based     (ii) Durovis Dive

Tabletop          Information  Projection   based
SAR [18]            sharing       based     (i) Handheld
                                            projecting device
                                            (ii) Physical object
                                            (iii) 6DoF Tracking
                                            (iv) Projector

Tag It! [19]      Information  See-through  (i) Glass
                    sharing       based     (ii) Glass touchpad
                                            (iii) Tango
                                            (iv) Target

Social spatial    Information    Mobile     (i) Google Project
mashup              sharing       based     Tango
(SSM) [20]

AR interior      Architecture    PC based   (i) PC + Mouse
designer [21]                               (ii) Microsoft

PimpMy           Architecture  Projection   (i) Projector
Kitchen [22]                      based     (ii) Blanks
                                            (white cabinet)
                                            (iii) Card
                                            (iv) SAR Swatch
                                            (v) Stylus

Table 5: Tangible interaction technique method.

Interaction       Virtual        Selection of the
medium           furniture      virtual furniture's
               representation         feature

                  Cabinet         Cabinet surface
Cube               Stove              Colour
                    Sink              Colour
                  Kitchen             Colour

                                 Colour of cabinet
Card                               Accessories
                                    of cabinet
COPYRIGHT 2018 Hindawi Limited
No portion of this article can be reproduced without the express written permission from the copyright holder.
Copyright 2018 Gale, Cengage Learning. All rights reserved.

Article Details
Printer friendly Cite/link Email Feedback
Author:Nizam, Siti Soleha Muhammad; Lam, Meng Chun; Arshad, Haslina; Suwadi, Nur Asylah
Publication:Advances in Multimedia
Article Type:Report
Date:Jan 1, 2018
Previous Article:Beijing Opera Synthesis Based on Straight Algorithm and Deep Learning.
Next Article:A Perception-Driven Transcale Display Scheme for Space Image Sequences.

Terms of use | Privacy policy | Copyright © 2019 Farlex, Inc. | Feedback | For webmasters