Extending the Tracking Distance of Fiducial Markers for Large Indoor Augmented Reality Applications.
I. INTRODUCTIONAugmented Reality (AR) uses computer vision, image processing and computer graphics techniques to augment the real-world by digital information [1]. This provides a real-time interaction with the real and virtual objects. AR is related to a mediated reality, in which a view of reality is modified using a computer system. By contrast, virtual reality replaces the real world with a simulated one [2]. Augmented reality lies in-between the virtual and real world [3]. A considerable amount of work has been done in the area of augmented reality but there are still several challenges need to be addressed.
The challenges are categorized as performance challenges, alignment challenges, interaction challenges, mobility/portability challenges and visualization challenges [4]. Alignment challenges include registration and tracking. The proper alignment of virtual objects to the real-world scene is called registration. Tracking, the process of estimating the camera pose (virtual or real) in the environment where augmentation takes place, is the most important part of AR systems. The alignment of virtual objects and real-world objects depends on accurate tracking of the viewing pose, relative to the real environment and the annotated objects [5]. Therefore in AR applications, it is necessary to track the user movement in six Degrees of Freedom (6DOF) relative to the environment.
In AR applications, tracking can be performed using sensor-based, vision-based and/or hybrid techniques [6]. Currently, vision-based tracking approach is most widely used tracking method in AR applications. Vision-based tracking uses computer vision methods to calculate the camera pose relative to the real-world objects. Vision-based tracking approaches use marker-based and markerless techniques.
In marker-based tracking, artificial markers are placed in the real environment for the development of augmented reality applications. This approach provides fast, accurate and real-time tracking solution for controlled indoor AR applications. Due to the short tracking distance, marker-based tracking is rarely used in the development of large indoor AR applications. This paper presents the design and implementation of layered marker to extend the marker tracking distance. Using this approach, the marker-based tracking can easily be used in large indoor AR environments.
The related work is discussed in section-II. Section-III contains the analysis of existing marker tracking distance. The design of proposed layered marker and its tracking procedure is elaborated in section-IV. The implementation and evaluation is presented in section-V followed by discussion in section-VI. The section-VII contains the conclusion with future work.
II. RELATED WORK
Marker-based approach uses fiducial markers to be placed in the real environment for the development of augmented reality applications. Different toolkits such as ARToolKit [7], ARToolKitPlus [8], ARTag [9], and ALVAR [10] are developed to perform marker-based tracking. These toolkits provide a good framework for the development of an AR application. The challenges in marker-based approach are robustness, occlusion, jitter, blurredness, tracking stability and moving marker or camera in x or y directions. Researchers proposed various solutions to overcome these challenges to enhance the marker-based tracking.
Ababsa and Mallem [11] proposed a robust solution for marker-based tracking that used markers' corners information to estimate the camera pose. By using particle filtering technique, Uematsu and Saito [12] correctly estimate rotation parameters of the camera which are important to track the 3D coordinate system and thus improve the accuracy of the 3D coordinate system. Maidi et al. [13] developed a system that combined extended Kalman filter [14] and analytical method [15] with direct resolution of pose parameters computation. This system improved stability, convergence and accuracy of the pose parameters in marker-based system. The critical issues in marker-based AR applications are jitter and occlusion which are solved by Seo et al. [16]. They used multiple key-points to handle marker jitter and feature tracking for marker occlusion. This system takes more computational time for pose estimation. Rabbi et al. [17] extends the marker-based tracking to semi/uncontrolled environment.
Jun et al. [18] solved the challenge of occlusion by extending the fiducial marker planar. They extend the working space as large as a room but the marker tracking distance is still a challenge. The concept of nested marker is presented by Teteno et al. [19]. The main limitations in their nested marker are that they used four markers inside each marker and the maximum hierarchy is set to three. The upper-layer marker is identified by its lower-layer markers, which may produce inter-marker confusion between upper layer and its lower-layer markers. As the maximum hierarchy is upto level 3, the distance covered is also limited. Recently, Khan et al. [20] identified factors affecting the tracking of fiducial markers.
Due to this limited tracking distance of marker-based tracking approach, it is rarely used in the development of large indoor AR applications. This paper presents a new layered marker which is the combination of multiple markers. Different AR applications combine multiple markers to achieve robust geometric registration against partial occlusion. But these systems still have the limitation of individual marker recognition from long distance. In this paper the automatic design of layered marker for any large tracking environment is presented and the implementation of developed layered marker is explained. This method extends the marker tracking distance and can be used for the development of AR applications in large indoor environment.
III. MARKERS TRACKING DISTANCE ANALYSIS
Each marker-based toolkit uses their own pattern of markers as shown in Fig. 1.
These markers consist of square black borders having distinct encoding on each of them. The camera pose is calculated relative to these markers in real environment.
The minimum and maximum tracking distance of a marker from the camera are calculated by taking markers of different sizes. For this purpose, a module is developed using ARToolKit and different size markers are considered to be tracked at different distances. Ten markers of size 2 x 2 cm, 4 x 4 cm, 6 x 6 cm,...20 x 20 cmare taken to perform these experiments. Table I shows the results obtained from tracking these markers along with minimum and maximum distances.
The minimum tracking is taken on the basis of occlusion of marker from the camera due to zoom in. When the marker is getting closer to camera, the corners of the marker are occluded and tracking failure occurs. The maximum tracking is analyzed on the basis of marker blurredness due to large distance from the camera. When the marker is away from a certain limit (maximum tracking distance) the marker tracking failure occurs. For example, the marker having size of 0.2 x 0.2 meter can be tracked from minimum distance of 0.4 meter and maximum distance of 5.0 meters from the camera.
Using the above analysis, the size and distance relationship is find out as given in equation (1), (2) and (3)
MinD=MS x 2 (1)
MaxD=MS x 25 (2)
TR=MaxD - MinD (3)
Where
MinD and MaxD represent the minimum and maximum distance respectively, MS represents marker size and TR represents marker tracking range
A. Best Tracking Range for Markers
In this section the best tracking range of a marker is elaborated. This provides the minimum and maximum tracking distance of marker from the camera. For this purpose a module is designed using ARToolKit that will report the marker distance from camera and the tracking errors found while tracking the sample1 marker (See Fig. 2).
The marker having size of 0.06 x 0.06 meter is considered for this analysis. The minimum and maximum tracking distance for this marker is 0.12 and 1.5 meter respectively. The resultant data is plotted on a graph as shown in Fig. 3.
The graph in Fig. 3 indicates that marker tracking is performed successfully within the minimum and maximum range (0.12 and 1.5 meter respectively). The tacking failure occurs when the marker distance is beyond the limits.
IV. MODEL FOR DESIGNING A LAYERED MARKER
The above marker analysis is used to design a new marker that extends the marker tracking distance from the camera. The proposed marker consists of different layers. Each layer will be a separate marker itself. Single marker will be placed at each layer. Each marker has its unique pattern to minimize inter-marker confusion (See Fig.4).
To design a layered marker for a certain application, the following steps are used.
Step-1: Determine the minimum and maximum distance to be tracked in the environment (MinTD and MaxTD)
Step-2: Calculate the size of the inner most and outer most layer markers as
MSIL = MinTD/2 x MinTD/2 (4)
MSOL = MaxTD/25 x MaxTD/25 (5)
Where MSIL and MSOL represent marker size at inner and outer most level respectively
Step-3: Calculate total number of markers required for the scene as
n=1 as outer most layer/marker
While MSOL > MSIL perform
MSOL/2 and n is incremented by one
After completion of this step n contains the total number of layers / marker required
Step-4: Design n number of markers so that each marker has minimum similarity against the list (say L).
Initially, L contains only one marker i.e [L.sub.1] which is the outermost layer having size equal to MSOL.
- Take new marker [L.sub.i] for i = 2, 3, 4,...n
- Insert [L.sub.i] to list L if correlation coefficient r of [L.sub.i] and [L.sub.j] < 0.1 for i = 2, 3, 4... n and j = 1... i as
[mathematical expression not reproducible]
Where [x.sub.p] is the intensity of pth pixel in marker [L.sub.i] and [y.sub.p] is the intensity of pth pixel in marker [L.sub.j] [x.sub.m] and [y.sub.m] are mean intensities of markers [L.sub.i] and [L.sub.j]
Step-5: Design layered marker by placing L at the outermost layer and [L.sub.1] at the innermost layer having sizes as:
Size([L.sub.j]) = Size([L.sub.j+1])/2
For example, consider a large indoor environment that needs AR application. The required maximum and minimum tracking distances are 100meters and 0.2meter respectively. So the two variables are initialized as:
MaxTD = 100meters
MinTD = 0.2meter
Following the above procedure to design the required layered marker for this environment is as: First the marker size at inner-most level and outer-most level is calculated using equations (4) and (5).
MSIL = MinTD/2 x MinTD/2 meters
MSIL = 0.2/2 x 0.2/2 = 0.1 x 0.1m
And
MSOL = MaxTD/25 x MaxTD/25 meters
MSOL = 100/25 x 100/25 = 4 x 4m
Now find the total number 2 of layers required for the development of AR application in this environment. For this calculation the next step of the procedure is applied. The step produce that seven layers are required in this layered marker. The sizes of each layer from outer-most layer to inner-most layer are 4 x 4 m, 2 x 2 m, 1 x 1 m, 0.5 x 0.5 m, 0.25 x 0.25 m, 0.125 x 0.125 m and 0.0625 x 0.0625 m respectively. Take seven markers having less correlation coefficient among them to avoid inter-marker confusion. The layered marker is designed by placing the largest marker at outer-most layer and the next marker at next level and so on as shown in Fig.5.
A. Tracking Procedure for Layered Marker
The procedure given above is used to design layered markers for large AR environment. The marker at each layer is registered with the toolkit and pattern file is saved in the database. When the camera detects the layered marker, these multiple files will be searched for matching the patterns. Fig. 6 elaborates the tracking procedure of the proposed layered marker.
In the first step, the video path is initialized that calculates the camera parameters, the marker pattern files are read from the database and setup of the graphics window. The layered marker detection is performed when the camera visualize the layered marker. The pattern file associated with the specific layer is searched in the database. The system finds the specific tracked layer by using the distance of layered marker from camera.
Equations (1) and (2) are used to find the tracking range of any size of marker. Using these equations, the Minimum Marker Tracking Distance (MinTD) and Maximum Marker Tracking Distance (MaxTD) of each marker of the layered marker is calculated. The MaxTD of outer most layer will be the upper tracking distance of the proposed layered marker and the MinTD of the inner most layer will be the lower tracking distance of layered marker. So the layered marker tracking distance will range from MaxTD to MinTD. The other layers ranges are calculated as
[L.sub.j] = [[MaxTD.sub.j-1] + [MinTD.sub.j]/2 [MaxTD.sub.j] + [MinTD.sub.j+1]/2] (6)
This will be the minimum and maximum distance range for each layer. This improves the layered marker tracking in case of occlusion at any layer. The marker tracking is shifted to the adjacent lower layer. The virtual contents are placed according to the tracked layer.
Consider the layered marker designed in section IV. The minimum and maximum tracking distances for this layered marker are 0.125 meter and 100meter respectively. The tracking distance of each layer is calculated as:
[L.sub.7]= [[MaxTD.sub.7-1] + [MinTD.sub.7]/2,100]
[L.sub.7]= [[MaxTD.sub.6] + [MinTD.sub.7]/2,100]
[L.sub.7] = [50+8/2,100] = [58/2,100] = [29,100]
So, the minimum and maximum tracking distance for layer seven is set to 29 and 100 meters respectively.
Using equation (3), the tracking distances of others layers are calculated and given as:
[L.sub.6] = [14.5,29], [L.sub.5] = [7.25,14.5]
[L.sub.4] = [3.625,7.25], [L.sub.3] = [1.8125,3.625]
[L.sub.2] = [0.90625,1.8125], [L.sub.1] = [0.125,0.90625]
V. IMPLEMENTATION AND RESULTS
The proposed layered marker is tested using ARToolKit [7] a well known augmented reality library. A module is designed to track the layered marker prepared by using the above scenario.
Consider a layered marker with five layers having sizes as 20 x 20 cm, 10 x 10 cm, 5 x 5 cm, 2.5 x 2.5 cm and 1.25 x 1.25 cm (as shown in Fig.7). The layered marker is designed by placing the largest marker at outer-most layer and the next marker at next level and so on. This marker is used for tracking in relatively a large environment.
The distance tracking results of layered marker is compared with distance tracking of single marker having different sizes equal to the size of each layer. For this purpose, five single markers of different patterns having sizes of 1.25 x 1.25 cm, 2.5 x 2.5 cm, 5 x 5 cm, 10 x 10 cm and 20 x 20 cm are used. Fig. 8 shows the markers used for this comparison. Here A, kanji, sample1, sample2 and calib are markers names.
These experiments were conducted using Sony VAIO laptop having 2.4GHZ processor, 4GB RAM, NVIDIA graphics card and webcam (ArcSoft Webcam Companion 4) with resolution of 640 x 480 pixels.
During experiments the marker tracking errors along with the distance of marker from camera were recorded in a text file. The tracking errors range from zero to one. The zero means complete tracking is performed while one indicates complete marker tracking failure. The threshold of tracking errors for marker detection was set to 0.5. The marker tracking errors below the threshold value means that the marker tracking is successful and the values above means marker is not detected.
Fig. 9 shows the marker tracking errors of layered marker and single 'calib' marker having size 1.25 x 1.25 cm. The graph indicates that the successful tracking of single marker is achieved only in a small distance range while the layered marker produces less tracking errors in the entire distance range.
In the second experiment, the layered marker is compared with single 'sample2' marker having size 2.5 x 2.5 cm. Fig. 10 shows the graph of the results produce during this experiment. The tracking failure occurs at a distance of 50 cm and above for single marker. This is the maximum tracking distance of this size marker. The minimum tracking distance for this maker is 5 cm. The layered marker is easily tracked in the entire tracking range.
The tracking graph of the layered marker is compared with single 'sample2' marker of size 5 x 5 cm and results are presented in Fig. 11. The tracking of marker having size 5 x 5 cm is failed at the distance of 125cm and above from the camera. The camera cannot properly visualize this marker up to large distance. On the other hand the layered marker provides successful tracking upto 500 cm distance.
Similarly, the layered marker is compared with 10 x10 cm and 20 x 20 cm single markers. Fig. 12 and Fig. 13 show their respective results.
The above analysis shows that the layered marker extends the marker tracking distance to be used for tracking application in large indoor environments.
VI. DISCUSSION
Marker-based tracking provides fast, accurate and real-time tracking solution for indoor AR applications. The key challenge in marker-based approach is its tracking distance. Due to this short tracking distance, marker-based tracking technique is rarely used in large indoor AR applications. The design and implementation of a new layered marker is presented in this paper that extends the marker tracking distance. This layered marker uses on the shelf marker-based approach. This method extended the marker tracking distance and can be used for the development of AR applications in large indoor environments.
A step by step procedure is given to design the layered marker for a certain large indoor environment. The designed marker is placed in the environment to be tracked. The tracking procedure is also specified that extracts the exact tracked layer in the layered marker. The virtual information related to the tracked layer is displayed to the user.
The layered marker provides the following advantages while developing AR applications:
* The layered marker extends the tracking distance.
* It allows tracking in case of partial occlusion of the marker. When the marker at one layer is occluded by any object in the environment, the camera parameter can be easily estimated for the marker at another layer. The tracking module will automatically choose the non-occluded marker at the next higher or lower level.
* This marker can be used for zooming purposes as well.
* New interaction techniques in virtual environments can be developed using the layered marker.
VII. CONCLUSION AND FUTURE WORK
Vision-based tracking are the most active tracking techniques used in the development of AR applications. Marker-based tracking is a vision based approach in which fiducial markers are used in the environment for tracking the camera pose. This tracking methodology provides fast, accurate and real-time tracking solution for indoor AR applications but due to the short tracking distance of this approach, it is rarely used for large indoor AR applications. This paper introduced the design and implementation of a new layered marker that is based on existing marker-based approach that extended the marker tracking distance to be used in large indoor AR applications. The layered marker is designed using the proposed model and tracked using the proposed tracking methodology. This layered marker is tested in large indoor environment and the evaluation showed that the layered marker increases the marker tracking distance reasonably.
During the experiments, we learnt that the speed of marker or camera in x-axis and y-axis produces blurredness in the environment that can affect the marker tracking. Therefore, the future work will focus on the design of algorithm that provides a robust and accurate tracking solution for blurred markers.
REFERENCES
[1] S. Siltanen. "Theory and Applications of Marker-Based Augmented Reality". 2012.
[2] P. Milgram, D. Drascic, J. Julius, Grodski, A. Restogi, S. Zhai, C. Zhou, "Merging Real and Virtual Worlds," Proceedings of IMAGINA '95 (Monte Carlo), pp. 218-230, 1995.
[3] P. Milgram, H. Takemura, A. Utsumi, F. Kishino, "Augmented reality: a class of displays on the reality-virtuality continuum," pp. 282-292, 1995., [Online]. Available: http://dx.doi.org/10.1117/12.197321
[4] I. Rabbi, S. Ullah, "A survey on augmented reality challenges and tracking," Acta Graphica, Croatia, vol. 24, 2013.
[5] U. Neumann, A. Majoros, "Cognitive, performance, and systems issues for augmented reality applications in manufacturing and maintenance," in Proceedings IEEE Virtual Reality Annual International Symposium, pp. 4-11, 1998., [Online] Available: http://dx.doi.org/10.1109/vrais.1998.658416
[6] F. Zhou, H. B. L. Duh, and M. Billinghurst, "Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR," in 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, ISMAR'08, pp. 193-202, 2008., [Online]. Available: http://dx.doi.org/10.1109/ismar.2008.4637362
[7] H. Kato, M. Billinghurst, I. Poupyrev, K. Imamoto, and K. Tachibana, "Virtual object manipulation on a table-top AR environment," presented at the IEEE and ACM International Symposium on Augmented Reality, 2000.
[8] M. Fiala, "Comparing ARTag and ARToolkit Plus fiducial marker systems," in IEEE International Workshop on Haptic Audio Visual Environments and their Applications, 2005., [Online]. Available: http://dx.doi.org/10.1109/have.2005.1545669
[9] M. Fiala, "ARTag, a fiducial marker system using digital techniques," presented at the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR'05, 2005., [Online]. Available: http://dx.doi.org/10.1109/CVPR.2005.74
[10] ALVAR. "ALVAR--A Library for Virtual and Augmented Reality", Available: www.vtt.fi/multimedia/alvar.html January 16, 2013.
[11] F. Ababsa, M. Mallem, "Robust camera pose estimation using 2D fiducials tracking for real-time augmented reality systems," presented at the Proceedings of the ACM SIGGRAPH International Conference on Virtual Reality Continuum and its Applications in Industry, Singapore, 2004., [Online]. Available http://dx.doi.org/10.1145/1044588.1044682
[12] Y. Uematsu, H. Saito, "Improvement of accuracy for 2D marker-based tracking using particle filter," in 17th International Conference on Artificial Reality and Telexistence, pp. 183-189, 2007., [Online]. Available: http://dx.doi.org/10.1109/icat.2007.16
[13] M. Maidi, J.-Y. Didier, F. Ababsa, M. Mallem, "A performance study for camera pose estimation using visual marker based tracking," Machine Vision and Applications, vol. 21, pp. 365-376, 2010. [Online]. Available: http://dx.doi.org/10.1007/s00138-008-0170-y
[14] G. Bishop, G. Welch, "An Introduction to the Kalman Filter," in ACM SIGGRAPH '01, 2001.
[15] M. Dhome, M. Richetin, J. T. Lapreste, G. Rives, "Determination of the attitude of 3D objects from a single perspective view," IEEE Transactions on Pattern Analysis and Machine Intelligence., vol. 11, pp. 1265-1278, 1989., [Online]. Available: http://dx.doi.org/10.1109/34.41365
[16] J. Seo, J. Shim, J. Choi, J. Park, T.-d. Han, "Enhancing marker-based AR technology," in Virtual and Mixed Reality - New Trends. vol. 6773, R. Shumaker, Ed., ed: Springer Berlin Heidelberg, pp. 97-104, 2011., [Online]. Available: http://dx.doi.org/10.1007/978-3-642-22021-0_12
[17] I. Rabbi, S. Ullah, S. U. Rahman, A. Alam, "Extending the functionality of ARToolKit to semi-controlled/uncontrolled environment," INFORMATION, vol. 17, pp. 2823-2832, 2014.
[18] J. Jun, Q. Yue, Z. Qing, "An extended marker-based tracking system for augmented reality," in Second International Conference on Modeling, Simulation and Visualization Methods (WMSVM'10), 2010, pp. 94-97., [Online]. Available: http://dx.doi.org/10.1109/wmsvm.2010.52
[19] K. Tateno, I. Kitahara, Y. Ohta, "A nested marker for augmented reality," in IEEE Virtual Reality Conference, VR '07, pp. 259-262, 2007., [Online]. Available: http://dx.doi.org/10.1109/vr.2007.352495
[20] D. Khan, S. Ullah, I. Rabbi, "Factors affecting the design and tracking of ARToolKit markers," Computer Standards & Interfaces, vol. 41, pp. 56-66, 2015., [Online]. Available: http://dx.doi.org/10.1016/j.csi.2015.02.006
Ihsan RABBI (1,2), Sehat ULLAH (1)
(1) Department of Computer Science and IT, University of Malakand, Pakistan
(2) Department of Computer Science, University of Secience and Technololgy, Bannu ihsanrabbi@uom.edu.pk
Digital Object Identifier 10.4316/AECE.2015.02008
TABLE I. MINIMUM AND MAXIMUM TRACKING DISTANCE OF MARKERS
S/No Marker Size Minimum Distance Maximum Distance
(meter) (meter) (meter)
1 0.02 x 0.02 0.04 0.5
2 0.04 x 0.04 0.08 1.0
3 0.06 x 0.06 0.12 1.5
4 0.08 x 0.08 0.16 2.0
5 0.10 x 0.10 0.20 2.5
6 0.12 x 0.12 0.24 3.0
7 0.14 x 0.14 0.28 3.5
8 0.16 x 0.16 0.32 4.0
9 0.18 x 0.18 0.36 4.5
10 0.20 x 0.20 0.40 5.0
 Printer friendly
 Cite/link
 Email
 Feedback
| |
| Author: | Rabbi, Ihsan; Ullah, Sehat |
|---|---|
| Publication: | Advances in Electrical and Computer Engineering |
| Date: | May 1, 2015 |
| Words: | 4140 |
| Previous Article: | Incorporating the Avoidance Behavior to the Standard Particle Swarm Optimization 2011. |
| Next Article: | Robotic Arm Control Algorithm Based on Stereo Vision Using RoboRealm Vision. |
| Topics: | |