Development of an AR Navigation System “Smart Information & Mobility Town (SiMT)” in Commercial Facilities

Masataka Nakahara1, Graduate Student, Sojo University, Japan

Motoya Koga², Associate Professor, Sojo University, Japan

Satoshi Fujimoto3, R&D Strategy Team Manager, Knowledge Communication Co., Ltd., Japan

Article History:

Received March 29, 2021

Accepted May 17, 2021

Published Online December 26, 2021

https://doi.org/10.51596/sijocp.v1i2.21

Abstract

In recent years, with the development of IT technology, various information services in cities have become convenient and easily obtainable for everyone. However, information support provisions for users in cities using these IT technologies are still not sufficient. Many of these technologies are aimed at the public, and not all have been optimised for persons with disabilities, such as wheelchair users, the elderly, pregnant women, those with strollers, and so on. There is also insufficient support to provide information needed to move around in public transportation facilities such as stations, airports, and extensive commercial facilities, which have become increasingly common in recent years in Japan. Therefore, in this study, we used AR technology to develop an application, “Smart Info. & Mobility Town” (SiMT), aimed at supporting movement in SAKURA MACHI Kumamoto, a commercial complex built in Kumamoto City, Kumamoto Prefecture, Japan in 2019. The research method is as follows. 1; Summarised the outline of the target facility, SAKURA MACHI Kumamoto. 2; Used AR technology to develop an application, “Smart Info. & Mobility Town” (SiMT), aimed at supporting movement in the commercial complex. 3; Conducted demonstration experiments, proposed future developments and possibilities of SiMT, and summarised this study.

SiMT, which was developed on a trial basis, uses a 3D map created from 3D scan data of SAKURA MACHI Kumamoto and utilises the camera function of a smartphone to synthesise CG images of routes from the current location to the destination to support users’ movements. We also developed this technology and conceived future technology development, such as support in the event of a disaster and support for the mobility of vulnerable road users.

Keywords: AR, 3D scan, spatial recognition, digital twin, self-position estimation

Introduction
In recent years, with the development of IT, various information services in cities have become convenient and easily obtainable for everyone. For example, Google Maps can search for cafes and guide people to their destinations. In Japan, where there are many disasters such as earthquakes, the government can provide information to help residents evacuate safely to shelters in the event of a disaster. However, many of these technologies are aimed at the general public, and not all of them have been optimised for people with disabilities, such as wheelchair
users, the elderly, pregnant women, those with strollers, etc. There is also insufficient support information needed to move around in public transportation facilities such as stations, airports and extensive commercial facilities, which have become increasingly common in recent years in Japan. Therefore, in this study, we used AR technology to develop an application, “Smart Info. & Mobility Town” (SiMT), aimed at supporting movement in SAKURA MACHI Kumamoto, a large- scale commercial complex built in Kumamoto City, Japan, in 2019. The research method is as follows. Furthermore, the future development and possibility of SiMT are discussed.
Related Studies
AR has been used not only in games but also in a wide range of fields, such as construction, civil engineering, medicine, retail, etc. A typical navigation system using AR is one that uses GPS location information outdoors. Tanaka et al. developed a navigation system that superimposes arrows to indicate the direction of travel to the destination using GPS. However, it is difficult to use GPS location information indoors (Tanaka et al., 2014).
As an indoor location estimation technique, there are methods using Wi-Fi, beacons, and AR markers. Kurihara et al. proposed a navigation system that shares location information with other users by preparing AR markers and a server that registers the information. The use of AR markers has achieved high positioning accuracy, but this requires placing many markers, which increases the installation cost (Kurihara & Satou, 2014). In addition, installing Wi-Fi and beacons would likewise incur installation costs.
For marker-less methods that do not require the installation of sensors or markers, SLAM technology has been developed, which tracks feature points from camera images and simultaneously performs self-position estimation and map creation. Positioning is now possible in an area as large as a room with an error of only a few centimetres (Fujimoto et al., 2016). Varelas et al. developed a navigation system that uses anchors to estimate one’s position by capturing many feature points with a camera and proposed a new method of measuring position using anchors (Varelas et al., 2020). Liu et al. also surveyed research on indoor navigation using IFC-based BIM and concluded that no indoor navigation system using IFC and BIM has been completed and that research needs to be continued (Liu et al., 2021).
In contrast to these previous studies, this study proposes the development of a new navigation system with AR display, in addition to advanced self-positioning using 3D scan data, which is now readily available with the iPhone and iPad Pro, and the camera function of smartphones, which have high penetration rates.
Research Method
The following methods were used to conduct the study. 1; Summarised the outline of the target facility, SAKURA MACHI Kumamoto. 2; Used AR technology to develop a prototype application model, “Smart Info. & Mobility Town” (SiMT), aimed at supporting movement in the commercial complex. 3; Proposed future developments and possibilities of SiMT and summarise this study.
Sakura Machi Kumamoto Overview and Signature Arrangement
SAKURA MACHI Kumamoto (Figure 1) is a commercial facility with restaurants, hotels, public facilities, etc. It opened on September 14, 2019, due to an urban redevelopment project. Commercial facilities account for about a quarter of the total area of 162,000 square meters, with 149 stores. This accounts for most of the visitors. As a facility with many uses, the distance to the destination can be complex and lengthy, and the route to the destination can be challenging to find. Also, there are problems, such as difficulty in understanding the signs. Therefore, there are many inquiries for information and employees.
There are two kinds of signs: guides consisting of arrows and information signs such as floor guides. Currently, there are 124 guide signs, 34 of which are movable or printed signs which were added and were not there when the building opened. Similarly, with regard to information signs, 11 of the 44 signs were movable or printed and added after opening, indicating that it is difficult to find a route within the facility.
The red arrow in Figure 2 shows an example of going from the main entrance to the bus platform. To get from the entrance to the platform, it is necessary to go to the second floor or the basement floor. The sign from the entrance to the platform is placed near the escalator going up to the second floor, but the sign from the second floor to the platform is far away and difficult to notice because it is small and located high overhead. In addition, the stairs or escalators to the platform are conspicuously marked in blue, but it is difficult to notice them unless you are near them. Therefore, improving the signage or developing a navigation system is considered necessary.

Figure 1. Photos of SAKURA MACHI Kumamoto

Figure 2. Signing Status
Overview of SiMT and Development Procedures
As shown in Figure 1, SiMT is an application that uses a smartphone to guide people around SAKURA MACHI Kumamoto. First, when the app starts, the smartphone’s camera function opens, and you can choose your destination, such as exits, multipurpose toilets, elevators, parking lots, and cafes. Second, the application’s starting point is fixed to the information area sign, so if you point your camera at the sign, you will be guided to your destination. (The guide shows an arrow mark on the road surface of the space projected by the camera.) The route search utilises Dijkstra’s algorithm, displaying the shortest route to the user. In this study, although the function is limited because it is a prototype model, all store entrances and exits can be selected as destinations in the future. In addition, not only the shortest route but also people with disabilities, wheelchair users, the elderly, pregnant women, those with strollers, and so on will be given
appropriate routes. It will also provide the safest evacuation route in the event of a disaster.
We performed 3D scans of SAKURA MACHI Kumamoto with the 3D Scanner App on iPad Pro. This application can export 3D scan data (Colourpoint group data, colour image, depth image, camera posture, etc.). The entire building was divided into floors and scanned in 3D, and then Unity, a 3D game development software, was used to integrate the point cloud. Figure 1 shows an example of a second-floor floor scanned in 3D. Points on the route (turning points and destinations) were set on the created 3D map. The shortest path from the spatially recognised starting point to the destination was created and displayed in AR with arrows to stick to the floor. Azure Spatial Anchors, a spatial recognition service, was used to identify the location on the 3D map.

Figure 3. Operation Method & Development Procedure
Future Developments Using the System
The plan of SiMT developed experimentally in this study is shown in Fig. 2. 1; We will develop SiMT, and we develop a system to guide users to evacuation exits in the event of a disaster (Support for Evacuation Activities). In doing so, search for the nearest evacuation door closest to the current location by a safe route, avoiding fire and dangerous areas. 2-1: A disaster prevention plan workshop using Mixed Reality (MR) technology will be held by expanding the device used. MR technology is a technology that projects CG images into the real world by wearing a HoloLens. This technology is used to reproduce the situation in the event of a disaster in large complexes, and a workshop is carried out in anticipation of a disaster. The system will then be expanded to include the elderly, wheelchair users, pregnant women and strollers. 2-2: We are committed to supporting the mobility of people with disabilities. The system finds and suggests the best route when they move around the facility. For example, wheelchair users are guided to elevators, not escalators. 3; Also, a self-driving system for wheelchairs using WHILL Model CR (A device that controls a wheelchair by sending input signals from an external device) is introduced. 4; Finally, the target area will be expanded from a commercial complex to the whole city. Seamless connection between facilities and outdoors and applications can be used to support seamless movement, evacuation in the event of a disaster, and accessibility to public transportation.
Conclusions
Figure 4. Future plans
In this study, we developed a trial application, SiMT, which supports users’ mobility by synthesising CG images of the route from the current location to the destination with images projected on the screen using the camera function of a smartphone large-scale commercial complex. We also conceived future technological developments such as support in times of disaster and mobility support for vulnerable road users. The following is a summary of our findings.
1. We have developed a prototype model of “Smart Info. & Mobility Town (SiMT)”, an application that supports mobility in facilities using AR technology. Conventional navigation applications are usually used outdoors, where GPS is required. However, by using the 3D scanning function, indoor mobility support is also possible. In addition, by using a highly versatile device such as an iPad for 3D scanning, the target area can be easily expanded.
2. By applying the mobility support of this application, it will be possible to provide mobility support according to the situation and the user, such as showing evacuation routes in times of disaster and barrier-free routes for wheelchair users.
3. For future tasks, we will propose ways to expand the target area to the whole city. We will also support both able-bodied people and vulnerable road users so that they can comfortably enjoy their activities in the city.

Acknowledgements

This work was supported by JSPS KAKENHI Grant Numbers 17H01948, 20K20690. I am grateful to the Foundation for Promoting Personal Mobility and Ecological Transportation for collaboration in the early stages of this work.

Conflict of Interests

The author declares no potential conflict of interest was reported by the author.

Endnotes

This paper has been presented at the SPACE International Conference 2021 on City Planning and Urban Design.

References

Tanaka, S., et al. (2014). Development of Smartphone Application for Augmented Reality Navigation. Proceedings of the 76th National Convention, 221-222.

Kurihara, K. & Sato, F. (2014). Marker Based Indoor Navigation System Sharing Location Information.

DICOMO 2014, 1099-1103.

Fujimoto, S., et al. (2016). ORB-SLAM map initialisation improvement using depth. Proceedings of the 76th National Convention, 2016 IEEE International Conference on Image Processing (ICIP). DOI: https://doi.org/10.1109/ICIP.2016.7532359

Liu L., et al. (2021). Indoor navigation supported by the Industry Foundation Classes (IFC): A survey. Automation in Construction, 103436.

Varelas, T., et al. (2020). An AR Indoor Positioning System Based on Anchors. International Journal of Science and Technology, 6(3), 43–57. DOI: https://doi.org/10.20319/mijst.2020.63.4357

Corresponding Author: Masataka Nakahara, Graduate Student, Sojo University, 1714055arc@gmail.com