Camera pose estimation is an enabling technology for example for autonomous robots, augmented reality, virtual reality, mixed reality and interactive image sharing service.
“My PhD thesis targets on two research problems. First, how to efficiently and robustly estimate the camera pose of a query image with a map that contains street-view snapshots and point clouds. Second, given the estimated camera pose of a query image, how to create meaningful and intuitive applications with the map data,” Junsheng Fu clarifies.
With the increased popularity of mobile camera phones and emergence of smart glasses, the camera sensor has become one of the most ubiquitous sensors. There are billions of images generated each day. Visual localization means performing the localization task by estimating the position and orientation (camera pose) of a given image. What are the benefits of knowing the camera pose of a given image? In his research work Junsheng Fu explains how the camera pose estimation contributes to autonomous vehicles, image sharing service and mixed and augmented reality.
Autonomous vehicles
To let a vehicle drive autonomously, the autonomous driving system needs to be able to localize the vehicle in centimetre level. Camera is a popular sensor, and it is widely used in different kinds of autonomous vehicles.
“Once the camera pose can be estimated, the vehicle position and orientation can be indirectly computed since the camera is usually fixed on the vehicle,” Fu says.
Image sharing service
Capturing images with a camera phone and sharing photos with friends on social media are gradually becoming parts of our daily activities. Do you have the experience of seeing an interesting photo of a tourist attraction either from your friends or online, and you would like to see more of the surroundings? For example, you would like to see what is on the left, what is on the right or even what is on the opposite side of the captured scene.
“This can be done by utilizing the camera pose estimation and a 3D map. With the estimated camera pose, we know exactly where the image was captured in the 3D map. Then, the user gains the possibility to explore the surroundings by utilizing the camera pose together with the point of interest information in the 3D map. Therefore, the ordinary image capturing and sharing experience can be greatly enriched by leveraging the estimated camera pose,” he explains.
Fu has developed an interactive image sharing application which can be seen on video.
Mixed reality and augmented reality
One key challenge for both mixed reality and augmented reality is how to efficiently compute the camera pose. The traditional real-time approach is the inertial sensor-based approach, e.g., GPS and IMU, but the purely inertial sensor-based approaches can be sensitive to environmental noise.
“Image-based camera pose estimation can strengthen the sensor-based approach. Once the camera pose is successfully estimated, many intuitive mixed reality applications become possible,” Fu states.
Fu has also developed an intuitive augmented reality application for video playback.
The doctoral dissertation of MSc (Tech) Junsheng Fu in the field of Computer Vision titled Camera Pose Estimation from Street-view Snapshots and Point Clouds will be publicly examined in the Faculty of Information Technology and Communication Sciences at Tampere University at 12 o'clock on 3rd June 2022. The Opponent will be Dr. Arto Kaarna from LUT University, Finland. The Custos will be Professor Joni Kämäräinen from the Faculty of Information Technology and Communication Sciences at Tampere University.
The event can be followed only via remote connection (Zoom).
The dissertation is available online at the http://urn.fi/URN:ISBN:978-952-03-2426-1