3D scene understanding in two glances

Introduction

The human mind understands visual scenes. We can usually tell what objects are present in a scene, we can imagine what the hidden parts of objects look like, and we can imagine what it would look like if we or an object moved.

Segmentation in Visual Understanding

The first step of visual scene understanding is segmentation, in which our brain tries to infer which parts of the scene belong to which objects. Adults can do this in photographs – but photographs are not how we learned to see as infants. We learned to see by moving around in a 3D world.

Information Processing

The way that scenes project into our eyes, how light is affected by the optics of our eyes, how our photoreceptors sample the light, and how we move our eyes all provide rich information about our environment. However, we do not know how adults combine all this information to perceive segmented scenes, and we do not know how infants learn this combination.

Challenges in Research

Two reasons for this are:

1. Standard visual display devices cannot precisely mimic these factors.
2. It is unethical to manipulate these factors in human infants.

Project Goals

The goals of this project are to:

• Understand how adults use the rich information present in active 3D vision to perform segmentation.
• Understand how this is learned.

Methodology

We will develop a new display device and experimental methods to study how adults segment scenes when realistic visual information is available. Additionally, we will develop groundbreaking new technologies using advanced computer graphics and machine learning to simulate the inputs to the visual system from early development to adulthood.

Experiments and Comparisons

We will then conduct in silico experiments in artificial neural networks to understand segmentation learning by systematically restricting or manipulating different factors. We will compare the learned behaviors of different artificial networks to adults performing segmentation during active exploration of 3D scenes.

Conclusion

We will use similarities and differences to better understand a fundamental puzzle of perception: how the mind makes sense of scenes.