Harmonising Observations and Underlying Principles for Visual Data Association

Introduction

Visual data association aims to find task-specific mappings involving visual data. Two significant examples are the mapping of physics models to complex scenes for planning overtaking manoeuvres in autonomous driving, or matching collections of 3D shapes for medical analysis.

Challenges in Visual Data Association

Despite the high relevance of visual data association, its progress has not kept pace with the revolutionary developments fueled by recent deep learning advances. Existing data association machinery lacks theoretical guarantees, such as:

• Global optimality
• Structure, such as geometric consistency in 3D shape matching

These guarantees are critical for high-stakes settings, or the machinery suffers from poor scalability. Moreover, current procedures fall short of understanding complex interconnections across different observable entities, such as collections of objects or scenes.

Vision of Harmony

The vision of Harmony is to tackle these shortcomings by harmonizing the complex interconnections between observable entities and underlying fundamental principles, including:

• Geometry
• Physics

This research direction is challenging, largely unexplored, and will require breaking substantially new ground at conceptual, algorithmic, and practical levels simultaneously.

Organised Challenges

Harmony is organized into four complementary challenges:

1. Challenge A: Addresses global optimality and scalability for 3D shape matching.
2. Challenge B: Addresses structure and dynamics inference from static images.
3. Challenge C: Addresses non-linear synchronization in data collections defined over graphs.
4. Challenge D: Will exploit synergies and cross-fertilize insights across Harmony.

Overall Impact

Overall, Harmony will benefit both researchers and practitioners by providing solutions to more complex tasks in practically relevant settings, such as:

• Geometrically consistent medical shape analysis
• Physics-based scene understanding