DEep COgnition Learning for LAnguage GEneration

Introduction

In recent years, transformer-based deep learning models such as BERT or GPT-3 have led to impressive results in many natural language processing (NLP) tasks, exhibiting transfer and few-shot learning capabilities.

Limitations of Current Models

However, despite faring well in benchmarks, current deep learning models for NLP often fail badly in the wild. The main issues include:

• Poor out-of-domain generalization
• Inability to exploit contextual information
• Poor calibration
• Non-traceable memory

These limitations stem from their monolithic architectures, which are good for perception but unsuitable for tasks requiring higher-level cognition.

Project Goals

In this project, I attack these fundamental problems by bringing together tools and ideas from various fields:

1. Machine Learning
2. Sparse Modeling
3. Information Theory
4. Cognitive Science

This interdisciplinary approach will involve several key strategies:

Utility-Guided Controlled Generation

First, I will use uncertainty and quality estimates for utility-guided controlled generation. This will combine the control mechanism with the efficient encoding of contextual information and integration of multiple modalities.

Sparse and Structured Memory Models

Second, I will develop sparse and structured memory models, along with attention descriptive representations aimed at conscious processing.

Mathematical Models for Sparse Communication

Third, I will build mathematical models for sparse communication that reconcile discrete and continuous domains. These models will support end-to-end differentiability and enable a shared workspace where multiple modules and agents can communicate.

Application of Innovations

I will apply the innovations mentioned above to highly challenging language generation tasks, including:

1. Machine translation
2. Open dialogue
3. Story generation

Collaboration and Impact

To reinforce interdisciplinarity and maximize technological impact, collaborations are planned with cognitive scientists and with a scale-up company in the crowd-sourcing translation industry.