4 deep thoughts on deep learning in 2022

We are leaving behind another year of exciting developments in artificial intelligence (AI) deep learning – a year full of remarkable progress, controversy and, of course, disputes. As we wrap up 2022 and prepare to embrace what 2023 has in store, here are some of the most notable overarching trends that marked this year in deep learning.

1. Scale remains an important factor

One theme that has remained constant in deep learning in recent years is the urge to create larger neural networks. The availability of compute resources allows for the scaling of neural networks, specialized AI hardware, large data sets, and the development of scale-friendly architectures such as the transformer fashion model-.

Today, companies are getting better results by scaling neural networks to larger sizes. In the past year, DeepMind announced: Gopher, a large language model with 280 billion parameters (LLM); Google has announced Pathways Language Model (Palm), with 540 billion parameters, and Generalist Language Model (GLaM), with up to 1.2 trillion parameters; and Microsoft and Nvidia have the Megatron-Turing NLGan LLM of 530 billion parameters.

One of the interesting aspects of scale is: emerging powers, where larger models manage to perform tasks that were impossible with smaller ones. This phenomenon is especially intriguing in LLMs, where models show promising results across a wider range of tasks and benchmarks as they grow in size.

However, it is worth noting that some fundamental problems of deep learning remain unsolved even in the largest models (more on this later).

2. Unsupervised learning continues to pay off

Many successful deep learning applications require people to label training examples, also known as guided learning. But most of the data available on the web doesn’t come with the clean labels needed for supervised learning. And annotating data is expensive and slow, creating bottlenecks. This is why researchers have long sought advances in unsupervised learning, training deep learning models without the need for human annotated data.

Huge progress has been made in this area in recent years, especially in LLMs, which are usually trained on large sets of raw data collected over the Internet. While LLMs continued to advance in 2022, we also saw other trends in unsupervised learning techniques gain momentum.

For example, there were phenomenal advances in text-to-image models this year. Models like OpenAI’s DALL-E 2, Google’s imageand stability AIs Stable diffusion have shown the power of unsupervised learning. Unlike older text-to-image models, which required well-annotated pairs of images and descriptions, these models use large datasets of loose captions that already exist on the web. The sheer size of their training datasets (which is only possible because no manual labeling is required) and the variability of the captioning schemes enable these models to find all kinds of intricate patterns between textual and visual information. As a result, they are much more flexible in generating images for different descriptions.

3. Multimodality takes big steps

Text-to-image generators have another interesting feature: they combine multiple data types into one model. By being able to handle multiple modalities, deep learning models can take on much more complicated tasks.

Multimodality is very important for the kind of intelligence found in humans and animals. For example, if you see a tree and hear the wind rustle in its branches, your mind can quickly relate them. Likewise, when you see the word “tree,” you can quickly recall the image of a tree, remember the smell of pine after a rain shower, or recall other experiences you’ve had before.

It is clear that multimodality has played an important role in making deep learning systems more flexible. This was perhaps best portrayed by DeepMind’s gato, a deep learning model trained on a variety of data types, including images, text, and proprioception data. Gato performed well in multiple tasks, including captioning images, interactive dialogues, controlling a robotic arm, and playing games. This is in contrast to classic deep learning models, which are designed to perform a single task.

Some researchers have taken the idea so far as to propose that a system like Gato is all we need to achieve artificial general intelligence (AGI). Although many scientists do not agree with this view, it is certain that multimodality has produced important results for deep learning.

4. Fundamental problems with deep learning persist

Despite the impressive results of deep learning, some problems in the field remain unsolved. Among them are causalitycomposition, common sense, reasoning, planning, intuitive physics and abstraction and making analogies.

These are some of the mysteries of intelligence that are still being studied by scientists in various fields. Pure scale and data-based deep learning approaches have helped make step-by-step progress on some of these problems, while providing no definitive solution.

For example, larger LLMs can maintain coherence and consistency over longer stretches of text. But she fail on tasks that require meticulous step-by-step reasoning and planning.

Similarly, text-to-image generators create stunning images, but make basic mistakes when asked to draw images that require composition or have complex descriptions.

These challenges are discussed and explored by several scientists, including some of the pioneers of deep learning. Prominent among them is Yann LeCun, the Turing Award-winning inventor of convolutional neural networks (CNN), who recently wrote a lengthy essay on the limits of LLMs that learn from text only. LeCun is researching a deep learning architecture that learns world models and can address some of the challenges currently facing the field.

Deep learning has come a long way. But the more progress we make, the more we become aware of the challenges of creating truly intelligent systems. Next year is sure to be as exciting as this one.