Today’s AI landscape is very different from when I started. As a young graduate student at Indiana University (and later University of Alberta), the introduction of Deep Q-Networks [cite] and AlphaGo [cite] were fresh and we were in for half a decade of research at the intersection of deep learning and reinforcement learning. Today, the field is dominated by large language models, something I would have thought unimaginable back in 2015, and honestly I caught myself flatfooted in being able to transition to this new world of foundation models. Fortunately, the revolution of LLMs, while impressive, is looking quite like the revolution of AlphaGO and related works from deep mind, where the methods are useful under a narrow set of circumstances (i.e., large datasets that can be trained and used for inference in non-time critical scenarios).

What is the future of RL?

There is no salient answer to this question. Perhaps, it is just as the cherry on-top for large foundation models as the pioneers of self-supervised learning have stated [cite], it could be the pathway to AGI as those with a vested interest in RL have put forward [cite], or maybe it is a relic of the past that should be abandoned as others have stated [cite]. Anyone who states confidently they know the answer almost definitely does not, no matter if they have been right in the past. No against my better judgement, I’m going to make some general statements on my beliefs about the future of the field.

Reinforcement learning has two trajectories that will co-exist for some time before an eventual unification.

1. RL for foundation models and general intelligence
2. RL for specific

What does RL have that other ML framings do not

The crux of today’s machine learning is that of datasets. Large language models and foundation models need massive amounts of data to work, and need a large amount of quasi-labeled data to finetune for the chat interface [cite]. If you have a dataset, you can learn almost anything (albeit, it must be things which are captured in the data distribution of your dataset)! But once you no longer have a dataset, or a dataset is too costly or impossible to generate, we have a problem.

Reinforcement learning, from its inception, confronts the need for data head-on; the agent generates its own data through acting in an environment. This process is referred to as exploration.