Why Does Reinforcement Learning Matter?

• The agent
• The environment
• A reward

What you will learn

• Describe how an agent learns from rewards.
• Compare reinforcement learning with learning from labeled data.
• Explain why this way of learning matters for robots and games.

Before you start

You only need the idea of machine learning from the first article of this topic: a computer learning from data.

How does an agent learn from rewards?

An agent learns by trial and error. Think of a child learning to ride a bike: each try shows what works and what does not.

MIT News describes researchers assigning rewards to "good" behaviors that help achieve a goal, and the agent adapting to maximize that reward. This is how the article describes its example, not a guarantee that every agent becomes an expert.

IBM adds a detail about the cycle. If an action obtains a reward signal from the surrounding environment, the agent is encouraged to take that action again when in a similar future state. IBM distinguishes an immediate reward signal from long-term benefit; the agent maximizes cumulative rewards from the environment.

• The agent tries
• The result of the try
• A reward

How is it different from learning with labeled data?

Supervised learning uses labeled examples to provide target outputs, such as categories or values to predict. During the reinforcement-learning phase, the agent receives feedback through reward signals instead of a label for each action saying whether it was correct or incorrect. This describes the feedback used in that learning phase; it does not mean every phase of a system uses no labeled data.

In IBM's description of an agent without manually labeled input data guiding it, the agent explores its environment and tries actions to discover which receive rewards.

The focus is different too. Whereas traditional machine learning has mostly focused on pattern mining, reinforcement learning shifts the focus to decision making.

• Labeled examples
• The agent
• A reward

Why does this way of learning matter?

It matters because it lets machines learn to act, not only to spot patterns. Stanford University says AI needs autonomous systems that learn to make good decisions, and reinforcement learning is one powerful paradigm for doing so.

The idea is useful in many places. A Stanford course describes it as relevant to an enormous range of tasks, including robotics, game playing, consumer modeling and healthcare.

A famous case is a game. A Stanford report says the success of AlphaGo, a program that beat the human Go champion, was due in large part to reinforcement learning. The report says the agent was initialized with a human-expert database, then refined by playing many games against itself and applying reinforcement learning.

Robots may gain too. The AI100 report says reinforcement learning may help bridge the gap left by the lack of large labeled data sets in robotics. This approach also requires systems to explore safely without committing errors that harm the system itself or others.

• Robotics
• Game playing
• Healthcare

Real-world example

Think of teaching a dog a new trick with treats. The dog tries things, gets a treat for the right move and repeats it. An agent in reinforcement learning follows the same loop, with a reward in place of a treat. No computer is needed for this thought experiment.

Common misunderstandings

The machine is told the right answer each time. In the reinforcement learning described here, the agent receives feedback through reward signals instead of labeled examples that identify each behavior as correct or incorrect.

Reinforcement learning is brand new. The Stanford report says it has existed for several decades as a framework for experience-driven sequential decision-making. Deep learning then gave it a "shot in the arm".

Only game programs use it. The Stanford course page lists robotics, consumer modeling and healthcare as well as game playing.

Frequently asked questions

What is an agent in reinforcement learning?

IBM describes reinforcement learning as a process in which autonomous agents learn to make decisions by interacting with their environment. Robots and self-driving cars are examples of such autonomous agents.

Does reinforcement learning need labeled data?

During the reinforcement-learning phase, the agent receives reward signals and explores the environment instead of being given a correct/incorrect label for each action. Other phases of a system can still use labeled data: the Stanford report says AlphaGo was initialized with an expert database before self-play and reinforcement learning.

Why did deep learning matter for reinforcement learning?

The Stanford report says the advent of deep learning provided reinforcement learning with a "shot in the arm". AlphaGo's success is its main example.

Summary

• An agent learns by trial and error, adapting its behavior to maximize cumulative reward.
• Unlike supervised learning, which uses labeled target outputs such as categories or values, reinforcement learning uses reward signals as feedback for actions.
• It matters because it teaches machines to make decisions, in areas such as robotics and game playing.