Scaling Reinforcement Learning toward RoboCup Soccer

Peter Stone
University of Texas Austin

RoboCup simulated soccer presents many challenges to reinforcement
learning methods, including the presence of multiple agents learning
simultaneously.  In addition, it involves a large state space, hidden
and uncertain state, and long and variable delays in the effects of
actions.  I will describe our application of episodic SMDP
Sarsa(lambda) with linear tile-coding function approximation and
variable lambda to learning higher-level decisions in a keepaway
subtask of RoboCup soccer.  In keepaway, one team, ``the keepers,''
tries to keep control of the ball for as long as possible despite the
efforts of ``the takers.''  The keepers learn individually when to
hold the ball and when to pass to a teammate, while the takers learn
when to charge the ball-holder and when to cover possible passing
lanes.  Our agents learned policies that significantly out-performed a
range of benchmark policies.  We demonstrate the generality of our
approach by applying it to a number of task variations including
different field sizes and different numbers of players on each team.

(joint work with Richard Sutton)