|
| Learning of Soccer Player Agents Using a Policy Gradient Method : Coordination Between Kicker and Receiver During Free Kicks
|
|
Full
text: |
 PDF(830.6KB) |
|
|
Source |
International Journal of Artificial Intelligence and Expert Systems (IJAE) |
|
Table of Contents |
|
|
Download
Complete Issue PDF(1.18MB) |
|
Volume: 2 Issue: 1 |
| |
Pages: 1-22 |
|
Publication
Date: March / April 2011 |
|
ISSN
(Online): 2180-124X |
|
|
|
|
|
Pages |
1 - 13 |
|
Author(s) |
|
|
|
Published
Date |
04-04-2011 |
|
Publisher |
CSC
Journals, Kuala Lumpur,
Malaysia |
|
ADDITIONAL
INFORMATION |
| Keywords Abstract References Cited by Related Articles Collaborative
Colleague |
| |
|
| |
KEYWORDS: Reinforcement Learning, Soccer Simulation, Policy Gradient, Multiagent |
|
|
| |
|
|
| This Manuscript is indexed in the following databases/websites:- |
|
| 1. Scribd |
| 2. Docstoc |
| 3. Google Scholar |
| |
|
| |
|
|
| As an example of multi-agent learning in soccer games of the RoboCup 2D Soccer Simulation League, we dealt with a learning problem between a kicker and a receiver when a direct free kick is awarded just outside the opponent's penalty area. We propose how to use a heuristic function to evaluate an advantageous target point for safely sending/receiving a pass and scoring. The heuristics include an interaction term between a kicker and a receiver to intensify their coordination. To calculate the interaction term, we let a kicker/receiver agent have a receiver's/kicker's action decision model to predict a receiver's/kicker's action. Parameters in the heuristic function can be learned by a kind of reinforcement learning called the policy gradient method. Our experiments show that if the two agents do not have the same type of heuristics, the interaction term based on prediction of a teammate's decision model leads to learning a master-servant relation between a kicker and a receiver, where a receiver is a master and a kicker is a servant. |
| |
|
| |
|
| |
| 1 |
G. Weiss, S. Sen, editors. “Adaption and Learning in Multi-agent Systems”, Springer-Verlag, Germany, 1996
|
|
|
| 2 |
S. Sen, G. Weiss, “Learning in Multiagent Systems”. In G. Weiss, editor, “Multiagent Systems: A Modern Approach to Distributed Artificial Intelligence”, pp. 259-208, The MIT Press, 1999
|
|
|
| 3 |
RoboCup 2D Soccer Simulation League, http://sourceforge.net/apps/mediawiki/sserver/ index.php?title=Main_Page ( access time: 14.03.2011 )
|
|
|
| 4 |
S. Arai, K. Miyazaki. “Learning Robust Policies for Uncertain and Stochastic Multi-agent Domains”. In Proceedings of 7th International Symposium on Artificial Life and Robotics, pp.179-182, 2002
|
|
|
| 5 |
W.S. Lovejoy. “A survey of algorithmic methods for partially observed Markov decision processes”. Annals of Operations Research, 28(1): 47-65, 1991
|
|
|
| 6 |
R. S. Sutton, A. G. Barto.”Reinforcement Learning”, The MIT Press, 1998
|
|
|
| 7 |
L. P. Kaelbling, M. L. Littman and A. W. Moore. “Reinforcement Learning: A Survey”. Journal of Artificial Intelligence Research, 4:237-285, 1996
|
|
|
| 8 |
H. Kimura, K. Miyazaki and S. Kobayashi. “Reinforcement Learning in POMDPs with Function Approximation”. In Proceedings of the 14th International Conference on Machine Learning, pp. 152-160, 1997
|
|
|
| 9 |
T. Andou. “Refinement of Soccer Agents' Positions Using Reinforcement Learning”. In H. Kitano, editor, “RoboCup-97: Robot Soccer World Cup I”, pp. 373-388, Springer-Verlag, Berlin, 1998
|
|
|
| 10 |
M. Riedmiller, T. Gabel. “On Experiences in a Complex and Competitive Gaming Domain – Reinforcement Learning Meets RoboCup–”. In Proceedings of the 2007 IEEE Symposium on Computational Intelligence and Games (CIG2007), pp.17-23, 2007
|
|
|
| 11 |
P. Stone, G. Kuhlmann, M. E. Taylor and Y. Liu. “Keepaway Soccer: From Machine Learning Testbed to Benchmark”. In A. Bredenfeld, A. Jacoff, I. Noda and Y. Takahashi, editors, “RoboCup 2005: Robot Soccer World Cup IX”, pp. 93-105, Springer-Verlag, New York, 2006
|
|
|
| 12 |
S. Kalyanakrishnan, Y. Liu and P. Stone. “Half Field Offense in RoboCup Soccer –A Multiagent Reinforcement Learning Case Study”. In G. Lakemeyer, E. Sklar, D. G. Sorrenti, T. Takahashi, editors, “RoboCup-2006: Robot Soccer World Cup X”, pp.72-85, Springer-Verlag, Berlin, 2007
|
|
|
| 13 |
R. J. Williams. “Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning”. Machine Learning, 8(3-4): 229-256, 1992
|
|
|
| 14 |
H. Igarashi, S. Ishihara and M. Kimura. “Reinforcement Learning in Non-Markov Decision Processes -Statistical Properties of Characteristic Eligibility-”, IEICE Transactions on Information and Systems, J90-D(9):2271-2280, 2007 (in Japanese). This paper is translated into English and included in The Research Reports of Shibaura Institute of Technology, Natural Sciences and Engineering (ISSN 0386-3115), 52(2): 1-7, 2008
|
|
|
| 15 |
S. Ishihara, H. Igarashi. “Applying the Policy Gradient Method to Behavior Learning in Multiagent Systems: The Pursuit Problem”, Systems and Computers in Japan, 37(10):101- 109, 2006
|
|
|
| 16 |
L. Peshkin, K. E. Kim, N. Meuleau and L. P. Kaelbling. “Learning to cooperate via policy search”. In Proceedings of 16th Conference on Uncertainty in Artificial Intelligence (UAI2000), pp. 489-496, 2000
|
|
|
| 17 |
J. R. Kok. “UvA Trilearn”, http://remote.science.uva.nl/~jellekok/robocup/ ( access time: 05.01.2011 )
|
|
|
| |
|
| |
|
| |
| |
|
| |
|
| |
| |
|
| |
|
| |
|
| Harukazu Igarashi : Colleagues
|
|
| Koji Nakamura : Colleagues
|
|
| Seiji Ishihara : Colleagues
|
|
