Home   >   CSC-OpenAccess Library   >    Manuscript Information
Full Text Available

This is an Open Access publication published under CSC-OpenAccess Policy.
Publications from CSC-OpenAccess Library are being accessed from over 74 countries worldwide.
Learning of Soccer Player Agents Using a Policy Gradient Method : Coordination Between Kicker and Receiver During Free Kicks
Harukazu Igarashi, Koji Nakamura, Seiji Ishihara
Pages - 1 - 13     |    Revised - 31-03-2011     |    Published - 04-04-2011
Volume - 2   Issue - 1    |    Publication Date - March / April 2011  Table of Contents
Reinforcement Learning, Soccer Simulation, Policy Gradient, Multiagent
As an example of multi-agent learning in soccer games of the RoboCup 2D Soccer Simulation League, we dealt with a learning problem between a kicker and a receiver when a direct free kick is awarded just outside the opponent's penalty area. We propose how to use a heuristic function to evaluate an advantageous target point for safely sending/receiving a pass and scoring. The heuristics include an interaction term between a kicker and a receiver to intensify their coordination. To calculate the interaction term, we let a kicker/receiver agent have a receiver's/kicker's action decision model to predict a receiver's/kicker's action. Parameters in the heuristic function can be learned by a kind of reinforcement learning called the policy gradient method. Our experiments show that if the two agents do not have the same type of heuristics, the interaction term based on prediction of a teammate's decision model leads to learning a master-servant relation between a kicker and a receiver, where a receiver is a master and a kicker is a servant.
CITED BY (6)  
1 Leskelä, C. L. H. (2014). Learning for RoboCup Soccer.
2 Oliveira, M. A. P. D. (2012). High level coordination and decision making of a simulated robotic soccer team.
3 Igarashi, H., Fukuoka, H., & Ishihara, S. (2011). Policy Gradient Approach for Learning of Soccer Player Agents. In Intelligent Control and Computer Engineering (pp. 137-148). Springer Netherlands.
4 Almeida, F., Lau, N., & Reis, L. P. (2010). A Survey on Coordination Methodologies for Simulated Robotic Soccer Teams. In MALLOW.
5 Igarashi, H., Fukuoka, H., & Ishihara, S. (2010). Learning of soccer player agents using a policy gradient method: pass selection. In Proceedings of the International MultiConference of Engineers and Computer Scientists (Vol. 1).
6 Igarashi, H., Masaki, J., Suzuki, T., Tonegawa, N., Sano, N., Imaizumi, T., ... & Fukuoka, H. Fifty-Storms: Team Description 2009.
1 Google Scholar 
2 CiteSeerX 
3 refSeek 
4 Scribd 
5 SlideShare 
6 PdfSR 
1 G. Weiss, S. Sen, editors. “Adaption and Learning in Multi-agent Systems”, Springer-Verlag, Germany, 1996
2 S. Sen, G. Weiss, “Learning in Multiagent Systems”. In G. Weiss, editor, “Multiagent Systems: A Modern Approach to Distributed Artificial Intelligence”, pp. 259-208, The MIT Press, 1999
3 RoboCup 2D Soccer Simulation League, http://sourceforge.net/apps/mediawiki/sserver/ index.php?title=Main_Page ( access time: 14.03.2011 )
4 S. Arai, K. Miyazaki. “Learning Robust Policies for Uncertain and Stochastic Multi-agent Domains”. In Proceedings of 7th International Symposium on Artificial Life and Robotics, pp.179-182, 2002
5 W.S. Lovejoy. “A survey of algorithmic methods for partially observed Markov decision processes”. Annals of Operations Research, 28(1): 47-65, 1991
6 R. S. Sutton, A. G. Barto.”Reinforcement Learning”, The MIT Press, 1998
7 L. P. Kaelbling, M. L. Littman and A. W. Moore. “Reinforcement Learning: A Survey”. Journal of Artificial Intelligence Research, 4:237-285, 1996
8 H. Kimura, K. Miyazaki and S. Kobayashi. “Reinforcement Learning in POMDPs with Function Approximation”. In Proceedings of the 14th International Conference on Machine Learning, pp. 152-160, 1997
9 T. Andou. “Refinement of Soccer Agents' Positions Using Reinforcement Learning”. In H. Kitano, editor, “RoboCup-97: Robot Soccer World Cup I”, pp. 373-388, Springer-Verlag, Berlin, 1998
10 M. Riedmiller, T. Gabel. “On Experiences in a Complex and Competitive Gaming Domain – Reinforcement Learning Meets RoboCup–”. In Proceedings of the 2007 IEEE Symposium on Computational Intelligence and Games (CIG2007), pp.17-23, 2007
11 P. Stone, G. Kuhlmann, M. E. Taylor and Y. Liu. “Keepaway Soccer: From Machine Learning Testbed to Benchmark”. In A. Bredenfeld, A. Jacoff, I. Noda and Y. Takahashi, editors, “RoboCup 2005: Robot Soccer World Cup IX”, pp. 93-105, Springer-Verlag, New York, 2006
12 S. Kalyanakrishnan, Y. Liu and P. Stone. “Half Field Offense in RoboCup Soccer –A Multiagent Reinforcement Learning Case Study”. In G. Lakemeyer, E. Sklar, D. G. Sorrenti, T. Takahashi, editors, “RoboCup-2006: Robot Soccer World Cup X”, pp.72-85, Springer-Verlag, Berlin, 2007
13 R. J. Williams. “Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning”. Machine Learning, 8(3-4): 229-256, 1992
14 H. Igarashi, S. Ishihara and M. Kimura. “Reinforcement Learning in Non-Markov Decision Processes -Statistical Properties of Characteristic Eligibility-”, IEICE Transactions on Information and Systems, J90-D(9):2271-2280, 2007 (in Japanese). This paper is translated into English and included in The Research Reports of Shibaura Institute of Technology, Natural Sciences and Engineering (ISSN 0386-3115), 52(2): 1-7, 2008
15 S. Ishihara, H. Igarashi. “Applying the Policy Gradient Method to Behavior Learning in Multiagent Systems: The Pursuit Problem”, Systems and Computers in Japan, 37(10):101- 109, 2006
16 L. Peshkin, K. E. Kim, N. Meuleau and L. P. Kaelbling. “Learning to cooperate via policy search”. In Proceedings of 16th Conference on Uncertainty in Artificial Intelligence (UAI2000), pp. 489-496, 2000
17 J. R. Kok. “UvA Trilearn”, http://remote.science.uva.nl/~jellekok/robocup/ ( access time: 05.01.2011 )
Professor Harukazu Igarashi
Shibaura Institute of Technology - Japan
Mr. Koji Nakamura
Shibaura Institute of Technology - Japan
Professor Seiji Ishihara
Kinki University - Japan