Home > CSC-OpenAccess Library > Manuscript Information
EXPLORE PUBLICATIONS BY COUNTRIES |
 |
 |
EUROPE |
|
MIDDLE EAST |
|
ASIA |
|
AFRICA |
| ............................. | |
|
United States of America |
|
United Kingdom |
|
Canada |
|
Australia |
|
Italy |
|
France |
|
Brazil |
|
Germany |
|
Malaysia |
|
Turkey |
|
China |
|
Taiwan |
|
Japan |
|
Saudi Arabia |
|
Jordan |
|
Egypt |
|
United Arab Emirates |
|
India |
|
Nigeria |
Learning of Soccer Player Agents Using a Policy Gradient Method : Coordination Between Kicker and Receiver During Free Kicks
Harukazu Igarashi, Koji Nakamura, Seiji Ishihara
Pages - 1 - 13 | Revised - 31-03-2011 | Published - 04-04-2011
MORE INFORMATION
KEYWORDS
Reinforcement Learning, Soccer Simulation, Policy Gradient, Multiagent
ABSTRACT
As an example of multi-agent learning in soccer games of the RoboCup 2D Soccer Simulation League, we dealt with a learning problem between a kicker and a receiver when a direct free kick is awarded just outside the opponent's penalty area. We propose how to use a heuristic function to evaluate an advantageous target point for safely sending/receiving a pass and scoring. The heuristics include an interaction term between a kicker and a receiver to intensify their coordination. To calculate the interaction term, we let a kicker/receiver agent have a receiver's/kicker's action decision model to predict a receiver's/kicker's action. Parameters in the heuristic function can be learned by a kind of reinforcement learning called the policy gradient method. Our experiments show that if the two agents do not have the same type of heuristics, the interaction term based on prediction of a teammate's decision model leads to learning a master-servant relation between a kicker and a receiver, where a receiver is a master and a kicker is a servant.
| 1 | Leskelä, C. L. H. (2014). Learning for RoboCup Soccer. |
| 2 | Oliveira, M. A. P. D. (2012). High level coordination and decision making of a simulated robotic soccer team. |
| 3 | Igarashi, H., Fukuoka, H., & Ishihara, S. (2011). Policy Gradient Approach for Learning of Soccer Player Agents. In Intelligent Control and Computer Engineering (pp. 137-148). Springer Netherlands. |
| 4 | Almeida, F., Lau, N., & Reis, L. P. (2010). A Survey on Coordination Methodologies for Simulated Robotic Soccer Teams. In MALLOW. |
| 5 | Igarashi, H., Fukuoka, H., & Ishihara, S. (2010). Learning of soccer player agents using a policy gradient method: pass selection. In Proceedings of the International MultiConference of Engineers and Computer Scientists (Vol. 1). |
| 6 | Igarashi, H., Masaki, J., Suzuki, T., Tonegawa, N., Sano, N., Imaizumi, T., ... & Fukuoka, H. Fifty-Storms: Team Description 2009. |
| 1 | Google Scholar  |
| 2 | CiteSeerX |
| 3 | refSeek |
| 4 | Scribd |
| 5 | SlideShare |
| 6 | PdfSR |
| G. Weiss, S. Sen, editors. “Adaption and Learning in Multi-agent Systems”, Springer-Verlag, Germany, 1996 | |
| H. Igarashi, S. Ishihara and M. Kimura. “Reinforcement Learning in Non-Markov Decision Processes -Statistical Properties of Characteristic Eligibility-”, IEICE Transactions on Information and Systems, J90-D(9):2271-2280, 2007 (in Japanese). This paper is translated into English and included in The Research Reports of Shibaura Institute of Technology, Natural Sciences and Engineering (ISSN 0386-3115), 52(2): 1-7, 2008 | |
| H. Kimura, K. Miyazaki and S. Kobayashi. “Reinforcement Learning in POMDPs with Function Approximation”. In Proceedings of the 14th International Conference on Machine Learning, pp. 152-160, 1997 | |
| J. R. Kok. “UvA Trilearn”, http://remote.science.uva.nl/~jellekok/robocup/ ( access time: 05.01.2011 ) | |
| L. P. Kaelbling, M. L. Littman and A. W. Moore. “Reinforcement Learning: A Survey”. Journal of Artificial Intelligence Research, 4:237-285, 1996 | |
| L. Peshkin, K. E. Kim, N. Meuleau and L. P. Kaelbling. “Learning to cooperate via policy search”. In Proceedings of 16th Conference on Uncertainty in Artificial Intelligence (UAI2000), pp. 489-496, 2000 | |
| M. Riedmiller, T. Gabel. “On Experiences in a Complex and Competitive Gaming Domain – Reinforcement Learning Meets RoboCup–”. In Proceedings of the 2007 IEEE Symposium on Computational Intelligence and Games (CIG2007), pp.17-23, 2007 | |
| P. Stone, G. Kuhlmann, M. E. Taylor and Y. Liu. “Keepaway Soccer: From Machine Learning Testbed to Benchmark”. In A. Bredenfeld, A. Jacoff, I. Noda and Y. Takahashi, editors, “RoboCup 2005: Robot Soccer World Cup IX”, pp. 93-105, Springer-Verlag, New York, 2006 | |
| R. J. Williams. “Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning”. Machine Learning, 8(3-4): 229-256, 1992 | |
| R. S. Sutton, A. G. Barto.”Reinforcement Learning”, The MIT Press, 1998 | |
| RoboCup 2D Soccer Simulation League, http://sourceforge.net/apps/mediawiki/sserver/ index.php?title=Main_Page ( access time: 14.03.2011 ) | |
| S. Arai, K. Miyazaki. “Learning Robust Policies for Uncertain and Stochastic Multi-agent Domains”. In Proceedings of 7th International Symposium on Artificial Life and Robotics, pp.179-182, 2002 | |
| S. Ishihara, H. Igarashi. “Applying the Policy Gradient Method to Behavior Learning in Multiagent Systems: The Pursuit Problem”, Systems and Computers in Japan, 37(10):101- 109, 2006 | |
| S. Kalyanakrishnan, Y. Liu and P. Stone. “Half Field Offense in RoboCup Soccer –A Multiagent Reinforcement Learning Case Study”. In G. Lakemeyer, E. Sklar, D. G. Sorrenti, T. Takahashi, editors, “RoboCup-2006: Robot Soccer World Cup X”, pp.72-85, Springer-Verlag, Berlin, 2007 | |
| S. Sen, G. Weiss, “Learning in Multiagent Systems”. In G. Weiss, editor, “Multiagent Systems: A Modern Approach to Distributed Artificial Intelligence”, pp. 259-208, The MIT Press, 1999 | |
| T. Andou. “Refinement of Soccer Agents' Positions Using Reinforcement Learning”. In H. Kitano, editor, “RoboCup-97: Robot Soccer World Cup I”, pp. 373-388, Springer-Verlag, Berlin, 1998 | |
| W.S. Lovejoy. “A survey of algorithmic methods for partially observed Markov decision processes”. Annals of Operations Research, 28(1): 47-65, 1991 | |
Professor Harukazu Igarashi
Shibaura Institute of Technology - Japan
arashi50@sic.shibaura-it.ac.jp
Mr. Koji Nakamura
Shibaura Institute of Technology - Japan
Professor Seiji Ishihara
Kinki University - Japan
|
|
|
|
| View all special issues >> | |
|
|
