Home   >   CSC-OpenAccess Library   >    Manuscript Information
When to Ask Participants to Think Aloud: A Comparative Study of Concurrent and Retrospective Think-Aloud Methods
Thamer Alshammari, Obead Alhadreti, Pam J. Mayhew
Pages - 48 - 64     |    Revised - 30-06-2015     |    Published - 31-07-2015
Volume - 6   Issue - 3    |    Publication Date - July / August 2015  Table of Contents
MORE INFORMATION
KEYWORDS
Usability Testing, Think-aloud Studies, Verbal Protocols.
ABSTRACT
This paper presents the results of a study that compared two think-aloud usability testing methods: the concurrent think-aloud and the retrospective think-aloud methods. Data from task performance, testing experience, and usability problems were collected from 30 participants equally distributed between the two think-aloud conditions. The results suggest that while the thinking aloud method had no impact on task performance and testing experience, participants using the concurrent think-aloud method reported a larger number of problems with the test interface than participants using the retrospective think-aloud method. These findings suggest a reason for preferring the concurrent think-aloud method to the retrospective one.
1 Google Scholar 
2 CiteSeerX 
3 refSeek 
4 Scribd 
5 SlideShare 
6 PdfSR 
Cotton, D. and Gresty, K. ‘Reflecting on the think-aloud method for evaluating elearning’. British Journal of Educational Technology, 37 (1), 2006, pp. 45-54.
Dumas, Joseph S., and Janice Redish. A practical guide to usability testing. Intellect Books, 1999.
Eger, N., Ball, L. J., Stevens, R., & Dodd, ‘Cueing retrospective verbal reports in usability testing through eye-movement replay’. In Proceedings of the 21st British HCI Group Annual Conference on People and Computers: HCI... but not as we know it-Volume 1 (pp. 129137). British Computer Society, 2007.
Ericsson, K. A. and Simon, H.A., (1993) Protocol Analysis: Verbal Reports as Data. Revised ed. Cambridge: MIT Press.
Fagan, J. C. Usability studies of faceted browsing: A literature review. Information Technology and Libraries, 2013, 29(2):58–66.
Gray M., Wardle H. ‘Observing gambling behaviour using think aloud and video technology: a methodological review’. NatCen Social Research. Available at: www.natcen.ac.uk. 2013, [Accessed: 22 January 2015).
Gray, W. D., & Salzman, M. C. Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction, 13, 1998 203-261.
Griffiths, M.D. ‘The role of cognitive bias and skill in fruit machine gambling’. British Journal of Psychology, 1994, 85: 351-369.
Haak V, Maaike J., Menno DT de Jong, and Peter Jan Schellens. ‘Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: a methodological comparison.’ Interacting with computers 16.6: 2004, 1153-1170.
Haak V, Menno D, and Peter J. ‘Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue.’ Behaviour & Information Technology 22.5, 2003, 339-351.
Haak, V, (2008). ‘A penny for your thoughts – investigating the validity and reliability of think-aloud protocols for usability testing’(PhD dissertation).
Hertzum, M., Hansen, K.D. and Andersen, H.H.K. ‘Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload?’. Behaviour & Information Technology, 28 (2). 2009, pp. 165-181.
Khajouei, Reza, Arie Hasman, and Monique WM Jaspers. ‘Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system.’ international journal of medical informatics 80.5, 2011, 341-350.
Lewis, C. & Rieman, J. ‘Task-Centered User Interface Design: a Practical Introduction’. 1993. [Online). Available from: http://hcibib.orgltcuidl. [Accessed: 22 November 2014).
Lindgaard, G. and Chattratichart, J. ‘Usability testing: what have we overlooked?’ In Proceedings of the SIGCHI conference on Human factors in computing systems, 2007, pages 1415–1424. ACM.
McDonald, S., Edwards, H. and Zhao, T. ‘Exploring think-alouds in usability testing: an international survey’. IEEE Transactions on Professional Communication, 55(1), 2012, pp.117.
McDonald, Sharon, and Helen Petrie (2013). ‘The effect of global instructions on think- aloud testing.’ Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM.
Molich, R., Ede, M. R., ‘Kaasgaard, K., & Karyukin, B. Comparative usability evaluation’ Behaviour & Information Technology, 23(1), 2004, 65-74.
Nielsen, J. (2000). Why you only need to test with 5 users. Nielsen Norman Group. Department of Computer Science. Machine Learning. Available at: bit.ly/1gpks7w [Accessed 25-04-2014].
Nielsen, J. Usability engineering. 1994, Elsevier.
Peute, L. W., de Keizer, N. F., & Jaspers, M. W. ‘Effectiveness of Retrospective and Concurrent Think Aloud in Formative Usability Testing; Which Method Performs Best?’. Human factors methods in health information systems’ design and, 2013, 65.
Seffah, A., Donyaee, M., Kline, R., and Padda, H. ‘Usability measurement and metrics: A consolidated model.’ Software Quality Journal, 14(2): 2006, 159–178.
Virzi, R.. Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 1992, 34(4):457–468.
Mr. Thamer Alshammari
School of Computing Sciences University of East Anglia Norwich, NR4 7TJ, UK - United Kingdom
t.alshammari@uea.ac.uk
Mr. Obead Alhadreti
School of Computing Sciences University of East Anglia Norwich, NR4 7TJ, UK - United Kingdom
Dr. Pam J. Mayhew
School of Computing Sciences University of East Anglia Norwich, NR4 7TJ, UK - United Kingdom