Home > CSC-OpenAccess Library > Manuscript Information
This is an Open Access publication published under CSC-OpenAccess Policy.
When to Ask Participants to Think Aloud: A Comparative Study of Concurrent and Retrospective Think-Aloud Methods
Thamer Alshammari, Obead Alhadreti, Pam J. Mayhew
Pages - 48 - 64 | Revised - 30-06-2015 | Published - 31-07-2015
MORE INFORMATION
KEYWORDS
Usability Testing, Think-aloud Studies, Verbal Protocols.
ABSTRACT
This paper presents the results of a study that compared two think-aloud usability testing methods: the concurrent think-aloud and the retrospective think-aloud methods. Data from task performance, testing experience, and usability problems were collected from 30 participants equally distributed between the two think-aloud conditions. The results suggest that while the thinking aloud method had no impact on task performance and testing experience, participants using the concurrent think-aloud method reported a larger number of problems with the test interface than participants using the retrospective think-aloud method. These findings suggest a reason for preferring the concurrent think-aloud method to the retrospective one.
| 1 | Google Scholar |
| 2 | CiteSeerX |
| 3 | refSeek |
| 4 | Scribd |
| 5 | slideshare |
| 6 | PdfSR |
| 1 | Seffah, A., Donyaee, M., Kline, R., and Padda, H. ‘Usability measurement and metrics: A consolidated model.’ Software Quality Journal, 14(2): 2006, 159–178. |
| 2 | McDonald, Sharon, and Helen Petrie (2013). ‘The effect of global instructions on think- aloud testing.’ Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM. |
| 3 | Lewis, C. & Rieman, J. ‘Task-Centered User Interface Design: a Practical Introduction’. 1993. [Online). Available from: http://hcibib.orgltcuidl. [Accessed: 22 November 2014). |
| 4 | Gray M., Wardle H. ‘Observing gambling behaviour using think aloud and video technology: a methodological review’. NatCen Social Research. Available at: www.natcen.ac.uk. 2013, [Accessed: 22 January 2015). |
| 5 | Cotton, D. and Gresty, K. ‘Reflecting on the think-aloud method for evaluating elearning’. British Journal of Educational Technology, 37 (1), 2006, pp. 45-54. |
| 6 | Ericsson, K. A. and Simon, H.A., (1993) Protocol Analysis: Verbal Reports as Data. Revised ed. Cambridge: MIT Press. |
| 7 | Haak, V, (2008). ‘A penny for your thoughts – investigating the validity and reliability of think-aloud protocols for usability testing’(PhD dissertation). |
| 8 | McDonald, S., Edwards, H. and Zhao, T. ‘Exploring think-alouds in usability testing: an international survey’. IEEE Transactions on Professional Communication, 55(1), 2012, pp.117. |
| 9 | Haak V, Menno D, and Peter J. ‘Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue.’ Behaviour & Information Technology 22.5, 2003, 339-351. |
| 10 | Hertzum, M., Hansen, K.D. and Andersen, H.H.K. ‘Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload?’. Behaviour & Information Technology, 28 (2). 2009, pp. 165-181. |
| 11 | Griffiths, M.D. ‘The role of cognitive bias and skill in fruit machine gambling’. British Journal of Psychology, 1994, 85: 351-369. |
| 12 | Haak V, Maaike J., Menno DT de Jong, and Peter Jan Schellens. ‘Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: a methodological comparison.’ Interacting with computers 16.6: 2004, 1153-1170. |
| 13 | Khajouei, Reza, Arie Hasman, and Monique WM Jaspers. ‘Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system.’ international journal of medical informatics 80.5, 2011, 341-350. |
| 14 | Molich, R., Ede, M. R., ‘Kaasgaard, K., & Karyukin, B. Comparative usability evaluation’ Behaviour & Information Technology, 23(1), 2004, 65-74. |
| 15 | Gray, W. D., & Salzman, M. C. Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction, 13, 1998 203-261. |
| 16 | Eger, N., Ball, L. J., Stevens, R., & Dodd, ‘Cueing retrospective verbal reports in usability testing through eye-movement replay’. In Proceedings of the 21st British HCI Group Annual Conference on People and Computers: HCI... but not as we know it-Volume 1 (pp. 129137). British Computer Society, 2007. |
| 17 | Dumas, Joseph S., and Janice Redish. A practical guide to usability testing. Intellect Books, 1999. |
| 18 | Virzi, R.. Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 1992, 34(4):457–468. |
| 19 | Nielsen, J. (2000). Why you only need to test with 5 users. Nielsen Norman Group. Department of Computer Science. Machine Learning. Available at: bit.ly/1gpks7w [Accessed 25-04-2014]. |
| 20 | Fagan, J. C. Usability studies of faceted browsing: A literature review. Information Technology and Libraries, 2013, 29(2):58–66. |
| 21 | Lindgaard, G. and Chattratichart, J. ‘Usability testing: what have we overlooked?’ In Proceedings of the SIGCHI conference on Human factors in computing systems, 2007, pages 1415–1424. ACM. |
| 22 | Nielsen, J. Usability engineering. 1994, Elsevier. |
| 23 | Peute, L. W., de Keizer, N. F., & Jaspers, M. W. ‘Effectiveness of Retrospective and Concurrent Think Aloud in Formative Usability Testing; Which Method Performs Best?’. Human factors methods in health information systems’ design and, 2013, 65. |
Mr. Thamer Alshammari
School of Computing Sciences
University of East Anglia
Norwich, NR4 7TJ, UK - United Kingdom
t.alshammari@uea.ac.uk
Mr. Obead Alhadreti
School of Computing Sciences
University of East Anglia
Norwich, NR4 7TJ, UK - United Kingdom
Dr. Pam J. Mayhew
School of Computing Sciences
University of East Anglia
Norwich, NR4 7TJ, UK - United Kingdom
