Home   >   CSC-OpenAccess Library   >    Manuscript Information
Full Text Available

(903.55KB)
This is an Open Access publication published under CSC-OpenAccess Policy.
Publications from CSC-OpenAccess Library are being accessed from over 158 countries worldwide.
When to Ask Participants to Think Aloud: A Comparative Study of Concurrent and Retrospective Think-Aloud Methods
Thamer Alshammari, Obead Alhadreti, Pam J. Mayhew
Pages - 48 - 64     |    Revised - 30-06-2015     |    Published - 31-07-2015
Volume - 6   Issue - 3    |    Publication Date - July / August 2015  Table of Contents
MORE INFORMATION
KEYWORDS
Usability Testing, Think-aloud Studies, Verbal Protocols.
ABSTRACT
This paper presents the results of a study that compared two think-aloud usability testing methods: the concurrent think-aloud and the retrospective think-aloud methods. Data from task performance, testing experience, and usability problems were collected from 30 participants equally distributed between the two think-aloud conditions. The results suggest that while the thinking aloud method had no impact on task performance and testing experience, participants using the concurrent think-aloud method reported a larger number of problems with the test interface than participants using the retrospective think-aloud method. These findings suggest a reason for preferring the concurrent think-aloud method to the retrospective one.
1 Google Scholar 
2 CiteSeerX 
3 refSeek 
4 Scribd 
5 SlideShare 
6 PdfSR 
1 Seffah, A., Donyaee, M., Kline, R., and Padda, H. ‘Usability measurement and metrics: A consolidated model.’ Software Quality Journal, 14(2): 2006, 159–178.
2 McDonald, Sharon, and Helen Petrie (2013). ‘The effect of global instructions on think- aloud testing.’ Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM.
3 Lewis, C. & Rieman, J. ‘Task-Centered User Interface Design: a Practical Introduction’. 1993. [Online). Available from: http://hcibib.orgltcuidl. [Accessed: 22 November 2014).
4 Gray M., Wardle H. ‘Observing gambling behaviour using think aloud and video technology: a methodological review’. NatCen Social Research. Available at: www.natcen.ac.uk. 2013, [Accessed: 22 January 2015).
5 Cotton, D. and Gresty, K. ‘Reflecting on the think-aloud method for evaluating elearning’. British Journal of Educational Technology, 37 (1), 2006, pp. 45-54.
6 Ericsson, K. A. and Simon, H.A., (1993) Protocol Analysis: Verbal Reports as Data. Revised ed. Cambridge: MIT Press.
7 Haak, V, (2008). ‘A penny for your thoughts – investigating the validity and reliability of think-aloud protocols for usability testing’(PhD dissertation).
8 McDonald, S., Edwards, H. and Zhao, T. ‘Exploring think-alouds in usability testing: an international survey’. IEEE Transactions on Professional Communication, 55(1), 2012, pp.117.
9 Haak V, Menno D, and Peter J. ‘Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue.’ Behaviour & Information Technology 22.5, 2003, 339-351.
10 Hertzum, M., Hansen, K.D. and Andersen, H.H.K. ‘Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload?’. Behaviour & Information Technology, 28 (2). 2009, pp. 165-181.
11 Griffiths, M.D. ‘The role of cognitive bias and skill in fruit machine gambling’. British Journal of Psychology, 1994, 85: 351-369.
12 Haak V, Maaike J., Menno DT de Jong, and Peter Jan Schellens. ‘Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: a methodological comparison.’ Interacting with computers 16.6: 2004, 1153-1170.
13 Khajouei, Reza, Arie Hasman, and Monique WM Jaspers. ‘Determination of the effectiveness of two methods for usability evaluation using a CPOE medication ordering system.’ international journal of medical informatics 80.5, 2011, 341-350.
14 Molich, R., Ede, M. R., ‘Kaasgaard, K., & Karyukin, B. Comparative usability evaluation’ Behaviour & Information Technology, 23(1), 2004, 65-74.
15 Gray, W. D., & Salzman, M. C. Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction, 13, 1998 203-261.
16 Eger, N., Ball, L. J., Stevens, R., & Dodd, ‘Cueing retrospective verbal reports in usability testing through eye-movement replay’. In Proceedings of the 21st British HCI Group Annual Conference on People and Computers: HCI... but not as we know it-Volume 1 (pp. 129137). British Computer Society, 2007.
17 Dumas, Joseph S., and Janice Redish. A practical guide to usability testing. Intellect Books, 1999.
18 Virzi, R.. Refining the test phase of usability evaluation: How many subjects is enough? Human Factors, 1992, 34(4):457–468.
19 Nielsen, J. (2000). Why you only need to test with 5 users. Nielsen Norman Group. Department of Computer Science. Machine Learning. Available at: bit.ly/1gpks7w [Accessed 25-04-2014].
20 Fagan, J. C. Usability studies of faceted browsing: A literature review. Information Technology and Libraries, 2013, 29(2):58–66.
21 Lindgaard, G. and Chattratichart, J. ‘Usability testing: what have we overlooked?’ In Proceedings of the SIGCHI conference on Human factors in computing systems, 2007, pages 1415–1424. ACM.
22 Nielsen, J. Usability engineering. 1994, Elsevier.
23 Peute, L. W., de Keizer, N. F., & Jaspers, M. W. ‘Effectiveness of Retrospective and Concurrent Think Aloud in Formative Usability Testing; Which Method Performs Best?’. Human factors methods in health information systems’ design and, 2013, 65.
Mr. Thamer Alshammari
School of Computing Sciences University of East Anglia Norwich, NR4 7TJ, UK - United Kingdom
t.alshammari@uea.ac.uk
Mr. Obead Alhadreti
School of Computing Sciences University of East Anglia Norwich, NR4 7TJ, UK - United Kingdom
Dr. Pam J. Mayhew
School of Computing Sciences University of East Anglia Norwich, NR4 7TJ, UK - United Kingdom