Cassidy & Hamilton (2016) A Design Science Research Approach to Website Benchmarking

Authors: Cassidy, Leonie J. and Hamilton, John
Citation Leonie J. Cassidy, John Hamilton (2016). A Design Science Research Approach to Website Benchmarking. Benchmarking: An International Journal, vol. 23 no. 5, pp. 1–28. URL

BibTex entry for this article:

BibTex entry for this article:

author = {Cassidy, Leonie J. and Hamilton, John},
isbn = {0720140064},
journal = {Benchmarking: An International Journal},
number = {5},
pages = {1--28},
title = {{A Design Science Research Approach to Website Benchmarking}},
url = {},
volume = {23},
year = {2016}

Key ideas


Design Science Research

This study adopts a design science research (DSR) approach to design and develop a new [website benchmarking] (WB) approach for business: (p. 3)

Design science research is a research paradigm in which a designer answers questions relevant to human problems via the creation of innovative artifacts - thereby contributing new knowledge to the body of scientific evidence. The designed artifacts are both useful and fundamental in understanding that problem.

- Design Science Research in Information Systems (2010) by Alan R. Hevner et al. p. 5

Under DSR we apply its seven guidelines to develop our technology-based solution (artifact) to solve this WB issue (Hevner et al., 2004). (p. 3)

From the literature, we first explain the gaps that exist, and that no theory shows overall relevance to this WB problem. The artifact can be a construct, or a model, or a method, or an instantiation. In this study we engage the artifact as a website analysis model (termed WAM). The WAM artifact is supported by clear, verifiable, and rigorous method, and is applied at the construction and evaluation stages of the artifact. (p. 3)

Table 1 summarizes the six DSRM activity stages. (Adapted from Peffers et al. (2008) A Design Science Research Methodology for Information Systems Research)

The introduction and literature review completes DSRM activity stages one and two, identifies WB approaches, identifies the limitations in the literature, identifies and discusses problems for organizations, and then defines objectives for the solution to the identified WB business problem. (p. 6)

Tables and figures

Table 1. Design Science Research Methodology – adapted from Peffers et al., 2007 Table 1 ~ p.?

Table 2. The Ride Guide Homepage at each change Table 2 ~ p.?

Table 3. Aesthetic Domain calculations Table 3 ~ p.?

Table 4. Domain and Stage Calculations Table 4 ~ p.?

Figure 1. WAM (developed from ?)

Figure 1. WAM Figure 1 ~ p.?

Figure 2. Effects recognition in website benchmarking Figure 2 ~ p.?

Figure 3. Effects recognition ( Figure 3 ~ p.?

Figure 4. Longitudinal tracking:'s website changes Figure 4 ~ p.?

References in cassidy2016design

  • Abascal, J., Arrue, M., Fajardo, I., Garay, N. and Tomás, J. (2004), “The use of guidelines to automatically verify web accessibility”, Universal Access in the Information Society, Vol. 3 No. 1, pp. 71-79.
  • Adler, P.S., Benner, M., Brunner, D.J., MacDuffie, J.P., Osono, E., Staats, B.R., Takeuchi, H., Tushman, M.L. and Winter S.G. (2009), “Perspectives on the productivity dilemma”, Journal of Operations Management, Vol. 27 No. 2, pp. 99–113.
  • Albaum, G., Evangelista, F. and Medina, N. (1998), “Role of response behavior theory in survey research: a cross-national study”, Journal of Business Research, Vol. 42 No. 2, pp. 115-125.
  • Albert, T.C., Goes, P.B. and Gupta, A. (2004), “GIST: a model for design and management of content and interactivity of customer-centric web sites”, MIS Quarterly, Vol. 28 No. 2, pp. 161-182.
  • Barnes, S.J. and Vidgen, R.T. (2002), “An integrative approach to the assessment of e- commerce quality”, Journal of Electronic Commerce Research, Vol. 3 No. 3, pp. 114- 127.
  • Benner, M.J. and Ranganathan, R. (2012), “Offsetting illegitimacy? How pressures from securities analysts influence incumbents in the face of new technologies”, Academy of Management Journal, Vol. 33 No. 1, pp. 213-233.
  • Beirekdar, A., Vanderdonckt, J. and Noirhomme-Fraiture, M. (2002), “KWARESMI— Knowledge-based Web Automated Evaluation with REconfigurable guidelineS optiMIzation”, in the 9th International Workshop on Design, Specification, and Verification of Interactive Systems DSV-IS2002, 2002, Springer-Verlag London, UK.
  • Boisvert, H. and Caron, M. (2006), “Benchmarking web site functions”, Benchmarking: An International Journal, Vol. 13 No. 1/2, pp. 174-189.
  • Bowers, N. (1996), “Weblint: quality assurance for the world wide web”, Computer Networks and ISDN Systems, Vol. 28 No. 7 to 11, pp. 1283-1290.
  • Calero, C., Ruiz, J. and Piattini, M. (2005), “Classifying web metrics using the web quality model”, Online Information Review, Vol. 29 No. 3, pp. 227-248.
  • Cao, M., Zhang, Q. and Seydel, J. (2005), “B2C e-commerce web site quality: an empirical examination”, Industrial Management and Data Systems, Vol. 105 No. 5, pp. 645- 661.
  • Cassidy, L. (2010), Website Benchmarking: Evaluating Scaled and Dichotomous Approaches. (Unpublished honours thesis). James Cook University, Cairns, Australia.
  • Cassidy, L. and Hamilton, J. (2011a), “Website benchmarking: evaluating scaled and dichotomous approaches”, in Rotchanakitumnuai, S. and Kaewkitipong, L. (Eds.), The 11th International Conference on Electronic Business, Borderless E-Business for the Next Decade, Bangkok, Thailand, Thammasat University, pp. 408-412.
  • Cassidy, L. and Hamilton, J. (2011b), “Website benchmarking: theoretical and measurement aspects”, in Rotchanakitumnuai, S. and Kaewkitipong, L. (Eds.), The 11th International Conference on Electronic Business, Borderless E-Business for the Next Decade, Bangkok, Thailand, Thammasat University, pp. 413-416.
  • Chang, Y-S. and Fang, S-R. (2013), “Antecedents and distinctions between online trust and distrust: Predicting high and low risk internet behaviors”, Journal of Electronic Commerce Research, Vol. 14 No. 2, pp. 149-166.
  • Chen, Y.-H. and Barnes, S. (2007), “Initial trust and online buyer behavior”, Industrial Management and Data Systems, Vol. 107 No. 1, pp. 21-36.
  • Choi, S. and Morrison, A.M. (2005), “Website effectiveness for bricks and mortar travel retailers”, International Journal of Tourism and Hospitality Research, Vol. 16 No. 1, pp. 63-78.
  • Cox, J. and Dale, B.G. (2002), “Key quality factors in web site design and use: an examination”, International Journal of Quality and Reliability Management, Vol. 19 No. 7, pp. 862-888.
  • Deng, J. (1989), “Introduction to grey system theory”, The Journal of Grey System, Vol. 1 No. 1, pp. 1-24.
  • Deng, L. and Poole, M.S. (2010), “Affect in web interfaces: a study of the impacts of web page visual complexity and order”, MIS Quarterly, Vol. 34 No. 4, pp. 711-730.
  • Elling, S., Lentz, L. and De Jong, M. (2007), “Website evaluation questionnaire: development of a research-based tool for evaluating information websites”, in Electronic Government 2007 proceedings of 6th International Conference in Berlin, Heidelberg, pp. 293-304.
  • Evangelista, F., Poon, P. and Albaum, G. (2012), “Using response behavior theory to solicit survey participation in consumer research: an empirical study”, Journal of Marketing Management, Vol. 28 No.9/10, pp. 1174-1189.
  • Falcon, A. (2011), “Aristotle on Causality”, in The Stanford Encyclopedia of Philosophy, E. N. Zalta (ed.) available at: causality (accessed 2 June 2013).
  • Fang, Y., Qureshi, I., Sun, H., McCole, P., Ramsey, E. and Lim, K.H. (2014), “Trust, satisfaction, and online repurchase intention: the moderating role of perceived effectiveness of e-commerce institutional mechanisms”, MIS Quarterly, Vol. 38 No. 2, pp. 407-428.
  • Fink, D. and Nyaga, C. (2009), “Evaluating web site quality: The value of a multi paradigm approach”, Benchmarking: An International Journal, Vol. 16 No. 2, pp. 259-273.
  • Flavian, C., Guinaliu, M. and Gurrea, R. (2006), “The role played by perceived usability, satisfaction and consumer trust on website loyalty”, Information and Management, Vol. 43 No. 1, pp. 1-14.
  • Ghasemaghaei, M. and Hassanein, K. (2013), “Consumers’ satisfaction with online information quality: the moderating roles of consumer decision-making style, gender and product involvement”, in the 2013 proceedings of the 21st European Conference on Information Systems in Utrecht, the Netherlands, pp. 1-7.
  • Gefen, D., Karahanna, E. and Straub, D.W. (2003), “Trust and TAM in online shopping: an integrated model”, MIS Quarterly, Vol. 27 No. 1, pp. 51-90.
  • Grigoroudis, E., Litos, C., Moustakis, V.A., Politis, Y. and Tsironis, L. (2008), “The assessment of user-perceived web quality: Application of a satisfaction benchmarking approach”, European Journal of Operational Research, Vol. 187 No. 3, pp. 1346- 1357.
  • Hamilton, J.R. (2010), “Rating the website: an objective approach”, International Journal of Electronic Business, Vol. 9 No. 4, pp. 1-24.
  • Hamilton, J.R. and Tee, S. (2013), “Understanding social network site consumer engagements”, in Proceedings of the 24th ICIS Conference, Melbourne, Australia, 4-6 December, Vol. 24 No. 1, pp. 1-10.
  • Herrero, A. and San Martin, H. (2012), “Developing and Testing a Global Model to Explain the Adoption of Websites by Users in Rural Tourism Accommodations”, International Journal of Hospitality Management, Vol. 31 No. 4, pp. 1178-1186.
  • Ho, C.-T. and Wu, Y.-S. (2006), “Benchmarking performance indicators for banks”, Benchmarking: An International Journal, Vol. 3 No. 1/2, pp. 147-159.
  • Hu, Y-C. (2009), “Fuzzy multiple-criteria decision making in the determination of critical criteria for assessing service quality of travel websites”, Expert Systems with Applications, Vol. 36 No. 3 part 2, pp. 6439-6445.
  • Io Storto, C. (2013), “Evaluating ecommerce websites cognitive efficiency: An integrative framework based on data envelopment analysis”, Applied Ergonomics, Vol. 44 No. 6, pp. 1004-1014.
  • Io Storto, C. (2014), “Benchmarking website performance in the public sector: A non parametric approach”, Journal of Computers, Vol. 9 No. 3, pp. 636-643.
  • Ivory, M.Y. and Hearst, M.A. (2001), “The state of the art in automating usability evaluation of user interfaces”, ACM Computing Surveys, Vol. 33 No. 4, pp. 470-516.
  • Ivory, M.Y. and Megraw, R. (2005), “Evolution of web site design patterns”, ACM Transactions on Information systems, Vol. 23 No. 4, pp. 463-497.
  • Kaplan, R. and Norton, D. (1992), “The balanced scorecard - measures that drive performance”, Harvard Business Review, Vol. 70 No. 1, pp. 71-79.
  • Kim, H.-W., Chan, H.C. and Gupta, S. (2007), “Value-based adoption of mobile internet: an empirical investigation”, Decision Support Systems, Vol. 43 No. 1, pp. 111-126.
  • Kim, J., Kim, M. and Kandampully, J. (2009), “Buying environment characteristics in the context of e-service”, European Journal of Marketing, Vol. 43 No. 9/10, pp. 1188- 1204
  • Kim, C. and Mauborgne, K. (2005), Blue Ocean Strategy, Harvard Business School Press: Boston, MA.
  • Kline, S.F., Morrison, A.M. and St. John, A. (2004), “Exploring bed and breakfast websites: a balanced scorecard approach”, Journal of Travel and Tourism Marketing, Vol. 17 No. 2-3, pp. 253-267.
  • Krosnick, J.A. (1991), “Response strategies for coping with the cognitive demands of attitude measures in surveys”, Applied Cognitive Psychology, Vol. 5 No. 3, pp. 213-236.
  • Krosnick, J.A. and Alwin, D.F. (1987), “An evaluation of a cognitive theory of response- order effects in survey measurement”, Public Opinion Quarterly, Vol. 51 No. 2, pp. 201-219.
  • Krosnick, J.A., Holbrook, A.L., Berent, M.K., Carson, R.T., Hanemann, W.M., Kopp, R.J., Mitchell, R.C., Presser, S., Rudd, P.A., Smith, V.K., Moody, W.R., Green, M.C. and Conaway, M. (2002), “The impact of “no opinion” response options on data quality: non-attitude reduction or an invitation to satisfice?”, Public Opinion Quarterly, Vol. 66 No. 3, pp. 371-403.
  • LaRose, R. and Eastin, M.S. (2004), “A social cognitive theory of internet uses and gratifications: toward a new model of media attendance”, Journal of Broadcasting and Electronic Media, Vol. 48 No. 3, pp. 358-377.
  • Lee, Y., Chen, A.N.K. and Ilie, V. (2012), “Can online wait be managed? The effect of filler interfaces and presentation modes on perceived waiting time online”, MIS Quarterly, Vol. 36 No. 2.
  • Lee, Y. and Kozar, K.A. (2009), “Designing usable online stores: a landscape preference perspective”, Information and Management, Vol. 46 No. 1, pp. 31-41.
  • Lee, J. and Morrison, A.M. (2010), “A comparative study of web site performance”, Journal of Hospitality and Tourism Technology, Vol. 1 No. 1, pp. 50-67.
  • Lee, S.-M., Ungson, G.R. and Russo, M.V. (2011), “What determines an engaging website? An empirical study of website characteristics and operational performance”, Journal of High Technology Management Research, Vol. 22 No. 1, pp. 67-79.
  • Lin, K.-Y., and Lu, H.-P. (2011), “Why people use social networking sites: an empirical study integrating network externalities and motivation theory”, Computers in Human Behavior, Vol. 27 No.3, pp. 1152-1161.
  • Loiacono, E.T., Watson, R.T. and Goodhue, D.L. (2007), “WebQual: an instrument for consumer evaluation of web sites”, International Journal of Electronic Commerce, Vol. 11 No. 3, pp. 51-87.
  • Luo, J., Ba, S. and Zhang, H. (2012), “The effectiveness of online shopping characteristics and well-designed websites on satisfaction”, MIS Quarterly, Vol. 36 No. 4, pp. 1131- 1144.
  • Luo, M.M., Chea, S. and Chen, J.-S. (2011), “Web-based information service adoption: a comparison of the motivational model and the uses and gratifications theory”, Decision Support Systems, Vol. 51 No. 1, pp. 21-30.
  • McKinney, V., Yoon, K. and Zahedi, F.M. (2002), “The measurement of web-customer satisfaction: an expectation and disconfirmation approach”, Information Systems Research, Vol. 13 No. 3, pp. 296-315.
  • Manyika, J., Chui, M., Bughin, J., Dobbs, R., Bisson, P. and Marrs, A. (2013), “Disruptive Technologies: Advances that will transform life, business, and the global economy”, McKinsey Global Institute, May, pp. 1-23.
  • Moriarty, J. (2011), “A theory of benchmarking”, Benchmarking: An International Journal, Vol. 18 No. 4, pp. 588-612.
  • Nathan, R.J. and Yeow, P.H.P. (2011), “Crucial web usability factors of 36 industries for students: a large-scale empirical study”, Electronic Commerce Research, Vol. 11 No. 2, pp. 151-180.
  • Nel, D., Niekerk, R. v., Berthon, J.-P. and Davies, T. (1999), “Going with the flow: web sites and customer involvement”, Internet Research: Electronic Networking Applications and Policy, Vol. 9 No. 2, pp. 109-116.
  • Nusair, K.K. and Kandampully, J. (2008), “The antecedents of customer satisfaction with online travel services: a conceptual model”, European business Review, Vol. 20 No. 1, pp. 4-19.
  • O'Cass, A. and Ngo, L.V. (2012), “The antecedents of customer satisfaction with online travel services: a conceptual model”, Industrial Marketing Management, Vol. 41 No. 1, pp. 125-135.
  • Olsina, L. and Rossi, G. (2002), “Measuring web application quality with WebQEM”, Multimedia IEEE, Vol. 9 No. 4, pp. 20-29.
  • Page, C. and Lepkowska-White, E. (2002), “Web equity: a framework for building consumer value in online companies”, Journal of Consumer Marketing, Vol. 19 No. 3, pp. 231- 248.
  • Pang, M.-S., Suh, W., Kim, J. and Lee, H. (2009), “A benchmarking-based requirement analysis methodology for improving websites”, International Journal of Electronic Commerce, Vol. 13 No. 3, pp. 119-162.
  • Rashid, M.A.A. and Othman, M.N.A. (2010), “User adoption and use of e-government online service”, Gading Business and Management Journal, Vol. 14 June, pp. 55-66.
  • Schwarz, N. (2007), “Cognitive aspects of survey methodology”, Applied Cognitive Psychology, Vol. 21 No. 2, pp. 277-287.
  • Simmons, G., Armstrong, G.A. and Durkin, M.G. (2008), “A conceptualization of the determinants of small business website adoption: setting the research agenda”, International Small Business Journal, Vol. 26 No. 3, pp. 351-389.
  • Steelman, Z.R., Hammer, B.I. and Limayem, M. (2014), “Data collection in the digital age: innovative alternatives to student samples”, MIS Quarterly, Vol. 38 No. 2.
  • Udo, G.J., Bagchi, K.K. and Kirs, P.J. (2010), “An assessment of customers' e-service quality perception, satisfaction and intention”, International Journal of Information Management, Vol. 30 No. 6, pp. 481-492.
  • Vilenkin, N.Y. (1986), Stories about Sets, Academic Press: New York, NY. Wakefield, R.L., Wakefield, K.L., Baker, J. and Wang, L.C. (2011), “how website socialness leads to website use”, European Journal of Information Systems, Vol. 20 No. 1, pp. 118-132.
  • Webb, H.W. and Webb, L.A. (2004), “SiteQual: an integrated measure of website quality”, The Journal of Enterprise Information Management, Vol. 17 No. 6, pp. 430-440.
  • Weijters, B., Cabooter, E. and Schillewaert, N. (2010), “The effect of rating scale format on response styles: The number of response categories and response category labels”, International Journal of Research in Marketing, Vol. 27 No. 3, pp. 236-247.
  • Wells, J.D., Valacich, J.S. and Hess, T.J. (2011), “What signal are you sending? How website quality influences perceptions of product quality and purchase intentions”, MIS Quarterly, Vol. 35 No. 2, pp. 373-396.
  • Wolfinbarger, M. and Gilly, M.C. (2003), “eTailQ: dimensionalizing, measuring and predicting eTail quality”, Journal of Retailing, Vol. 79 No. 3, pp. 183-198
  • Xu, J., Benbasat, I. and Cenfetelli, R.T. (2013), “Integrating service quality with system and information quality: an empirical test in the e-service context”, MIS Quarterly, Vol. 37 No. 3, pp. 777-794.
  • Yang, T., Linder, J. and Bolchini, D. (2012), “DEEP: design-oriented evaluation of perceived usability”, International Journal of Human-Computer Interaction, Vol. 28 No. 5, pp. 308-346.
  • Yoo, B. and Donthu, N. (2001), “Developing a scale to measure the perceived quality of an internet shopping site (SITEQUAL)”, Quarterly Journal of Electronic Commerce, Vol. 2 No. 1, pp. 31-47.
  • Yoon, C. and Kim, S. (2009), “Developing the causal model of online store success”, Journal of Organizational Computing and Electronic Commerce, Vol. 19 No. 4, pp. 265-284.
  • Zhu, W. (2007), “Topological approaches to covering rough sets”, Information Sciences, Vol. 177 No. 6, pp. 1499-1508.
  • Zhu, Y., Basil, D.Z. and Hunter, M.G. (2009), “The extended website stage model: a study of Canadian winery websites”, Canadian Journal of Administrative Sciences, Vol. 26 No. 4, pp. 286-300.

Design Science, Design Science Research, Website Benchmarking