Prat et al. (2014) Artifact Evaluation in Information Systems Design Science Research

Artifact Evaluation in Information Systems Design Science Research - A Holistic View

Abstract

Design science in Information Systems (IS) research pertains to the creation of artifacts to solve real- life problems. Research on IS artifact evaluation remains at an early stage. In the design-science research literature, evaluation criteria are presented in a fragmented or incomplete manner. This paper addresses the following research questions: which criteria are proposed in the literature to evaluate IS artifacts? Which ones are actually used in published research? How can we structure these criteria? Finally, which evaluation methods emerge as generic means to assess IS artifacts? The artifact resulting from our research comprises three main components: a hierarchy of evaluation criteria for IS artifacts organized according to the dimensions of a system (goal, environment, structure, activity, and evolution), a model providing a high-level abstraction of evaluation methods, and finally, a set of generic evaluation methods which are instantiations of this model. These methods result from an inductive study of twenty-six recently published papers.

Citation Nicolas Prat, Isabelle Comyn-Wattiau, Jacky Akoka (2014). Artifact Evaluation in Information Systems Design Science Research - A Holistic View. PACIS 2014 Proceedings, vol. Paper 23, pp. 1–16. URL

BibTex entry for this article:

BibTex entry for this article:

@article{prat2014artifact,
author = {Prat, Nicolas and Comyn-Wattiau, Isabelle and Akoka, Jacky},
journal = {PACIS 2014 Proceedings},
pages = {1--16},
title = {{Artifact Evaluation in Information Systems Design Science Research - A Holistic View}},
url = {http://aisel.aisnet.org/pacis2014/23/},
volume = {Paper 23},
year = {2014}
}

Key ideas

Notes

Tables and figures

Figure 1: Hierarchy of criteria for IS artifact evaluation. Figure 1 ~ p.6


Figure 2: DSR artifact evaluation criteria used in the sample of twenty-six papers. Figure 2 ~ p.7


Figure 3: A model of generic evaluation methods. Figure 3 ~ p.9


Figure 4: Mapping system dimensions used in three approaches into system properties suggested by Skyttner (2005). Figure 4 ~ p.12


Appendix: Sample of Design-Research papers

MIS Quarterly

  1. Abbasi, A., Albrecht, C., Vance, A. and Hansen, J., “Metafraud: a meta-learning framework for detecting financial fraud”, 36:(4), 2012, 1293-1327.
  2. Abbasi, A. and Chen, H., “CyberGate: a design framework and system for text analysis of computer-mediated communication”, 32:(4), 2008, 811-837.
  3. Abbasi, A., Zhang, Z., Zimbra, D., Chen, H. and Nunamaker, J. F., “Detecting fake websites: the contribution of statistical learning theory”, 34:(3), 2010, 435-461.
  4. Adipat, B., Zhang, D. and Zhou, L., “The effects of tree-view based presentation adaptation on mobile web browsing”, 35:(1), 2011, 99-122.
  5. Adomavicius, G., Bockstedt, J. C., Gupta, A. and Kauffman, R. J., “Making sense of technology trends in the information technology landscape: a design science approach”, 32:(4), 2008, 779-809.
  6. Lau, R. Y. K., Liao, S. S. Y., Wong, K. F. and Chiu, D. K. W., “Web 2.0 environmental scanning and adaptive decision support for business mergers and acquisitions”, 36:(4), 2012, 1239-1268.
  7. Lee, J., Wyner, G. M. and Pentland, B. T., “Process grammar as a tool for business process design”, 32:(4), 2008, 757-778.
  8. McLaren, T. S., Head, M. M., Yuan, Y. and Chan, Y. E., “A multilevel model for measuring fit between a firm's competitive strategies and information systems capabilities”, 35:(4), 2011, 909-930.
  9. Parsons, J. and Wand, Y., “Using cognitive principles to guide classification in information systems modeling”, 32:(4), 2008, 839-868.
  10. Pries-Heje2008design Pries-Heje, J. and Baskerville, R., “The design theory nexus”, 32:(4), 2008, 731-755.
  11. Sahoo, N., Singh, P. V. and Mukhopadhyay, T., “A hidden Markov model for collaborative filtering”, 36:(4), 2012, 1329-1356.
  12. VanderMeer, D., Dutta, K. and Datta, A., “A cost-based database request distribution technique for online e-commerce applications”, 36:(2), 2012, 479-507.

Journal of Computer Information Systems

  1. Apostolou, D., Mentzas, G. and Abecker, A., “Managing knowledge at multiple organizational levels using faceted ontologies”, 49:(2), 2008, 32-49.
  2. Deane, J. and Agarwal, A., “Scheduling online advertisements to maximize revenue under non-linear pricing”, 53:(2), 2012, 85-92.
  3. Du, H.-J., Shin, D.-H. and Lee, K.-H., “A sophisticated approach to semantic web services discovery”, 48:(3), 2008, 44-60.
  4. Hong, S.-Y., Kim, J.-W. and Hwang, Y.-H., “Fuzzy-semantic information management system for dispersed information”, 52:(1), 2011, 96-105.
  5. Hou, J.-L. and Huang, C.-H., “A model for document validation using concurrent authentication processes”, 49:(2), 2008, 65-80.
  6. Kim, Y. S., “Multi-objective clustering with data- and human-driven metrics”, 51:(4), 2011, 64-73.
  7. Li, S.-H., Huang, S.-M. and Lin, Y.-C., “Developing a continuous auditing assistance system based on information process models”, 48:(1), 2007, 2-13.
  8. Li, S.-T. and Chang, W.-C., “Design and evaluation of a layered thematic knowledge map system”, 49:(2), 2008, 92-103.
  9. Li, S.-T. and Tsai, F.-C., “Concept-guided query expansion for knowledge management with semi-automatic knowledge capturing”, 49:(4), 2009, 53-65.
  10. Montero, J. D., Kim, Y. S. and Johnson, J. J., “A rapid mapping conversion methodology to a commercial-off-the-shelf system”, 50:(4), 2010, 57-66.
  11. Sheikh, M. and Conlon, S., “A rule-based system to extract financial information”, 52:(4), 2012, 10-19.
  12. Ullah, A. and Lai, R., “Modeling business goal for business/IT alignment using requirements engineering”, 51:(3), 2011, 21-28.
  13. Wang, H. and Wang, S., “Ontology-based data summarization engine: a design methodology”, 53:(1), 2012, 48-56.

References

Keywords:

Design Science, artifact evaluation, design science, evaluation criterion, general systems, generic evaluation method, information systems research, theory

Print/export