Larsen & Bong (2016) A Tool for Addressing Construct Identity in Literature Reviews and Meta-Analyses

Abstract

The problem of detecting whether two behavioral constructs reference the same real- world phenomenon has existed for over 100 years; we term discordant naming of constructs the Construct Identity Fallacy (CIF). We designed and evaluated the Construct Identity Detector (CID), the first tool with large-scale construct identity detection properties and the first tool that does not require respondent data. Through the adaptation and combination of different natural language processing (NLP) algorithms, six designs were created and evaluated against human expert decisions. All six designs were found capable of detecting construct identity, and a design combining two existing algorithms significantly outperformed the other approaches. A set of follow-up studies suggests the tool is valuable as a supplement to expert efforts in literature review and meta-analysis. Beyond Design Science contributions, this article has important implications related to the taxonomic structure of social- and behavioral-science constructs, for the jingle and jangle fallacy, the core of the Information Systems (IS) nomological network, and the inaccessibility of social and behavioral science knowledge. In sum, CID represents an important, albeit tentative, step towards discipline-wide identification of construct identities.

Citation Kai R. Larsen, Chih How Bong (2016). A Tool for Addressing Construct Identity in Literature Reviews and Meta-Analyses. MIS Quarterly no. January, pp. 1–86. PDF

BibTex entry for this article:

BibTex entry for this article:

@article{larsen2016tool,
author = {Larsen, Kai R. and Bong, Chih How},
journal = {MIS Quarterly},
number = {January},
pages = {1--86},
title = {{A Tool for Addressing Construct Identity in Literature Reviews and Meta-Analyses}},
year = {2016}
}

Key ideas

The first step of science is to know one thing from another. This knowledge consists in their specific distinctions; but in order that it may be fixed and permanent, distinct names must be given to different things, and those names must be recorded and remembered. - Carolus Linnaeus, Systema Naturae (1738).

CIF ~ Construct identity fallacy
CID ~ Construct Identity Detector
NLP ~ Natural Language Processing
LSA ~ Latent Semantic Analysis

Notes

The Construct Identity Fallacy

Jangle fallacy

If you commit a jangle fallacy your work is akin to the reinvention of the wheel. Either a lack of knowledge or of concern about the work of other scientists or a lack of required data demonstrating divergent validity between new and established measures/constructs is the cause of the fallacy (Wilhelm 2009, p. 146).

Jingle fallacy

The jingle fallacy applies to any case where two constructs with identical names measure different latent constructs. If you commit a jingle fallacy you transfer available evidence to a new measure although in reality what you conceive as being an instance of an established construct somehow diverges (Wilhelm 2009, p. 146).

Consequences of the Construct Identity Fallacy


Proposed Methods For Construct Identity Detection


Evaluation And Findings


Design Science Process

  • Step 1. Procuring constructs and items.
  • Step 2. Developing new designs or importing existing designs.
Table 3. Design Science Contributions.
Guideline
(Hevner et al. 2004)
Contribution
Design as an Artifact The research outcomes:
1. CID methodology;
2. The extension and operationalization of jingle and jangle fallacies;
3. Tool for integration of theories.
Problem Relevance Large-scale construct identity problem was identified over 100 years ago with no viable solution to date, in spite of increasing relevance due to publication volume.
Providing tools for recognition of overlaps between theories is essential for true scientific progress (Noar and Zimmerman 2005).
Design Evaluation Utility and efficacy of design outcome evaluated using expert gold standard, experiment with Ph.D. students, a meta-analytic evaluation of common constructs, and a conceptual analysis of uncommon constructs.
Research Contributions The first design tool, showing promise for detecting construct correspondence and independence.
Expanded knowledge about IS constructs.
Research Rigor Comparison of six designs using a carefully developed gold standard, as well as other evaluations, resulting in triangulation.
Design as a Search Process Discovery of an effective solution in the form of CID1.
Evaluation of two different types of similarity designs and the creation of hybrid design.
Communication of Research Detailed information on how to build replicable CID technologies and implications for behavioral research.
Keywords:

Construct Identity Detector, Construct identity fallacy, Design Science, Latent Semantic Analysis, Natural Language Processing, Ontologies, construct validity, inter-nomological network, jingle and jangle fallacy, nomological networks, synonymy and polysemy

Print/export