Iversen & Ngwenyama (2006) Problems in Measuring Effectiveness in Software Process Improvement

Problems in Measuring Effectiveness in Software Process Improvement: A Longitudinal Study of Organizational Change at Danske Data
Iversen, Jakob H. and Ngwenyama, Ojelanki K.


Software process improvement (SPI) is a widely recognized approach that software companies implement to improve quality, productivity, and time-to-market. Assessing and analyzing performance improvements are important SPI activities. However, many SPI managers have found it difficult to develop and implement effective performance measurement programs for SPI, in part because guidelines for conducting SPI measurements are scarce. We address this gap in the SPI literature by examining major problems that SPI change agents encounter when developing and implementing SPI measurement programs. We report on a longitudinal study of an SPI change initiative and the challenges that the SPI Team faced in dealing with the issues of measuring effectiveness of the initiative. We systematically analyze an SPI performance measurement program to understand its limitations and the problems that the SPI Team encountered when implementing it. We used an organizational change theory framework to derive theoretical and practical insights that can help managers and researchers develop and implement better SPI performance measurement programs.

Citation Jakob H. Iversen, Ojelanki K. Ngwenyama (2006). Problems in Measuring Effectiveness in Software Process Improvement: A Longitudinal Study of Organizational Change at Danske Data. International Journal of Information Management, vol. 26 no. 1, pp. 30–43. DOI

BibTex entry for this article:

BibTex entry for this article:

author = {Iversen, Jakob H. and Ngwenyama, Ojelanki K.},
doi = {10.1016/j.ijinfomgt.2005.10.006},
isbn = {0268-4012},
issn = {02684012},
journal = {International Journal of Information Management},
number = {1},
pages = {30--43},
title = {{Problems in Measuring Effectiveness in Software Process Improvement: A Longitudinal Study of Organizational Change at Danske Data}},
volume = {26},
year = {2006}

Key ideas

SPI ~ Software Process Improvement
CMM ~ Capability Maturity Model

Many business firms view SPI as a strategic issue and are involved in organizational change initiatives to improve their software development practices. The fundamental objective of the SPI movement is to change the organizational practices of software development firms, in order to improve software quality and reliability, employee and customer satisfaction, and profitability. (p. 31)

The measurement and analysis of organizational effectiveness is an important activity in any SPI initiative. Measuring aspects of organizational effectiveness in software development can provide management with critical information upon which to base decisions about allocating resources to effect change in specific areas. (p. 31)

Many researchers view the lack of a systematic and reliable measurement methodology as a major factor contributing to the high failure rate of SPI initiatives. (p. 31)

The main objective in this paper is to develop an understanding of how to measure software development performance and how these measures can inform software development managers, SPI practitioners, and senior management. (p. 31)


1. Introduction

Some firms are outsourcing software development to Asia, and others are setting up Asian offices to take advantage of the lower cost of production. Still others are implementing various forms of organizational change, such as business process re-engineering; more flexible approaches to organizing work, such as teams and autonomous work-groups; and SPI. (p. 30)

The theoretical foundation of our analysis is organizational change theory, and our research methodology is based on longitudinal process analysis. The focus of our discussion is a longitudinal study of an SPI change initiative in the Danish software company, Danske Data. (p. 31)

2. Research Methodology

The research project was a longitudinal process study (from 1996 to 2001) of the SPI initiatives intended to advance the four companies up the CMM scale. (p. 31)

Most of the researchers were engaged in the four companies and actively participated in the SPI projects, and were thus able to understand the SPI initiatives from the perspective of participant observers. (p. 31)

The major strength of longitudinal process research (LPR) is the opportunity to get close to the organization and find out what is really going on. However, close involvement can also be problematic if researchers do not systematically collect data to generalize finding. (p. 31)

The data collection for this study started in December 1996 and continued through December 2000. One of the authors communicated daily with the members of Danske Data, and made regular visits to the organization (at least one each month) to participate in regular project meetings. We also collected a broad range of organizational artifacts relevant to the SPI initiative at Danske Data. In addition to the participant observation, we recorded meeting minutes, and collected e-mail, memos, reports, project plans, and so on (Table 1). This multiple-source data collection strategy supports triangulation, systematic gathering, and reliable recording and transcription of data, which in turn helps to ensure the validity of our empirical findings. (p. 31)

List of data sources: (p. 32)

  • Meeting notes
  • Meeting minutes
  • Meeting transcriptions
  • Electronic messages
  • SPI artifacts
  • Company artifacts
  • Maturity assessments
  • Performance measurement

3. The organizational change perspective

Since SPI attempts to fundamentally alter the way the organization develops software, it is relevant to study SPI under the lens of an accepted organizational change framework. (p. 32)

The theoretical framework for organizational change upon which our study is based was adapted from Applegate (1994), who suggests that to achieve effective and lasting organizational change, change agents must attend to more than just management and technical systems. To transform taken-for-granted work routines, behaviors and cultural habits, organizational actors must continuously learn and change. Therefore, change agents must attend to social and cultural issues, which are often the most difficult to address. (p. 32)

Applegate (1994) outlines an integrative longitudinal process model for organizational change that addresses the following three dimensions: (p. 32)

  1. Aspects of organizational change;
  2. The change episode;
  3. Organizational effectiveness measures.

4. The case organization, Danske Data

Danske Data was a subsidiary of Den Danske Bank Group, and their primary business function was the development of business management software for Den Danske Bank Group. (p.35)

The primary goal for the SPI effort at Danske Data was to improve productivity by 10% (p.35)

5. Research findings

A careful analysis of the empirical data reveals the key problems that managers, practitioners, and the SPI group encountered with the measurement program. They were: (p. 38)

  1. Technical systems: size measurement
  2. People: customer and employee satisfaction
  3. Management practice: data discipline
  4. Management practice: using the data
  5. Organizing structure: project organization

6. Discussion

Measuring the effectiveness of SPI change initiatives is a complex undertaking, as these initiatives target various dimensions of the organization. Evidence from this longitudinal study suggests that the development, implementation and success of such a measurement program can benefit from a comprehensive authoritative framework. Such a framework could help guide SPI managers in their work, saving much time and cost. It can also enable early convergence on the central issues of the various stakeholders. (p. 41)

Our research suggests that although Applegate's framework for organizational change (Applegate, 1994) is general, it does address the critical dimensions SPI change. As such, it can be adopted and specialized for software organizations. (p. 41)

Early assessments from our studies of SPI suggest that (Applegate, 1994) can be useful for guiding both field studies and research and development of SPI measurement programs. (p. 42)

Tables and figures

Table 1: List of data sources Table 1 ~ p.32

Table 2: Organizational effectiveness measures (Applegate, 1994) Table 2 ~ p.35

Table 3: Indicators of the measurement program in Danske Data Table 3 ~ p.37

Table 4: Measures in Danske Data Table 4 ~ p.38

Figure 1: Organizational framework (Applegate, 1994) Figure 1 ~ p.33

Figure 2: Longitudinal perspective of organizational change (Applegate, 1994) Figure 2 ~ p.34

Figure 3: Organization of the SPI project in Danske Data Figure 3 ~ p.36

Figure 4: Data discipline in the performance measurement program Figure 4 ~ p.39


INF6001W, Longitudinal process research, Organizational change, Software metrics, Software process improvement, organizational change theory