Email: D.J.Leigh@soc.staffs.ac.uk

Email: C.J.Theaker@soc.staffs.ac.uk

Email: nb@terrafix.co.uk

Email: R.Mason@terrafix.co.uk

David John Leigh, Professor Colin J Theaker
Staffordshire University, Stafford, England

Neil Blackwood, Dr Robert Mason
Terrafix Limited, Stoke-on-Trent, England

The practical application of metric measurements to an evolving project is described. The initial implementation in a C-based non O-O environment has been assessed using a number of techniques. The migration to a Java-based O-O implementation is then discussed. The approach to measurements being taken at equivalent points is discussed. Design and temporal aspects are included.

1. Introduction

This paper reports on a collaborative venture between the company Terrafix Limited and researchers in software engineering at Staffordshire University. The company Terrafix produces vehicle location systems, high-performance data communication systems, and command and control room systems. Typical applications include support for the despatch and monitoring of ambulances in response to emergency calls. Whilst much of the development in the company is concerned with systems integration, there are complex real-time, multi-tasking, distributed and communications intensive requirements which prove challenging in developing systems of high integrity.

Historically, software development in the company has followed a pattern typical in many organisations. Written procedures provide the guidelines which software developers are expected to observe. There has been little or no use of CASE tools to support the software development. Most of the code has been written in C, and this has been modified incrementally for over a decade.

The company has recognised the need for a radical change of approach to software development, and this led naturally to collaboration with the software researchers at nearby Staffordshire University. A Teaching Company was established, with support from the UK Department of Trade and Industry, to focus on technology transfer into the company. The move to an object oriented paradigm was identified as an important first stage, particularly as the flexibility that it offered was considered important for building systems that had to be customised to each users' requirements. UML and Java were adopted as the new implementation vehicles, and staff training was started as part of the migration process. In 1998, an ESSI supported Process Improvement Experiment was started (Programme 27719 - PIOJAVA). This involved establishing a framework for the collection of metrics relating to the migration to an object oriented paradigm, and also to the improvements in both the development process and the resulting products.

The use of metrics within this environment was therefore seen as a way of monitoring activity to genuine productivity improvement.

2. Experimental design

There is an obvious risk when trying to assess the impact of a paradigm shift of ending up comparing 'apples' with 'pears' and concluding merely that they are different. The design of the experimental environment was therefore a very important activity "up front". The experimental monitoring is to take a period of approximately 18 months in total. During this time, a new product demonstrator for the company is being designed and implemented, and metrics are being collected at all stages of this development. This presents a number of opportunities for evaluating the impact of the paradigm change:

• An existing product with similar functionality was identified as a baseline. Although there was little data concerned with the process of its design, as it had evolved over many years, there existed a very large software base that could provide a reference datum.
• The new product demonstrator was being generated using UML as the design notation and Java as the implementation language. From the outset, it was decided to collect metrics for both the design and implementation processes in as automated a way as possible.
• It was recognised that probably the most significant impact of the paradigm shift was on the developers themselves, and that staff training was to play an increasingly important role as product development took place. As the project is scheduled to last 18 months, this also provided an opportunity to monitor the impact on the developers of experience gained at different stages. I.e., one would expect observable differences in 'quality' in components developed immediately after initial training to those produced after 12 months experience. It was decided to investigate this temporal aspect by the periodic snapshot of the system under development and its associated metrics. By the integration of metric measurements within all stages of the project, changing values of those measurements may be used to provide profiles of achievement and worth.

This led to an experimental structure, as illustrated in the following figure:

 

---

---

---

Figure 1 Experimental Structure

3. Practical considerations

As with any experimental process, constraints of a practical nature can have a major impact on the effectiveness of the experiment and the validity of its results. Several issues have been identified in this respect.

1. Configuration Management

As a management decision for the development of the product demonstrator, it was concluded that a strict configuration management regime should be used for all aspects of the new product (documentation, design, implementation, etc.). As high integrity was a major factor, the traceability of the design and implementation were of high priority from a commercial and marketing point of view. This could only be achieved through a controlled and automated regime. The benefit of this, however, is that it provided from the outset an opportunity for configuring and extending the chosen suite of configuration management tools, to enable the collection of metrics on the development process.

2. Multi-implementor considerations

Within any development team, there is inevitably considerable variability in the skill and experience of the team members. As the team itself is quite small, there is the potential for distortion of the experimental results, if only due to the size of the sample of designers and implementors being monitored. This could call into question any results that emerged, particularly as to whether they could transfer to a different development environment. It was concluded that this was a fact of life that could not be removed in the experiment design, and that the experiment was still of value.

The configuration management system was set up to allow the monitoring of individuals' activities. In this respect, the company culture is very important. The development team is close-knit and culpability is not an issue. This aspect has been raised in many other metrics programmes, where the emphasis is on quality and company improvement rather than on individual's performance.

3. Choice of metrics and metric packages

The issue arose as to whether the requirements of the programme could be met using commercially available metrics packages, or whether specially written metric software needed to be developed. A number of packages were identified and evaluated, capable of functioning with both 'classical' implementation languages and the object oriented languages projected for the product demonstrator. The base-line project was used as the test vehicle for these tools. Whilst no tools were completely satisfactory, and in some cases the results that they generated could best be described as bizarre, it was decided that the use of commercially available packages was the best return on investment, and that special software, particularly providing an interface into the configuration management suite, would only be developed as needed.

The choice of metrics is itself a major issue, particularly as it relates to both process and product, and to object oriented development as well as the classical approach in the base-line product. This will be reported elsewhere.

4. Experience in the comparison of implementations
1. Migration from the classical base-line implementation

The base-line project had been carried out in a number of ways, in a mixture of formal and informal approaches. This had been found, pragmatically, to be appropriate to the mixture of applications and interfaces used by the project.

Measurement of appropriate metrics for the design stage of the base-line project, while not at all impossible, is difficult to carry out consistently on non-computer-system-based data. Techniques used on pictorially presented or semi-formal descriptions can be time-consuming, and not favoured for continuous monitoring in a commercially-based environment. Information may be gathered on a "one-off" snapshot basis, for comparison with metrics from the demonstrator project, but it is not generally thought viable to re-evaluate metrics when changes to the base-line system are being made.

The tools appropriate to O-O design, particularly CASE tools, enabled a more robust and consistent approach to be taken. In this way, the semi-automatic measurement of formal metrics becomes possible at earlier stages in the software life-cycle using files of design information. The advantages of monitoring changes without undue extra effort were considered paramount in the commercially-based environment, as more relevant information can be acted upon in a timely fashion.

2. Temporal aspects of monitoring

A number of possible measurement scenarios were evaluated, so that proper advantage could be taken of the information becoming available from the metrics. The overriding consideration was of commercially viable replication of outcomes, so that advantage of the understanding gained could be taken in the future. At the same time, it was important that the on-going implementation effort should not be jeopardised.

The approach taken was that there should be two aspects of the effort: the implementor’s and the monitor’s. These would be kept separate, unless a significant deviation became apparent in the implementation, which had become plain because of the measurements being taken. At the same time, more traditional project management techniques would be employed, so that the possibility of intervention on such a basis would be minimised.

Alternative possibilities for taking measurements were explored, based on extent of measurement and frequency. It was decided that metric measurements were to be taken on a regular snapshot basis, expected to be every two weeks, with corresponding archiving of relevant source and design files. This represented a compromise between the diversion of overmuch effort to outcomes, which were essentially non-productive, and the need to gain information at a suitably granular level to inform future project activities to be undertaken. This is an ongoing activity.

3. Continued evaluation of the base-line project

It was felt to be important that measurements of suitable project worth already in use should be continued, to provide a basis for comparison. This would be in terms of effort and efficiency; although less formally gathered, as the configuration management system was not being applied to the existing product lines, it was hoped that the corresponding profiles would be in general agreement.

Code inspections form the foundation of this work. They are typically taken at one or two stages during the continued enhancement of the existing system. They inform the existing quality control and assurance activities associated with the project, and reflect the feedback mechanisms that have proved successful over a prolonged period.

5. Summary

This paper presents the structure of experimental work in progress, into the migration to an object oriented paradigm, for a company involved in the development of high integrity and advanced technology systems. The project includes comparison with a base-line project, using classical development techniques, and also investigates the temporal aspects of the migration to object orientation that accompanies the adoption of new tools and languages and the training of staff in the new paradigm.

The initial collection of metrics for the base-line project has already been performed and the development of the product demonstrator is under way. Specific measurements taken within the project will be reported at the workshop.