Terrafix Limited

Version 1.0

This is the mid-term report for project PIOJAVA – 27719. The project is concerned with evaluating the process improvements that can be gained through migration to the use of JAVA and its object oriented paradigm.

At this point in the programme, a reference data set has been generated by collecting metrics from a large number of existing software modules from the company’s current product range. These are largely written in C. A study of appropriate metrics collection tools has been undertaken, and a tool set, able to collect metrics for both object oriented and non object oriented software has been identified. The programme is now entering the phase where JAVA code for the new product line is being generated.

To avoid the situation where the collection of metrics would actually interfere with the work of the software professionals within the company, an attempt has been made to automate the data collection, using a configuration management system.

The primary lessons learnt so far from this project have been concerned with the infrastructure for software development, more than related to the actual paradigm shift. This includes experience in the use of configuration management software for data collection, and the vagaries of ‘off-the-shelf’ packages for collecting software metrics. Results from the main point of the experiment, namely relating to the software structure and the development experience in generating object oriented software will start to appear in the next few months.

The results of this experiment will be of interest to all businesses and developers considering a change to JAVA and an object oriented paradigm, particularly where the software to be generated is real-time and subject to severe operational constraints.

The work reported here was undertaken by Terrafix Limited, with consultancy support from Staffordshire University School of Computing. The project is an ESSI Process Improvement Experiment (PIOJAVA 27719) fully funded by the European Commission.

Terrafix is a UK SME that produces world leading vehicle location, data communication and control room systems. The main application areas are for the emergency services and other organisations that require a command and control capability, including facilities for mapping and vehicle tracking. These systems are highly software dependent, with tight constraints that involve complex real-time, multi-tasking, distributed and communications intensive requirements spanning diverse platforms.

The customer base is also distinctive, and this impacts on the specific functionality of the systems, which are tailored to the individual service requirements, and the needs to support thousands of individual mobile/portable units. This has obvious implications on the maintainability of the system components, and particularly on the ability to upgrade as new technologies are introduced. It has also been noted that clients often request software related changes to the functionality at short notice. To meet these market needs, it is important that the company is responsive to this very specialised market, and consequently the ease of software production and change is of significant importance to the business. At the same time, it is important to retain high quality, reliability and performance in the most cost-effective way.

Historically, the software development process at Terrafix has been controlled by written procedures, which the software professionals are required to observe. Most existing code is written in C and has been incrementally modified for over a decade. This has resulted in highly functional but difficult to maintain and modify software modules. Portability across platforms is also an issue. The company has recognised the need for a radical change in its software approach and to this end, the company is adopting object oriented design techniques within its new development programmes.

This experiment is designed to complement this move by assessing the impact of introducing an object oriented programming language for systems implementation. For the experiment the transition to JAVA has been chosen. The rationale is that JAVA is designed to be modular (due to its enforced object oriented structure), multi-tasking (due to its user controllable multi-threading capability), platform independent (due to a fixed strict binary interface and virtual machine approach) and tightly structured (being defined in such a way that some of the vagaries of C and C++ are not allowed). It can be compiled onto diverse platforms for speed advantages and processors directly running JAVA byte-code are available. It is becoming the de-facto standard for mobile and SMART card applications, and most importantly, provides the capability to integrate simply with communications networks.

The baseline project is a subproject within the development of an advanced command, control and communication system. It addresses the mobile/portable element used for vehicle and person location, data messaging and database functions. This mobile/portable development requires real-time, multi-tasking, multi-media, communications intensive and resilient programming in a cost sensitive environment. The project is to produce a pre-production demonstrator, the code from which will be implemented on diverse platforms.

Objective assessment of the impact of JAVA implementation will be achieved by comparison of measures of both costs and software structure. These measures have been applied prior to the commencement of the baseline project to existing company products with similar functionality, thereby providing a reference datum. Metric measurements of coupling, cohesion and complexity are being used to allow direct analysis of any process improvements, particularly relating to code structure. In addition, continuous assessment throughout the experiment provides a further evaluation of the impact of the paradigm shift, particularly in relation to development costs (as evaluated by actual costs and code re-use).

The specific objectives of the project are therefore:

These factors affect both technical performance of the software (a, b and c above) and business performance in terms of product costs, time to market and market share (a, b, c and d above). The results will determine if the measurable gains in better software structure arising from the use of JAVA do indeed also have an impact on development costs and in the longer term such aspects as reliability and maintainability. In addition, issues regarding the performance of JAVA in an intensive environment will be highlighted.

The origins of the experiment stem largely from the drive within the company to introduce advanced methods of software engineering, in order to sustain a competitive advantage over similar system suppliers. Historically the company has considerable experience at developing custom built systems, largely driven by high performance and customised hardware. As the technology base has changed, there is an increasing emphasis on software production and the quality standards that go with it. This has been a significant motivating factor behind the move towards higher quality software production.

The baseline project is part of the development of a new product demonstrator based on the next generation of enhanced real-time command and control systems. The functionality of these includes real-time mapping, communications and database manipulation, controlled through multi-media interfaces. They are therefore heavily dependent on the software sub-systems. As a SME providing complex and often customer specific equipment to critical services, the need for quality, reliability, maintainability and robustness of the products is paramount. Part of this product demonstrator provides the baseline for the PIE experiment. It is expected that the performance of the PIE will allow assessment of a major change in programming paradigm in a critical and highly demanding application. Improvement in software practices and product cost effectiveness are the immediate business outcomes.

The experiment is primarily concerned with evaluating the impact of a shift to a JAVA based object oriented paradigm. The experimental programme is designed to evaluate:

(a) Object oriented concepts for real-time system development.

(b) JAVA for small and portable embedded systems.

(c) The impact of a virtual machine approach.

Each of these can be evaluated by the single process change to using JAVA. Additionally, long-term aspects of an object oriented approach such as re-use and maintainability are being investigated, but these are not within the scope of the present experiment but short term characteristics of these properties will be reported.

The experimental team comprises one member of staff dedicated to establishing the necessary tools and support environment for the collection of experimental data and performing the analysis of the metric data. Reference data is derived from existing product modules and will be compared with experimental data collected during the base-line project. A number of staff are involved in developing the base-line product and consultancy advice is being given by Staffordshire University. This latter source of expertise has provided assistance with the parts of the work plan concerned with planning and experiment design and the pre-assessment phase where existing product modules are examined. Relevant training in JAVA and object orientation is also the responsibility of the University.

Terrafix produce vehicle location and data communication equipment varying in complexity from simple in-car locators to large tracking and communications systems for emergency and public services. The systems are diverse and incorporate data terminals and communications systems in mobiles interlinked with (static and mobile) command and control centres.

The company has approximately 40 employees. Historically its electronics engineering expertise has been the foundation on which the company has been built. It has always sought to achieve high quality of its manufactured products, particularly given the customer base, which is largely in the emergency service industries.

The Company recognises the importance of quality, particularly from a business standpoint as this is a major marketing issue. There is an ongoing strategy for process improvement; accreditation to quality standards and procedures defined by ISO9001 and TickIT has contributed to this. However, it also recognises that technological developments have changed the face of the business, with systems integration using high performance off-the-shelf components becoming more of a core activity. This in turn increases the need for flexible configuration of systems and an increasing emphasis on the engineering of the software components that need to be customised to specific requirements. The need to have demonstrable quality is therefore a very strong motivation. The experimental structure described in this report is an important and ongoing aspect of the company’s desire to improve the management and monitoring of its systems development.

The management structure within the company is relatively flat, with senior managers being in close contact with the technical decision making, and all staff being aware of the business implications of the day-to-day activities. This is considered to be an important factor for the introduction of a metrics programme, as staff are aware that this is concerned with process improvement, and consequently the longer term well being of the company, rather than a way of monitoring individual performance, which might have connotations of culpability.

The baseline project is a subproject within the larger product development (TERRAFIX 2000) outlined earlier. This development is addressing the provision of the next generation of command, control and communication systems encompassing fixed and mobile resources. Command elements and controlled elements may both be mobile and will integrate advanced techniques for location, data, voice and other multimedia processing and communications. It is expected that this will be the basis of the main company product lines for the near future.

The baseline project is the development of new mobile/portable systems within this framework. These systems will replace the current company products with similar functionality. The current systems (based on embedded processors) provide asynchronous local communications to navigation equipment, vehicle sensors and actuators, provide user data terminal facilities, database access, geodetic calculations and global communications via various (and sometimes multiple) media. All operations are real-time, inherently multi-tasking, communication intensive and require high reliability.

This project uses an embedded processor platform (Motorola Power PC) with a real-time kernel running a JAVA virtual machine in which the multi-threaded applications will operate. The baseline project will produce an initial pre-production demonstrator of the system whose operational software will be eventually implemented on diverse platforms as part of the larger development. Code portability is therefore of high importance. At the time of commencing the experiment, the requirements of the demonstrator have been identified, although the design and coding has yet to start.

The baseline project is expected to involve up to six software professionals who will be applying their differing expertise to various aspects of the mobile software systems. It is planned to be of ten months duration. In terms of turnover of staff at the mid-point stage, one professional left the project team immediately before software development commenced and one professional has just joined the development team.

The experiment will provide data to evaluate the linkage between structure metrics and development costs. For assessment of the impact of the process improvement caused by JAVA implementation it is assumed, in the absence of other data, that a percentage improvement in software structure will have a related percentage reduction in development costs; the experiment will clarify this assumption.

During the baseline development a regime of data collection and analysis is being applied to allow objective measures of software structure and development process times (and hence costs) to be produced. These measures will be compared with the reference data set drawn from existing product lines, and their variation with project time assessed to evaluate both step-change improvements and derive the time profile for learning and other effects.

To allow quantitative analysis of all these effects it is necessary to have objective means of measurement and an accurate but non-invasive method of data gathering. To this end therefore metrics will be applied to the reference and development software to measure coupling, cohesion and complexity throughout the experiment. The use of these metrics has been validated on the reference data at the beginning of the experiment and will be monitored for the base line project during the experiment.

It is important that the data collection should not interference with the work of the software professionals. Either paper or log-file based recording would be invasive, potentially inaccurate and might also disrupt the normal development work therefore distorting the experimental results. It was therefore decided that use of a configuration management system to perform the data acquisition was important.

A configuration management tool was identified to assist in the management of a software repository. A number of tools were evaluated, on the basis of a set of ‘critical success factors’. Cost was also a consideration here, particularly as SMEs have probably greater constraints where support infrastructures are concerned. Two evaluations of these are available in [Fish 1999, Lawson 1999]. Ultimately the Perforce system was selected [Perforce 1999], particularly as it presented an opportunity to extend its potential for metrics collection using scripts.

As the baseline software is still at an early stage of its development and the Perforce software has only recently been installed, this capability has yet to be fully tested. An important aspect of the experiment uses monitoring of the changes made to the software modules at different times in the development lifecycle to assess the impact of the paradigm shift and the changes that have occurred within the staff involved in the development. This monitoring will further identify re-use of software components. There is some confidence that the Perforce tool will provide an appropriate capability; this will be described upon in the final report.

A very important issue that has been addressed concerns the actual collection of metrics. The experiment design, reported in [PLAN], identifies the precise measures that should be taken for both the reference data set, based on the original company products, and the new software based on the object oriented paradigm. The issue was also addressed as to whether the requirements of the programme could be met using commercially available metrics packages, or whether specially written metric software needed to be developed.

A number of packages have been identified and evaluated, capable of functioning with both 'classical' implementation languages and object oriented JAVA for the baseline project. The reference software modules were used as the test vehicle for these tools. Whilst no tools are completely satisfactory, and in some cases the results that they generated could best be described as bizarre, it has been decided that the use of commercially available packages is the best return on investment, and that special software, particularly providing an interface into the configuration management suite, would only be developed as needed. However, it was identified that caution is necessary in the interpretation of all metrics produced by standard software packages, because in some circumstances calculations were incorrect and terminology varies widely across vendors.

The metrics package identified as best meeting the needs of this project is KRAKATAU, produced by Powersoft [Powersoft 1999]. A number of very important lessons were learnt from this evaluation exercise and these are reported in section 6 of this report.

Training has been, and is being, undertaken in both JAVA and UML. Although the project focuses on the implementation language, as far as the migration to object orientation is concerned, a number of design methods appropriate for the development were also assessed. In particular, the MOOSE method, used within Esprit project OMI/DE 6909 [Morris et al 1996] was considered because of its capability for hardware/software co-design. However, the lack of industry strength tools precluded its adoption. The growing support for UML, particularly with its extensions for real-time development, were considered a major attraction over other object oriented approaches for design.

Some initial training has been undertaken in both JAVA and UML, although this experience has still to propagate through the entire design team. This is planned for the immediate future.

The original work plan identified eight work packages:

In terms of progress within the project, WP0 and WP1 are ongoing, and will continue for the lifetime of the project.

The planning stage WP2 has been completed, to identify the experimental process and the metrics and stages at which the measures will be collected. A deliverable [PLAN] has been submitted. An important issue that was raised in developing the plan, was identifying appropriate metrics that could be used in the assessment, particularly their relevance (or otherwise) for evaluating O-O and non O-O software.

The Pre-assessment stage WP3 has been completed, with a deliverable [REF] that reports on the measures derived from the reference data, namely the existing product range. This includes approximately 150 modules of software, with around 1300 methods (procedures) developed in C.

The training aspects for JAVA and object oriented techniques of WP4 are ongoing and are provided as necessary to the staff involved.

The main metrics collection occurs during WP5. This has been delayed within the base-line project, for reasons beyond the scope of the PIE project. However, the development phase is now under way in the base-line project, following the recruitment of new staff, and metrics will start to be generated within the new quarter. WP6 will be undertaken during and particularly on the conclusion of WP5.

The dissemination of results has already commenced, using the experiment design and preliminary findings as the basis for conference and workshop papers. In particular, the experiment design was reported in [Leigh et al, 1999] and a paper, based on the initial software evaluations, has been submitted [Theaker et al, 1999].

Assistance was required by the Company throughout the experiment, but particularly at its inception, to advise and assist with configuration control, metrics to be used and data analysis. The School of Computing at Staffordshire University was chosen for this task due to their expertise and facilities in the relevant areas, their ongoing co-operation with industry and their proximity to the Company site.

The consultants have been involved at various stages of the project so far, in particular assisting in the definition of the experimental plan and in supporting company staff in the metric analysis leading to the REF reference data set. The training of staff in both UML and JAVA involves University academics as identified in the project plan.

As this report covers the work to the mid term, the impact has still to be fully measured and evaluated. However, there are some immediate effects that can be reported, based on the introduction of a metrics programme within the company and the initial analysis of the reference data set based on the current product line.

The importance of the paradigm shift has been accepted by the management of the company, even though it is currently unsubstantiated by solid measurements. It is firmly believed that results generated by the programme will verify the commitment to JAVA and object oriented approaches for software development.

The true business benefits have yet to be realised. However, with a very discerning customer base that itself is under public scrutiny, the importance of being able to deliver quality software systems will provide the company with a market edge.

The introduction of new methods for software development and for configuration management and control are regarded as a formalisation of the quality assurance methods that the company currently embody within its other development activities, and its wish to ensure conformity to recognised standards and ‘best practice’. This will have an impact on the organisation and related processes for software development.

The cultural impact is perhaps less than might be observed within more rigid, top-down developmental organisations, as the recognition of a programme for process improvement might be viewed as beneficial for the company as a whole, and in the context of a close knitted organisation, to the benefit of all its employees. No resistance to change has so far been observed. This is still an interim finding, as the full impact of introducing new technology, such as object orientation and support through configuration management systems, has still to hit the main software development teams.

The skills impact is probably the most significant, as the paradigm shift to an object oriented paradigm involves both the design and implementation technologies. The company is committed to this development, but would clearly like to assess the impact in terms of productivity and quality of the delivered product.

This section is again premature in terms of considering the impact of the experimental process, at the mid-term stage.

There have been some considerable lessons learnt about the use of metrics tool sets, particularly with respect to their suitability for quantifying the characteristics of traditional software development approaches, using languages such as C. The variability of measures derived from ‘off the shelf’ tool sets is very noticeable. However, the following observations have been made by staff involved in analysis of the metrics so far:

• ‘Quality’ is not a straightforward concept as far as the interpretation of metrics is concerned and suitable metrics are not always available.
• The most dependable metrics (and easiest to produce, such as Lines of Code, LOC) are of relatively little use.
• Metrics at the design stage are difficult to acquire, except from textual based material.
• Dependability of the metrics is a major issue, as packages that are, say 95% reliable, still require considerable additional scrutiny to ensure an acceptable level of confidence. Fortunately, when erroneous results are returned, they tend to be extremely out of alignment.
• The data returned from standard packages needs considerable additional post processing and analysis to be of use – the output is not ‘user friendly’.
• Correlation between O-O and non O-O metrics is an interesting issue for further investigation. Many programs do not truly measure O-O concepts.

An obvious conclusion that can also be drawn from this work is that an integrated metrics system that includes the design stages of software development is very important, particularly if it can process designs in machine-readable form.

Now that the baseline project is synchronised with the ESSI programme, it is expected that tangible results will start to flow from this programme. The whole basis of metrics collection has been clarified with respect to object oriented software development and the tools available, and the baseline project is now experiencing a major drive with a view to delivering a product demonstrator well within the timeframe of the metrics programme. Experience in using the metrics tools has been gained, and this will be transferred to the analysis of the software generated within the base line project.

The dissemination of results is an ongoing activity, with preliminary papers about the experiment design and the reference data set having already been generated for access within the community.