1. INTRODUCTION *

2. REFERENCE DATA SET *

3. METRICS APPLIED *

4. DATA ANALYSIS *

5. RESULTS AND DISCUSSION *

6. OBSERVATIONS AND CONCLUSIONS *

ADDENDUM – METRICS DERIVED FROM JAVA MODULES *

ANNEX 1 METHOD METRICS *

ANNEX 2 REFERENCE DATA SUMMARY STATISTICS *

ANNEX 3 REFERENCE DATA ANALYSIS BREAKDOWN *

ANNEX 4 CLASS METRICS *

ANNEX 5 JAVA DATA SUMMARY STATISTICS (METHOD METRICS) *

ANNEX 6 JAVA DATA ANALYSIS BREAKDOWN (METHOD METRICS) *

ANNEX 7 JAVA DATA SUMMARY STATISTICS (CLASS METRICS) *

ANNEX 8 JAVA DATA ANALYSIS BREAKDOWN (CLASS METRICS) *

1. INTRODUCTION
1. General

This report forms part of the ESSI project PIOJAVA, which is an investigation of the effects on the software process of applying the Object-Oriented JAVA Language.

To allow evaluation of the change of process, the experiment applies metrics to existing non-Object-Oriented ‘C’ language programs to form reference measures for comparison during the execution of the baseline project.

The report identifies the metrics to be applied and provides reference data derived by their application. An addendum to this report includes an assessment with the selected metrics of a small number of JAVA modules.

2. Objectives

The objectives of this part of the experiment are:

1. To evaluate appropriate metrics using existing ‘C’ code modules produced within the Company.
2. To form a set of reference measures based on the existing ‘C’ code.
3. To identify any valid groupings which will assist in the efficient realisation of the experimental plan.
3. Definitions

For the purposes of this report, the following definitions have been adopted.

A Software Product is constructed by the compilation and linking of one or more Source Files. These Source Files are also referred to as Modules where they contain either function or method code.

Each Source File, which may contain either program code or other information, contains at least one Function or Declaration.

 

2. REFERENCE DATA SET

The Reference Data Set comprises software products from existing Terrafix products. The six software products which make up this set have been developed in the ‘C’ programming language.

The first four products contain embedded code; whilst the remaining two are non-embedded applications. Table 1 summarises the basic content of these software products.

	OPERATING SYSTEM	LANGUAGE	NUMBER OF SOURCE FILES	NUMBER OF METHODS
EMBEDDED
IIU	RTOS	C	20	197
MIP	RTOS	C	10	63
CIU-CHANNEL	RTOS	C	6	34
CIU-CONTROL	RTOS	C	8	78
NON-EMBEDDED
DGPSSERVER	Windows NT	C/C++	17	157
BASESTAT	DOS	C/C++	98	811

Table 1. Software products used for reference data evaluation

3. METRICS APPLIED

Software metrics were applied to this data to:

1. Check the accuracy of function and method metrics;
2. Provide reference metric measures to compare with the function and method metrics of the JAVA files measured at the conclusion of the baseline project.

 

1. List of method metrics

The following method metrics were applied:

LOC –Lines Of Code per Method.

V(G) – Cyclomatic Complexity.

OC – Operational Complexity.

CONTROL – Number of control statements.

EXEC – Number of executable statements.

NP – Number of Parameters per Method.

NION – Number of Input / Output Nodes.

CALLS – Number of Calls

BRANCH – Number of Branching Nodes.

NEST – Maximum Number of Levels.

Halstead's Software Sciences

n1 – Number of Unique Operators.

n2 – Number of Unique Operands.

n – Halstead Program Vocabulary (n1 + n2).

N1 – Total Number of Operators.

N2 – Total Number of Operands.

N – Halstead Program Length (N1 + N2).

V – Halstead Program Volume, calculated as V = N × (log₂n).

More detailed descriptions of the metrics are given in Annex 1.

 

2. Metrics classification

The metrics have been classified, where appropriate, into broader categories. It should be noted that some metrics appear in more than one category. The rationale and justification for this classification arise from the descriptions given in the experimental plan.

1. Structural complexity

LOC (Lines Of Code); Executable Statements; Halstead Software Science measures

2. Module complexity

V(G); OC (Operational Complexity); Control; Branch; Nesting

3. Cohesion

NION (Number of Input Output Nodes); Calls

4. Coupling

Calls; NOP (Number Of Parameters)

 

4. DATA ANALYSIS

Following the gathering of the metric measurements, three analyses have been performed on the resulting data.

1. Standard statistical calculations

A standard set of statistical calculations has been made on each of the original metrics: mean and standard deviation. This has enabled file-wide comparisons to be made.

2. Normalisation

The original measures were then normalised by dividing each value by the mean of the values for the file under consideration. This has normalised the mean to 1, and assisted in allowing the different metrics to be compared for values of standard deviation.

3. Comparison between and within module groups

A comparison has been made between the metric results for the embedded and non-embedded groups of modules, to determine if any significant differences are observable. The metrics within the groups were also examined.

5. RESULTS AND DISCUSSION

 

1. Method metrics

17 measurements have been taken for each of the 6 software products. The standard statistical calculations described in section 4 have been carried out to provide summary statistics. These summary statistics from the individual software products have been aggregated according to the categories described in section 2. These statistics are shown for the individual and aggregate categories in annex 2. The values for the aggregations are summarised in table 2.

	EMBEDDED			NONEMBEDDED			COMBINE
METRIC	MEAN	SD	NORM SD	MEAN	SD	NORM SD		MEAN	SD	NORM SD
LOC	62.49	173.05	2.76	34.77	73.24	2.11		42.47	111.02	2.61
NP	0.95	1.12	1.18	1.42	2.53	1.11		1.30	1.49	1.15
V(G)	8.26	28.31	3.42	4.93	9.60	1.95		5.86	17.05	2.91
OC	78.26	238.3	3.05	49.34	102.72	2.08		57.37	153.37	2.67
NEST	1.25	2.98	2.39	0.79	1.06	1.34		0.91	1.82	1.99
CALLS	1.72	4.29	2.50	2.11	4.52	2.14		2.00	4.46	2.23
BRANCH	2.09	10.61	5.08	1.23	4.38	3.56		1.47	6.72	4.57
NION	2.16	0.79	0.36	2.39	1.09	0.45		2.33	1.02	0.44
CONTROL	7.61	25.55	3.36	5.09	10.98	2.16		5.79	16.41	2.83
EXEC	28.94	90.64	3.13	16.82	35.73	2.12		20.19	56.81	2.81
Halstead Software Science
N1	117.7	351.12	2.98	77.76	164.08	2.11		88.85	232.21	2.61
N2	87.06	254.75	2.93	57.22	120.65	2.11		65.50	169.33	2.59
n1	13.45	7.57	0.56	11.60	7.94	0.68		12.11	7.88	0.65
n2	30.33	52.94	1.75	22.91	28.02	1.22		24.97	36.80	1.47
N	204.76	605.25	2.96	134.97	284.36	2.11		154.35	401.09	2.60
n	43.78	57.93	1.32	34.51	34.58	1.00		37.09	42.55	1.15
V	1413.2	5256.1	3.72	839.57	2108.4	2.51		998.82	3306.2	3.31

Table 2. Summary statistics

To evaluate differences between and within the aggregate groups, pairwise Student’s "t" tests were applied in the following manner:

• For each individual metric, the values for the aggregate groups were compared;
• Within each aggregate group, for each individual metric, the values for each software product were compared.

The results of these comparisons are shown in tables 3 and 4 for significance levels of p < 0.05 and p < 0.1 respectively. Differences, where they are significant at these levels, are shown as "D", with a subscript indicating the numbers of significant pairwise differences where appropriate. For information, further breakdowns of all of these results are given in annex 3.

As a further consistency check, the same calculations were performed on data sets in which outliers beyond twice the standard deviation had been removed.

METRIC	Between EMBEDDED and NONEMBEDDED	Within EMBEDDED GROUP (6 pairwise comparisons)	Within NONEMBEDDED GROUP
LOC	D	N	D
NP	D	D3	N
V(G)	D	N	N
OC	D	D3	D
NEST	D	N	N
CALLS	N	D2	D
BRANCH	N	D1	N
NION	D	N	D
CONTROL	D	N	N
EXEC	D	N	D
Halstead Software Science
N1	D	D1	D
N2	D	D2	D
n1	D	D1	N
n2	D	D2	D
N	D	D2	D
n	D	D2	D
V	D	D2	D

D – significant difference (subscript gives number of significant pairs); N – no significant difference

Table 3. Significant difference at p<0.05 level

METRIC	Between EMBEDDED and NONEMBEDDED	Within EMBEDDED GROUP (6 pairwise comparisons)	Within NONEMBEDDED GROUP
LOC	D	D2	D
NP	D	D3	N
V(G)	D	D3	D
OC	D	D3	D
NEST	D	N	D
CALLS	D	D3	D
BRANCH	D	D3	D
NION	D	D2	D
CONTROL	D	D2	N
EXEC	D	D3	D
Halstead Software Science
N1	D	D2	D
N2	D	D3	D
n1	D	D3	N
n2	D	D3	D
N	D	D3	D
n	D	D3	D
V	D	D3	D

D – significant difference (subscript gives number of significant pairs); N – no significant difference

Table 4 Significant difference at p<0.1 level

6. OBSERVATIONS AND CONCLUSIONS
1. Initial outcomes

The conclusions derived from these results are:

In all cases, a significant difference was observed between the embedded and the non-embedded aggregate groups. (It should be noted that the values for all of the metrics except CALLS and BRANCH were more highly significant.)

Within the embedded aggregate group, variations in significance were observed at varying levels which depended on software product and metric.

Within the non-embedded aggregate group, although variations in significance were observed, on this small sample significant differences were observed with certain metrics.

2. Application of conclusions

On the basis of these conclusions, the reference data set has been grouped into non-embedded, embedded and pooled subsets. This provides effectively 3 sets of reference data for the purposes of the project. These reference data set measurements are given in table 2, repeated below.

	EMBEDDED			NONEMBEDDED			COMBINE
METRIC	MEAN	SD	NORM SD	MEAN	SD	NORM SD		MEAN	SD	NORM SD
LOC	62.49	173.05	2.76	34.77	73.24	2.11		42.47	111.02	2.61
NP	0.95	1.12	1.18	1.42	2.53	1.11		1.30	1.49	1.15
V(G)	8.26	28.31	3.42	4.93	9.60	1.95		5.86	17.05	2.91
OC	78.26	238.3	3.05	49.34	102.72	2.08		57.37	153.37	2.67
NEST	1.25	2.98	2.39	0.79	1.06	1.34		0.91	1.82	1.99
CALLS	1.72	4.29	2.50	2.11	4.52	2.14		2.00	4.46	2.23
BRANCH	2.09	10.61	5.08	1.23	4.38	3.56		1.47	6.72	4.57
NION	2.16	0.79	0.36	2.39	1.09	0.45		2.33	1.02	0.44
CONTROL	7.61	25.55	3.36	5.09	10.98	2.16		5.79	16.41	2.83
EXEC	28.94	90.64	3.13	16.82	35.73	2.12		20.19	56.81	2.81
Halstead Software Science
N1	117.7	351.12	2.98	77.76	164.08	2.11		88.85	232.21	2.61
N2	87.06	254.75	2.93	57.22	120.65	2.11		65.50	169.33	2.59
n1	13.45	7.57	0.56	11.60	7.94	0.68		12.11	7.88	0.65
n2	30.33	52.94	1.75	22.91	28.02	1.22		24.97	36.80	1.47
N	204.76	605.25	2.96	134.97	284.36	2.11		154.35	401.09	2.60
n	43.78	57.93	1.32	34.51	34.58	1.00		37.09	42.55	1.15
V	1413.2	5256.1	3.72	839.57	2108.4	2.51		998.82	3306.2	3.31

Table 2. Metric reference data set

ADDENDUM – METRICS DERIVED FROM JAVA MODULES

1 Software modules from published JAVA product

To investigate the application of metrics to JAVA based software a small number of software products were assessed. The method metrics of section 3 and class metrics applicable to OO structured code were applied to these products.

Table A1 summarises these JAVA based software products.

 

PRODUCT	NUMBER OF SOURCE FILES	NUMBER OF METHODS	NUMBER OF CLASSES
JCVS	16	130	17
CVCS	21	432	19

Table A1. Software products used for JAVA data evaluation

The metrics measured from these data were used to:

1. Check the application of method and class metrics
2. To provide comparison data for the method and class metrics measured in the baseline project.

2 List of class metrics

The following class metrics were be applied to the JAVA modules:

CSA – Class Size in Attributes

CSO – Class Size on Operations (Methods)

PPPC – Percentage of Package, Public and Protected members in a Class

NOCC – Number Of Child Classes

DIT – Depth of Class Inheritance Tree

CBO – Coupling Between Objects

LOCM – Lack Of Cohesion of Methods

NOOC – Number of Operations Overridden by a Class

NOAC – Number of Operations Added by a Class

WMC – Weighted Methods per Class

LOC – Lines Of Code comprising a Class

PA – Public Accessors

More detailed descriptions of the metrics are given in Annex 4.

3 Metrics classification

The metrics have been classified, where appropriate, into broader categories. It should be noted that some metrics appear in more than one category. The rationale and justification for this classification arise from the descriptions given in the experimental plan.

3.1 Structural complexity

CSA (Class Size in Attributes); NOCC (Number Of Child Classes); DIT (Depth of class Inheritance Tree); NOAC (Number of Operations Added by a Class)

3.2 Module complexity

CSO (Class Size in Operations); NOOC (Number of Operations Overridden by a Class); WMC (Weighted Methods per Class)

3.3 Cohesion

NOCC (Number Of Child Classes); LOCM (Lack Of Cohesion of Methods); NOOC (Number of Operations Overridden by a Class)

3.4 Coupling

PPPC (percentage of Package, Public and Protected members in a Class); CBO (Coupling Between Objects); PA (Public Accessors)

4. Data analysis

 

4.1 Standard statistical calculations

A standard set of statistical calculations has been made on each of the original metrics: mean and standard deviation. This has enabled file-wide comparisons to be made.

4.2 Normalisation

The original measures were then normalised by dividing each value by the mean of the values for the file under consideration. This has normalised the mean to 1, and assisted in allowing the different metrics to be compared for values of standard deviation.

4.3 Comparison between and within module groups

A comparison has been made between the metric results obtained, and those for 3 groups of non Object-Oriented reference data (pooled, embedded and non-embedded groups of modules; main document section 6), to determine if any significant differences are observable. The metrics within the groups were also examined.

5 Results and discussion

5.1 Method metrics

17 measurements have been taken for each of the 2 available JAVA software products. The standard statistical calculations described in section 4 of the addendum have been carried out to provide summary statistics for the JAVA products. These summary statistics from the individual software products have been aggregated into categories of Embedded ‘C’, Non-Embedded ‘C’, Pooled ‘C’ (from the main document section 5) and JAVA. These statistics are shown for the individual and aggregate categories in annex 5. The values for the aggregations are summarised in table A2.

	EMBEDDED				NONEMBEDDED				COMBINE				JAVA
METRIC	MEAN	SD	NSD	MEAN		SD	NSD	MEAN		SD	NSD	MEAN		SD	NSD
LOC	62.49	173.05	2.76	34.77		73.24	2.11	42.47		111.02	2.61	22.09		48.57	2.20
NP	0.95	1.12	1.18	1.42		2.53	1.11	1.30		1.49	1.15	0.99		1.16	1.17
V(G)	8.26	28.31	3.42	4.93		9.60	1.95	5.86		17.05	2.91	2.76		5.20	1.88
OC	78.26	238.3	3.05	49.34		102.7	2.08	57.37		153.3	2.67	11.08		29.21	2.64
NEST	1.25	2.98	2.39	0.79		1.06	1.34	0.91		1.82	1.99	0.68		1.58	2.34
CALLS	1.72	4.29	2.50	2.11		4.52	2.14	2.00		4.46	2.23	0.41		1.27	3.11
BRANCH	2.09	10.61	5.08	1.23		4.38	3.56	1.47		6.72	4.57	1.08		2.07	1.91
NION	2.16	0.79	0.36	2.39		1.09	0.45	2.33		1.02	0.44	2.27		0.97	0.43
CONTROL	7.61	25.55	3.36	5.09		10.98	2.16	5.79		16.41	2.83	4.66		10.56	2.26
EXEC	28.94	90.64	3.13	16.82		35.73	2.12	20.19		56.81	2.81	4.12		10.38	2.52
Halstead Software Science
N1	117.7	351.1	2.98	77.76		164.1	2.11	88.85		232.2	2.61	43.86		106.8	2.44
N2	87.06	254.7	2.93	57.22		120.7	2.11	65.50		169.3	2.59	25.07		63.39	2.53
n1	13.45	7.57	0.56	11.60		7.94	0.68	12.11		7.88	0.65	8.61		6.04	0.70
n2	30.33	52.94	1.75	22.91		28.02	1.22	24.97		36.80	1.47	12.71		20.43	1.61
N	204.8	605.6	2.96	135.0		284.4	2.11	154.4		401.1	2.60	68.93		169.5	2.46
n	43.78	57.93	1.32	34.51		34.58	1.00	37.09		42.55	1.15	21.32		25.62	1.20
V	1413	5256	3.72	839.6		2108	2.51	999		3306	3.31	401.8		1227	3.05

Table A2. Summary statistics (Method metrics)

To evaluate differences between and within the groups, pairwise Student’s "t" tests were applied in the following manner:

• For each individual metric, the values were compared against the corresponding non object-oriented measurements;
• Within the JAVA aggregate group, for each individual metric, the values for each of the 2 software products were compared.

The results of these comparisons are shown in tables A3 and A4 for significance levels of p < 0.05 and p < 0.1 respectively. Differences, where they are significant at these levels, are shown as "D", with a subscript indicating the numbers of significant pairwise differences where appropriate. For information, further breakdowns of all of these results are given in annex 6.

 

METRIC	Between C CODE and JAVA	Between EMBEDDED and JAVA	Between NONEMBEDDED and JAVA	Within JAVA GROUP
LOC	D	D	D	N
NP	D	N	D	D
V(G)	D	D	D	N
OC	D	D	D	N
NEST	D	D	N	N
CALLS	D	D	D	N
BRANCH	D	D	N	N
NION	N	D	N	N
CONTROL	D	D	N	N
EXEC	D	D	D	N
Halstead Software Science
N1	D	D	D	N
N2	D	D	D	N
n1	D	D	D	N
n2	D	D	D	D
N	D	D	D	D
n	D	D	D	D
V	D	D	D	N

Table A3. Significant difference at p<0.05 level (Method metrics)

 

 

METRIC	Between C CODE and JAVA	Between EMBEDDED and JAVA	Between NONEMBEDDED and JAVA	Within JAVA GROUP
LOC	D	D	D	N
NP	D	N	D	D
V(G)	D	D	D	N
OC	D	D	D	N
NEST	D	D	D	N
CALLS	D	D	D	N
BRANCH	D	D	N	N
NION	N	D	D	N
CONTROL	D	D	N	N
EXEC	D	D	D	N
Halstead Software Science
N1	D	D	D	N
N2	D	D	D	N
n1	D	D	D	N
n2	D	D	D	D
N	D	D	D	D
n	D	D	D	D
V	D	D	D	N

 

Table A4. Significant difference at p<0.1 level (Method metrics)

5.2 Class metrics

12 measurements have been taken for each of the 2 available software products. The standard statistical calculations described in section 4 have been carried out to provide summary statistics. These summary statistics from the individual software products have been aggregated according to the categories described in section 3 of this addendum. These statistics are shown for the individual and aggregate categories in annex 7. The values for the aggregations are summarised in table A5.

 

	JAVA
METRIC	MEAN	SD	NSD
LOC	427.67	819.11	1.92
CSA	9.58	15.01	1.57
CSO	15.33	21.25	1.39
PPPPC	81.47	17.06	0.21
NOCC	0.17	1	6
DIT	0.17	0.38	2.27
CBO	1.56	2.48	1.59
LOCM	72.96	35.05	0.49
NOOC	0.03	0.17	6
NOAC	15.31	21.25	1.39
WMC	42.83	77.69	1.81
PA	0	0	*

Table A5. Summary statistics (Class metrics)

To evaluate differences between and within the groups, pairwise Student’s "t" tests were applied in the following manner:

• Within each aggregate group, for each individual metric, the values for each of the 2 software products were compared.

The results of these comparisons are shown in table A6 for significance levels of p < 0.05 and p < 0.1. Differences, where they are significant at these levels, are shown as "D", with a subscript indicating the numbers of significant pairwise differences where appropriate. For information, further breakdowns of all of these results are given in annex 8.

 

METRIC	P < 0.05	P < 0.1
LOC	N	D
CSA	D	D
CSO	D	D
PPPC	N	N
NOCC	N	N
DIT	D	D
CBO	N	N
LOCM	D	D
NOCC	N	N
NOAC	D	D
WMC	D	D
PA	*	*

 

Table A6. Significant difference at p<