«This research has been partially sponsored by the Dutch Joint Academic and Commer- cial Quality Research & Development (Jacquard) program on Software ...»
We observe no speciﬁc impact of the number of sites involved onto the perceived usefulness of this architectural knowledge management practice. This is reﬂected by a correlation coefﬁcient of -0.041. The perceived usefulness of this practice is not as high as we had expected from previous research; in (Clerc et al., 2007a) we showed that the ‘yellow pages’ containing directory information were the most popular of all system-related information.
9. Have a shared infrastructure for work products and source code (conﬁguration management) – This practice intends to have a shared environment or infrastructure where project members can share work products like documents, plans, and source code.
This architectural knowledge management practice is in general perceived as useful, regardless of the number of sites involved. A prime reason for this may be that this practice is highly supported by the IT service provider’s software development methodology, as described in §7.4.1.
7.6 Discussion of the Results 7.6.1 Analysis of the number of sites So far we have presented the perceived usefulness of each architectural knowledge management practice related to the number of development sites. Now we can combine these results to explain differences observed. Fig. 7.1 shows these combined results.
We make a number of observations based on these results:
• First, the perception of the usefulness of the architectural knowledge management practices does not differ between projects with one development site (i.e.,
local) and projects that involve two sites (i.e., distributed). Only the architectural knowledge management practices ‘using email, mailing list or telephone to quickly get information’ and ‘have a clear organization/project structure’ show a drop of more than 10 % in perceived usefulness at two sites. The difference in perceived usefulness of the remaining seven practices does not differ signiﬁcantly between one or two sites involved.
• Second, we observe a peak in perceived usefulness at software development projects that involve three sites. Only the architectural knowledge management practice ‘frequent interaction’ shows a drop; all other practices show an increase in perceived usefulness. Five out of nine architectural knowledge management practices show an increase of more than 10 %.
We have statistically analyzed the observed differences as listed above to verify whether the difference in perceived usefulness observed at software development projects with three sites is signiﬁcant. We assume that the population follows a normal distribution and applied the chi-squared test (Greenwood and Nikulin, 1996) to test whether the observed variance in the results is statistically signiﬁcant. To this end, we formulated
the following hypotheses:
• H0 – The responses of the group with three sites does not differ signiﬁcantly from each of the other groups.
• H1 – The responses of the group with three sites differ signiﬁcantly from each of the other groups.
102 7.6. Discussion of the Results
For each architectural knowledge management practice, we compared the responses of each group (one site, two sites, three sites, and four or more sites) with the other groups. In general, we observe very high p-values (above 0.5) which result in not being able to discard the null hypotheses; the observed difference in perceived usefulness of a practice per number of sites involved does not appear to be signiﬁcant. However, we do observe signiﬁcantly lower p-values (about 0.1) when we observe the usefulness of architectural knowledge management practices focusing on visiting other teams and traveling to other project locations of three sites. Although the p-value observed is still above the 0.05 threshold that is common in empirical research, these lower p-values do urge us to investigate this more thoroughly (see §7.6.3). The main reason for not having obtained statistically signiﬁcant result is that the data is highly skewed (the number of responses across the different number of sites varies, cf. Table 7.2)3.
7.6.2 Analysis of architectural knowledge management practices As with the difference in perceived usefulness of practices depending on the number of sites involved, we combined the results to make some observations. First of all, a difference exists between the perceived usefulness of codiﬁcation practices and personalization practices. In Chapter 6, we described the codiﬁcation practice ‘have a single repository for architecture artifacts’ and this research added ‘have a shared infrastructure for work products and source code (conﬁguration management)’, as described in §7.4.2. These codiﬁcation practices are not perceived as being as useful as the practices that focus on personalization. Rather than a focus on codiﬁcation, the respondents tend to rely on other forms of communication in which practitioners are linked directly to each other; examples include sharing knowledge during meetings or through other communication mechanisms such as email or mailing lists.
Second, we can relate the observed increase in perceived usefulness of certain architectural knowledge management practices with the patterns observed at other practices.
It is important to meet each other at the start of the software development project, during its kick-off. This, in turn, may reduce the importance of knowing ‘who is who’ across the project; a kick-off meeting already caters for this. In addition, when people already know each other, they are more willing to respond to each other’s email or provide each other with information through a mailing list. Hence, we conjecture that a temporal relationship exists between the architectural knowledge management practices (“ﬁrst do a, because it next reduces the need for b”).
7.6.3 What’s special about three sites?
Analysis of the questionnaire results shows that no signiﬁcant differences in the perceived usefulness can be identiﬁed depending on the number of sites. However, as noted before, we did observe a difference in perceived usefulness of practices at respondents participating in software development projects that involve three sites.
3 For this reason, we have chosen not to list all p-values in this section.
1037. The Usefulness of AKM Practices in GSD
In order to analyze the difference, we validated the results described in §7.6.1 and §7.6.2 with key representatives of the IT service provider during Organization D’s annual event for Dutch software and systems engineers.
Of the 102 participants to this event, 15 joined a session that was organized for the purpose of the validation. We presented the ﬁndings as listed in §7.6.1 and §7.6.2 and
conducted a facilitated group discussion on these ﬁndings. We list the results below:
• Projects with four or more sites appear to be more organized. E.g., according to the participants, these projects need to organize communication across sites, because it will be more complex and expensive when it is not organized. Hence, communication often occurs through a ‘hub-and-spoke’ model, which results in complexity O(n) instead of O(n2 ), where n is the number of sites involved.
• Traveling does not scale when the number of sites involved in the software development project increases.
• Software development projects are initially thought to be set up either with more than three sites or with only one or two sites. Projects with three sites typically evolved to that stage in the course of the project, i.e., they started as a one- or two-site project. Following that, the main conclusion of the participants of the validation session is that software development projects with three sites organize work in a different way than projects with a different number of sites.
To validate the above statements, we directly contacted the 18 respondents that were involved in a software development project with three sites and the 11 respondents involved in projects with four sites. We asked them a number of questions to investigate
further characteristics of these projects:
• How was the software development project organized when it started? What was the size and number of sites at the start?
• How is work currently divided in the software development project? Is there a central site that divides the work across the three sites?
We collected responses from three different software development projects that involved three sites and one project that involved four sites.
We learned that two software projects with three sites did not start as such but evolved to that stage after having run for several months with two sites. The two sites that were involved at the start of the project included the customer’s location and one of the development centers of Organization D near the customer’s location. After a couple of months, project management decided to move software development activities to the service provider’s off shore development centers. At that time, communication was not structurally reorganized, which resulted in less communication between the sites. In addition, the practitioners involved in software projects with three sites indicated an increased perceived usefulness for some architectural knowledge management practices because they encountered several problems in their project. The practitioners indicated that the architectural knowledge management practices were not implemented
104 7.7. Conclusions
but, rather, thought that applying these practices would help in addressing these problems.
Conversely, the software development project with four sites was designed as such at the start of the project and remained stable in number of sites since then. Most architectural knowledge management practices were either implemented or discarded on beforehand; e.g., traveling does not scale with four geographically distributed development sites, so other measures had already been taken.
In conclusion, software development projects that encounter an increase in number of sites should be keen to implement additional architectural knowledge management practices in time to prevent problems from surfacing.
Internal validity – Numerous other factors than the number of sites of a software development project may affect the perceived usefulness of architectural knowledge management practices. Although in this research we primarily focus on the number of sites, we used the validation session at Organization D to identify several other, underlying reasons, as described in §7.6.3.
External validity – We conducted our questionnaire at one organization. We asked the respondents selected from this organization to answer the questionnaire with a certain software development project in mind. When this questionnaire is repeated over time, the same group of respondents may be involved in different software development projects and answer the questions according to that project. Given the nature of our population (the complete population of Dutch architects of the service provider) and the variety of software development projects encountered in this questionnaire, we do not expect signiﬁcant deviations. However, when the questionnaire is completed by another organization, different results may be obtained.
7.7 Conclusions We have conducted empirical research on the relation between the usefulness of architectural knowledge management practices in global software development. Speciﬁcally, we related the usefulness of these practices to the number of sites involved in several GSD projects at a large IT service provider in the Netherlands. We sent out a questionnaire to the Dutch architects of this organization to obtain our results. Using the questionnaire, we related the number of sites to the perceived usefulness of nine practices for architectural knowledge management in GSD, most of which were identiﬁed in earlier research (described in Chapter 6). Seven of these nine practices support a personalization strategy towards knowledge management, the remaining two support a codiﬁcation strategy towards knowledge management.
1057. The Usefulness of AKM Practices in GSD
The architectural knowledge management practices in general are perceived as useful (about 40 % of the respondents regards the architectural knowledge management practices for GSD as useful). Personalization practices in general are perceived as more useful than codiﬁcation practices. Some architectural knowledge management practices show an increase in usefulness as the number of sites increases. This increase peaks at software development projects that involve three sites. Other practices are regarded as useful regardless of the number of sites involved. We learned that the main reason for the increase is that practitioners involved in projects with three sites have a need for additional architectural knowledge management practices because certain problems are encountered. Several three-site software development projects were not started with three sites but evolved into that stage after having run for some time with two sites.
The high perceived usefulness for some architectural knowledge management practices in three-site software development projects denotes a need to have these practices implemented proactively to prevent problems with architectural knowledge sharing from arising.