«This research has been partially sponsored by the Dutch Joint Academic and Commer- cial Quality Research & Development (Jacquard) program on Software ...»
7.4 Research Approach 7.4.1 Organization context We conducted our research at the Dutch branch of a large, international IT service provider, Organization D. The Dutch branch employs over 6,000 employees spread across a number of business sectors, e.g., public, ﬁnance, and energy.
Organization D aims for standardization of its software development processes1 and associated tools. Associated tools include a central conﬁguration management system, standardized electronic project workplaces, and standardized collaboration and communication tooling.
The software development methodology in use by the service provider can be characterized as ‘mixed delivery’. Accordingly, the service provider chooses for an appropriate proportion of activities performed on-site at the customer and works towards an ideal mix between on-site, off-site (at one of the service provider’s ofﬁces), near- and off shored software development.
7.4.2 Approach We conducted seven semi-structured interviews at prominent multi-site projects at Organization D to collect an initial view of the software development practices that are in place at the organization. The interviews focused on the context of the software development project, the size of the software development project (both in number of project members and workload in terms of necessary man days), and the development methodology. This helped us to construct the questionnaire.
Based on the insight obtained during the semi-structured interviews, we decided to make several changes to the way the architectural knowledge management practices were initially described in Chapter 6. Main reasons for this were to speciﬁcally adapt the practices to the jargon of Organization D to create an understandable questionnaire.
Hence, for some practices, we decided to reformulate the practice; in these cases,the exact phrase used in the questionnaire is listed between brackets in Table 7.1. We made
the following overall changes:
7 Travel to other project locations – We explicitly added this practice on travelling. Where the practice “cross-site delegation” focuses on traveling to another development to actually do (a substantial amount of) work for a longer period, this practice focuses on traveling for e.g., attending meetings of for project management coordination. Organization D was primarily interested in identifying the best means of project management coordination (e.g., via video-conferencing or via physical visits).
1 The organization aims for CMMI level three (Poort et al., 2007).
96 7.4. Research Approach
8 Know who is who across the project (directory) – Organization D was experimenting with developing project intranet sites to contain all information relevant to the project (including all the project stakeholders). Obtaining a view on the perceived usefulness of knowing ‘who is who’ would help in better deﬁning what information should be on these intranet sites. So, after consultation with Organization D, we decided to add this practice.
9 Have a shared infrastructure for work products (e.g., documents, scripts, plans) and source code (conﬁguration management) – Organization D runs several projects with international partners (consortia) in which access to a shared conﬁguration management system was not always possible due to legal constraints. The organization was interested to ﬁnd out to what extent this was perceived as a problem, and to identify possible differences with non-partnered projects. Hence, we added this practice.
– Collocated high-level architecture phase – The practice to develop the highlevel architecture in a collocated manner, a practice distilled from the requirements engineering literature (see Chapter 6), is not directly supported by Organization D or their quality management system. In order not to interfere too much with the management practices implemented at Organization D, we decided to remove this practice.
As a basis for the population of the questionnaire, we decided to use the members of a mailing list to which all Dutch architects of Organization D are subscribed. In this way, we are able to gather information on the complete portfolio of software development projects and thus have variations in the number of sites involved in these projects. In addition, we added relevant roles (architects and designers) from the speciﬁc multisite projects where we conducted the semi-structured interviews. We sent out an email invitation to the architecture mailing list, in which we invited the members to complete the questionnaire.
We asked the participants to complete the questionnaire with a certain (past or ongoing) software development project in mind. Consequently, by indicating the perceived usefulness of a speciﬁc architectural knowledge management practice, a respondent does this for the software development project at hand.
Table 7.1 lists the parts of our questionnaire relevant to this chapter.
We inquired a list of all the locations where software development occurs for possible future geographic analysis. Furthermore, we decided to use a three-point Likert-scale as described by (Jacoby and Matell, 1971; Likert, 1932) with values “Yes”, “Neutral or Don’t Know”, and “No”. Speciﬁcally, we grouped “Neutral” and “Don’t Know” because we are primarily interested in the signiﬁcant outlying results, forcing the respondents to a clear statement.
Table 7.1: Survey excerpt with those questions relevant to this research
8. Please list all locations where project members were performing software development activities.
10. What practices for architectural knowledge management can be useful within your project?
(“Yes”, “Neutral or Don’t know”, “No”)
1. Frequent interaction across sites
2. Cross-site delegation (representatives of each team visit other teams)
3. Face-to-face project kick-off meeting
4. Urgent request (email, mailing list or telephone to quickly get information)
5. Have a clear organization/project structure with communication responsibilities
6. Have a single repository for architecture artifacts (establish a repository for architecture artifacts)
7. Travel to other project locations
8. Know who is who across the project (directory)
9. Have a shared infrastructure for work products and source code (conﬁguration management)
7.5 Analysis and Results 7.5.1 Demographic information The total population consisted of 363 employees of Organization D. These employees, all members of the architecture mailing list, are performing architecting or architectingrelated activities. We received 132 responses, which corresponds to a response rate of 36.4 %. Not all of these 132 responses were complete. We selected only the responses that provided answers to the questions that are relevant to our research (see Table 7.1).
This resulted in a set of 114 respondents, corresponding to an adjusted response rate of 31.4 %. We used these responses as a basis for our research. Although the data does not permit to determine the total number of projects due to conﬁdentiality purposes, analysis of the responses reveals that the responses include at least 70 different software development projects.
The average experience in IT industry of the respondents was 15.2 years. The soft
ware development projects on average included 48 members. The average duration of the software development project was 193 working days, the average duration of the involvement of the respondents in the software development was 68 working days.
7.5.2 Data preparation Table 7.2 shows the distribution of the 114 respondents over the number of sites in the software project they were involved in. A site is a separate, geographic location (possibly the customer’s location) where a team performs software development activities (Sangwan et al., 2006).
We regard the number of respondents in each individual category of 4, 5, 6, and 7 sites too low to draw conclusions from. Consequently, we sum up the invidual responses of these categories, and form a new group (“4–7 sites”), with 17 responses.
For the analysis of the usefulness of a speciﬁc architectural knowledge management practice, we only took into account respondents who rated the perceived r of that practice by answering the appropriate subquestion of question 10. Hence, the response rate per architectural knowledge management practice may differ slightly because not all participants answered all subquestions of question 10 in Table 7.1.
7.5.3 Perceived usefulness of architectural knowledge management practices Following our research question, we relate the perceived usefulness of each architectural knowledge management practice to the number of sites involved in the software development project of that respondent. We describe our results in the remainder of this section.
We list each practice and start by summarizing the intention of the practice from Chapter 6. Next, we summarize the results of the perceived usefulness of the practice by the respondents. The Appendix shows the responses for each architectural knowledge management practices graphically.
1. Frequent interaction across sites – This practice intends to let practitioners from different sites interact frequently with each other. Interaction may be done through a
variety of means, e.g., by using collaboration software such as video-conferencing or wikis, or through on-site, face-to-face interaction. This practice is primarily important for architects to share their views on the architecture with other project stakeholders.
Other research further distinguishes between e.g., audio support and video support (Thomas et al., 2007). Based on the standardization in Organization D’s software development methodology and the combined use of audio support and video support, we have not made this distinction.
We observe a positive attitude towards the usefulness of this practice in GSD. Regardless the number of sites, more than 50% of the respondents regards frequent interaction between the teams involved to be valuable.
2. Cross-site delegation (representatives of each team visit other teams) – This practice intends to obtain better integration between teams from different development sites involved in the software development project by having teams visit each other (e.g., during joint architecture meetings or pair software design or development).
The respondents value this practice, although not as much as some other practices.
Furthermore, we observe a clear (positive) peak in the perceived usefulness of this practice at projects that involve three sites, and a lowered value from respondents involved in four or more sites.
3. Face-to-face project kickoff meeting – This practice intends to establish initial relationships across sites by bringing together all project members to hold a joint kickoff meeting.
The respondents clearly value this practice: regardless of the number of sites involved, more than 70 % of the respondents value this practice; its perceived usefulness shows a slight increase with the number of sites increasing, but a peculiar drop at three sites.
4. Urgent request (email, mailing list or telephone to quickly get information) – This practice intends to quickly collect information on a given topic of interest.
Organization D has implemented a group mailing address that includes all Dutch architects2. The results show that our respondents clearly value this practice; again, more than 70 % of the respondents perceive this practice as being useful, regardless of the number of sites. The pattern, however, is different. We observe a drop in perceived usefulness at two sites, and a steady increase towards three and four or more sites.
5. Have a clear organization/project structure with communication responsibilities – As indicated in Chapter 6, we deﬁne ‘organization as ‘project’ in the validation of the usefulness of this practice at Organization D, since we are primarily interested in identifying differences across projects in terms of the number of development sites involved in the projects.
We observe that the respondents value this practice less than the previously mentioned practices. We would have expected to observe an increase in perceived usefulness since projects with a higher number of sites often include more project members and a larger variety of roles involved. Instead, we observe that only 30.8 % of the respondents who are involved in a project with four sites or more value this practice.
2 In fact, we sent out our questionnaire using this list.
100 7.6. Discussion of the Results
6. Travel to other project locations – This practice intends to obtain better integration between teams involved in the software development project by having architects or architecting roles from different development sites visit each other physically.
About 40 % of the respondents value this practice. One exception exists: 66.7 % of the respondents involved in a project with three sites value the practice.
7. Have a single repository for architecture artifacts (establish a repository for architecture artifacts) – This practice intends to build a repository to store architectural decisions including the rationale of these decisions.
The respondents value the perceived usefulness of this practice lowest of all architectural knowledge management practice; at most 40 % of the respondents at each site perceive this practice as useful. In addition, we observe a drop in perceived usefulness at the group of respondents working in software development projects that include 4 or more sites. This is reﬂected by a moderately low correlation of -0.114.
8. Know who is who across the project (directory) – This practice intends to allow project members to quickly know who is working within the project. This can be done by providing a directory, consisting of e.g., a mugshot and phone or email contact information.