«This research has been partially sponsored by the Dutch Joint Academic and Commer- cial Quality Research & Development (Jacquard) program on Software ...»
architectural compliance. The questionnaires and interviews may reveal potential issues in securing architectural compliance. For that reason, we send out the questionnaires to participants individually and conduct individual interviews (Obbink et al., 2002).
4. The suggestions collected in the interviews in turn are used to provide recommendations to the software development management of Organization A, including team leaders and product management.
5. Insight into what architectural rules are most important for the participants’ daily work enables us to study these rules. We can determine the structure that seems most appropriate to disseminate architectural rules. This may result in an improved classiﬁcation of the architectural rules. Finally, the architecture team uses the list of most important architectural rules to see if these rules have a higher relevance and require a different way of securing compliance with them.
3.4.1 Analysis of the population The potential population that has to comply with the architectural rules consists of all employees that are engaged in the architecture for which the architectural rules are compulsory.
The actual usage of the architectural rules across development sites provides information on that part of the population that actually reads them. We distinguish between different sites to see whether differences exist between those sites.
Organization A develops its software using four main development sites. Each site has a main conﬁguration manager role assigned. One of the tasks of the main conﬁguration manager is to grants employees access to the source code tree. We contacted the
46 3.4. Performing the Assessment
main conﬁguration manager of each site to obtain the number of employees involved in software development that reside at that site.
We used the log ﬁles of the intranet-site web server of the archnotes to get insight into the actual usage of archnotes. The analysis spans the period from January, 2005 until April, 2006. Fig. 3.3 shows the usage of archnotes at the four main development sites. The total number of employees involved is 515. Of these potential users of archnotes, 288 have accessed at least one archnote.
Although the number of potential users is roughly the same for the two major development sites (sites A and B), signiﬁcant differences can be observed in the actual number of users from those sites that access the archnotes. The site with the highest ratio of potential versus actual users (site A) is the site at which the architecture team resides.
The actual usage of the architectural rules provides information on the number of queries (‘hits’) on architectural rules from different sites. This helps in identifying what kinds of architectural rules are accessed most often. In the next step of the method, among other things, we determine if these architectural rules are indeed the architectural rules deemed most important for the participants’ daily work (and thus, whether existing rules satisfy the knowledge need).
Using the same log ﬁles, we counted the number of hits on archnotes. To prevent a bias, we regard multiple queries of a single user to a single archnote ﬁle within a period of two minutes as one hit. This two-minute threshold was chosen after a discussion with several architects – they indicated that every user of an archnote should be able to ﬁnd the information in it within two minutes.
We wanted to know which archnotes were accessed more often than others and distinguish between development sites. To this end, we plotted the number of hits per development site for the archnotes accessed most often – see Fig. 3.4. The usage of archnotes
provided valuable information as to what knowledge of the software architecture is accessed most often. Archnotes accessed most often pertain to software artifacts (such as subsystems1, naming conventions, and branching of subsystems) and coding standards.
These topics are typically covered in the development view or code view as covered by (Kruchten, 1995) and (Soni et al., 1995), respectively.
We used the analysis of the audience of the archnotes in two ways:
• To select participants for questionnaire-guided interviews to obtain insight into the actual status of securing architectural compliance and suggestions on how architectural compliance can be improved within Organization A.
• To determine whether archnotes that are accessed more often are also regarded to be more important to inform of architectural rules.
Analysis of the intranet website usage does not provide an undisputed view on the actual usage of architectural rules throughout Organization A: other mechanisms besides accessing a repository can be used to share architectural knowledge, such as informal chats and team presentations. In the interviews we verify the importance of the intranet website in disseminating architectural rules.
1 Especially the archnotes on subsystem lifecycle, describing the possible states of a subsystem, and subsystem releases, describing the process and conventions used when making a release of a subsystem for integration purposes, are important in GSD organizations. Compliance with these archnotes enables efﬁcient integration of subsystems.
3.4.2 Conduct questionnaires The next phase of our assessment method starts with developing a questionnaire based on the results collected so far. We use a questionnaire to gain understanding of the underlying reasons for the usage of the architectural rules. Furthermore, we try to obtain evidence to support our hypotheses stated earlier.
The questionnaire consists of the following elements:
1. Demographic data – Obtain name, role, years of experience, and subsystem experience of the participant.
2. Knowledge sharing – Identify practices and supporting mechanisms used by the participant to share knowledge, architectural knowledge in particular.
3. Opinion on the architecture team – Identify the participant’s opinion on the role of the architecture team and the activities the architecture team undertakes to disseminate architectural knowledge and secure compliance.
4. Importance of architectural rules – Identify the importance of the architectural rules for the daily activities of the participant. In this step, we tailor the questionnaire to contain the actual list of architectural rules that are applicable to Organization A.
5. Missing architectural rules – Identify what topics exist that require architectural rules, but do not have them.
The questionnaire is designed to obtain feedback from the assessment participants on each of the topics mentioned above. This feedback results in information on both the architectural rules themselves as well as the activities that are in place within the organization to disseminate and secure the architectural rules. In addition, the questionnaire allows participants to bring forward suggestions for improving the the current situation.
Based on the analysis of the audience of the archnotes we identiﬁed a representative subset of 20 employees. These employees include the most important roles from the four main development sites. We sent out the questionnaire to these employees and obtained
the following results:
1. We included 15 participants in either the subsystem architect or engineer roles.
The remainder of the participants were 2 conﬁguration managers, an event manager, a quality assurance ofﬁcer, and a requirements manager. The participants covered 58 % of all subsystems and three of the four main development sites. We selected 9 participants from site A, 9 from site B, and 2 from site C.
2. Table 3.1 shows the use of knowledge sharing practices at Organization A. We found that the most important mechanism to share knowledge is the formal mechanism of a change request. A change request is issued using the conﬁguration management system. The importance of other mechanisms for knowledge sharing is signiﬁcantly lower than the importance of a change request. These other
mechanisms all are less formal than using a change request. Although these informal mechanisms are important in general, they are applied less often due to the nature of the GSD organization. This has also been reported in (Ovaska et al., 2003).
3. All participants regarded the role of the architecture team as pivotal for disseminating the architectural rules. Table 3.3 lists the opinion of the participants on the architecture team. The table shows the degree to which the participants agree
with the statement. We observed the three statements with the lowest score (indicated in boldface in the table):
(a) participants felt that they were less regularly updated on the subjects addressed by the architecture team.
(b) participants indicated an improvement point in the response time of the architecture team on a speciﬁc request for the team issued directly or by delegation.
(c) participants felt that the architecture team can improve on the point of being in control of architectural activities. We investigated these reasons in order to foster suggestions for improvement during the follow-up interviews.
4. We asked the participants to indicate the ten most important archnotes for their daily activities. Table 3.2 shows the results. We see that most of the important archnotes are the ones that were accessed most often using the intranet website (see Fig. 3.4). Consequently, we concluded that the intranet website serves as an adequate mechanism to disseminate the architectural rules.
5. As a ﬁnal result of the questionnaire, we collected a list of missing architectural rules. Missing architectural rules concern architectural topics for which architectural rules are requested, but not present. The participants indicated to need
architectural rules for the following topics:
• Dynamic aspects (execution, memory consumption, inter-process calls) of the software architecture.
• Deadlocks and preventing deadlocks in software.
• Dealing with state changes in an end product.
• Integration strategies of different subsystems.
These architectural topics pertain to dynamic aspects of the software architecture.
Addressing these aspects may be done by introducing a process view according to (Kruchten, 1995). In the absence of architectural rules for these topics, the knowledge is currently obtained through informal communication. Although informal communication is important, it reduces traceability of compliance with these rules.
The questionnaire results show no difference in opinion at different development sites on important knowledge sharing mechanisms, the architecture team, or the most important archnotes. We already observed a low usage rate from site B, compared to the potential number of users of archnotes. We attributed this to the fact that the architecture team resides at site A. Employees at site B received notiﬁcations of new architectural rules in the form of a new or updated archnote. Consequently, site B might regard the archnotes as ‘not-invented-here’ and has a lower urge to read them and comply with the architectural rules. We used follow-up interviews to validate this hypothesis.
3.4.3 Conduct follow-up interviews The questionnaire results are collected and analyzed to formulate additional questions for the interviews. Possible questions arise from contradictions in the answers of participants and contradictions between the questionnaire response and the analysis of the population (see §3.4.1). In order to obtain unbiased feedback from the participants, the interviews are held with the participants individually. The interview results were studied and combined to determine overall improvement points.
We interviewed seventeen interviewees from the list of participants. In case multiple employees work on the same subsystem in the same role, we selected only one of these employee. While conducting the interviews, we speciﬁcally focused on the role of the architecture team to ﬁnd reasons for the scores listed in Table 3.3. Studying the interview results led to insight into the reasons for the scores and suggestions to improve the
scores. The interview results offered the following suggestions related to the entries in
boldface in Table 3.3:
• Several participants experienced uncertainty on certain architectural topics. The participants see projects deviate from the architectural rules for the sake of the project. The architecture team often participates in discussions and is aware of the deviations. Yet, these deviations are not communicated throughout the development organization. This lack of communication results in the opinion that the architecture team is not in control and trailing on projects.
• The visibility of the architecture team is further reduced by a lack of resources in capturing the architectural rules in archnotes. At the moment, nineteen archnotes have the status ‘planned’ and have not yet been written. Although a list of missing architectural topics was identiﬁed, these are not expected to be addressed within a few months.
• The effectiveness of the architecture team’s meetings can be improved. Members of the team that were interviewed indicated that both the current frequency of the meeting and the attendance should be higher. They felt that this would improve the visibility of the architecture team to the rest of the software development community.
Aside from these speciﬁc suggestions to improve the role of the architecture team, we identiﬁed other problems experienced by the participants. The most important problems
that prevent the organization from securing the architectural rules at different development sites are listed in Table 3.4.