«13TH INTERNATIONAL PUBLIC RELATIONS RESEARCH CONFERENCE “Ethical Issues for Public Relations Practice in a Multicultural World” Holiday Inn ...»
Sample and Descriptive Statistics of the International Survey The data collection of the international survey was accomplished by Watson Wyatt during 2007-2008 as part of the Watson Wyatt Communication ROI Study. However, the researchers were not involved in the questionnaire design and the data collection process.
Therefore, detailed description about the sample profile and survey procedure is missing in the article. Based on limited access to the online survey, the researchers confirmed that 264 senior 525 communication executives representing different regions and diverse industries participated in the ROI study and shared their opinions. The majority of the survey participants were from North America, with 182 in the U.S. (68.94%) and 44 in Canada (16.67%). Other participating regions included Europe (n=19, 7.20%) and Asia (n=19, 7.20%). The sizes of participants’ firms varied, ranging from medium-sized (1,000-2,500) to large corporations with more than 25,000 employees. Their range of services included financial advising/planning, health services, manufacturing, utility and transportation, wholesale, and others. Table 1 summarizes some demographic information of the participants by region, industry, and organization size.
Table 1. Categorical Demographic Profiles for the International Survey Participants
Study 2: In-depth Interviews During the second stage of the research project, a series of in-depth interviews were carried out with senior communication executives. With the support of the IABC Research Foundation, an electronic email invitation to participate in the in-depth interview was sent to IABC Gold Quill award recipients in the past five years (2004-2008). Of the 65 invitations that were sent out, 18communication professionals responded and indicated their interest in participating in the study. Eventually, five respondents dropped from the study due to changed schedule. Thus, final in-depth interviews were conducted with 13 senior business communicators who were Gold Quill award recipients.
Sample Profiles of Participants The research sample for the second project included 13 senior business communicators (7 women and 6 men) in different regions: six from the U.S., five in Canada, one from Mexico, and 526 one from Brazil. Although they are from different regions, they represented senior communication professionals in their region, with an average of 15 years of communications working experiences. Their job responsibilities ranged from public affairs, corporate communication, and strategic employee communications, to corporate reputation management and independent consulting.
All 13 interviews were finished via telephone. The interviews averaged 35.4 minute in length; the shortest lasted 26 minutes, and the longest was 57-minute in length. A qualitative research analytical technique—thematic analysis—was used to analyze the transcripts (Goulding, 2005). Because the objective of the in-depth interviews was to identify patterns and trends of the best practices and metrics that award-winning business communicators have used to address communication effectiveness and to gain support from senior organizational leaders, the researchers argued that the application of thematic interpretation is appropriate in this case.
Along with the results that have been generated in the survey data analysis, the research efforts from a qualitative perspective continue contributing to the exploration of effective business metrics in communication measurement.
Findings and Results Finding #1: Though communication effectiveness has been an important concern for organizational leaders, the assessment of communication effectiveness has not been widely applied by using business outcome metrics in organizations.
Analysis of the international survey revealed that nearly 46.6% (n=123) of respondents indicated that their company has used no formal assessment to measure the effectiveness of internal communication initiatives. Almost 36% of respondents in the survey indicated that less than 50% of their internal communication initiatives are assessed by business outcome metrics.
Only 17.2% reported that more than 50% of their internal communication initiatives are measured by business outcome metrics.
Similarly, the percentage of communication initiatives that have been measured by using business outcome metrics varied by organizational size. As the organization’s size increases, the percentage of formal measurement of communication effectiveness increases. For instance, due to the resources and financial advantages, organizations with more than 25,000 employees are more likely to develop business outcome metrics to measure communication effectiveness, and the percentage of no formal measurement/assessment of internal communication initiatives is 32%. On the other hand, organizations of small to medium size (1,000-5,000) showed a high percentage (60.9%-73.1%) of internal communication initiatives that are not measured.
Analysis of the in-depth interviews revealed that almost every participant agreed that measuring communication effectivenss was valued by senior leaders in their organizations. They also indicated that evaluation is “a daily job” and “a challenging job” for them. Participants
every one of them and sell the function and demonstrate that what you are doing can really bring benefits and changes with people internally.” “I think the biggest challenge in measurement continues to be convincing clients to spend, not so much the money, but to spend the time. As the industry develops, I don’t have a hard time in convincing them about the validity of measurement, but they are reluctant to actually take the time away from business to actually administer surveys or focus groups or some other measurement tools.” “We have to use different approaches to prove everyday that we are important and that we give results to the organization. So it has become a daily job for us to make sure they [senior organization leaders] understand that.” As a consequence, participants agreed that “seeing things built up and seeing the results” have been the most rewarding part of their efforts in measurement.
Previously, Watson Wyatt had grouped participating companies into two broader categories (high-effectiveness vs. low-effectiveness companies) based on its six years of communication ROI studies. The differences in perceptions and execution of measurement efforts between the two categories are also reflected in this study. Overall, high-effectiveness organizations are always willing to measure the effectiveness of the internal communication initiatives by using business outcome metrics (n=70, 26.8%) when compared to those loweffectiveness organizations (n=25, 9.8%). In addition, the majority of low-effectiveness organizations do not have a formal measurement to use to assess the effectiveness of communication initiatives (n=158, 59.8%).
Finding #2: Though most respondents agreed that measuring the effectiveness of internal communication initiatives should be part of standard operating practice in the organization, other factors such as scorecard balance, practice justification, and leadership direction also contribute to the use of business outcome metrics to the measurement process.
Almost half of the respondents indicated that the application of business metrics to the measurement of communication effectiveness is part of the standard operating practice within the organization (n=120, 45.5%). This feature is heavily reflected in organizations based in Europe (n=205, 77.8%) and those organizations in the finance/insurance industry (n=171, 65.0%).
However, respondents also indicated other reasons that drive them to put metrics in place to measure the effectiveness of communication initiatives. For instance, achieving a balanced scorecard was the second most frequently mentioned reason for measuring communication initiatives (n=44, 16.8%), followed by current practice and budget justification (n=42, 16.1%), and CEO/leadership directives (n=35, 13.3%).
It is clear that using measurement efforts as part of standard operating practice in the organization is often a strategy to demonstrate the value of public relations/communication
practices to achieving business results. Participants in the in-depth interviews explained:
528 “It is both a proof point of our value and also a challenge to us, because we are not as proficient in all of those areas as we probably need to be as a best profession. Clients are looking for value; they are looking for support; and they are looking for, I think, ways in which they can stay connected to all those important stakeholders.” “It is challenging that we have to explain sometimes once, sometimes more than once;
but the key thing is to make sure they understand the importance of those communication programs and strategies. It is really important to see us communicators helping the company to understand how stakeholders see them and what the best ways of relating with them are.” The survey results also indicated that low-effectiveness organizations had different reasons for putting metrics in place to measure the communication effectiveness. For instance, in high-effectiveness organizations, the measurement of communication effectiveness was validated as a standard operating practice and is used for creating a balanced scorecard. In loweffectiveness organizations, however, measurement of communication effectiveness has been used as an approach to acquire additional budget and/or staff, or it is a result of personal interest.
See Figure 1.1 for the graphic presentation of the percentages.
Respondents indicated that, although organizations use metrics to measure different aspects of internal communication efforts, five aspects or areas are measured more frequently.
These include: (1) increased employee awareness or understanding after the information has been delivered (80.5%, n=213); (2) whether delivered information helps employees do their job better (73.2%, n=193); (3) the extent to which internal communication initiatives impact employee behaviors (55.0%, n=145); (4) the affect of internal communication efforts on employee engagement (49.7%, n=131); and (5) the relationships between communication effectiveness and business performance, such as the revenue growth and customer satisfaction (33.6%, n=89). These five measurement areas are reflected most often in high-effectiveness organizations (see Figure 1.2 for details).
91.5% 83.1% 66.1% 62.2% 61.0% 59.5% 55.9% 45.9% 40.5% 10.8%
Survey results also indicated that geographic location and organizational size didn’t affect use of these measures. Thus, at the present time, these areas appear to be the core for measurement efforts in organizations. This is especially true for the areas of increased employee awareness or understanding, the effect of communication on employee engagement, and improved job performance.
Similar results were reflected in the interviews. Participants agreed that there is no one perfect approach to measure communication effectiveness. The situations for applying relevant business metrics to assess communication effectiveness are often multidimensional. Most participants said they focused on measuring employee participation, increased awareness and understanding of new programs or policies, increased workforce productivity, and higher levels
of employee engagement. One participant commented:
530 “I think the profession has gotten much more sophisticated over the years. We have evolved in the measuring process. Look at how we have moved: comprehension, attitude, behavior; all of those things are now measured. [Measurement] is part of what we do now.” Participants emphasized that for any communication project, the first important thing is to understand the client’s performance goals before adopting communication measures. They also mentioned the possibility of using outside resources or existing tools and metrics developed
by other professional firms, though they were leery of using templates:
“In terms of formalizing a metric, we don’t suggest using a template. The template or the training that someone developed would reflect more on his work. What we do is we start from our clients with goal setting and make sense of whatever budget they may carry for the project. From there, then we develop our measurements based upon their goals. What it does for us is to align us in accomplishing not what we want to accomplish but what our clients are trying to accomplish. These are two different things in measurement.” “What’s the most effective way? There’s no one way. You really need to bring in both quantitative and qualitative aspects because some clients are very much science-based and evidence-based or the other way. So what we do here is very heavily invested in research. We don’t generally go forward on anything significant unless we’ve got a good baseline of research.” Most participants also addressed the importance of conducting measurement at different levels. They mentioned that, to measure the effectiveness of internal communications, all involved groups have to be examined, e.g., top leaders, VPs and above, director-level employees, and employees in general.
Findings #4: The study highlighted a number of potential barriers to measurement initiatives in organizations. The three potential issues or barriers cited most often were: 1) insufficient resources (e.g., money and staff), 2) difficulties determining a specific causeand-effect relationship between communication initiatives and business results, and 3) time constraints.