«13TH INTERNATIONAL PUBLIC RELATIONS RESEARCH CONFERENCE “Ethical Issues for Public Relations Practice in a Multicultural World” Holiday Inn ...»
Wall, M. (2005). 'Blogs of war': Weblogs as news. Journalism, 6(2), 153-172.
Wilcox, D. L., & Cameron, G. T. (2008). Public Relations: Strategies and Tactics (9 ed.).
Boston: Allyn & Bacon.
Yang, S.-U., & Lim, J. S. (2009). The Effects of Blog-Mediated Public Relations (BMPR) on Relational Trust. Journal of Public Relations Research, 21(3), 341-359.
Zaichkowsky, J. L. (1985). Measuring the involvement construct. Journal of Consumer Research, 12(3), 341-352.
Zaichkowsky, J. L. (1994). The personal involvement inventory: Reduction, revision, and application to advertising. Journal of Advertising, 23(4), 59-70.
614 Doing Measurement Right: On the Road to ROI Mark R. Phillips, VP of Corporate Communications, USO, and doctoral candidate, Department of Communication, University of Maryland
This is the second report from the USO’s (United Service Organizations) longitudinal program of communication metrics development and evaluation. It is driven by the organization’s desire to understand the effectiveness of its communication program and to be able to use that understanding to fine tune communication programs, activities and resources.
A year ago, we presented the background design and goals of the USO’s new communication measurement and evaluation program. The gist of that presentation was that we had identified the strengths and weaknesses of the USO’s in-house communication capabilities, developed a long-term measurement and evaluation schema, completed a year-long data collection process for mainstream media data and conducted a national public awareness survey.
These last two activities provided baseline assessments for the longitudinal study.
During the past year, we continued to collect and analyze data for the mainstream media database and began collecting data that would enable correlations between communication activities and the USO’s overall key performance indicators. Such data include USO web traffic, social media activity, public service announcement (PSA) program data, customer satisfaction, organizational climate and employee satisfaction, online donation and other revenue data. These data sets are now being integrated into the dashboard, so that we can identify correlations and determine actionable meaning buried within the data.
Developing a Metrics Schema For us, the first step in developing a metrics schema was to very clearly identify those groups of people with whom we need to communicate and about what. To do this, we asked the question, “At the most fundamental level, who does the USO depend upon for its existence and operation?” We also asked, “Who depends upon the USO?” The answers to these two questions identified the constellation of strategic stakeholders for the USO. Furthermore, we noted that the more important that organization-stakeholder interdependence was to either or both parties, the more strategically important that stakeholder relationship was to the USO.
The resulting constellation of stakeholders is presented in Figure 1. This constellation represents those groups of people who are vital to the organization. By “vital,” we mean that they organization cannot exist or thrive without these groups of people. Serving the customers (i.e., American troops and their families) is the USO’s reason for being. For the most part, the other groups of stakeholders have an affinity for these customers and, therefore, are allied with the USO as they help the organization fulfill its mission of lifting the spirits of the troops and their families.
Figure 1. The USO’s constellation of strategic stakeholders.
Once the constellation of strategic stakeholders had been identified, we had answered the “Who” question. Next, we needed to answer the “What” and “Why” questions. Essentially, these were, “For what do our stakeholders depend on us and for what do we depend on them?” and “Why is this important to each of us?” These questions help us confirm that the level of interdependence means that these relationships are strategically important to each party.
These questions also help us understand what it is that we need to communicate about. In a nutshell, the core of the USO’s communication with each stakeholder group should focus on our interdependence. Using the relationship with our customers as an example, they need to know programs and services the USO has to offer and how to access these programs and services. The USO needs to know whether it is offering the programs and services that customers need and want, along with the degree to which customers are satisfied with these offerings.
With the constellation of strategic stakeholders identified, along with a clear understanding of what constitutes each group’s stake in the USO and vice versa, the next step was to develop a metrics schema that would guide the collection, assessment and reporting of communication data. Our presentation at the 2009 IPRRC conference, along with the accompanying paper, laid out these metrics in detail; we have consolidated that into a more easily digested form, presented in Figure 2.
Figure 2. USO metrics schema overview.
The metrics laid out in Figure 2 should be read from left to right, with importance to the organization increasing as one moves from outputs to impacts. It should be noted that there is not universal agreement on communication metrics terminology (e.g., some organizations use outputs, outcomes and outgrowths), but these are the terms that work for us.
Outputs represent “busyness;” they are the things that keep organizational communicators, especially those working in a technician role, busy. These activities include writing, coordinating and disseminating press releases and social media releases, producing brochures and other collateral materials, building web pages, engaging in social media communities, shooting and editing photographs and video, and so forth. Measuring these outputs is essentially a simple accounting of the numbers of things produced. These could be numbers of press releases written and sent, media events staged and collateral materials (e.g., fact sheets, brochures, white papers and annual reports) produced. When viewed alone, output metrics are not very useful, but they serve as the first step in assessing the efficiency of an organization’s communication programs. It also sets the stage for determining the ROI (return on investment) of communication programs.
618 Measuring processes is an attempt to understand the level of efficiency of an organization’s communication programs. These process metrics can focus on the time between an event and a corresponding press release, social media release, blog post or tweet on Twitter.
Process metrics can also focus on the accuracy of communication products; an example of this could be the assessment of error rates in press release drafts that have been sent out for coordination. Another process metric can focus on the degree to which organizational publications or collateral materials (e.g., brochures, fact sheets, web pages and so forth) are current. In this context, currency refers to the degree to which these products contain up-to-date facts, figures, photos, video and messages. As with output metrics, these process metrics are useful mainly in assessing the efficiency of an organization’s communication activities.
Outcomes are intermediate results that lie somewhere between the things produced and the effects that those things create. For the USO, outcome metrics include media coverage from pitches and releases, as well as PSA (public service announcement) program media values that also result from producing high-quality PSAs and pitching them to public service directors.
It is a more thorough understanding of the impact of our communication program that we are really trying to develop via this evaluation process. For the USO’s purposes, impact can be assessed in terms of changes in stakeholder perceptions, behaviors and the resulting quality of the relationships between the organization and its stakeholders. Within this model, perception includes awareness and understanding of the USO, its mission and how that is connected to stakeholders’ lives. Perception also includes stakeholder opinions (i.e., what they think about the USO) and attitudes (i.e., how they feel about the USO).
Behaviors are unique to each stakeholder group, but each is vitally important. For example, vital donor behavior is in the act of donating money or in-kind support. Volunteers and celebrity entertainers donate their time. The board of governors provide strategic guidance and access for business development. Members of the Congressional caucus may vote one way or another on appropriations bills that provide some financial support to the organization. Thus, each stakeholder group behaves in ways that are either supportive of the organization or not, just as the organization behaves in ways that might or might not support its stakeholders.
A simplified way of viewing this model is that we produce things (Outputs: Releases, videos, photos, tweets, blog posts, web pages, and so forth), that people have the opportunity to experience (Outcomes: Opportunities to see) and that can change their perceptions, behaviors and our relationships (Impacts).
Ultimately, goal of the USO’s communication program is to help the organization foster long-term, stable, mutually beneficial relationships with its strategic stakeholders. The communication metrics schema reflects this, as the output, process, outcome and other impact metrics are designed to support relationship building.
one measures things that are more important to the organization, it is increasingly difficult to attribute changes in those measurements to specific communication inputs.
Feeling the Elephant: Communication Measurement in the Round We now have two full years of data from mainstream media coverage of the USO. This rich data was collected by monitoring 385 publications and broadcast outlets in the top 200 DMAs (i.e., Designated Market Areas) across the United States. Mentions of the USO in these media outlets are identified, captured and filtered by a third party media monitoring service.
Mentions of “USO” that are about other organizations or uses of the acronym (e.g., references to Universal Service Obligations, U.S. Open tennis tournament, and so forth) are filtered out. Also filtered out are incidental mentions, such as obituaries that mention that the deceased had been a USO volunteer or that they had met their spouse at a USO decades earlier.
These data collected from mainstream media coverage allow us to understand the number of articles, tone of coverage (Figure 3), and subjects covered (Figure 4); which departments were covered (Figure 5) and the level of exposure (i.e., opportunities to see or “reach” of the coverage); the extent to which key messages were present in the coverage (Figure 6); the accuracy and completeness of USO messages in coverage (Figure 7); the degree to which the coverage was focused on the USO (Figure 8); where the USO coverage occurred in the publication or broadcast (i.e., whether the USO was the lead story or was buried); and what share of coverage the USO garnered as compared to peer organizations. These assessments were explained in the authors’ 2009 paper, “Doing Measurement Right: One Organization’s Experience Creating A Best-In-Class Measurement Program From Scratch,” which is available on the IPRRC website. Additionally, this data set enables analysis of mainstream media coverage by geographic location, media type, communication campaign, and may be aggregated by day, week, month or year.
620 Figure 3. Monthly articles in mainstream media by tone.
this is by looking, over time, at the volume of opportunities to see, tonality of coverage, the degree of USO visibility in media coverage, and the presence of key messages (Figure 9).
Figure 9. Key performance indicators for mainstream media coverage.
Few if any organizations, whether for-profit, non-profit or governmental, can afford to do communication measurement and evaluation simply for the sake of doing it. Data collection, evaluation and reporting is a time-consuming and typically not inexpensive process. Ultimately, measurement has to be able to inform decision-making and help organizations more effectively plan and execute their communication and other operations.
One example is at Figure 10, which shows the relationships between opportunities to see (OTS) and numbers of articles by quarter in a comparison of 2008 and 2009 media coverage.
For non-profit organizations, which depend on donations for operating capital, the last quarter of the year is the most important fund-raising period of the year. This is due to a combination of holidays, during which time people tend to be more generous, and the approach of the end of the tax year, which drives charitable contributions that can be counted as tax deductions. During the fourth quarter of 2009, the USO made a deliberate, concerted effort to communicate as broadly as possible. This included a significant outreach effort to national media. If one simply looks at the number of stories in Q4 of 2009 as compared to 2008, there was some increase, but it was not 624 dramatic. Conversely, if one compares the level of exposure (i.e., OTS) of the same periods, the increase is dramatic. It is not until one considers both changes in numbers of stories and OTS that the true picture emerges: The Q4 2009 push for coverage in national media was successful.
Figure 11. Year-over-year comparison of OTS and articles by quarter (2008-2009).
Another example is seen in Figure 11, which correlates donations received through the USO website (www.uso.org) with episodic national media coverage. In the summer of 2009, the USO conducted an Operation USO Care Package stuffing event on the south lawn of the White House with President Obama and the Pittsburgh Steelers, a similar care package event on NBC’s Today Show plaza with Dr. Jill Biden and the show’s hosts, and then took The Colbert Report to Iraq, where a week of shows were taped with a military audience in Baghdad. These events generated substantial national media coverage and Figure 11 correlates online donations with media coverage.