We are always on the lookout for different methods and approaches for evaluation. Here is a new method that we haven’t come across before: “the evaluation café“.

Following is a brief description:

The Evaluation Café is a method for group facilitation that allows stakeholders of a project or programme to evaluate its impact in an informal brief session. The purpose of the Evaluation Café is to build and document stakeholders’ views on success and impacts after a planned activity.

More information>>

I recently conducted a one day training workshop for the staff of Gellis Communications on communications evaluation. We looked at several aspects including:

• How to evaluate communication  programmes,  products and campaigns;
• How to use the “theory of change” concept;
• Methods specific to communication evaluation including expert reviews, network mapping and tracking mechanisms;
• Options for reporting evaluation findings;
• Case studies and examples on all of the above.

Gellis Communications and myself are happy to share the presentation slides used during the workshop – just see  below (these were combined with practical exercises – write to me if you would like copies)

Online tools, such as corporate websites, members’ directories or portals increasingly play an important role in communications’ strategies.  And of course,  they are increasingly important to evaluate.

I just concluded an evaluation of an online tool, created to facilitate the exchange of information amongst a specific community. The tool in question, the Central Register of Disaster Management Capacities is managed by the United Nations Office for the Coordination of Humanitarian Affairs.

The evaluation methodology that I used for evaluating this online tool is interesting as it combines:

• Content analysis
• Network mapping
• Online survey
• Interviews
• Expert review
• Web metrics

And for once, you can dig into the methodology and findings as the evaluation report is available publicly: View the full report here (pdf) >>

Last week I gave a presentation on evaluation for communicators (pdf) at the International Federation of Red Cross and Red Crescent Societies. A communicator asked me what trends had I seen in communications evaluation, particularly relevant to the non-profit sector. This got me thinking and here are some of the trends I have seen in 2008 that I believe are an indication of some directions in 2009:

Measuring web & social media: as websites and social media increasingly grow in importance for communication programmes, so to is the necessity to have the capacity to measure what their impact is. Web analytics has grown in importance as will the ability to measure social media.

Media monitoring not the be-all and end-all: after many years of organisations only focusing on media monitoring as the means of measuring communications, there is finally some realisation that media monitoring is an interesting gauge of visibility but not more. Organisations are now interested more and more in having some qualitative analysis of data collected (such as looking at how influential the media are, the tone and the importance).

Use of non-intrusive or natural data:  organisations are also now considering “non-intrusive” or “natural” data – information that already exists – e.g. blog / video posts, customer comments, attendance records,  conference papers, etc.  As I’ve written about before, this data is underated by evaluators as everyone rushes to survey and interview people.

Belated arrival of results-based management: Despite existing for over 50 years, results-based management or management by objectives is just arriving in many organsations. What does this mean for communicators? It means that at the minimum they have to set measurable objectives for their activities – which is starting to happen. They have no more excuses(pdf) for not evaluating!

Glenn

I’ve written before about survey respones and the use of “don’t know” as an option on a Likert scale. What I said was that in some situations, a person may not have an opinion on a subject – and cannot say if they agree or disagree – so it may be wise to include a “don’t know” option. Well, i just read an interesting article that suggests that people who respond “don’t know” may actually have an opinion – it’s just that they may require a longer amount of time to develop confidence or awareness of their choice. The article gives an example of how the opinion of undecided people can be acurately predicted by creative means:

In a recent study, 33 residents of an Italian town initially told interviewers that they were undecided about their attitude toward a controversial expansion of a nearby American military base. But researchers found that those people’s opinions could be predicted by measuring how quickly they made automatic associations between photographs of the military base with positive or negative words.

Read the full article here>>

I came across this useful table from the Devcom blog which explains how research can be used at different stages of communication projects. There are many elements that will be familiar to readers, but what caught my eye was the first method “audience analysis” – which is often ignored by communicators in their rush to create materials and campaigns. The blog also has an example of an audience analysis (pdf) for readers. And method 3 – pretesting of prototype material – is another step often skipped over.

Read the full post here >>

No doubt you have heard of the Millenium Development Goals (MDGs), eight broad goals on poverty, ill-health, etc, agreed upon by all countries to try and reach by 2015.

From a monitoring and evaluation point-of-view, what is interesting is that these goals are broad sweeping statements, such as:

Goal 1: Eradicate Extreme Hunger and Poverty

Goal 3: Promote Gender Equality and Empower Women

One could ask – how can these broad goals be possibly monitored and evaluated?

As detailed on this MDGs monitoring website, what has been done is to set specific indicators for each goal, for example:

Goal 3: Promote Gender Equality and Empower Women

Description: Eliminate gender disparity in primary and secondary education, preferably by 2005, and in all levels of education no later than 2015

Indicators:
3.1 Ratios of girls to boys in primary, secondary and tertiary education
3.2 Share of women in wage employment in the non-agricultural sector
3.3 Proportion of seats held by women in national parliament

So from broad goals, the MDGs focus on two to seven specific indicators per goal that they are monitoring. That’s an interesting approach, as often we see broad goals set by organisations and then no attempt made to actually detail any indicators.

the MDGs monitoring website plays an active role in monitoring these indicators combining quantitative data (statistics) and qualitative data (case studies) – also an interesting approach to show how such indicators can be tracked.

Glenn

Organizations spend millions on communication products every year. Brochures, annual reports, corporate videos and promotional materials are produced and distributed to a wide variety of audiences as part of broader communication programmes or as “stand alone” products.

However, working with many different types of organizations, I’ve noticed that little systematic follow-up is undertaken to evaluate how these products are used and what is their contribution to achieving communication or organizational goals.

I recently worked on a project where did just that – we evaluated specific communication products and attempted to answer the following questions:

• Is the product considered to be of high quality in terms of design and content?
• Is the product targeted to the right audiences?
• Is the product available, accessible and distributed to the intended target audiences?
• Is the product used in the manner for which it was intended – and for what other unintended purposes?
• What has the product contributed to broader communication and organizational goals?
• What lessons can be learnt for improving future editions of the product and design, distribution and promotion in general?

The results were quite interesting and surprising. We were also able to map out the use of a given product, like in this example:

You can read more about this approach in this fact sheet (pdf) >>

Glenn

MUDGE’s Musings

So this is the third episode of what has turned out to be a quadruple-duty blog post.

1. I endeavor, as always, to edify faithful reader of this nanocorner of the ‘Sphere© on a daily basis. I take very seriously the blogger’s Prime Directive: Thou Shalt Blog Daily!

2. The event at which I am attending, the Enterprise 2.0 conference in Boston, Massachusetts, eats its own dog food in the sense that it provides space for participants to blog, contribute to a wiki, etc. So, I am posting these efforts in the blog space, although I am certain that for most participants, after a day filled with conference activities, and an evening filled with beyond the venue dining and/or entertainment (conjecturing about them, not describing your diligent and well behaved correspondent), the idea of writing, much less reading other people’s blogs in whatever time remains is probably far-fetched.

3. I am also posting these efforts on one of our fledgling blogs at the Heart of Corporate America, where, as I’ve mentioned, I am working with the technical review board considering which of the tools in this evolving market we should be adopting. I naturally gravitated to that space during the time I have worked with the team, and probably have been as active blogging in their test spaces as anyone (read: not very). The overall effort is crying out for a user champion who does more than attend meetings; so far we haven’t much evidence of one.

4. As a business traveler, I have what apparently is an old-fashioned self-impetus to file a trip report for management. I say old fashioned, because when I sent my department head last year’s report (I don’t get out very much) apologizing for it taking about a week after I returned, she replied that so far, of the more than half-dozen people from the department that had attended the event, mine was the only such report thus far received. Under those circumstances, I hereby declare that these several days’ efforts will serve for that report.

Once again, I recorded six hand-printed (as a left-handed person — the title of the blog, after all! — I gave up cursive writing as soon as I could get away with it) for the day’s lengthy sessions. As I begin to write this, at 5:00pmEDT, there is still one final session to go before the day is over, although as it is scheduled for one of the break-out rooms, and is likely to be oversubscribed, a pretty common occurrence this week, I may break to attend, and promptly return after being shut out. Not a total disappointment for a person who’s been sitting in sessions since 8:00am….

Okay, now it’s well after 8:30pm, and I’ve added more than another full page of notes from the final session. Whew! Hope my stamina is up to the challenge!

13. Enterprise 2.0 Tools: A Critical Evaluation: Tony Byrne, CMS Watch

CMS Watch is a software rating consultancy, and Tony Berne, its founder spoke quite eloquently despite the 8:00am starting time. Some of my fellow attendees, coming off a conference evening that might not have been as boring as mine (although, rest assured faithful reader, that I am always inspired and energized blogging for you!), questioned the necessity, not to say appropriateness of an 8am start time. This was just one element of a logistics topic that there will perhaps be an appropriate time to consider.

First point Mr. Byrne raised is that this enterprise 2.0 technology is immature; this is just the third annual of these conferences, after all. But, while technology does indeed matter, a business’s culture and governance are even more important.

He notes that purpose built social software, which he defined as collaboration and networking within and beyond the enterprise, differs from socializing existing software. An example of the latter he gave was the recent addition of the ability to add tags to a transaction within SAP.

Further he makes an interesting distinction that networking is unformalized whereas collaboration is formal. He used IBM software to make the comparison. IBM Lotus Connections is network enabling, and thus unformal. For collaboration purposes (i.e., finite document output?), IBM provides Quickr. Both operate atop Websphere Portal.

On the other hand, many vendors have chosen to build upon Microsoft Sharepoint. Hit some of those at the end of the day.

The market is highly fragmented. but can be basically classified as:

Platform vendors, providing infrastructure (Oracle, IBM, Microsoft and Google [he snarkily commented that Google provided an effective keynote, but felt their software offering in this space is, as yet, primitive]).

Social software suites, including Connectbeam, Traction, Jive, Awareness, Drupal (which is open source).

White label community services (which are hosted tools with custom branding), including Ning, Lithium and Pluck.

Blog/Wiki tools (pure play), Blogger, Six Apart, Automattic (home of WordPress.com, yay!), Atlassian, Socialtext.

Public networks, which include LinkedIn, Xing and Facebook.

Byrne recommends analyzing the appropriateness of each potential vendor’s product based on what will be a good fit. Base that evaluation on what he describes as canonical scenarios. Create a scenario that represents the enterprise’s needs. He divides the individual attributes to be measured into classifications such as business services, tools capabilities, application services, administrative services and vendor intangibles (i.e., do you believe that they’ll be around next year?).

Of all these, the scenario fit is probably the most important. He urged us to develop use cases and test them against each other in a “bake-off.” Test use case scenarios with users, not just a checklist on a spreadsheet.

He made some observations particularly germane to the current evaluation activity your correspondent is contributing to in his small way. Sharepoint is built on a stack of technologies. The blog and wiki tools are not good. Technologically capable customers might be better off building their own based on the portal/application platform Sharepoint provides. Some vendors build their products on the Sharepoint platform. IBM Lotus Connections consists of five different products branded together, implying potential issues with interoperability.

14. Driving Business Innovation through Communities: Mark Woollen, VP, Oracle Corp.

Mr. Woollen noted that employee disengagement is a crisis, and had a Gallup survey to back him up. Organizations are by their nature top-down hierarchically organized, but individual people work through social networks. He believes that social tools including Web 2.0 applications mash-ups and widgets and gadgets added to existing tools will help engage employees, in such specific areas as sales, talent pool management in HR, and mobile productivity. Oracle tools provide the means for developers to create products that are device/browser agnostic.

15. Realizing Business Value through Social Networking within Wachovia: Pete Fields, SVP, Wachovia

Fields’ presentation was engaging. Based on his experience at Wachovia, often it’s necessary to roll out social network tools to “validate your intuition” about their business value. He rolled out a comprehensive, integrated framework that incorporates the usual blogs and wikis, extended for pervasive instant messaging including 1:1 video messaging, as well as video blogs. 60,000 (about half) of employees have had the tools rolled out to date, built with Microsoft Office Sharepoint System (MOSS).

Business rationale for all this difficult to quantify in advance, but includes collaboration vs. travel, working more effectively across time and distance, better connected and engaged employees. His analogy: Web 2.0 is this generation’s equivalent of his generation’s company picnic and bowling leagues. Additional critical rationale: mitigate the impact of the maturing, retiring workforce, i.e., the attrition of knowledge assets.

There’s an enterprise wiki, “Wachovia Wisdom.”

16. (we’ve made it to 10:10am) Enterprise 2.0 in Action: Pfizer, Simon Revell, Manager, Pfizer.

Revell, of the UK told us that adoption of web 2.0 tools, as occurs so often in his company, was spearheaded by the R&D organization. He started by putting up a blog in the UK, in order to get through the “fear barrier.” The blog quickly became the place to go, and so the grassroots effort was centralized there. Due to the nature of the organization, there was lots of nervousness. They forced themselves to post and comment (anonymously!).

The Pfizerpedia wiki began in a similar grassroots way in the U.S. It’s accessible and editable by everybody in the organization and at present hosts 10,000 articles. It includes video podcasts.

He observes that blogging is the most difficult to get off the ground [the difference in a blog and wiki has to do, this writer thinks, between the first person voice of a blog, and the third person voice of a wiki article. In a buttoned up culture, third person is definitely more comfortable]. The best blog practitioners use a blog as a conversation, not a lecture.

Regulatory Affairs, usually a most buttoned down group, embraced Pfizerpedia, as it has been able to capture the continuous improvement process. Thus, lots of small success stories lead to a cumulative large achievement.

17. Real Enterprise 2.0 @ Sony Computer Entertainment’s World Wide Studios, Ned Lerner, Director, Sony

My least favorite presentation of the day. Mr. Lerner began slowly, experiencing some logistic issues with PowerPoint (guess you can’t run it on a PlayStation after all!), and was not a tremendously emotive speaker. It was simply difficult to relate to his small group of video game software writers, and his results sounded more like project management using open source tools, another conference altogether.

18. Enterprise 2.0 Reality Check, Andrew McAfee, Harvard Business School, moderator.

Included on this panel were the various customer presenters of the previous two days of plenary presentations. The Wachovia and Pfizer and Sony guys, as well as the two CIA guys from the day before.

This was essentially Q&A from the audience, hampered (another logistical issue) by insufficient microphones and runners for same posted in the large space.

Some highlights only:

When asked, how much does management really want honest dialog, Pete Fields (Wachovia) responded, they spend zillions on McKinsey and Accenture because employees won’t speak truth to power. Blogs might save some of that expense.

At Pfizer, the question from management wasn’t how do we stop this, but how do we do this safely?

19. Launch Pad: Stowe Boyd (my hero speaker from Tuesday), The/Messengers

So, this was a contest, consisting of brief (three minute) elevator stories presenting their products by five of this week’s vendors. The winner was to be voted on by the audience (“crowdware”) using cell phone texting. The first four presenters did high-speed runthroughs of their highlight PowerPoint presentations. The last one, Veodia, a provider of high definition video as a service, featured live video of the event and the audience, very high definition. By far the coolest, and the audience agreed, voting them the winner with 39% of the overall vote, a 2:1 margin over the next highest score. An exhilarating way to end the morning.

Lunch, as the day before, was hosted amongst the vendor demonstration booths. Talked to a couple of interesting people, although in some respects, I feel hamstrung by the fact that the major vendor decisions have already been made.

It’s 10:10pmEDT, and I’ve still got two afternoon and one evening session to go. These will have to wait for the next post.

As I just IM’d my son a few minutes ago, anyone who believes business travel is a vacation has never taken a business trip with me!

But, this has been an edifying event (and it still has another half-day to run!) full of very bright and vibrant people. Enterprise 2.0 can’t happen soon enough in the world of work.

It’s it for now. Thanks,

–MUDGE

In evaluation projects, we often feel the strong need to talk to people – to assess a situation or judge a phenomena by surveying or interviewing people. However, this is “manufacturing” data – we are framing questions and then putting them to people – and perhaps in doing so are influencing how they respond.

Alternatively, there is a lot to say for “found” or “natural” data – information that already exists – e.g. media reports, blog posts, conference papers, etc. We often forget about this type of data in our rush to speak to people.

Take this example. I recently saw a paper presenting “current challenges in the PR/communications field”. After surveying PR/comm. professionals, a list of five current challenges were presented by the authors. This is “manufactured” data. An approach using “found” data would be to examine recent PR/comm. conference papers and see what challenges are spoken about – or study the websites of PR/comm. agencies and see what they are presenting as the main challenges.

Another example. Imagine you would like to study the experiences of US troops in Iraq. Of course you could survey and interview military personnel. However, a rich body of data certainly exists online in blog posts, videos and photos from military personnel describing their experiences.

Of course, there are limitations to using “found” data (such as it may present only the views of a select part of a population/phenomena) – but an evaluation project combining both “manufactured” and “found” would certainly make its findings more solid.

Examples of “found” data:

• blog posts
• discussion forums
• websites
• website statistics
• photo/video archives (online or offline)
• media reporting
• conference papers
• policy documents
• records (attendance, participation, complaints, sales, calls, etc.)

If you are interested to read further on this subject, this book “A Very Short, Fairly Interesting and Reasonably Cheap Book about Qualitative Research” by David Silverman provides more examples and information on this concept.

Glenn

Well I do, actually – granularity is more important than we think for many fields…no, it’s not some sort of breakfast cereal- it’s the size or scale that characterizes an object or activity. And often we see errors made in placing activities at the same level that are not actually at the correct level … An example – I recently noticed a survey that featured the following question:

In which region are you working?
- Sub-Saharan Africa
- North Africa and Middle East
- Europe
- South Asia
- East Asia
- Russia and North Asia
- China
- North America
- Central and South America
- Australasia and Japan
- UK
- Other region

The problem is that the countries and regions mentioned are at different levels – and this is a problem of granularity. “UK” and “China” are not at the same level as “Central and South America” and “South Asia”. This creates problems for people completing the survey – If I live in the UK what do I select? UK or Europe? Both are correct.

In this example, there would be three possible solutions; 1) list all countries of the world using an ISO standard list, 2) list countries applicable to the project (using “other country” for those exceptions that will certainly arise) or 3) use broad, widely accepted regions, such as Europe, Asia, Africa, etc. These solutions resolve the issue of granularity by placing the countries/regions at the same level.

It may seem banal but if these issues are not resolved before the questions are asked, the analysis will prove difficult. This is just one example – granularity is important for many fields such as information management (libraries), website design, software and retail (you never see supermarket aisles marked “vegetables, cereals, bananas” do you?)

Glenn

I’ve been looking into the best practices for using the Likert scale type of question, probably the most widely used response scale featured in surveys – often used to measure attitudes and other factors (e.g. “Excellent” to “Poor”). Created by Rensis Likert (pictured above) in the 1930s, his original scale featured five points. Over time, there has been many discussions and disagreements focused on one central question: What works best with the Likert scale to give you the most accurate responses?

I have read a number of studies on this question (sorry, I don’t link to them as they are all books or academic journals (that require a fee) but if you are interested write to me and I’ll give you the references) and the following are the points that most (but not all) scholars agree on:

More than seven points on a scale are too much. Studies show that people are not able to place their point of view on a scale greater than seven. So go for seven or less. What is the perfect number? Studies are not conclusive on this, most commonly mentioned are five, four or three point scales.

Numbered scales are difficult for people. For example, scales that are marked “1 to 5, with 5 being the highest” result in less accurate results than scales with labels such as “good” or “poor”. If numbered scales are used, signposts are recommended (e.g. put “poor” above 1, “satisfactory” above 3 and “excellent” above 5).

Labelled scales need to be as accurate as possible. Commonly uses labels such as “often” or “sometimes” often result in inaccurate responses. As these terms mean different notions of engagement from person to person, culture to culture (not to add the complexity of translating these terms). Scholars recommend using time-bound labels for frequency measures such as “once a week” (although problems of correct recall are also an issue). In addition, studies show that people find it difficult to differentiate between “very good” and “good” – better to use “good” and “excellent”.

And that’s it! Basically, there are inconclusive results on the use of a middle or neutral point (e.g. four point vs. a five point scale). Some scholars advocate a five point scale where respondents can have a “neutral” middle point whereas others prefer to “force” people to select a negative or positive position with a four point scale. In addition, the use of a “don’t know” option is inconclusive. I personally believe that a “don’t know” option is essential on some scales where people may simply not have an opinion. However, studies are inconclusive on if a “don’t know” option increases accuracy of responses.

Further information on the Likert Scale:

Examples of commonly-used Likert Scales >>

More examples of commonly-used Likert Scales >>

“Designing a useful Likert Scale” (pdf)>>

“The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information”>>

Glenn

Often in evaluation, we are asked to evaluate projects and programmes from several different perspectives: the end user, the implementer or that of an external specialist or “expert”. I always favour the idea that evaluation is representing the *target audiences* point of view – as is often the case in evaluating training or communications programmes – we are trying to explain the effects of a given programme or project on target audiences. However, often a complementary point of view from an “expert” can be useful. A simple example – imagine if you making an assessment of a company website – a useful comparison would be comparing the feedback from site visitors with that of an “expert” who examines the the website and gives his/her opinion.

However, often opinions of “experts” are mixed in with feedback from audiences and comes across as unstructured opinions and impressions. A way of avoiding this is for “experts” to use checklists – a structured way to assess the overall merit, worth or importance of something.

Now many would consider checklists as being a simple tool not worthy of discussion. But actually a checklist is often a representation of a huge body of knowledge or experience: e.g. how do you determine and describe the key criteria for a successful website?

Most checklists used in evaluation are criteria of merit checklists – where a series of criteria are established and given a standard scale (e.g. very poor to excellent) and are weighed equally or not (e.g. one criteria is equal or more crucial than the next one). Here are several examples where checklists could be useful in evaluation:

• Evaluating an event: you determine “success criteria” for the event and have several experts use a checklist and then compare results.
• Project implementation: a team of evaluators are interviewing staff/partners on how a project is being implemented. The evaluators use a checklist to assess the progress themselves.
• Evaluating services/products: commonly used, where a checklist is used by a selection panel to determine the most appropriate product/services for their needs.

This post by Rick Davies actually got me thinking about this subject and discusses the use of checklists in assessing the functioning of health centres.

Glenn

I recently came across a short article in a London newspaper which I summarise as following:

On Friday 21 September, Jose Mourinho, coach of the UK Chelsea football team resigned. On this day, a jump was seen in the number of people visiting a certain job vacancies website. It was proposed by the website marketing team that this jump was due to people learning of Mr Mourinho’s resignation and prompting them to think about changing careers and looking for a new opportunity – thus the rise in number of visits to the job vacancies website.

Doesn’t that strike you as a slightly spurious claim of cause and effect? They appear to be connecting the unrelated and proposing a series of changes that seem slightly unlikely: 1) a change to knowledge (knowing that Chelsea’s coach resigns) to 2) change to attitude (I should change jobs) to 3) behaviour (I actively look for a new job). Possibly of even more concern is that there are no alternative explanations offered as to the jump in website visitors. We can imagine several alternative explanations:

- On Fridays there is always a jump in website visitors (like patterns often seen in purchasing or visits to museums).

- A publicity campaign, a change is site referencing or link campaign drove more visitors to the website on that day.

- Or the more mundane, a technician installed a new monitoring software on the website that led to a more accurate or inaccurate reporting of website visitors.

In addition, no comparison was given to traffic on other job vacancies websites or even global statistics of web traffic (perhaps all sites experienced a jump on 21 September?)

Thinking it through, I think it’s safe to say that there is not enough evidence to point to an association between the resignation of Chelsea’s coach and the jump in visitors to a job vacancies website. The cause (resignation) does seem very distant from the effect (people seek to change jobs). We can point to other cases where an association does seem to exist between a public event and a particular change in behaviour as I’ve written about before: the effect of a reality TV program on the number of people willing to donate an organ and the effect of Kylie Minogue’s treatment for breast cancer on the number of young women taking a scan appointment. In both cases, an association between the cause and effect could be demonstrated and alternative causes were ruled out.

Glenn

It is often difficult to get organisations away from simply measuring “outputs” – what is produced – to measuring “outcomes” – what are the effects of outputs.

Funny enough, many organisations want to go from the very superficial measuring of output (e.g. how many news articles did we generate) to the very in-depth measuring of impact (e.g. the long term effect of our media visibility on audiences). Impact is feasible but difficult to measure, as I’ve written about before. However, instead of focusing on the two ends of the measurement scale, organisations would perhaps be wise to focus on “outcome” measurement.

I think this quote from a UN Development Programme Evaluation Manual (pdf) sums up why outcome is an appropriate level to measure for most organisations:

“Today, the focus of UNDP evaluations is on outcomes, because this level of results reveals more about how effective UNDP’s actions are in achieving real development changes. A focus on outcomes also promises a shorter timeframe and more credible linkages between UNDP action and an eventual effect than does a focus on the level of overall improvement in people’s lives, which represent much longer-term and diffuse impacts .”

The notion of the shorter timeframe and more credible linkages is certainly appealing for many organisations considering their focus of evaluation.

Glenn

As I mentioned in an earlier post, people often confuse “impact” with “results”. Is it possible to measure “long term impact” of projects? It is, however for most projects it is unrealistic to do so for two reasons: time and cost.

To evaluate impact, you would usually need to wait some 12 months after the major elements of a project have been implemented. Many organisations cannot simply wait that long. In term of costs, an impact study requires a triangulation methodology that uses various quantitative and qualitative research methods which could be costly. However, if time and cost are not issues, an impact evaluation is possible, keeping in mind the following points:

Was the impact desired defined at the beginning of the project?

For example, greater organisation efficiency; change in the way a target audience and/or an organisation behaves; or improvements in how services for a given audience are managed?

What have been the other elements influencing the impact you want to measure?

Your project cannot be viewed in isolation; there must have been other factors influencing the changes being observed. Identifying these factors will help you to assess the level of influence of your project compared to other factors.

Do you have a mandate to measure impact?

When assessing impact, you will be looking at long term effects that probably go outside of your own responsibilities and into the realms of other projects and units – you are looking at an area of the wider effects of your organisation’s activities and this needs to be taken into consideration. For example, if you are looking at the longer term effects of a training program, you would want to look at how individuals and the organisation as a whole are more efficent as a result of the training. Do you have the political mandate to do so? – As you may discover effects that go way beyond your own responsibilities.

Evaluating impact is a daunting but not impossible task. For most projects, it would be more realistic to focus on measuring outputs and preferably outcomes – and think of short term outcomes as I have written about previously.

Glenn

When speaking of achieving objectives for a project, I’ve heard a lot of people speak of the “intended impact” and I’ve read quite some “impact reports”. I know it’s a question of language, but often people use the word “impact” when in fact they should use the word “results”. Impact in the evaluation field has a specific application to long term effects of a project. The DAC Glossary of Key Terms in Evaluation and Results Based Management (pdf) produced by the OECD contains the most widely accepted definitions in this field. Impact is defined as:

“Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended”.

And “results” is defined as

“The output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention”.

Consequently I believe that when we produce a report that shows media visibility generated by a project, this is a short term output and should be called “results” rather than “impact” which applies to more long-term effects.

Glenn

I did appreciate the following quote from Alberto Gonzales, US Attorney General who when defending the work of his department said:

“Good, if you look at the output”

Regardless of what you think of Mr Gonzales and his department’s performance, I find it interesting the use of the word output – it has sneaked in from management-by-objective speak… but output is usually a poor measure for performance, as it represents the products or services produced. It is just like..

A press officer judges her performance by the number of press releases she writes
A training office judges his performance by the number of people that attends his training sessions

What is far more important are outcomes – the effects and changes that are a result of the outputs:

A press officer should judge her performance by how her press activities change the knowledge and attitudes of audiences
A training officer should judge his performance by how the people he trains uses what they have learnt

Like Mr Gonzales, most people prefer to look at outputs to judge performance as they are much easier to control and monitor compared to outcomes, which I’ve written about previously. But increasingly activities are assessed on what they achieve (outcome) rather than what they produce (output).

Glenn

This month, the Dutch authorities reported that people registering as organ donors had tripled compared to previous months. What caused this sudden jump in registrations – a fantastic awareness programme?

In fact, they trace the increase to the now infamous “Dutch TV Kidney Hoax“, a reality TV show where real patients in need of a kidney “competed” for one.

From the communications evaluation point of view, it is an interesting example of how a communications activity can bring about a rapid change in behaviour (in this case donor registration) and perhaps one that was not intended.

In evaluating our own communication activities, we should try and identify other factors that could have influenced the change being seen – in the kidney TV hoax it was obvious but it will not be for many of the more day-to-day communication activities that we run.

Which reminds me of another example – in August 2005, the number of appointments made for manograms (to detect breast cancer) jumped by 101% in Australia. Was this the result of a successful communications campaign? No, in fact that month, pop singer and fellow Melburnian Kylie Minogue was diagnosed with breast cancer resulting in mass media coverage about the issue which I’ve written about previously.

The identification of other possible explanations for changes being observed (rather than just saying “look our communications campaign worked”) is important in maintaining a credible and balanced approach to evaluation.

Glenn

Here is something from the mainstream media, the International Herald Tribune has published an interesting article about the legitimacy or not of online polling – basically the use of online survey tools to conduct polling with broad audiences.

The article pits two main players in the field – Yougov against TNS.

The representative from TNS says about internet polling:

“Internet polling is like the Far West, with no rules, no sheriff and no reference points.”

Of course, he has a point although the counter argument is that *offline* polling – usually done by calling people on their fixed phone line – is fastly becoming obsolete as countries create “no call” lists and people use increasingly their mobile phones.

From my perspective, I would see the debate from a slightly different angle:

• We shouldn’t forget the approach of triangulation, that is the combination of different research methods to bring us closer to the *truth*. Internet polling, if combined with other research methods (interviews, focus groups, observations) can become more useful.
• The whole article focuses only on surveying *unknown* audiences – that is members of the public as a whole. However, most evaluation that I see undertaken is done with *known” audiences, e.g. staff, partners, customers, members, etc. For *known* audiences, the use of internet polling is efficient and reliable, assuming of course that your audience does have access to the Internet. Of course, if you are trying to gauge the opinions of audiences that are obviously not using the Internet, then another approach would be appropriate.

Glenn

In the past months I’ve had many discussions with project managers about evaluation where they clearly feel uncomfortable linking their objectives to outcomes – and now after reading a recent article in the Evaluation journal, the penny has dropped (i.e. I am able to link up the dots and understand a little more…).

In an article by John Mayne, “Challenges and Lessons in Implementing Results-Based Management”, he discusses the issue of accountability and outcomes:

“People are generally comfortable with being accountable for things they can control. Thus, managers can see themselves as being accountable for the outputs produced by the activities they control. When the focus turns to outcomes, they are considerably less comfortable, since the outcomes to be achieved are affected by many factors not under the control of the manager.”

And that’s it – a communications manager prefers to be accountable for the number of press releases s/he publishes and not the changes to an audiences knowledge or attitude, a training manager prefers to be accountable for the number of courses s/he organises and not the impact on an organisation’s efficency, etc.

So is there a solution? John Mayne speaks of a more sophisticated approach to accountability, notably to look at the extent to which a programme has influenced and contributed to the outcomes observed. And that leads us to the next question – how much influence is good enough?

Food for thought…you can read an earlier version of the article here (pdf).

Glenn

When I’m asked to take on an evaluation project, I usually categorise it in my own mind as “micro” or “macro”. Let me explain. I see evaluation projects falling into these two categories:

Macro: evaluation of an overall project or programme (e.g. training programme, communications project)

Micro: evaluation of an element that is part of a larger programme (e.g. evaluation of online communications – part of larger communications project, evaluation of an event that is part of larger campaign).

I find that a lot of evaluation projects that I get involved with are at a “micro” level. And I’ve been wondering why is this so?

I believe that evaluation is more often approached at a “micro” level because it is easier to deal with and less daunting for an organisation to cope with. Many people do not have the resources, time and political authority to launch a “macro” evaluation of projects/programmes.

And a lot of the literature on evaluation recommends implementing evaluation aspects in small steps. And there’s certainly some merit in this – to start at the “micro” level and build up to the “macro” level. In this way, people can hopefully see the benefits of evaluation and will support larger evaluation efforts when needed.

Glenn

Evaluation is often thought of as a “concluding” activity – something that is done once a programme or project is finished. But evaluation has its role “before” and “during” an activity. A recent experience highlighted for me the importance that evaluation can play in the “before” phase.

I have been involved in setting-up a pan-European e-learning platform and prior to its launch, we decided to test the platform with a select group of users. In the learning or communications field that would be a standard procedure – to pre-test material before it is used with its target audiences. But I am amazed at how many organisations don’t pre-test material – a “before” evaluation activity.

The feedback we received from the test users was incredibly informative – they identified issues that we did not even think about; access, usablity and broader issues on motivation and incentives for using the platform. User tests for online websites/platforms do not have to be complicated and costly – Jakob Nielsen, the specialist in this field explains well why usability is not necessarily expensive.

The “before” evaluation phase is much broader than simply pre-testing material. The establishment of baseline data (e.g. attitude levels on issues), the gathering of existing research on a subject, benchmarking with comparable projects and ensuring that a project’s objectives are clear and measurable are some of the components of this phase.

Glenn

The Uk-based Overseas Development Institute have published a very comprehensive guide “Tools for Knowledge and Learning: A guide for development and humanitarian organisations” (pdf) which contains descriptions of 30 knowledge and learning tools and techniques.

It contains guidelines for several relatively new methodologies useful for evaluation, notably Social Network Analysis, Most Significant Change and Outcome Mapping. I believe these new methodologies could be useful in a lot fields, not only for the development / humanitarian sector.

Glenn

As I wrote in a previous post, research results are sometimes misused (that’s nothing new…) and we are often given scant details on how the results were gathered and analysed.

I came across a study undertaken by a bank in Geneva, Switzerland (where I am living) that makes a series of claims about e-banking, web surfing habits and computer use in general. I was surprised to learn that these claims were based on a sample of 300 residents. Now Geneva has some 440,000 residents and I seem to recall from Statistics 101 that 300 people doesn’t really make a representative sample of 440,000 (it would be closer to 600 people depending upon the confidence level and intervals you are aiming at).

I’m not such a stickler on samples given that often the audiences we are looking at can be broken down into sub-populations that are often relatively small in number (so we look for highest participation as possible) – but if you do have a uniform finite population, try using this online sample size calculator to estimate the sample needed, it’s quite useful.

Glenn