Most energy efficiency programs are required by regulators to be evaluated to ensure ratepayer money is being spent wisely and reported savings are being achieved.  If only such oversight were to happen for the millions/billions/gazillions being shelled out to state and local governments in the name of energy efficiency.

State and local governments have Amazon-wide budget gaps to fill, and I can assure you that earmarks (dirty word) for energy efficiency will find their way to plug budget holes to keep buildings open, replace roofs, buy new lawn mowers and pickup trucks, and avoid staff reductions.

We in Wisconsin have already experienced this during the last recession.  Starting in about 2000, most money collected by utilities for programs was turned over to Madison to be distributed from the ivory tower.  The recession of 2001 resulted in a major budget gap (major at that time – it probably looks like a hairline fracture compared to what we have now).  There, coming in from investor owned utilities, was a nice cash stream of $80 million per year.  The state government swiped half of it.  It pretty much eviscerated the energy efficiency programs and brought the industry to a slow crawl.  Incentives were pathetic.

Thankfully, the Public Service Commission has taken control of cash flow now to help ensure ratepayer money is used to save energy, reduce demand, and delay/avoid construction of power plants and transmission systems as intended, rather than filling in a tiny portion of a humongous budget hole.  Now energy efficiency incentives in the state are what I consider to be very attractive.

These federal funds should either be funneled through established credible program delivery channels such as utility programs or, in some cases, state governments (as long as it is out of reach of the legislative and executive branches), or there should be third party impact evaluation of projects emanating from block grants to local governments and other private sector grant writers.

If there is no oversight, vendors, consultants, engineers, architects, whoever can declare whatever savings they want. Or worse, as noted above, the funds will go toward new park benches and decorative street lights.

We welcome the oversight and technical review of our work because we are going to do things right regardless of whether others review our work.  In a competitive market, the more technically astute and persnickety the reviewers are, the better for us.  While LEED® takes its lumps for being too cumbersome, time consuming, and nit-picky, I think it would be a big mistake to slack off the review process.  It will weaken a strong brand.

The bottom line is, if you have no rigorous third party review, you can expect pennies on the dollar of proclaimed savings.

written by Jeffrey L. Ihnen, P.E., LEED AP

If Christianity is true then it ought to follow (a) That any Christian will be nicer than the same person would be if he were not a Christian. (b) That any man who becomes a Christian will be nicer than he was before. … Christian Miss Bates may have an unkinder tongue than unbelieving Dick Firkin. That, by itself, does not tell us whether Christianity works. The question is what Miss Bate’s tongue would be like if she were not a Christian and what Dick’s would be like if he became one. …

We must, therefore, not be surprised if we find among the Christians some people who are still nasty. There is even, when you come to think it over, a reason why nasty people might be expected to turn to Christ in greater numbers than nice ones. That was what people objected to about Christ during His life on earth: He seemed to attract “such awful people.” (C.S. Lewis, Mere Christianity [available in full on-line], Book 4, Chapter 10)

If you are not clear about the business value of your KM initiative, then you will also find it difficult to measure its impact. Selecting a KM initiative from gut feel, or from fascination about a new technology or from reading about a KM best practice, but not being clear about its link to business results your company desires will result in also not being clear about its business impact.

Here are examples of how you measure the impact of a KM initiative, based on the link (–>) between it and desired business results:

• Enhancements in the company intranet –> Less time wasted hunting for information
  (Impact measures: reduction in percentage of working hours used in looking for information, extra days per year saved and equivalent amount of monthly payroll saved, additional productivity from extra days saved; compare this with cost of installing/training in intranet enhancement; caveats: to ensure attribution, interview/survey each user if he had actually used the enhancement; it can happen that the financial gain is not entirely attributable to the intranet enhancement: see previous blog on “Interactivity and Context”)

• e-Orientation of new recruits, including training in effectively using company intranet –> Shorter learning curve
  (Impact measures: after most of the new recruits complete their learning curves, interviews/survey to estimate person-days they saved compared to former recruits who did not use e-Orientation, convert person-days to money values; money saved from shorter face-to-face briefing of new recruits; compare these savings with cost of developing/testing e-Orientation package)

• “Ask the Experts” program –> Faster and better resolution of a high-value business process problem
  (Impact measures: compared with pre-program baseline data, shorter resolution time and less frequency of rework, and money value of shorter downtime resulting from both; compare this with the cost of setting up the program including an official customized/detailed expertise directory plus the money value of experts’ time used up; caveat: these financial measures do not reflect the value of ego-boost to the company-acknowleged experts and enhancement of knowledge-sharing culture/habits)

A footnote: the Asian Productivity Organization or APO — an inter-governmental network of 19 Asian governments — is convening a third study meeting on KM Measurements at Taipei, Taiwan in the third week of November 2009. The first two were held in Yogyakarta, Indonesia and Manila, Philippines.

Cheers!

—

Note that there is an embedded link in this blog post. It shows up as colored text. While pressing “Ctrl” click on the link to create a new tab to reach the webpage pointed to.

=>Back to main page of Apin Talisayon’s Weblog
=>Jump to Clickable Master Index

Water, sanitation and hygiene interventions to combat childhood diarrhoea in developing countries

Hugh Waddington, Birte Snilstveit, Howard White, and Lorna Fewtrell
Corresponding author email: hwaddington [at] 3ieimpact.org
August 2009
Synthetic review 001
International Initiative for Impact Evaluation (3ie).

Full text

Summary

This report is a synthetic review of impact evaluations examining effectiveness of water, sanitation and hygiene (WSH) interventions in reducing childhood diarrhoea.

The review has been conducted to Campbell/Cochrane Collabo ration standards of systematic review, as well as employing mixed methods of data analysis to assess not only which interventions are effective, or not, but why and under what circumstances. The review provides an update of previous reviews conducted in this area, notably Fewtrell and Colford (2004).

A comprehensive search was conducted of published and unpublished materials. Studies were identified for inclusion which employed rigorous impact evaluation techniques, using experimental (randomised assignme nt) and quasi-experimental methods, and which evaluated the impact of water, sanitation and/or hygiene interventions on diarrhoea morbidity among children in low- and middle-income countries. 65 rigorous impact evaluations were identified for quantitative synthesis, covering 71 distinct interventions assessed across 130,000 children in 35 developing countries during the past three decades. Each study was coded for a range of variables relating to type of intervention, effect size and precision, internal validity (relating to evaluation quality) and external validity (relating to context and behavioural mechanisms). Interventions were grouped into five categories: water supply improvements, water quality, sanitation, hygiene and multiple interventions involving a combination of water and sanitation and/or hygiene. Data were collected and synthesised on both quantitative and qualitative information presented in the evaluations.

The results challenge the notion that water quality treatment in the household (at point -of-use) and sanitation ‘software’ (hygiene) interventions are necessarily the most efficacious and sustainable interventions for promoting reduction of diarrhoea.

While point-of-use water quality interventions appear to be highly effective – and indeed, more effective than water supply or source treatment in reducing diarrhoea – much of the evidence is from trials conducted over small populations and short time periods. More evidence is needed on sustainability, as water quality interventions conducted over longer periods tend to show smaller effectiveness, while compliance rates, and therefore impact, appear to fall markedly over time.

Hygiene interventions, particularly provision of soap for hand-washing, are effective in reducing diarrhoea morb idity, and there does not appear to be evidence that compliance falls over time. The analysis suggests that sanitation ‘hardware’ interventions are also highly effective. However, relatively few studies have been conducted in this area to-date and studies are particularly needed that quantify the possible environmental spillovers from sanitation provision.

Evidence on the combined impact of multiple interventions is mixed. Further primary studies employing factorial design – that is, comparing different interventions using multiple treatment arms – are needed for more conclusiveness on whether water and sanitation/hygiene interventions are substitutes or complements in the health production function.

The study highlights the importance of behavioural factors in determining up-take and sustainable adoption of WSH technologies. Insights from diffusion theory suggest that preventive interventions tend to be adopted more slowly as benefits are difficult to observe and users presumably discontinue treatment as they perceive that the costs of using the intervention outweigh the benefits. These problems are more relevant for interventions aiming to reduce disease prevalence which do not have additional benefits, for example time savings. Unfortunately, few impact evaluations addressing sustainability collect data on the reasons for the levels of compliance and acceptance found among beneficiaries. This information is an essential guide to fostering long-term impact.

Garandeau, R., Casella, D. and Bostoen, K. (2009). Evaluating & improving the WASH sector : strengthening WASH governance, learning about complexity, assessing change. (Thematic overview paper series ; 23). The Hague, The Netherlands, IRC International Water and Sanitation Centre. 101 p.
Download here

Over recent decades, several events and trends resulted in an increased focus on development agencies’ results, hence an increased demand for quantitative measurement in general, and impact evaluation in particular. As part of this trend, funding agencies started to request more objective, independent evaluations to prove that their resources are used transparently on interventions that improve people’s lives, health, education, and wealth. As a result, all development agencies try to measure in one way or another the performance and achievements of their interventions.

This TOP discusses evaluations, focusing on the Water, Sanitation and Hygiene (WASH) sector. It starts with a tale of three farms, representing three paradigms of development. Each of these paradigms is then discussed separately in the specific context of this sector. For each paradigm, the resulting evaluation methodologies are presented in a chronological order, illustrated by examples of WASH related evaluations. The TOP ends with useful books, websites and contacts.

Today I am flying to Indonesia to conduct interviews of staff in a company in Sukabumi, in the highlands of West Java. The Asian Productivity Organization provided them with training in 5S and kaizen a few years ago. My aim is to discover and evaluate the impacts of the training through interviews. After a couple of days I will fly to Thailand and do the same in Phitsanulok in Northern Thailand near the Laotian border.

As a knowledge-pull activity, the interview is the art and science of asking questions. The most unrepentant knowledge-pushers are professor-lecturers-cum-bloggers (like me!) and the best cure for them is to do interviews (ouch!).

From a previous blog post (“L14- Voicing”), the unproductive manners of voicing are: lack of respect, being judgmental and acting as if you know better, and talking more than listening. The interviewer is forced to unlearn these unproductive habits of voicing. The interviewer is also forced to learn the art of asking questions and interviewing.

What are some productive ways of interviewing?

1. The presumptions of the interviewer hide underneath every question he asks — presumptions which may not always be shared by the respondent. Hence the first duty of the interviewer is to reduce his presumptions to a bare minimum. This is the reason why it is important to start with open-ended questions, or “grand tour” questions (see Step 1 in the diagram below), before moving on to more structured questions.
2. A good open-ended question is the invitation to storytelling (e.g. “Tell us about it/them” in Steps 1-3 in the diagram below) which is a good device to elicit narration of experiences from a respondent in her own terms and using her own assumptions to interpret her experiences. The interviewer just listens, instead of asking questions which are often loaded — knowingly or unknowingly — by his own assumptions. The story teller’s experiences and her interpretation of those experiences reveal her mental models (see “12 Types of Learning”).
3. In general, the cultural, technical and other assumptions of the interviewer are not the same as those of the respondent. And so the duty of the interviewer is to discover the assumptions of the respondent. He can then develop his next questions “on the spot” following the assumptions, categories and mental models/structures of the respondent — not his own. This is the reason why many anthropologists and ethnographers shy away from structured questionnaires: these research tools insinuate the researcher’s own assumptions onto the respondents’ answers. The casualties are scientific validity and cultural fairness.
4. Interviews are expensive in terms of time, expertise and money. Open-ended interviews cannot generate the kind of data needed to make useful generalizations. In other words, open-ended interviews are good for hypothesis building, but structured questionnaires are better for hypothesis testing. If questionnaires are designed AFTER open-ended interviews and ACCORDING to mental models/structures of the respondent population, then structured questionnaires offer opportunities for generalizations that open-ended methods alone cannot offer. The two are complementary, but a structured questionnaire/survey should be a LATTER step (see Step 6 in the diagram below) in the research process.

The same principle applies in designing a questionnaire: more open-ended questions should precede the more structured questions. For some research purposes, it is not advisable to tell the respondent what is the purpose of a questionnaire until after she is done answering. A good technique for avoiding or postponing some “smart” respondents becoming “test-wise” (i.e. they guess the purpose of the questionnaire and that guess consciously influences their answers) is to start with more open-ended questions.

Structure is “a fact of research life” because having research objectives means adopting a research structure. In this particular interview, my aim is impact evaluation of projects. The table above is oriented towards that aim. For my field interviews in five Asian countries, the sponsor and user of the impact evaluation is the Asian Productivity Organization or APO.

The range of concerns in impact evaluations typically includes (starting from the most immediate): outputs of a project, results on the larger organizational context of the project, side effects, and broader social outcomes. Attribution is easiest to claim on outputs and most difficult on social outcomes. Because interviews are at the individual level and social outcomes are at the macro level, interview is not a suitable tool for delineating social outcomes. The interviewer can still ask outcome questions, but he must remember that the answers he will get are highly interpretive and contextual on the part of the respondent. Hence, the correct sequence in asking such questions in an interview is to start with concrete and immediate outputs, and then move towards the more macro questions and their interpretive and contextual answers (Steps 2 and 3, followed by Steps 4 and 5 in the diagram below).

I will appreciate comments and improvements from other interviewers, or other impact evaluators (I am practicing knowledge-pulling!).

—

Note that there are embedded links in this blog post. They show up as colored text. While pressing “Ctrl” click on any link to create a new tab to reach the webpages pointed to.

=>Back to main page of Apin Talisayon’s Weblog
=>Jump to Clickable Master Index

In The Use and Misuse of Computers in Education: Evidence from a Randomized Experiment in Colombia, by Felipe Barrera-Osorio and Leigh L. Linden, the authors examine a program that

aims to integrate computers, donated by the private sector, into the teaching of language in public schools. The authors conduct a two-year randomized evaluation of the program using a sample of 97 schools and 5,201 children. Overall, the program seems to have had little effect on students’ test scores and other outcomes. These results are consistent across grade levels, subjects, and gender. The main reason for these results seems to be the failure to incorporate the computers into the educational process. Although the program increased the number of computers in the treatment schools and provided training to the teachers on how to use the computers in their classrooms, surveys of both teachers and students suggest that teachers did not incorporate the computers into their curriculum.

Two thoughts on this:

1. This reminds us – and I’d say “as if we needed reminding” except that we do – that you cannot just dump inputs into schools and expect changes.  If inputs don’t get used well, they don’t matter.  Even though this seems like a no-brainer, many development programs are very narrow: build a school or give some books or ….  Same problem, I’m afraid.
2. That said, a quick look at the tables suggests to me that the authors may be confusing a noisy result with a narrowly bound zero result.  In other words, there seem to be differences in outcomes between kids who got computers and those who didn’t, but there is so much variation in both groups that we cannot be sure.  What this really means is that we don’t know if there is an effect, that there might be a heterogeneous effect, or there might not.  (Either way, clearly this program wasn’t a raging success.)

There is also some evidence from India (I haven’t evaluated the quality) that if you just let kids play with the computers, they’ll learn some stuff.  (One question is, Do they learn things that will help them?)

King and Behrman have a paper on Timing and Duration of Exposure in Evaluations of Social Programs. The paper gives a useful (if not adrenaline-fueled – this is a review, after all) discussion of a host of issues to consider when deciding when to look for results of social programs. [For example, what if the program ends up rolling out at different times in different places? etc] They then give lots of examples of papers that have dealt with the issues (and how they’ve done it). Instructive stuff.

Summary: Impact evaluations aim to measure the outcomes that can be attributed to a specific policy or intervention. Although there have been excellent reviews of the different methods that an evaluator can choose in order to estimate impact, there has not been sufficient attention given to questions related to timing: How long after a program has begun should one wait before evaluating it? How long should treatment groups be exposed to a program before they can be expected to benefit from it? Are there important time patterns in a program’s impact? Many impact evaluations assume that interventions occur at specified launch dates and produce equal and constant changes in conditions among eligible beneficiary groups; but there are many reasons why this generally is not the case. This paper examines the evaluation issues related to timing and discusses the sources of variation in the duration of exposure within programs and their implications for impact estimates. It reviews the evidence from careful evaluations of programs (with a focus on developing countries) on the ways that duration affects impacts.

Yesterday, I was at a validation forum in which results of research supported by public funds were presented. One of the projects which is relevant to Devcompage was about impact, rate and spatial dissemination of agricultural technologies. Effectiveness of communication strategies used was measured through perception data, categorized into highly effective, effective, and so on — while their assessment of communication strategies was a mere listing of diffusion methods used — meetings, agro-fairs, radio program and techno-demo. What the report terribly lacked were objective indicators of media effectiveness and a more informed analysis of the diffusion and uptake of agricultural technologies covered by the study. The ACIAR scoping study on policy linkages could very well provide the overall context that will help one understand diffusion and adoption of agricultural technologies in the Philippines.

The next day, I received a copy of the impact evaluation report on a media campaign in Vietnam. Undertaken by two IRRI staff –Dr. Debbie Templeton, former impact specialist, and Zenaida Huelgas, associate scientist — this impact evaluation report is a fine example of how the use of an assessment framework, objective indicators, and careful analysis could provide scientific rigor. The report titled, “Three Reductions, Three Gains (3R3G) Technology in South Vietnam: Searching for Evidence of Economic Impact” presented evidence of adoption of 3R3G in terms of lowering seed rates and the resulting changes in farm production/cost structure and farmer profits were measured.

What is impact evaluation?

The World Bank independent evaluation group defines impact evaluation as the systematic identification of the effects – positive or negative, intended or not – on individual households, institutions, and the environment caused by a given development activity such as a program or project. Alternatively, we can say that Impact evaluation is an approach which measures the outcomes of an intervention apart from other possible factors. It is intended to determine whether the program had the desired effects on individuals, households, and institutions and whether those effects can be attributed to program intervention. Impact evaluations can also ascertain unintended consequences, whether positive or negative, on beneficiaries.

External review team visit fertilizer omission plots demonstration, Sragen, Central Java

Impact evaluations can be costly Rigorous impact evaluations can be costly. The World Bank has estimated that the average cost of a rigorous impact evaluation can run up to US$200,000 considering the cost of consultants. But the World Bank Independent Evaluation Group (IEG) came up with a booklet that details how one can conduct quality impact evaluations under budget, money and time constraints. The IEG website is a gold mine of useful and reliable information on monitoring and evaluation which must be bookmarked by all Devcompage readers.