One of the common themes that has surfaced lately, in conversations from Helsinki to Indianapolis, to Washington DC to Norfolk (VA), has been the need to move from talking of ‘feelings’, when discussing the value of KM, and the development of the KM value proposition, to conversations driven by impact, cost and ROI.

Too many conversations around KM value start with ‘I feel’, ‘I think’ or ‘I’ve talked with someone who feels/thinks’.  Feelings are not enough in today’s risk averse environment, we need evidence; this is why we need to shift the conversation to impact and cost, as a precursor to ROI.

To help, we (K3-Cubed) have produced the KM Value pyramid.  This has been adapted from the Kirkpatrick/Phillips training evaluation model and has been informed by our K-Core research project; we are also using this on the KM Professional Development Certificate with the University of Edinburgh (now in-house with K3-Cubed – see www.KMeducation.com), as a way to stimulate evaluation processes that, in turn, can improve understanding of the business value of the concept.

The 5 stages of KM value measurement

On 20 and 21 February 2012, the  London-based Wellcome Collection is the stage for the final workshop organised by the Information Knowledge Management Emergent (IKM-Emergent or ‘IKM-E’) programme. Ten IKM-E members are looking at the body of work completed in the past five years in this DGIS-funded research programme and trying to unpack four key themes that are interweaving the work of the three working groups which have been active in the programme:

1. Linearity and predictability;
2. Participation and engagement;
3. Individual agency and organisational remit;
4. Accountability

This very rich programme is also a tentative intermediary step towards a suggested extension for the programme.

In this post I’m summarising quite a few of the points mentioned during the first day of the workshop, covering the first two points on the list above.

On linearity and predictability:

Linear approaches to development – suggesting that planning is a useful exercise to map out and follow a predictable causal series of events – are delusional and ineffective and we have other perspectives that can help plan with a higher degree of realism, if not certainty.

Linearity and predictability strongly emphasise the current (and desired alternative) planning tools that we have at our disposal or are sometimes forced to use, and the relation that we entertain with the actors promoting these specific planning tools.

Planning tools

After trying out so many ineffective approaches for so long, it seems clear thataspirational intent might act as a crucial element to mitigate some of the negative effects of linearity and predictability. Planning tools can be seen as positivist, urging a fixed and causal course of events, indeed focusing on one highlighted path – as is too often the case with the practice around logical framework – or can have an aspirational nature, in which case they focus on the end destination or the objective hoped for and strive to test out the assumptions underlying a certain pathway to impact (at a certain time).

Different situations require different planning approaches. Following the Cynefin framework approach, we might be facing simple, complicated, complex or chaotic situations and we will not respond the same way to each of those. A complex social change process may require planning that entails regular or thorough consultation from various stakeholder groups, a (more) simple approach such as an inoculation campaign may just require ‘getting on with the job’ without a heavy consultation process.

At any rate, planning mechanisms are one thing but the reality on the ground is often different and putting a careful eye to co-creating reality on the ground is perhaps the best approach to ensure a stronger and more realistic development, reflecting opportunities and embracing natural feedback mechanisms (the reality call).

There are strong power lobbies that might go against this intention. Against such remote control mechanisms – sometimes following a tokenistic approach to participation though really hoarding discretionary decision-making power – we need  distanced control checks and balances, hinting at accountability.

Managing the relationship leading to planning mechanisms

Planning tools are one side of the coin. The other side of the coin is the relationship that you maintain with the funding or managing agency that requires you to use these planning tools.

Although donor agencies might seem like ‘laggards’ in some way, managing the relationship with them implies that we should not stigmatise their lack of flexibility and insufficient will to change. In a more optimistic way, managing our relationship with them may also mean that we need to move away from the contractual nature of the relations that characterise much of development work.

Ways to influence that relationship include among others seeking evidence and using evidence that we have (e.g. stories of change, counter-examples from the past either from one’s own past practice or from others’ past practice etc.) and advocating it. Process documentation is crucial here to demonstrate the evidence around the value of process work and the general conditions under which development interventions have been designed and implemented. It is our duty to negotiate smart monitoring and evaluation in the intervention, including e.g.  process documentation, the use of a theory of change and about the non instrumentalisation (in a way that logical frameworks have been in the past). In this sense, tools do not matter much as such; practice behind the tools matters a lot more.

Finally, still, there is much importance in changing relationships with the donor to make the plan more effective: trust is central to effective relationships. And we can build trust with donors by reaching out to them: if they need some degree of predictability, although we cannot necessarily offer it, we can try, talk about our intent to reduce uncertainty. However, most importantly, in the process we are exposing them to uncertainty and forcing them to deal with it, which helps them feel more comfortable with uncertainty and paradox and find ways to deal with it. Convincing donors and managers about this may seem like a major challenge at first, but then again, every CEO or manager knows that their managing practice does not come from a strict application of ‘the golden book of management’. We all know that reality is more complex than we would like it to be. It is safe and sound management practice to recognise the complexity and the .

Perhaps also, the best way to manage our relationship with our donors in a not-so-linear-not-so-predictable way is to lead by example: by being a shining living example of our experience and comfort with a certain level of uncertainty, and showing that recognising the complexity and the impossibility to predict a certain course of events is a sound and realistic management approach to development. Getting that window of opportunity to influence based on our own example depends much on the trust developed with our donors.

Trust is not only a result of time spent working and discussing together but also the result ofsurfacing the deeper values and principles that bind and unite us (or not). The conception of development as being results-based or relationship-based influences this, and so does the ‘funding time span’ in which we implement our initiatives.

Time and space, moderating and maintaining the process

The default development cooperation and funding mechanism is the project, with its typically limited lifetime and unrealistic level of endowment (in terms of resources, capacities etc. available). In the past, a better approach aimed at funding institutions, thereby allowing those organisations to afford the luxury of learning, critical thinking and other original activities. An even more ideal funding mechanism would be to favour endemic (e.g. civic-driven) social movements where local capacities to self-organise are encouraged and supported over a period that may go over a project lifetime. If this was the default approach, trust would become a common currency and indeed we would have to engage in longer term partnerships, a better guarantee for stronger development results.

A final way to develop tolerance to multiple knowledges and uncertainty is to bring together various actors and to use facilitation in these workshops so as to allow all participants to reveal their personal (knowledge culture) perspective, cohabiting with each other. Facilitation becomes de facto a powerful approach to plant new ideas, verging on the idea  of ‘facipulation’ (facilitation-manipulation).

Beyond a given development intervention, a way to make its legacy live on is to plug those ideas onto networks that will keep exploring the learning capital of that intervention.

What is the value proposition of all this to donors? Cynically perhaps the innovativeness of working in those ways; much more importantly, the promise of sustainable results – better guaranteed through embedded, local work. The use of metaphors can be enlightening here, in the sense that it gives different ideas: what can you invest in projects and short term relationships? e.g. gardening for instance planting new initiatives in an existing soil/bed or putting fertilizer in existing plants…

Interesting links related to the discussion:

• Documenting change: an introduction to process documentation (a document explaining the theory and IRC’s practice around process documentation over seven years;
• Getting to social (blog post by Harold Jarche, February 2012);
• Mid-term assessment blues (blog post by Patrick Moriarty from IRC on a flexible learning-focused M&E approach in the Triple-S project.
• Results from this IKM-E discussion on the wiki;

On participation and engagement:

Sustainable, effective development interventions are informed by careful and consistent participation and engagement, recognising the value of multiple knowledges and valuing respect for different perspectives as part of a general scientific curiosity and humility as to what we know about what works and what doesn’t in development.

The second strand we explored on day 1 was participation and engagement with multiple knowledges. This boils down to the question: how to value different knowledges and particularly ‘local knowledge’, bearing in mind that local knowledge is not a synonym to Southern knowledge because we all possess some local knowledge, regardless of where we live.

A sound approach to valuing participation and engagement is to recognise the importance of creating the bigger picture in our complex social initiatives. The concept of cognitive dissonance is particularly helpful here: As communities of people we (should) value some of our practices and document them so that we create and recognise a bigger collective whole but then we have to realise that something might be missing from that collective narrative, that we might have to play the devil’s advocate to challenge our thinking – this is the ‘cognitive dissonance at play – and it is more likely to happen by bringing external views or alternative points of view, but also e.g. by using facilitation methods that put the onus on participants to adopt a different perspective (e.g. DeBono’s six-thinking hats). Development work has to include cognitive dissonance to create better conditions to combine different knowledges.

Participation and engagement is also conditioned by power play of course, but also by our comfort zones; e.g. as raised in a recent KM4Dev discussion, we are usually not keen on hiring people with different perspectives, who might challenge the current situation. We also don’t like the frictions that come about with bringing different people to the table: we don’t like to rediscuss the obvious, we don’t like to renegotiate meaning but that is exactly what is necessary for multiple knowledges to create a trustworthy space. The tension between deepening the field and expanding it laterally with new people is an important tension, in workshops as in development initiatives.

We may also have to adopt different approaches and responses in front of a multi-faceted adversity for change: Some people need to be aware of the gaps; others are aware but not willing because they don’t see the value or feel threatened by inviting multiple perspectives; others still are also aware and don’t feel threatened but need to be challenged beyond their comfort zone. Some will need ideas, others principles, others yet actions.

At any rate, inviting participation calls for inviting related accountability mechanisms. Accountability (which will come back on the menu on day 2) is not just towards donors but also towards the people we invite participation, or we run the risk of ‘tokenising’ participation (pretending that we are participatory but not changing the decision-making process). When one interviews a person, they  have to make sure that what they are transcribing faithfully reflects what the interviewee said. So with participation, participants have to be made aware that their inputs are valued and reflected in the wider engagement process, not just interpreted as ‘a tick on the participatory box’.

Participation and engagement opens up the reflective and conversation space to collective engagement, which is a very complex process as highlighted in Charles Dhewa’s model of collective sense-making in his work on traducture. A prerequisite in that collective engagement and sense-making is the self-confidence that you develop in your own knowledge. For ‘local knowledge’, this is a very difficult requirement, not least because even in their own context, proponents of local knowledge might be discriminated and rejected by others for the lack of rigor they display.

So how to invite participation and engagement?

Values and principles are guiding pointers. Respect (for oneself and others) and humility or curiosity are great lights on the complex path to collective sense-making (as illustrated by Charles Dhewa’s graph below). They guide our initiatives by preserving a learning attitude among each and every one of us. Perhaps development should grow up to be more about  ‘ignorance management’, an insatiable thirst for new knowledge. The humility about our own ignorance and curiosity might lead us to unravel ever sharper questions, on the dialectical and critical thinking path, rather than off-the-shelf (and upscaling-friendly) answers – which we tend to favour in the development sector. The importance here is thedevelopment of shared meaning.

A collective sensemaking framework (by Charles Dhewa)

As highlighted in the previous conversation, not every step of a development initiative requires multi-stakeholder participation, but a useful principle to invite participation and engagement isiteration. By revisiting at regular intervals the assumptions we have, together with various actors, we can perhaps more easily ensure that some key elements from the bigger picture are not thrown away in the process. This comes back to the idea of assessing the level of complexity we are facing, which is certainly affected by a) the amount of people that are affected by (or have a crucial stake in) the initiative at hand and b) the degree of inter-relatedness of the changes that affect them and connect them.

Iteration and multi-stakeholder engagement and participation are at the heart of the‘inception phase’ approach. This is only one model for participation and un-linear planning:

• On one end of the spectrum, a fully planned process with no room for (meaningful) engagement because the pathway traced is not up for renegotiation;
• Somewhere in the middle, a project approach using an inception period to renegotiate the objectives, reassess the context, understand the motivations of the stake-holders;
• At the other end of the spectrum, a totally emergent approach where one keeps organising new processes as they show up along the way, renegotiating with a variety of actors.

Seed money helps here for ‘safe-fail’ approaches, to try things out and draw early lessons and perhaps then properly budget for activities that expand that seed initiative. Examples from the corporate sector also give away some interesting pointers and approaches (see Mintzberg’s books and the strategy safari under ‘related resources’). The blog post by Robert Chambers on ‘whose paradigm’

“]Adaptive pluralism – a useful map to navigate complexity? [Credits: Robert Chambers

counts and his stark comparison between a positivist and adaptive pluralism perspectives are also very helpful resources to map out the issues we are facing here.

At any rate, and this can never be emphasised enough, in complex environments – as is the case in development work more often than not – a solid context analysis is in order if one is to hope for any valuable result, in the short or long run.

Related resources:

• Whose paradigm counts (blog post on Aid on the edge);
• Strategy safari - a guided tour through the wilds of strategic management;
• Henry Mintzberg’s books about management and other proofs from the corporate sector about the fact that top-down / command & control approaches do not work;
• Jerome Brumer’s two modes of thought in psychology.
• Results of this discussion on the IKM wiki

These have been our musings on day 1, perhaps not ground-breaking observations but pieces of an IKM-E collage that brings together important pointers to the legacy of IKM-Emergent. Day 2 is promising…

To be honest, I don’t care if we win the challenge. Instead, I’m just delighted that the IKM story is there with the others. And have a good look at them – I think there are about 45 entries – because I think you will find them as interesting and fascinating as I do. I’m also really looking forward to the synthesis of the findings.

This has led me to reflect a little on the methodology of the challenge which seems to be an excellent way of collecting stories from those who are undertaking evaluation of a wide variety of KM4D initiatives. I’m also hoping that the people behind the challenge can be encouraged to write up the approach they have used because it is such an inspiring way of facilitating learning across the development sector – such an important thing to do and one which doesn’t happen often because it’s so difficult. I won’t say it’s a best practice, I wouldn’t dare. After all, I’ve just submitted a story influenced by complexity and some proponents of complexity don’t believe that such a thing as best practice is possible at all. But I will say that I think it’s brilliant and something to be adapted and used again by others.

I’ve never been to a webinar before – or any virtual conference which may seem a bit surprising – so it was a new experience for me. Actually, I was rather sceptical about the form although I had high hopes of the content. But, in reality and for lots of reasons, it was a wonderful experience.

First of all, the subject was really interesting – monitoring and evaluation (M&E) of knowledge – and the content was great. There were presentations by IKM colleagues, Simon Hearn, Valerie Brown and Ewen le Borgne, who talked about their work on M&E of knowledge, and also about previous work on this subject, particularly papers written by Joitske Hulsebosch and colleagues on Monitoring and evaluating knowledge management strategies and Serafin Talisayon on Monitoing and evaluation of knowledge management for development. This was followed by an update from Marie-Ange Binagwaho, Louise Clark and Norma Garza of the findings to date on the KMIC. Main findings comprise:

• Stronger stories are those that incorporated M&E from the beginning of processes and / or used M&E assessment to generate actionable data which informed project development.
• Assessment processes designed to promote learning and improvement are stronger than those which respond to reporting requirements
• Simplicity is key, too much data makes things confusing
• There is some ambiguity between KM / M&E because both require information exchange and both are strengthened by functional feedback loops

Like @ithorpe wondering how presentations at #KMImpact and IKM webinar apply to my practical knowledge sharing work. Another webinar maybe?

I think this is a good point and something with which we are grappling at IKM Emergent. Out of this concern has come an initiative on Practice-based change which we are currently developing with Hannah Beardon, Ewen le Borgne, Mare Fort and others – which looks at the implications of the findings of IKM research directly in practice. One of the areas which we will be focusing on will be M&E, along with the implications of complexity and emergence, using and supporting local language processes, and also traducture. By the way, I’ll write another blog post on traducture as when I googled it, I only found a link to IKM Newsletter No. 3.

In the meantime, this tweet from Peter Ballantyne got me thinking about multiple knowledges – presented by Valerie at the webinar – and wondering about the relevance of this perspective for knowledge sharing practice. I think it is very relevant because it reminds us that there are many different sorts of knowledges with different values and different prejudices, and much more. Yet another blog post perhaps?

At the M&E cooking class, there's never too many cooks (Photo credits: vår resa)

On the KM4DEV mailing list, there has been a useful exchange on this topic of M&E of KM and this has triggered more reflections on our side to approach this paper. By the way, special thanks for Sarah Cummings, Roxane Samii and Patrick Lambe for getting this discussion going!

Simon just introduced in a blogpost one of our suggested theoretical models to address the different paradigms (what I profanely refer to as ‘world views’) on knowledge management, offering a spectrum from positivist to constructionist and from cognitivist to social learning).

In this post I’d like to share a refined version of the framework that we would like to offer to your scrutiny. This framework will eventually include a series of questions helping to crack the nuts for the M&E recipe, but for now let’s focus on the recipe itself.

When developing M&E activities around knowledge management, from the design phase to the evaluation of this whole process, we suggest following this approach with four distinct phases that play a role in cooking M&E for KM:

1. Initial appraisal – Sorting the hats and dishing out the invitation.
2. Framework design – Getting guests aligned.
3. Implementation – Serving hosts.
4. Post-assessment – Tasting the recipe and improving it.

The initial appraisal is usually overlooked but contains the precious assessments of the paradigms / world views used to guide the M&E approach as well as the roles and responsibilities that lead to a good governance of these M&E undertakings.

World views are by and large ill-considered as irrelevant philosophical discussions, vastly unspoken of, let alone addressed explicitly. However, they can have crucial repercussions on the perspective taken by various people involved in monitoring of knowledge (management) because the philosophical traditions on which they rest can vary significantly. If these perspectives are not explicitly tackled they can lead to differing opinions, potentially provoking defiance, mistrust and acrimony among the parties involved.

In turn, the misunderstandings and acrimony can lead to additional time to clarify expectations, to amplifying biases (of positive or negative interpretations on more subjective qualitative approaches), reducing the willingness to cooperate on M&E and ultimately to widening the gap that typically characterises Northern and Southern development agents. Addressing these world views is therefore a precondition for philosophical soundness, equilibrium in partnership as well as efficiency and effectiveness.

As for the roles and responsibilities involved in M&E activities, one could consider:

• Patrons: who are financing or commissioning the M&E activities. A typical example would be the donor of a project.
• Account handlers: who are steering the activities (sometimes it is the same as the funder, sometimes a team that usually ends up designing M&E activities).
• Implementers: who are the team leading the activities that we are monitoring;
• Monitors: who are the team effectively monitoring: collecting data,
• (Boundary) Partners: who are potentially affected by the activities we are monitoring and may therefore be contacted or interviewed for their inputs.
• Beneficiaries: who are the ultimate beneficiaries of the activities that we are monitoring, outside of the partners mentioned above.

These roles should not necessarily be discussed in details at this stage but knowing what stake each party has in conducting M&E is central to the success of these undertakings. The clearer the picture will be, the easier it will become to accept the purposes that guide the design of an M&E framework.

Past this stage – hopefully clarified by a joint discussion between parties involved – comes the framework design. This phase defines the recipe that will be served (i.e. the set of M&E activities) and offers a comprehensive view on all the ingredients required to cook the recipe:

• Purposes for which M&E is conducted;
• Levels of intervention at which M&E activities are targeting;
• Monitoring areas (the various aspects that will be monitored);
• Resource considerations (budget available for M&E, timeframe, capacities required);
• Tools and approaches to monitor KM activities (the playground!);
• Final roles and responsibilities (revised and specified in details after careful consideration of all the above).

These have been introduced in the original post but perhaps what matters here is to show our ambition to sort the set of M&E tools and methods in the following table:

The table of tools we propose to adopt

The third phase (implementation) is all about serving guests, i.e. implementing M&E activities: collecting data, analysing them, reflecting upon them to make recommendations and finally using the recommendations to improve decisions and inform activities.

Finally comes the ex-post assessment phase which will give hints on the savour of the dish served and provide ideas for further improvement. As with any sound learning approach, one has to review what happened and draw lessons for ongoing improvement. This stands true even after a collective initial appraisal which, if conducted correctly, should reduce the need for improvement.

Do you want to go on cooking class with us? We are learning by doing, and we believe this is a recipe for learning… You reckon?

Methodological paradigms

Assessing the value of KM requires the evaluator (or monitor) to take a particular perspective on how they will enquire into the reality they are confronted with. This means both their methods of enquiry and their approach to knowledge and learning. Their perspective is usually inherent and will depend on their worldview, namely their ontology (understanding of reality), epistemology (how this reality is ‘known’) and their ethic or moral purpose (Mowles).

In terms of methods; put simplistically, the evaluator identifies themselves on a spectrum with positivists at one end and constructionists at the other. Positivists are realists, they assume that ‘the reality’ exists and they can know and represent this reality accurately by taking measurements using natural science methods.  Constructionists, on the other hand, are interpretivists, who believe that reality is relative and observers constructs their own interpretation and make meaning together, and are therefore more interested in understanding perspectives of reality.

We can imagine a similar spectrum that describes the various approaches to knowledge and learning that the evaluator could take. At one end we have cognitivists who believe the mind is like a computer or information processor where knowledge is stored in a mental schema and learning is defined as a change in this schema. At the other end we have social learning theorists who understand learning to be a social process that arises between people at a given time in a given context.

Putting these two scales together, we can imagine a matrix that describes the different methodological paradigms that exist in the M&E of KM. We take a constructionist, social learning approach.

For Peter and I, it was very interesting to have the opportunity to interact with colleagues from the private sector. And one of the most notable observations we made was that many knowledge managers seem to be struggling with attitudes and behaviours of individual knowledge workers: “experts want to keep their knowledge to themselves”, “sharing is not taking place”, and “competition stops people sharing knowledge”.  This seems to be in striking contrast to the development sector where, sharing via knowledge networks and communities of practice, is really taking place, although limited by time pressures and other practicalities.

When I mentioned this stark difference between sectors, the reaction of the other participants was  “Oh, it’s because your sector is non-profit…” but I wondered if there is not a deeper explanation and went further with my probing. I came to the conclusion that the difference is that many development organisations are seriously committed to becoming learning organisations: without organisational learning, they can’t do better development. This commitment takes, of course, various forms, but there is usually a general agreement that it is necessary. Although I am, of course, making generalisations, this appears to be in stark contrast to the situation in the private sector where there seems to be the rhetoric of the learning organisation but that commitment of management does not always carry through: in good times, there is no need to change, and in bad times the resources aren’t there. I think that another difference is the common understanding within development – despite natural organisational rivalries and a host of other impediments – that we are all working towards the same objective, and the same humanitarian objective at that.

Kindly note that I am here not trying to bad-mouth my fellow participants here: they were all very committed and are grappling with the same issues. When I explained about IKM Emergent, and told that it was funded by the Dutch Ministry of Foreign Affairs, one participant reacted:

Oh, there is commitment to knowledge management at the top, no wonder knowledge management is more accepted!

If these results are to be taken at face value, they suggest that after a period in which interest in KM declined somewhat, there has been a substantial renewal and expansion of worldwide interest amongst enterprises of all types. But, in the face of this increasing interest, KM, as a discipline, is – it is fair to say – in a mess. [read more]