Success Bias versus Failure Case Analysis at the World Bank

At the end of my presentation on Monday this week, I was asked an interesting question from one of the participants: Would I agree that having collected the ethnographic data, which contained both positive and negative variance, I had simply disregarded the positive data and then focused only on the negative?

I said perhaps this was so, but there remained good and solid reasons for doing this — in so far as one is interested in actually improving the system, and this improvement represents a delta which is described by its absence, the very failure which we were seeking to map. The next day I felt this was a bit of a half-arsed answer and I tried to finesse it further. This time I suggested that perhaps one should understand that both effective delivery and delivery failure were both already always present in the course of conducting Failure Case Analysis — in fact they were folded upon one another.

In both cases, however, I think I failed to really respond in a full and effective way to the kind of perspective I was encountering. That’s what I’m going to try to do now. 

. I heard the term positive deviance multiple times over the course of the two days. This of course refers to the variation from the mean on the side favorable to the outcomes you might be interested in. Inquiring into the causes of positive deviance could be helpful if you wanted to normalize or regularize whatever is happening inside those cases.

. But this way of thinking about outcomes removes the observer from the system. We would rather advocate the intentional design of systems in such a way as to ensure we achieve expected outcomes. This ability needs to be within the responsibility and skill-set of the World Bank as whole.

. In any case, as Jim Kim, the President of the Bank, has himself noted, the search for positive deviance and of ‘best practices’ which may be replicated has not worked out so well. The principal reason for this is the reification of the case as opposed to the specific alignment of component factors that made up the case, including local culture, institutional networks, and other factors. It is therefore the system which must in each and every case designed, and not the broad approach or model which should become the hero. My old professor Arjun Appadurai characterized this phenomenon in terms of success bias when he visited us at our campus last year.

. The other challenge here is that if staff members of the Bank cannot and do not undertake systems analysis themselves, and if they do not require it of their implementation partners, then no one will do it, and it will not happen except perhaps as an accident. Building systems without anyone undertaking systems analysis in an effective and competent manner is a rather poor plan.

. In another presentation we saw a situation where the skills of impact evaluation and economic modeling were being deployed to try and search for an effective solution or service model in several developing country contexts. This amounted to really using a shovel for one’s soufflé — it was just not the correct instrument for the goal intended. Macro-quantiative modeling extremely necessary and valuable at the far end of a well-designed intervention and innovation, but it cannot be used to generate hypotheses around what sets of interventions to the current system might make it work better.

. My summary views, then, in response to the provocative question I’ve still been struggling to answer, is that Failure Case Analysis is indeed necessary and the focus on failure is absolutely essential to understand how the system actually works when it does work and what its limits are at present, in case we need to make it work better. This approach is NOT well characterized in quantitative terms as a deviance spread, for that amounts to grossly mischaracterizing the method and the quality of data that it describes. FCA describes the challenges which must be overcome to make the system work better at a minute scale and allows for the construction of a large series of micro-propositions which must then be assembled together to create new and innovative redesigns of the delivery system.

. In effect, what emerged from this two day workshop is the need for a blending of skills and methodologies, from the quantitative to the qualitative, from the design side plus the evaluation side. This is an exciting challenge to contribute towards, certainly from the perspective of the network of organizations at the Vihara Innovation Campus.

This entry was posted in Design!publiC. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *