World Bank

Note: For the official report released by the World Bank, please click here.

With an annual spend of nearly $38m and just over 100 staff, in 2020 WBIEG delivered over 250 evaluation and validation reports, covering over 80 countries.

This work allowed Octant AI to demonstrate that an AI-driven approach to project evaluation would provide benefits to WBIEG. This demonstrate would prove its potential usefulness to the project community more broadly, for instance to government departments and other private sector organisations maintaining large project portfolios (Quang, 2017).

Client Objectives

WBIEG had commenced its evaluation of the performance of its investments into a global portfolio of 392 nutrition projects from 64 countries, representing a total commitment over USD $28.8 billion over 20 years.

This portfolio is aligned with at least two of the Sustainability Development Goals (“SDG”):

• SDG Goal #3 Ensure healthy lives and promote well-being for all at all ages and
• SDG Goal #6 Ensure availability and sustainable management of water and sanitation for all.

Scope of Services

Octant AI’s work with WBIEG included:

• Data analytics
• Data pre-processing, cleaning, validation, and Extract Transfer and Load
• Natural language processing
• Unsupervised machine learning
• Vector space modelling
• Working with World Bank experts to interpret findings and integrate those into the overall study objectives
• Report writing

Challenges and Opportunities

Experts evaluate projects by meticulously analysing and synthesising numerical and text data from long project completion reports. These reports describe project goals, progress, outcomes, success and failure factors, and lessons learnt. This analysis and synthesis process is time intensive, needs deep expertise, and can sometimes be difficult to replicate because of how experts make interpretations. These costs can make large scale evidence syntheses across hundreds of projects prohibitive.

WBIEG engaged Octant AI to demonstrate our Octant AI Prevent product to use both supervised and unsupervised machine learning to:

1. Automate content labelling for statistical analysis
2. Find new insights into risks, success and failure factors that affect project success
3. Synthesise project performance insights to help with future project planning

Solutions and Innovations

A team of WBIEG experts read, extracted text and numerical data, and manually labelled content in nearly a thousand project reports. The data were tabulated and provided to Octant AI for input into Octant AI Prevent. Octant AI prevent then analysed the data and presented its findings for expert interpretation.

Overall the results were positive and showed strong potential for improving the speed, scale, and insights from large-scale project revaluations:

1. Octant AI showed that it performed only modestly at labelling report content at the most detailed level but performed strongly (90-95% accuracy) when labelling content at the highest level. World Bank’s high accuracy content extraction algorithms will help to scale this automation solution further
2. Unsupervised machine learning methods including natural language processing, unsupervised machine learning and t-SNE visualisations, allowed WBIEG to identify nine unique data points from five countries with over 14,000 data points. This allowed WBIEG to characterise the features that made these projects unique compared to the others around the world.
3. These unsupervised machine learning methods also revealed novel insights into success and failure factors that were shown to be strongly predictive of project success. Without any prior knowledge, Octant AI produced factors that not only aligned with global best practises for international development projects but revealed new structure in the factors that were novel to WBIEG. For instance, Octant AI showed how the factors smoothly varied between socio-political and technical factors, and from project-specific to country-specific factors. It captured and interpreted the smoothly varying nature of meaning between these topics in a way that was new, easy to visualise and grasp for WBIEG. This provides new insight into how factors relate to each other which is useful for richer and more effective project planning.

Further development of the algorithms will help to integrate these capabilities to create an end-to-end solution for future large-scale evaluations

Benefits

The pilot project with WBIEG showed strong potential for using AI to evaluate whether large project portfolios achieved their goals, and to provide evidence-based lessons learnt to improve future project planning. It can complement, and improve, traditional evaluation methods due to the speed, efficiency, scale, and replication benefits AI offers for large-scale evidence synthesis.

Learning from project history, identifying systemic risks, and understanding the critical success and failure factors for project performance are valuable for projects in all sectors, not just nutrition projects. This work with WBIEG shows that these are demonstrable capabilities for other sectors.

The final report of the nutrition portfolio evaluation is now publicly available here (Independent Evaluation Group, 2021). A second report by WBIEG that focuses specifically on Octant AI’s artificial intelligence work will be available soon (Franzen et. al., 2021, forthcoming).

 

References

Independent Evaluation Group. ‘World Bank Support to Reducing Child Undernutrition’. Washington, DC: World Bank, 2021. https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/files/Undernutrition.pdf

Franzen, Samuel, Cuong Quang, Lukas Schweizer, Alexander Budzier, Jenny Gold, Mercedes Vellez, Santiago Ramirez, and Estelle Raimondo. ‘Advanced Content Analysis: Can Artificial Intelligence Accelerate Theory-Driven Complex Program Evaluation?’ IEG Methods and Evaluation Capacity Development Working Paper Series. Washington, DC: World Bank, 2022.

Quang, Cuong. ‘Does Machine Learning Tame Complexity and the inside View?’ Masters dissertation, University of Oxford, 2017.