Australian Road Research Board and NACOE for Queensland Department of Transport and Main Roads

The client specifically wanted to test the suitability and performance of Octant AI to improve the management of project risk through early warning and accurate prediction of cost and time overruns to accurately forecast the total cost of its portfolio.

As a result of Octant AI application, TMR’s decision making was enhanced through the improved accuracy and precision of project cost and duration forecasting.

Application of Octant AI on TMR’s projects allowed for:

• Improved early warning by an average of four months for underruns and 2 months for overruns.
• Improved accuracy of portfolio cost forecast predictions by 87 per cent.
• Improved accuracy of project cost forecast predictions by 21 per cent.
• Improved accuracy of project duration forecast predictions by 11 per cent.
• An average forecast cost savings of $148,000 per project.
• More accurate tracking of actual value of the portfolio compared to the previous TMR approach.

Scope of Services

Octant AI was appointed by appointed by TMR and ARRB in March 2020 and to date, have provided:

• A definition of TMR’s business value and an articulation of where the value in the implementation of Octant AI existed.
• Analysis of data for use in a comprehensive and effective AI model.
• The building and testing of a unique and specifically trained AI model which would work and learn against TMR’s portfolio of projects.
• Predictive analytics carried out on TMR’s dataset, which presented evidence-based predictive and descriptive observations.
• The testing of the AI model predictions against actual results.
• Comprehensive reporting on findings, outcomes and benefits delivered through Octant AI.

Challenges and Opportunities

Improving the predictability of project cost and duration underruns and overruns is a challenging problem affecting major projects. As would be expected with a large portfolio of works, there were a range of challenges that needed to be overcome to allow for AI-benefit:

• To achieve an appropriate level of performance for machine learning, data used for the AI model construction needed to be of a high-quality, with sufficient quantity, within the required format and containing minimal missing data.
• TMR had a large and diverse set of data which spanned across hundreds of projects, making the interrogation of data a complex process, which would ultimately result in primary and secondary data collection phases. The availability of high-quality data in enough volume to produce useable data was a key challenge. Adding to the complexity was the fact that pre-2009 data contained less detail in comparison to data from more recent years, meaning that data from earlier projects was often unusable.
• The first data collection of 48 projects were interrogated, finding 18 projects suitable for the AI cost prediction model and 24 suitable for the time prediction model. A range of projects were unusable due to being currently active and not allowing for a full lifecycle assessment, atypical project cost behaviours and static forecast end dates through the project lifecycle. This led to a secondary data collection phase which was initiated to provide a more robust database with 116 historical projects, comprising 89 small (less than $10 million in value) and 26 large ($10 million – more than $100 million) budget projects.

Solutions and Innovations

Despite the challenges, the client was supportive and willing to embrace the technology which would contribute to the success of the application. Solutions were applied to challenges to ensure innovations were delivered:

• Identification of data extraction through new sources, which would ultimately see the introduction of the secondary data collection. The Octant AI team was able understand evidence-based behaviours which would highlight portfolio behaviours that would not have otherwise been clear to the client.
• Octant AI has delivered a unique DNA that characterises how TMR’s projects perform, which can be used as a benchmark internally but also as an industry comparison tool.
• The findings have resulted in an important social impact, in which novel technology is able to deliver significant community benefit through time and cost savings to major projects.
• TMR has become a world leader in pioneering the application of AI in transport projects, and is a clear innovator and early adopter of this home-grown technology.
• Octant AI has identified significant financial and delivery opportunities for the re‐distribution of capital budget and/or contingency back into its portfolio, given that forecast project costs and contingency may be predicted earlier and more accurately. Overall, Octant AI has demonstrated capabilities that enhance capital productivity, portfolio performance, early warning capability and a decrease in monitoring costs.