The legend John K. Hollmann, a well-recognized industry thought leader and Honorary Life Member of AACE® International, conducted a live webcast in collaboration with Project Control Academy on “How to Conduct Practical and Realistic Cost & Schedule Risk Analysis.”
John Hollmann shared his research on cost growth, schedule slip, and accuracy and presented an integrated set of project risk quantification methods “that work” for every project, program, and portfolio. The presentation was based on John’s recent book: “Project Risk Quantification: A Practitioner’s Guide to Realistic Cost and Schedule Risk Management”.
Project professionals across the globe enrolled for this informative webcast and learned a new way of looking at project risks.
Dinah Sylvester, PMI-RMP, PMI-SP, PMP, one of the webinar attendees provided a great summary of what was discussed.
According to Dinah, a wealth of knowledge was presented. The presenter, John Hollmann, mentioned that Probabilistic modeling should include the integration of both cost and schedule. It should also be integrated with Business (Capex) and Commercial aspects. Hollmann went on to share that one common occurrence in risk analysis is that we fail to quantify escalation risks. Escalation risks are economically driven price changes. However, “Systemic Risks”, or those risks that are inherent in our process should be our focus, as they are usually the most significant risks to the project.
Mr. Hollmann mentioned that there is research to support that if we improve our tools, strengthen our people, processes, etc., we can expect better outcomes. When quantifying the impact of risk events, the focus should be on the Risk Responses, meaning, “What will we do if a particular risk materializes/occurs?”; the response forms the scope of the impact estimate. Also, if a risk is treated, the treatment must be listed in both the risk register and in the estimate and schedule (i.e., coordinate with Change Management).
Although Mr. Hollmann stressed that critical path methods (CPM) can create a more robust schedule, it is not necessary for risk analysis. CPM and other quantitative methods also help the team to produce a better report rather than to create ‘risk register jockeys.”
Two key elements that should be integrated with quantitative Risk analysis during execution are:
- 1. Change management-ensuring that any occurrence that deviates from the baseline plan must be reported, analyzed, and dispositioned.
- 2. Forecasting-Take what we know and estimate looking forward to determining an accurate estimate to complete. Do a fresh, bottoms-up re-quantification (not hard to do with the right methods). Then, if more contingency is needed, we can secure it. If however, we determine that we have too much contingency, we can give it back. The main thing is that we stop, re-forecast, and reassess the risk with a fresh look.
Another interesting viewpoint of Mr. Hollmann was regarding construction claims. He summed up claims as being merely a symptom, and not a cause of risk. In other words, claims are a manifestation of poor performance.
Here are more takeaways from the audience:
In addition to the wealth of knowledge presented by John Hollmann, the audience raised great questions. Here are the Q&As that were addressed during the webcast. (All answers are provided by John Hollmann):
Why Project Cost & Schedule Risk Analysis?
Question # 1- Which industry and what type of projects are adapting Project Cost & schedule risk analysis to assess cost and time probabilities?
Nearly all the major owner companies, and their EPC partners, in the process world, use some method. Transport and other infrastructure has lagged but is catching up. The smaller the project or industry sector and the more cookie-cutter the scope, the less likely is the implementation of project risk quantification.
Question # 2- How is Project Cost & schedule risk analysis beneficial to the construction industry and practical implementation?
It provides better early alternative selection, investment decisions, and bidding on the front-end. It provides, and maintains, a better control basis during execution.
Question # 3- By using risk analysis, how much percentage of profit/ additional advantage we can get to the project? Since most of the management people think that, why we need to spend so much money on risk analysis software and sparing valuable time and professional efforts.
We do risk analysis so we understand the residual risk and how to price it; it provides better assurance of meeting whatever goal you set, whether that is added profit or hitting a completion date or something else. For a portfolio, realistic methods will kill the projects that need killing, or result in their being recycled for scope improvement; that results in more profit for the company (profit is the return on assets, and lower CapEx increases that measure).
Question # 4- Organizations usually don’t reveal the cost breakdown of material/labor/profit… etc. what is the use of cost risk analysis?
The methods covered here do not require any breakout other than for major equipment. The approach works at all phases including Class 5 when one may only have a single number from a cost-capacity estimate or similar.
Question # 5- What are the major setbacks of conducting practical and realistic cost and schedule of risks analysis in work environment?
Hard questions will be asked because the contingency at p50 or the mean and P90 will likely increase significantly from the old rules-of-thumb. It also raises serious questions for the business because systemic risks belong to them. But in the end, your company will develop a stronger system.
When to Conduct Project Risk Quantification?
Question #6- Do you do the Risk Quantification post-mitigation or before the mitigation?
Do risk quantification at the phase decision gate. If risk has not been treated by that point, I take the conservative assumption that treatment will not happen or work well. Only treatments embedded in the current plan, budget, and activities are real and count. One does no favors for the team by being optimistic and facilitating lax control and change management discipline-optimist is not the risk analyst’s role.
What Tools to Use for Project Risk Quantification?
Question #7- What tool do you use to integrate cost and schedule risk quantification?
The tools presented are integrated. For example, they support the Joint Confidence Level (JCL) analysis. CPM is not necessary. Assessing cost and schedule trade-off for risk responses integrates cost and schedule inherently, and the parametric method yields both cost and schedule outcomes
Question #8- Is there any method or thumb rule which would help us to validate the outputs from various analysis software?
Validation requires a historical analysis of actuals vs. estimate. These are simple to plot and one can track the metrics improvement over time. However, this is not a “software” issue, but a methods issue. If the software supports the principles (e.g., risk-driven, cost, and schedule integrated, realistically quantifies systemic risks) it can meet the realistic goal (if not the practical one).
Schedule Risk Analysis
Question #9– How can we standardize to assessing the probabilities of activity durations (optimistic, most likely, pessimistic durations) between person to person?
Don’t do a subjective activity or cost line-item ranging, use an empirically based parametric model. The only subjectivity then is the ratings of the systemic risk factors. Research has discredited ranging when there are significant risks.
Question #10– How to quantify cost growth due to schedule delay?
The basic approach in both CPM or EV is delay times burn rate. However, the reality is that because the start of revenue date is so critical to NPV, that businesses will spend money to preserve the completion date. In other words, they trade cost for schedule. The result is that cost overrun is disproportional to delay. For example, if the risk response is to double the workforce to accelerate the work so there is no delay, the impact may be a negligible delay, but huge costs. We need to model that dynamic or we are really not doing our job.
Question #11- Do we have any correlation, in the construction industry, between the cost of developing detailed risk analysis from the onset of the project and the cost of claims preparation at claims settlement at the end of the project?
Project risk quantification puts a distribution on the likely outcome of the risks as they are. It does not reduce or change anything (unless the quantification outcome leads to re-cycling the scope and plans). If planning is lousy, there will be lots of claims and you will need lots of contingency to cover them. Better planning=less risks=less claims=less need for contingency. Direct connection.
Question #12- Planning Apps Administrator often advises us that conducting SRA of a larger schedule (greater than10,000 activities) with Primavera Risk Analysis won’t give the right output due to its large size. We have been always asked to develop the rolled small schedule (less than 1000 activities) replicating the critical path of the original schedule. Is this a recommendable practical approach?
One must always build a manageable “study” schedule for these large projects (if one uses the CPM method which I do not). I think this question raises the issue of the “practicality” of the CPM method.
Question #13- How do you address the relationships and correlations between schedule and cost risk events?
With the EV method, one first plans the risk response if the event occurs, or range of responses. This is the scope of the impact estimates for cost and duration. It considers cost and schedule trading explicitly. Therefore, cost and schedule are integrated.
Unknown-unknown Risks
Question #14- Is unknown-unknown a management reserve?
I do not believe there are unknown-unknowns and I discourage that loose language. In my years of benchmarking and consulting I have never seen a risk occur that has not happened before. There are only those risks that we choose not to quantify. That said, a low probability, high impact risk is not amenable to contingency funding, so should be considered for reserve funding instead.
Question #15- Is it a good idea to throw a risk element in the risk register for unknown-unknown items that are not clearly visible now but may appear later in the execution phase?
No. There is no such thing as unknown-unknown. Systemic risk and parametric model of them covers all risk except critical events, which are usually knowable even at early phases. Use a systemic risk rating tool because most systemic risks will not be in the risk register (they are politically incorrect).
How To Assess Risks On Various Project Delivery Methods, Project Components, etc>?
Question #16- Depending upon the type of project delivery, the parties that share risk differs. Evolving PD methods like IPD – Integrated Project Delivery enables stakeholders to share the risk, for example, between Owner, Designer, and Contractor. Do you have data on IPD projects?
I do not have specific research. However, the risk here is around “Team Development” which is a known risk driver and root cause. Any method such as IPD or alliances that facilitate better teams and coordination is good. However, no contract type is a magic bullet-it comes back to people, cultures, and systems and if they are not aligned, no legal format will help much. Sharing risk is somewhat of a misnomer; contractors have very little financial capacity when push comes to shove, and when serious failure sets in, they will react accordingly and push the risk back on the owner one way or the other.
Question #17- Most, if not all, capital projects contain a significant IT/IM component – how would you address risk assessment for these components, which are increasingly executed in an agile manner?
Did not cover IT. However, I believe the principles and general methods would apply. If you are talking about IT as part of getting the project player’s control and accounting systems to talk to each other, that is a key systemic risk. The only place I see IT in the EPC world is in the control system software and that is usually defined software and not being “coded”. What non-IT projects write raw code?
Question #18- Do you have a comprehensive list of risks for various categories of projects based on past experience? Often risks hit in the middle of project execution as risks were not captured at the planning phase in the risk register.
Yes, the historical step captures risks that have occurred for use in later risk identification. There are numerous risks that can be found but develop your own.
Question #19– What is the size of small projects?
Small projects are typically plant-based. Usually, <US$10-20 million is the threshold where companies assign large projects to a capital projects organization.
Question #20- Your slide of the case study shows a reduction of variability around a sanction-value. Did the case study also look at the absolute cost? You could argue it is easy to reduce variability by sanctioning a much higher value and ensuring you spend the money.
The case study was for a company that has competitors, in that case, pipeline. If they simply added more money, they would lose in the market. Further, the case study company has used its comprehensive historical data to calibrate its base estimate to be the p25 of history; i.e., a competitive target. Most estimators start with a “most likely” base which is uncompetitive from the start. Establishing a “cost strategy” is part and parcel of my methodology. One’s goal cannot be just predictability or one will just burn money.
Question #21- How is the weather risk considered?
In the expected value there are two ways. If it is just uncertainty around the base weather assumption, it can be an event with probability=100% and -/+ impact distribution around a most likely value of zero cost and time. If it is a discrete event such as the hurricane, typically use probability x impact.
Question #22- What is your opinion on Joint Confidence Level…and where it works better (Which Decision point)?
JCL is not a method. It’s a chart that @Risk can print for any analysis including the hybrid covered here. It is not that it “works better”; it is simply is a way of stating risk tolerance more clearly considering both cost and schedule. Instead of specifying your policy is cost p50 and schedule p50, you specify a policy of JCL being p50, which will require say p70 (or some other higher p-value) for each of cost and schedule to achieve; Just more conservative decision-making. I like it.
Question #23- Risk quantification in terms of money can be shown as a line item in the Budget and drawn down over time. How do you show Risk quantification in the schedule?
AACE RP 70R-12 covers schedule contingency principles and rules. The same as cost, there should be a distinct schedule contingency or buffer that you draw down from. Float is not contingency.
What is the Impact of Technological Advancements such as BIM, 4D, 5D, and AI on Project Risk Quantification?
Question #24- How do you believe that BIM 4D, 5D, and so on will impact if any, construction risks, quantities, and what else? In what way?
Risks are about people and overall project systems in a dynamic environment. BIM and eventually AI will reduce some uncertainty in some parts of the process, but I think the improvement will be nominal in terms of risk (maybe more so for absolute cost or efficiency). People will still screw up. When risks happen they will still do dumb things. Systems will still not work well as a whole.
Question #25- What do you see the future with artificial intelligence? Is there any impact?
AI requires data. Right now that is broken. Nobody collects useful data. It will take 10-20 years to finally collect useful data with which to train the robots. At that point, it will put most project controls people out of work because most really just report generators.
Question #26- Where do you see the future of risk management professionals with the advent of artificial intelligence?
Like project controls, I believe true AI will make risk management (risk register jockeys) largely obsolete. However, we have 10-20 years before data capture catches up in our world.
How to Start in Project Risk Quantification & Risk Management?
Question #27- I am a planning engineer and not so involved in costing or any of this PRQ. How should I start?
Study every paper and book you can get your hands on. Write a paper or two. Then when an opportunity comes up, raise your hand.
Question #28- I have a keen interest in risk management but little to no experience doing it. Where would you advise I start from?
Read, read, and read. Join AACE International. Read every paper and Recommended Practice on the topic. In the end, you will know more than many who call themselves experts.
Also, consider taking a formal training on Quantitative Risk Assessments.
What Is Covered In The Project Risk Quantification Book
Every AACE recommended practice is “risk-driven”. I think you are asking can you combine parametric and CPM, the answer is yes. See the Project Risk Quantification book.
Question #30- Have you introduced any new approach in the Project Risk Quantification book about risk projectization and using trend analysis?
Two different topics. The method is centered on “Critical” risks so prioritization is a key step. However, if there are more than 10-15 critical risks, one should likely not be doing the project. As to trend analysis, this is an analytical project control process in forecasting. Once one finishes that analysis, ask what is the risk of my forecast; re-do your risk quantification. If you are speaking of semi-automated EVMS trend forecasting, stop doing it; apply your best intelligence in every forecast.
Question #31- In chapter 7 of the Project Risk Quantification book, you state that Activity Ranging (Monte Carlos Simulation based on 3 point activity durations) should not be used to quantify schedule uncertainty. There are risk tools on the market that make this method increasingly easy to apply. I understand how to quantify discrete schedule risks (based on risk events in my risk register) to determine schedule contingency, but can your recap the pitfalls of Activity Ranging and describe the best method for quantifying schedule uncertainty?
Activity ranging (assignment of subjective uncertainty) has no empirical basis and typically ignores systemic risks. It is only as realistic as the analyst who applies it, which is not a reliable method. Apply the parametric model for the uncertainty. Per my book, or see the paper by Colin Cropley for more elegant methods of dealing with uncertainty in CPM based analyses.
Question #32- Is the Project Risk Quantification book applicable to the construction industry?
It is about project costs including all of Capex, including engineering, procurement, and construction. If you only do the “construction” part as a subcontractor, the methods apply but would need to be calibrated using your historical data.
Question #33- How can I get a copy of the Project Risk Quantification book?
Please check http://www.decisions-books.com/PRQ
I highly recommend you get a copy of John Hollmann’s Project Risk Quantification book. It’s a must-have book in any Project Controls professional library.
If you were part of this webcast, please share below your top 3 takeaways from the training.
About John Hollmann
John regularly reviews major international estimates and conducts risk analyses in support of investment decision making and contingency setting.
Mr. Hollmann is a frequent speaker at international conferences, has written many papers, and was the lead author of the AACE® International text; the Total Cost Management (TCM) Framework.
John is an Honorary Life Member of AACE®, recipient of their Award of Merit and Lifetime Achievement Awards. He is former Director of Recommended Practices at AACE® International and also led the development of the AACE® Decision and Risk Management Professional certification.
Prior to founding Validation Estimating, Mr. Hollmann led the Cost Engineering Committee metrics and research efforts at Independent Project Analysis, Inc.
John is a registered Professional Engineer, a Certified Cost Professional, a Certified Estimating Professional, and Decision and Risk Management Professional.
To connect with John, please check his LinkedIn profile or his website.
About the Author, Shohreh Ghorbani
Shohreh is the founder and director of Project Control Academy, the leading provider of comprehensive online training programs in Project Controls. Shohreh has served tens of thousands of professionals and several international corporations build their technical Project Controls knowledge and shave off years of trial and error in learning the vital skills in controlling their projects.
Shohreh is a licensed project management professional (PMP) recognized by the Project Management Institute (PMI) and holds a Master of Science in Industrial Engineering.
Connect with Shohreh via Facebook, Linkedin, Google+, Instagram, iTunes, YouTube
