Which one? #2

I've criticised the common weighted scoring method of evaluating proposals as a type of voodoo: a ceremony that appears to be attached to reality, but, in reality, is not. It is so unprotected against common cognitive biases and misaprehensions that it may as well be 'voodoo'.

What to do, then?

Rather than subjective scoring and thinking that this produces meaning, let alone numbers that are able to be manipulated mathematically, a system of ranking by measures of actual criteria achieved, then setting hurdles for a stage 1 evaluation against requirement domains is more likely to be accurate.

The Stage 1 evaluation grid looks like this:

 The 'hurdle' is the minimum rank that must be achieved to be satisfactory, or 'in consideration'. I've set that at least half the domains must jump the hurdle for the proposal to proceed to the next stage.

The ranks are achieved objectively; for instance, under 'compliance' the proposal might need to contain satisfactory information about: board overview, corporate governance systems, means of complying with WHS requirements, means of ensuring compliance with local government approval conditions, method of ensuring sound procurement of sub-contractors, how relevant board and executive committees are employed in respect of this project, how legal and procedural obligations under the contract are met: that's at total of 7 areas of interest (in reality there should be quite a few more specified). Count the satisfactory ones, and give a rating.

Proposal 'b' has a rating of 2: it meets less than 81% and more than 60% of requirements:  that is, 5 items are satisfactory. This is a binary choice, no 'grades'; an item is either in or out.

Stage 2

At stage 2 we get serious. It would be rare for any of the contenders to be over the hurdle in all domains, so we work with only those who meet the 'clearance count': the count of number of domains for further consideration. The degree of criticality of the domain for project success is reflected in the hurdle rating.

Further consideration should be a probabilistic evaluation of the value (expected value) that the owner would obtain from the proposal.

Let's say the proposal 'b' is set at $10m. In the domain of 'urgency' for example, we see that the contractor will deploy on site much later than expected, increasing the risk of an overrun by, say 10%. The owner will lose $10k per day. The effective loss is therefore $1k per day...and so on.

This approach will give calulated and examinable numbers related to value produced.

In 'soft' projects, a similar approach would apply, but the estimating environment and related calculations would need to be handled differently.