We wrote here about the new guidance produced by the UK’s Cabinet Office around the use of the new Digital Outcomes and Specialists procurement framework. We admired much of the guidance material, although have some concerns about the practicality of running “mini-competitions” to choose the supplier when there may be a large number of capable suppliers who should be part of that final selection process – there are no less than 1200 suppliers on the list!
The guidance is very extensive covering the whole process, with an extensive section around evaluation. But it is in this area that we have another concern about the guidance. When it comes to evaluation criteria and evaluating price, the guidance recommends a route that many public sector organisations do follow (certainly in the UK) when it comes to converting a cost number into a “score”. That number can then be combined with “quality” type scores to arrive at a total mark for each supplier, which informs the selection decision.
The guidance says this (for fixed quotes).
To score fixed price quotes, you must divide the cheapest quote by each supplier’s quote.
- supplier A’s quote is £15,000
- supplier B’s quote is £10,000
- supplier C’s quote is £30,000
To calculate a score for supplier A, divide 10,000 by 15,000. Supplier A scores 0.667.
To calculate a score for supplier B, divide 10,000 by 10,000. Supplier B scores 1.
To calculate a score for supplier C, divide 10,000 by 30,000. Supplier A scores 0.333.
The problem with this is that the methodology is severely flawed, to the point that we believe a legal challenge could well succeed – if a bidder went that far of course. We have written about this problem several times before.
The method is non-linear in the relationship between cost and the score awarded which is basically illogical form a value perspective. Indeed, the simple example given in the guidance above highlights this illogicality perfectly. Supplier A loses 0.333 points by being £5000 more expensive than the cheapest bid (supplier B). But supplier C only loses an additional 0.333 points for being a further £15,000 more expensive. So actually the method favours really expensive bids over slightly expensive! This all is certainly open to serious questioning if it ever came to court.
On a similar note, here is the clever example given by Peter Marshall of Qinetiq Commerce Decisions on our Spend Matters UK/Europe website last year. He shows how a selection decision between two firms can be changed if a third enters the evaluation – even if that third firm does not win, its mere presence changes the winner in his example from one supplier to another from supplier A to supplier B, which is clearly crazy. It is also illogical that the mark given to one bidder is dependent on that given to another – it means that we are not assessing each bid in its own right. A “low-ball” bid can distort scoring for everyone else, for instance.
The problem is that this method is simple and easy to implement – other options are more complex or require a bit more effort. We won’t go into those alternatives today, but there is a problem here. We heard at a conference last year that this methodology is actually illegal (or specifically excluded in the national regulations) in Portugal. That may be going too far – but it is surprising to see such a dodgy methodology recommended by Crown Commercial Service. We’ve sent a copy of this article to the CCS policy head – we’ll let you know if we get any response.