Organizations need to pick winners and losers.  For example, a government must decide to whether to fund project X, Y or Z; a corporation only has the capital to build asset A, B or C. 

Most organizations have developed a portfolio selection and management methodology.  There are typically 2 parts to such a process: a set of criterion and a scoring scheme to rank the criteria.  In this blog, I want to focus on the second challenge, the scoring. 

The +/- Metric

Most scoring methods use a set of positive values along with a rubric.  A corporation, internal rate of return (IRR) is likely one criteria but regulatory compliance could be another.  Project ‘A’ improves regulatory compliance but costs money (e.g. a negative IRR) while Project ‘B’ has a strong IRR but its regulatory compliance is dodgy at best.  

Most scoring schemes only return positive numbers.  A negative IRR, in the above example, might be given a ‘zero’ or low score when it really ought to detract from that specific-criteria.  The +/- Metric recognizes that for specific criteria not every project can make a positive contribution.   Conceptually, this can be thought of with the following graphic.  

Six Degrees of Consideration

There are six possible whole-number scores that a specific criterion can be awarded: two-negative, three-positive and a zero-score when a criterion is not applicable.  The following table lists the generic rubric and the subsequent table provides an example of a scoring rubric.  

Number	Name	Description
3	Significant Benefit	Strongest possible score typically benefiting all considerations of that criteria.  Significantly or overwhelming criteria-benefit
2	Strong Benefit	Strong score typically benefiting a majority of the considerations for the criteria but usually not all.  Materially beneficial.
1	Some Benefit – Neutral	Typical or average criteria result.  Ideally most projects will be awarded this score if the criterion is only somewhat applicable.
0	Not Applicable	For this project, this attribute is not applicable or it would be nonsensical to attempt attribution.
NR	Not Received/ Incomplete	Applicable but the submission is incomplete, not received or otherwise not usable.
-1	Detriment	Project will definitively cause harm relative to this criterion.  This harm may be acceptable due to strong positive benefits in other criteria.
-2	Strong Detriment	Harmful or negative consequences are extreme and the project can only be considered by having the highest political/organizational acceptance.

An example of a scoring rubric that may be used for one criterion based on the +/- Scoring Metric. 

Criteria	Comments	#	Rubric
Project Timeline	The longer a project takes the greater its relative risks.	3	Project will be fully commissioned by 2095-12-31.
2	Project will be fully commissioned by 2094-12-31.
1	Project will be fully commissioned by 2093-12-31.
0	Not Applicable
-1	Project will be fully commissioned later than 2095-12-31.
-2	A timeline can not be estimated at this point.

Six and Tired of Analysis Paralysis

When scoring a particular criterion, only whole numbers are used but when analyzing the entire criteria, one or two decimal places can be reported.  There is nothing profound using six scores other than they represent a reasonable spread of possible values.  Adding positive or negative numbers on either end is perfectly acceptable – with a caveat.  Adding more numbers means increased differentiation between these values. Increased precision in the metric may simply lead to non-value-added analysis and ‘evaluation-churn’.  

Ultimately, this metric and the scoring criteria exists to help an organization compare disparate projects in a reasonably manner.  The final decision, however, must take into consideration other intangible considerations that are difficult to score or measure.  This includes the question, ‘Is this the right decision, will our stakeholders support it, is it ethical and economic?’.

Part of a Decision System

The +/- Metric fleshes out the decision process.  The reality is that a negative score will seldom be awarded.  A project with very strong negative results likely will not make very far in the decision process.  Nevertheless, awarding a negative score will generate the greatest degree of discussion and consideration.  Humans are profoundly loss-averse – a negative score is jarring to the senses.  This is a good thing because selecting Project A, B or C is a human activity that involves both mechanical metrics and those ephemeral qualities of leadership and good judgement. 

Notes and Further Reading 

• The 5 Classification Evaluation metrics every Data Scientist must know.  Worth reading for the first section on Accuracy, Precision, and Recall.
• Loss Aversion, Wikipedia.