The Perils of Metrics

Got blood?  Here's an elegant example of the positive power of data-informed decision-making.

Transfusions save lives, but they are also expensive and not without risks.  Each decision whether or not to transfuse is the output of some decision-making process, presenting the opportunity to reach a better balance by improving the algorithm.  It may one day be possible to run every lab test that might be relevant at a reasonable cost and feed the results into an extremely accurate ML model developed at a reasonable cost.  But we aren't there yet.

So these researchers built a system to compare readily available lab results to previously-developed guidelines, then inform the doctor whether or not a transfusion was indicated.  The data did not drive the decision, and the primary goal of great patient outcomes was not altered.

The result:  "By simply asking doctors to think twice about transfusions, the hospital had not only reduced costs, but also improved patient outcomes."

http://www.nature.com/news/evidence-based-medicine-save-blood-save-lives-1.17224 

Let's start the year with an example that is, at least superficially, on the light-hearted side:  librarians creating fake patrons to check out important books that would otherwise be discarded for lack of popularity.

Underneath are two deep and important dysfunctions.  The specific one -- aiming to show people only what they are likely to want to see based on popularity -- is how fake news spreads virally.  The generic one is the idea that metrics provide a straight line to decisions with no thought or interpretation required along the way.

"The problem here isn't the collection of data: it's the blind adherence to data over human judgment, the use of data as a shackle rather than a tool."

http://boingboing.net/2017/01/02/automated-book-culling-softwar.html

Here's an interesting way to take metrics dysfunction to an entirely different level.  Examples abound of optimizing for only easy-to-measure aspects of an activity and getting, well, what was measured.  The new and innovative idea described here is to select among activities based on which of them have easy-to-obtain metrics. 

Another way to look at this is to view it as a version of the problem of measuring inputs rather than outputs.  Choosing a course of action should be based on some model of expected value -- and metrics should be chosen based on the actionable insight they are likely to yield.

http://www.business2community.com/marketing/beware-dark-side-marketing-performance-measurement-01710377#s5tQGEW4LwYeumYr.97

During a presidential debate in 1980 Ronald Reagan famously asked "Are you better off than you were four years ago?".  His next sentence is less well-known but it shows how quickly we move to conflate measurement of human well-being with financial and economic metrics:  "Is it easier for you to go and buy things in the stores than it was four years ago?"

A previous post here (https://www.onionsauce.com/mole_end/metrics/it-s-not-work-if-you-don-t) pointed to the omission of unpaid work -- most egregiously household work often done by women -- from national accounts.  The key problem is that there is no money transaction involved, and that's all the processes can account.  Ironically, if the stores in Reagan's second sentence had started giving everything away for free, GDP would have gone down and people could have logically answered "no, it has become impossible to buy anything".

Which brings us to today's item.  While it is relatively easy for economists to continue to gloss over the contributions of homemakers everywhere it is harder to explain how some of the most prominent -- and, according to stock markets, most valuable -- global companies add nothing.  And, in this case, the gap between the information needed and the metric used has international political consequences.

https://www.washingtonpost.com/news/innovations/wp/2016/10/24/google-and-facebook-contribute-zero-economic-value-thats-a-big-problem-for-trade 

If Wells Fargo had set out to write a cautionary tale, a case study of data-driven dysfunction, they could not have succeeded better.

The first article posted here showed how creating strong incentives to hit numerical targets drove people over the edge.  Entirely predictable, and straight out of Austin's Measuring and Managing Performance in Organizations.  (The first post in this blog links to that book.)

But why?  Why push for more and more accounts, particularly from existing customers?  If someone is already a customer, is there really enough margin in that person having two checking accounts rather than one?  Now we have a report pointing to Data Science getting used as a cudgel by someone who didn't understand the relationship between correlation and causation:

"It all stems from Wells Fargo's internal goal of selling at least eight financial products per customer. It's what Wells Fargo calls the "Gr-eight initiative." Currently, Wells Fargo boasts an average of about six financial products per customer."

While I have no direct inside knowledge, I'll bet dollars to doughnuts that someone found that customers with eight financial products were the most profitable.  Without enquiring further into why that might be so, someone dreamed up Gr-eight.  (A plausible explanation, by the way, is that relationship customers are more profitable than transactional customers and that people with a strong relationship with the bank are more likely to turn to it as their financial needs change.)

http://money.cnn.com/2016/09/09/investing/wells-fargo-phony-accounts-culture/