The Perils of Metrics

A colleague 

mentioned the hazard of biasing a research investigation towards the data that are readily available over the data that are most desirable.

That seemed a good cue to post a link to this old chestnut.

http://quoteinvestigator.com/2013/04/11/better-light/

"How deeply a problem is understood and how concretely it is defined sets an upper bound for the quality of any subsequent solutions."

https://medium.com/intercom-inside/growth-hacking-is-bullshit-60aae95f9caa

True progress is hard; it is always easier to move the goalposts than to score more goals.  Human beings are very good at rationalization and self-deception, so adjustments made in ambiguous or complicated situations for varied motivations are not always obviously directly aimed at shifting the reference in order to inflate the metric.

On the other hand, some are more transparent than others:  "Los Angeles initially required that students earn at least a C in those classes, but the number of students on track to graduate plummeted. Now grades of D or higher are accepted."

While I'm not an unquestioning fan of Arne Duncan, in this case he gets the big picture exactly right:

"But `the goal is not just high school graduation,' Arne Duncan, the departing secretary of education, said in a telephone interview. `The goal is being truly college and career ready.'"

http://www.nytimes.com/2015/12/27/us/as-graduation-rates-rise-experts-fear-standards-have-fallen.html

"It’s easier to measure if that change led to increased engagement than to measure if it also made your users hate you" --Benedict Evans

https://twitter.com/BenedictEvans/status/673633094113484800

Metrics are used to answer questions, and questions are like fleas:

So, naturalists observe, a flea

Has smaller fleas that on him prey;

And these have smaller still to bite 'em,

And so proceed ad infinitum.

(Jonathan Swift, On Poetry:  A Rhapsody, 1733)

So a change in a single metric should be assessed both in the context of other metrics that address the same question, but also in the context of other peer subquestions that address a wider issue.  If the wider focus is not maintained, well, "... one little old metric goes down, and everyone loses their minds!". 

Here's an excellent example from one of my favorite financial markets commentators.

http://www.capitalspectator.com/dont-confuse-manufacturing-with-the-broad-macro-trend/