A Cruel Metric for Journalism, and Why Almost Nothing Scores

Journalism is judged by prizes, by ratings, by the prestige of the outlet, and by whether the piece felt important. None of these measure what journalism is supposed to do, which is to make the public know something it did not know before. So here is a metric that does, one half for questions and one half for answers. It is simple to apply, and applying it is a bleak experience, which is why nobody applies it.

The Metric

A question scores if it extracts information the subject did not want to give.

That is the whole of the questioner's job. A subject who wanted the information out could have issued a press release. The interviewer exists for the residue: the fact the minister would rather not confirm, the number the executive would rather not state, the contradiction the candidate would rather not have pointed out. A question that produces something the subject was happy to say has produced a press release with a microphone in front of it.

An answer scores if it is falsifiable and changes what we know.

Falsifiable: it could turn out to be wrong, and we would be able to tell. "We are committed to the security of every citizen" cannot be wrong; it is not an answer. "The unit was withdrawn on the ninth" can be wrong, and is therefore information. Changes what we know: after hearing it, some belief a reasonable listener held is now different. If everyone who knew anything already knew it, nothing was reported.

Two tests, each a yes or a no. Now run them.

Running It on an Interview

Take an ordinary political interview, the kind broadcast every evening.

The first question is about the news of the day. The subject gives the position his office prepared. Question: did it extract anything he did not want to give? No. Answer: falsifiable? No, it is a statement of values. Score: zero.

The second question asks him to respond to something an opponent said. He responds with a criticism of the opponent, also prepared. Zero.

The third question is the tough one, the one the interviewer is proud of: "But isn't it true that..." The subject says the premise is wrong and repeats the position. Nothing was extracted; the answer is unfalsifiable. Zero.

A ten-minute segment, eight questions, and the public knows exactly what it knew at the start, plus the fact that the interviewer was tough, which is not information about the world. The typical score of the typical interview is zero. Not low. Zero.

Why It Is Cruel

The metric is cruel because it makes no allowance for effort, for difficulty, or for how the segment felt.

A journalist who prepared for a week, asked sharp questions, and was stonewalled scores the same as one who read the questions off a card. That seems unfair, and it is meant to. The public did not receive the week of preparation. It received the stonewalling. Effort that produces no information is, from the reader's side, indistinguishable from no effort.

It is cruel to the subjects too. An answer that is careful, articulate, and unfalsifiable scores zero, and a rude one that leaks a fact scores one. The polished performer who has never once said anything checkable has, by this measure, never given an interview.

And it is cruel to the audience, because it tells them what they have been consuming. Most of what is called news, measured this way, is the exchange of positions between people who already had them.

What Scores

It is worth saying what does pass, because the list is instructive.

Documents obtained and published. Numbers the institution did not release. The follow-up question that pins a vague answer to a specific claim, and then the second follow-up when the specific claim is dodged. The interview with the person who was in the room and was not supposed to talk. Investigative work generally, which is expensive and slow and scores, and commentary generally, which is cheap and fast and does not.

Notice that the scoring work is almost all adversarial and almost all invisible. The visible journalism, the studio, the panel, the interview, is where the scores are lowest, because it is where the subjects have the most control over what is said.

What the Metric Is For

I do not expect newsrooms to adopt this. It would empty the evening schedule.

It is for the reader. Apply it while watching and the effect is immediate: you stop hearing the tone and start counting. Did that question get anything? Could that answer have been false? The count at the end of the hour is the amount of journalism you received, and once you have counted a few times you cannot go back to grading on how it felt.

The cruelty is the accuracy. Every gentler metric measures something else and calls it journalism.