Scoring Insights: What to Believe

Author: Tony Page

In the past, I’ve avoided talking much about my previous life in the intelligence community —not because “if I told you, I’d have to kill you,” but because, being British by birth, I’m burdened with excessive modesty when it comes to talking about myself. At the most recent MAPS meeting, however, I was struck by how well some of my anecdotal examples resonated and how relevant people found them. So I’ve decided to take the plunge and systematically share where my brain goes, as a former professional intelligence officer, when I think about common problems in insights.

The intelligence community has been doing “insights” for a long time, and many of their approaches provide useful models for how to think about insights in the pharmaceutical industry.

Intelligence Scoring Metrics 

As a start, I want to elaborate on something that came up in the recent Insights MasterClass at the annual MAPS meeting in Denver. During a discussion about how to evaluate the reliability of insights, I made a passing comment about how the intelligence community does this. I was quite surprised at how many people wrote it down and asked me about it later, so I thought it would be good to share it again, with a bit more detail.

So how does the intelligence community decide what to believe? In an industrial‑scale system with millions of data points coming in, you need a consistent methodology. When it comes to scoring specific pieces of incoming reporting, the intelligence community looks at two things:

1. the source of the insight (for example, a person), and
2. the information itself.

So every individual piece of intelligence is rated, with the highest quality being “A1” and the lowest “E5.” 

A Practical Example for Pharma

A practical example would be an individual criticizing something about your product on social media. The first question would be: “Who is this person?”

If the person posting is a KOL or an HCP who’s published in the area, treats a lot of patients, and is someone you trust based on past interactions, then this source would be an “A.” If it turns out the person doesn’t even work in the disease area and is just posting a seemingly unfounded personal opinion, they might be an “E.”

Likewise, if the information is consistent with what you’ve heard from advisors, it might be a “1,” but if it’s contradicted by other sources and just doesn’t make logical sense, then it’s a “5.”

This scoring methodology—separating the information from the source—is a proven approach, but execution requires being able to cross‑check a lot of data that typically lives in different silos within a company. Operationalizing it also requires some work with whatever CRM or data systems are being used.

When to Act

This scoring approach only applies to individual pieces of intelligence (i.e., MSL field observations, social media posts, conference notes, etc.). At the reporting level, where you communicate an opinion or recommendation to decision makers, these individual source validations play into the overall confidence you attribute to the finding and the reasonableness of acting on it. In the intelligence community, this subjective confidence level is typically described simply as “Very Low” to “Very High” confidence.

So if you have three good sources saying the same thing about a significant market shift, you can indicate that you have very high confidence in the reporting and reasonably say, “We need to turn the ship right now!” Whereas less reliable data points might be reported as low or medium confidence, and you might instead say, “This is a potentially serious concern that we need to keep an eye on and prepare a contingency plan for.”

When I ran a competitive intelligence agency, I included this scoring in footnotes to our reports so that clients could judge for themselves the reliability of our sources and findings. We also integrated this into an infographic at the end of each section showing our confidence level in the conclusions and indicating what types of sourcing the report was based on (interviews, social media, conference collection, etc.).

A Pet Peeve About Field Insights

One related pet peeve that I noticed immediately when I entered the industry (that complicates scoring) is that field reporting in industry often combines what the person reporting thinks with what was actually said by the source. In the intelligence community, field reporting is very clearly separated into two parts:

1. what the source said, and
2. any comments the reporter wants to add.

If a source says something that doesn’t seem to make much sense, that gets reported, as is. The reporter can comment that they think they probably meant something else but you never edit the source’s comments. Combining these two components injects a lot of reporter bias into the data and makes it difficult to distinguish later on what a KOL or HCP actually said from what the reporter thinks they meant to say—or the reporter’s opinions about it.

This complicates analysis because you can’t look for trends in “HCP language” when it’s conflated with “MSL language” and opinion. Part of the reason I think this happens is that we train MSLs to report “insights,” so we ask them to be “reporters” and “analysts” at the same time.

This is fine, as long as we separate the two. It’s great to have MSLs contribute to analysis, but the individual observations also need to be aggregated, compared, validated, and then analyzed before we can decide what’s an actionable insight and what isn’t. You’re rarely going to act on a single field observation. Likewise, an MSLs dire opinions about a single observation would also have to be taken with a grain of salt until all the data has been analyzed. 

It’s like the difference between lab results and a doctor’s initial intake observations about a patient. Both are important, but more labs and more observations are usually needed before that patient is diagnosed. Similarly, a lot of data from a lot of patients is going to be needed before anyone can say, “Hey, there’s an epidemic!”

So, I think best practice would be for MSLs to separate these two equally valuable parts of their reporting.

End transmission. 

Article written by Tony Page, Expert Insights Consultant, MAPS Insights Competency Co-Lead, & Former Intelligence Officer