Last month, MDigitalLife Lead Analyst Kayla Rodriguez introduced a new network scoring algorithm intended to take conference and conversation metrics beyond the blunt instruments of size and so-called “reach.” [The Evolution of ASH – Measuring the Strength of Online Networks]
Over the course of the last month, Kayla and our lead data scientist Dr. Yash Gad have worked together and with a group of our colleagues to take that algorithm apart, test it, tweak it and put it back together again many times. And over the past week, we’ve been “test-driving” the model as we look at the evolution of the online health ecosystem’s conversation around the JP Morgan Healthcare Conference (#JPM16).
Rather than just looking at the number of posts (size) or the theoretical “reach” of the authors of those posts (an aggregation of the number of followers each poster has – a misleading measurement), we’re measuring online conversations with 9 different metrics:
- Size Index: Our index incorporates the number of posts & the number of unique health ecosystem authors (validated & indexed stakeholders in the health ecosystem like doctors, patients, hospitals, reporters, healthcare companies, caregivers, etc.)
- Author Diversity: How many different stakeholder groups are represented in the conversation? Is it just doctors? Or just healthcare company executives?
- Content Diversity: How broad are the topics that are discussed at the conference?
- Connectivity: How well-connected are the participants in the conversation to each other?
- Equitability: Are there one or two people who dominate the conversation, or is there a more equal share among participants?
- Originality: How many posts are original vs shared (or retweeted) posts?
- Temporality: How long is the nose (pre-conference) and tail (post conference) relative to the conversation occurring in real time during the conference?
- Impact: How many of the participants have real influence in the online health ecosystem?
- Conversationality: How many posts represent actual conversations between participants as opposed to simple posts?
Those who have studied online conversation measurement for some time will hopefully recognize the importance of understanding each of these elements when studying the online back-channel at a conference. When we look at JP Morgan, for instance, we can see that the Size Index has grown impressively over the last 3 years (all subsequent data points relate to #JPM13, #JPM14 and #JPM15).
The growth in both number of posts and number of unique authors is impressive – and reflects the growing influence of JPM as a must-do conference for many of the worlds largest and most innovative healthcare brands.
Of all the stakeholder groups participating, the one with the most impressive growth in number of participants between 2013-2015 were physicians, who now make up 11% of the total participants in the conversation. It’s also worth noting that what was in 2015 largely an industry conversation has flipped … in 2015, there were more media outlets and reporters posting from the conference than the healthcare industry accounts themselves. The network that exists around the conference has also gotten a lot tighter. At some conferences, you’re likely to see a lot of disconnected posts, but at #JPM15, we saw the participants’ level of connectedness rise sharply. In fact, in 2015 more than 40% of the participants were mentioned by at least one other participant. Finally, the impact of the conference has increased dramatically. In healthcare circles, JPM has become a must-attend event where some of the most significant news is announced, where CEOs, CFOs, reporters and industry analysts rub elbows in settings both formal and informal. And rather than using a simplistic measurement like “reach” to describe how important the event is, we want to study whether the real “heavy hitters” are participating in the conference actively. As you can see, they are doing so in dramatically increasing numbers.
This level of sophistication in terms of being able to really understand and quantify the effectiveness of networks, conversations or events was a long time coming, and we know it has a long way to go. In fact, as my team was working through the current iteration with our brilliant colleague Lucas Galan in London, he pointed out an obvious flaw in our methodology: It’s too complicated. In tomorrow’s post, I’ll illustrate how we propose to simplify it. In the meantime, though, tell me through your comments and tweets: What questions do you have about the metrics? Where do you think we’ve missed the mark? What should we be doing to be more accurate and meaningful in the way we’re measuring? I’ll address your questions and comments in tomorrow’s post!
Connect with us on Twitter:
JPM16 Hashtag Image courtesy of Global Biodefense