technoscepticism – A Medical Education

From Mike Caufield, a piece that reminds me of the adage Garbage In, Garbage Out:

For many years, the underlying thesis of the tech world has been that there is too much information and therefore we need technology to surface the best information. In the mid 2000s, that technology was pitched as Web 2.0. Nowadays, the solution is supposedly AI.

I’m increasingly convinced, however, that our problem is not information overload but information underload. We suffer not because there is just too much good information out there to process, but because most information out there is low quality slapdash takes on low quality research, endlessly pinging around the spin-o-sphere.

Take, for instance, the latest news on Watson. Watson, you might remember, was IBM’s former AI-based Jeopardy winner that was going to go from “Who is David McCullough?” to curing cancer.

So how has this worked out? Four years later, Watson has yet to treat a patient. It’s hit a roadblock with some changes in backend records systems. And most importantly, it can’t figure out how to treat cancer because we don’t currently have enough good information on how to treat cancer:

“IBM spun a story about how Watson could improve cancer treatment that was superficially plausible – there are thousands of research papers published every year and no doctor can read them all,” said David Howard, a faculty member in the Department of Health Policy and Management at Emory University, via email. “However, the problem is not that there is too much information, but rather there is too little. Only a handful of published articles are high-quality, randomized trials. In many cases, oncologists have to choose between drugs that have never been directly compared in a randomized trial.”
This is not just the case with cancer, of course. You’ve heard about the reproducibility crisis, right? Most published research findings are false. And they are false for a number of reasons, but primary reasons include that there are no incentives for researchers to check the research, that data is not shared, and that publications aren’t particularly interested in publishing boring findings. The push to commercialize university research has also corrupted expertise, putting a thumb on the scale for anything universities can license or monetize.

In other words, there’s not enough information out there, and what’s out there is generally worse than it should be.

You can find this pattern in less dramatic areas as well — in fact, almost any place that you’re told big data and analytics will save us. Take Netflix as an example. Endless thinkpieces have been written about the Netflix matching algorithm, but for many years that algorithm could only match you with the equivalent of the films in the Walmart bargain bin, because Netflix had a matching algorithm but nothing worth watching. (Are you starting to see the pattern here?)

In this case at least, the story has a happy ending. Since Netflix is a business and needs to survive, they decided not to pour the majority of their money into newer algorithms to better match people with the version of Big Momma’s House they would hate the least. Instead, they poured their money into making and obtaining things people actually wanted to watch, and as a result Netflix is actually useful now. But if you stick with Netflix or Amazon Prime today it’s more likely because you are hooked on something they created than that you are sold on the strength of their recommendation engine.

Let’s belabor the point: let’s talk about Big Data in education. It’s easy to pick on MOOCs, but remember that the big value proposition of MOOCs was that with millions of students we would finally spot patterns that would allow us to supercharge learning. Recommendation engines would parse these patterns, and… well, what? Do we have a bunch of superb educational content just waiting in the wings that I don’t know about? Do we even have decent educational research that can conclusively direct people to solutions? If the world of cancer research is compromised, the world of educational research is a control group wasteland.

Ok, it may seem somewhat arbitrary to bring up a column on MobiHealthNews, a website which promises the latest in digital health news direct to your inbox. However this particular column, and also some of the responses which Versel provoked (collected here), struck a chord with me at the time and indeed largely inspired my presentation at this workshop at the 2013 eChallenges conference.

In 2012 I had beta tested a couple of apps in the general health field (I won’t go into any more specifics) – none of which seemed clinically useful. My interest in healthcare technology had flowed largely from my interest in technology in medical education. Versel’s column, and the comments attributed to “Cynical” in the follow up column by Brian Dolan, struck a chord. I also found they transcended the often labyrinthine structures of US Healthcare.

The key paragraph of Versel’s original column was this

What those projects all have in common is that they never figured out some of the basic realities of healthcare. Fitness and healthcare are distinct markets. The vast majority of healthcare spending comes not from workout freaks and the worried well, but from chronic diseases and acute care. Sure, you can prevent a lot of future ailments by promoting active lifestyles today, but you might not see a return on investment for decades.

..but an awful lot of it is worth quoting:

Pardon my skepticism, but hasn’t everyone peddling a DTC health tool focused on user engagement? Isn’t that the point of all the gamification apps, widgets and gizmos?

I never was able to find anything unique about Massive Health, other than its Massive Hype. It had a high-minded business name, a Silicon Valley rock star on board — namely former Mozilla Firefox creative lead Asa Raskin — and a lot of buzz. But no real breakthroughs or much in the way of actual products.

….

Another problem is that Massive Health, Google Health, Revolution Health and Keas never came to grips with the fact that healthcare is unlike any other industry.

In the case of Google and every other “untethered” personal health record out there, it didn’t fit physician workflow. That’s why I was disheartened to learn this week that one of the first twodevelopment partners for Walgreens’ new API for prescription refills is a PHR startup called Healthspek. I hate to say it, but that is bound to fail unless Walgreens finds a way to populate Healthspek records with pharmacy and Take Care Health System clinic data.

Predictably enough, there was a strong response to Versel’s column. Here is Dr Betsy Bennet:

As a health psychologist with a lot of years in pharma and healthcare, I am continually frustrated with the hype that accompanies most “health apps”. Not everyone enjoys computer games, not everyone wants to “share” the issues they’re ashamed of with their “social network”, not everyone is interested in being a “quantified self”. This is not to say that digital health is futile or a bad idea. But if we took the time to understand why so many doctors hate EHRs and patients are not interested in paying to “manage their health information” (What does that mean, anyway?) we would come a long way towards finding digital interventions that people actually want to use.

The most trenchant (particularly point 1) comment was from “Cynical”

Well written. This is one of the few columns (or rants) that actually understands the reality of healthcare and digital health (attending any health care conference will also highlight this divide). What I am finding is two fold:

1. The vast majority of these DTC products are created by people who have had success in other areas of “digital” – and therefore they build what they know – consumer facing apps / websites that just happen to be focused in health. They think that healthcare is huge ($$), broken, and therefore easily fixed using the same principals applied to music, banking, or finding a movie. But they have zero understanding of the “business of healthcare”, and as a result have no ability to actually sell their products into the health care industry – one of the slowest moving, convoluted, and cumbersome industries in the world.

2. Almost none of these products have any clinical knowledge closely integrated — many have a doctor (entrepreneur) on the “advisory board”, but in most cases there are no actual practicing physicians involved (physician founders are often still in med school, only practiced for a limited time, or never at all). This results in two problems – one of which the author notes – no understanding of workflow; the other being no real clinical efficacy for the product — meaning, they do not actually improve health, improve efficiency, or lower cost. Any physician will be able to lament the issues of self-reported data…

Instead of hanging out at gyms or restaurants building apps for diets or food I would recommend digital health entrepreneurs hang out in any casino in America around 1pm any day of the week – that is your audience. And until your product tests well with that group, you have no real shot.

This perspective from Jim Bloedau is also worth quoting., given how much of the rhetoric on healthcare and technology is focused on the dysfunctionality of the current system:

Who likes consuming healthcare? Nobody. How many providers have you heard say they wish they could spend more time in the office? Never. Because of this, the industry’s growth has been predicated on the idea that somebody else will do it all for me – employers will provide insurance and pay for it, doctors will provide care. This is also the driver of the traditional business model for healthcare that many pundits label as a “dysfunctional healthcare system.” Actually, the business of healthcare has been optimized as it has been designed – as a volume based business and is working very well.

Coming up to four years on, and from my own point of viewing having had further immersion in the health IT world, how does it stack up? Well, for one thing I seem not to hear the word “gamification” quite that much. There seems to be a realisation that having “clinical knowledge closely integrated” is not a nice to have have but an absolute sine qua non. Within the CCIO group and from my experience of the CCIO Summer school, there certain isn’t a sense that healthcare is going to be “easily fixed” by technology. Bob Wachter’s book and report also seem to have tempered much hype.

Yet an awful lot of Versel’s original critique and the responses he provoked still rings true about the wider culture and discussion of healthcare and technology, not in CCIO circles in my experience but elsewhere. There is still often a rather inchoate assumption that the likes of the FitBit will in some sense transform things. As Cynical states above, in the majority of cases self-reported data is something there are issues with, (there are exceptions such as mood and sleep diaries, and Early Warning Signals systems in bipolar disorder, but there too a simplicity and judiciousness is key)

Re-reading his blog post I am also struck by his lede, which was that mobile tech has enabled what could be described as the Axis of Sedentary to a far greater degree than it has enable the forces of exercise and healthy eating. Versel graciously spent some time on the phone with me prior to the EuroChallenges workshop linked to above and provided me with very many further insights. I would be interested to know what he makes of the scene outlined in his column now.