Hype, The Life Study and trying to do too much

A while back I reviewed Helen Pearson’s, “The Life Project” in the TLS. I had previously blogged on the perils of trying to do too much and mission creep and overload.

From the original draft of the review (published version differed slightly):

Pearson is laudably clear that the story of the birth cohorts is also a study of failure; the failure of the NHS to improve the inequality of health incomes between social classes, the failure of educational reforms and re-reforms to broach the similar academic achievement gap. Indeed, the book culminates in a failure which introduces a darker tone to the story of the birth cohort studies.

Launched in January 2015, the Life Study was supposed to follow 80,000 babies born in 2015 and intended to be a birth cohort for the “Olympic Children.” It had a government patron in David Willetts, who departure from politics in May 2015 perhaps set the stage for its collapse. Overstuffed antenatal clinics and a lack of health visitors meant that the Life Study’s participants would have to self-select. The optimistic scenario has 16,000 women signing up in the first eighteen months; in the first six months, 249 women did. By October 2015, just as Pearson was completing five years of work on this book, the study had officially been abandoned.

Along with the cancellation of the National Institute for Health’s National Children’s Study in December 2014, this made it clear that birth cohorts have been victims of their own success. An understandable tendency to include as much potentially useful information as possible seemed to have created massive, and ultimately unworkable cohorts. The Life Study would have generated vast data sets: “80,000 babies, warehouses of stool samples of placentas, gigabytes of video clips, several hundred thousand questionnaires and much more” (the history of the 1982 study repeated itself, perhaps.) Then there is the recruitment issue. Pregnant women volunteering for the Life Study would “travel to special recruitment centres set up for the study and then spend two hours there, answering questions and giving their samples of urine and blood.” Perhaps the surprise is that 249 pregnant women actually did volunteer for this.

Pearson’s book illustrates how tempting mission creep is. She recounts how birth cohorts went from obscure beginnings to official neglect with perpetual funding issues to suddenly becoming a crown jewel of British research. Indeed, as I observe in the review, while relatively few countries  have emulated the NHS’ structure and funding model, very many have tried to get on the birth cohort train.

This situation of an understandable enthusiasm and sudden fascination has parallels across health services and research. It is particularly a risk in eHealth and connected health, especially as the systems are inherently complex, and there is a great deal of fashionability to using technology more effectively in healthcare. It is one of those mom-and-apple-pie things, a god term, that can shut down critical thinking at times.

Megaprojects are seductive also in an age where the politics of funding research loom large. The big, “transformative” projects can squeeze out the less ambitious, less hype-y, more human-scale approaches. It can be another version of the Big Man theory of leadership.

Whatever we do, it is made up of a collection of tiny, often implicit actions, attitudes, near-reflexes, and is embedded in some kind of system beyond ourselves that is ultimately made up of other people performing and enacting a collection of tiny, often implicit actions, attitudes, and near-reflexes.

 

Helen Pearson, “The Life Project”, Review in TLS 29/03/17

I have a review of Helen Pearson’s “The Life Project” on the UK birth cohort studies in the current TLS. The full article is behind a paywall so here is the preview:

Born to fail

To a non-Briton, the oft-repeated assertion that the NHS is “the envy of the world” can grate. If imitation is the sincerest form of envy, the world’s laggardly adoption of free-at-point-of-use health care is perhaps the truest mark of how much emotional investment the rest of the world really has in the UK’s health system. Early in The Life Project, her book on the British birth cohort studies, Helen Pearson describes them as “the envy of scientists all over the world”. In this case, envy is easier to precisely pinpoint; birth cohort studies have become all the epidemiological and social scientific rage in recent decades, especially around the turn of the millennium. My own daughter, born in 2008, is a member of the Economic and Social Research Institute’s “Growing Up in Ireland” birth cohort.

1946 is the Year Zero of birth cohorts. The low interwar birth rate had caused much…

 

 

Evidence based medicine and evidence based policy

There was a fair bit of media coverage of the finding that teenage-pregnancy-prevention programmes using simulated babies are associated with an increased rather than decreased teenage pregnancy rate. Some of the media discussion focused on the role of evidence in public policy.

Via Twitter, I came across this article on evidence-based policy by Howard White in The Independent

Evidence-based medicine has transformed medical practice. TheCochrane Library has published more than 6,000 studies summarising high quality evidence for health interventions. Notable cases include breast screening, which used to be recommended for women from the age of 40 until the evidence showed that the number of false positives recorded was in fact doing more harm than good. The risks from unnecessary surgery were greater than the often small benefits from early treatment forbreast cancer.

Hormone replacement therapy (HRT) is another example. It was routinely used to reduce heart disease, but then became far less common when evidence showed adverse effects. There is now a more nuanced understanding of which women will benefit from HRT and which will not.

Prior to Cochrane, doctors based their advice on out-of-date knowledge, personal experience and the influence of drug reps. Today, doctors have access to evidence-based guidelines. Decisions on what the NHS can and should fund are informed by the advice of the National Institute for Health Clinical Excellence after a review of the evidence.

So why can’t we do the same for social and economic policy?

For those who are interested, I engaged (or am engaging) in a twitter exchange on this with Howard White, whose replies have been very gracious. As in so many of these exchanges I suspect that we agree on more than we disagree on (and possibly agree on everything with a difference in emphasis)

Of course policy should be based on evidence, where available. This not only seems extremely reasonable and  rational – it is eminently reasonable and rational. I also write as an admirer of the Cochrane Collaboration.

However, I always feel a sense of caution when clinical concepts are introduced into political discourse. The best definitions of EBM always include the word “judicious”, as here

“Evidence-based medicine is the conscientious explicit and judicious use of current best evidence in making decisions about the care of individual patients.”

Judicious is key – judgment and reflection are required. Does this body of evidence apply to my patient, this individual person in front of me, or does it not? As I wrote on another point:

it is one thing to have an evidence base for a specific therapy in medical practice, quite another for a specific intervention in society itself.

My review of Helen Pearson’s The Life Project is still to be published, when it does I will perhaps write a little more on “evidence-based policy”, a concept which began to enjoy great vogue in the 1990s.What Pearson’s book shows, however, is that the devil can cite evidence for his own purpose; “evidence” can be wielded with agendas.

Obviously the Campbell Collaboration aims to address this, by being transparent about the evidence used and the methodology used to synthesise it.

A further point is that evidence-based policy tends to presuppose consensus on the ends of policy  – and emphasise technocratic means of getting there. Thereby the focus on specific interventions, rather than any wider sense of not merely social goals but of social meaning. Of course, this very much in keeping with a time in which we are all supposed to be beyond “grand narratives” – which is of course itself a “grand narrative.” I would suggest that many recent events in politics around the world are best understood as testing this notion to destruction.

Bringing it all back to a question I asked a while ago about the best kind of evidence for health informatics innovations, perhaps what this illustrates is that the way we do evidence now tends to be to focus on specific interventions and, as far as possible, measure their effects as specific interventions and without reference to an overall system. Indeed, this is obviously necessary for assessing therapies and treatments. But is it necessarily missing something when it comes to a system?

Engaging clinicians and the evidence for informatics innovations

A few weeks ago Richard Gibson from Gartner spoke to members of the CCIO group. It was a fascinating, wide-ranging talk – managing the time effectively was a challenge. Dr Gibson talked about the implications for acute care and long term care of technological innovations – as might be obvious from my previous post here, I have a concern that much of the focus on empowerment via wearables and consumer technology misses the point that the vast bulk of healthcare is acute care and long term care. As Dr Gibson pointed out, at the rate things are going healthcare will be the only economic, social, indeed human activity in years to go

One long term concern I have about connected health approaches is engaging the wide group of clinicians. Groups like the CCIO do a good job (in my experience!) of engaging the already interested, more than likely unabashedly enthusiastic. At the other extreme, there always going to be some resistance to innovation almost on principle. In between, there is a larger group interested but perhaps sceptical.

One occasional response from peers to what I will call “informatics innovations” (to emphasise that this not about ICT but also about care planning and various other approaches that do not depend on “tech” for implementation) is to ask “where is the evidence?” And often this is not a call for empirical studies as such, but for an impossible standard – RCTs!

Now, I advocate for empirical studies of any innovation, and a willingness to admit when things are going wrong based on actual experience rather than theoretical evidence. In education, I strongly support the concept of Best Evidence Medical Education and indeed in following public debates and media coverage about education I personally find it frustrating that there is a sense that educational practice is purely opinion-based.

With innovation, the demand for the kind of RCT based evidence is something of a category error. There is also a wider issue of how “evidence-based” has migrated from healthcare to politics. In Helen Pearson’s Life Project we read how birth cohorts went from ignored, chronically underfunded studies ran by a few eccentrics to celebrated, slightly less underfunded, flagship projects of British epidemiology and sociology. Since the 1990s, they have enjoyed a policy vogue in tandem with a political emphasis on “evidence-based policy.” My own thought on this is that it is one thing to have an evidence base for a specific therapy in medical practice, quite another for a specific intervention in society itself.

I am also reminded of a passage in the closing chapters of Donald Berwick’s Escape Fire (I don’t have a copy of the book to hand so bear with me) which essentially consists of a dialogue between a younger, reforming doctor and an older, traditionally focused doctor. Somewhat in the manner of the Socratic dialogues in which (despite the meaning ascribed now to “Socratic”) Socrates turns out to be correct and his interlocutors wrong, the younger doctor has ready counters for the grumpy arguments of the older one. That is until towards the very end, when in a heartfelt speech the older doctor reveals his concerns not only about the changes of practice but what they mean for their own patients. It is easy to get into a false dichotomy between doctors open to change and those closed to change; often what can be perceived by eager reformers as resistance to change is based on legitimate concern about patient care. There are also concerns about an impersonal approach to medicine. Perhaps ensuring that colleagues know, to as robust a level as innovation allows, that patient care will be improved, is one way through this impasse.

 

The perils of trying to do too much: data, the Life Study, and Mission Overload

One interesting moment at the CCIO Network Summer School came in a panel discussion. A speaker was talking about the vast amount of data that can be collected and how impractical this can be. He gave the example of – while acknowledging that he completely understood why this particular data might be interesting – the postcode of  the patients most frequent visitor. As someone pointed out from the audience, the person in the best position to collect this data is probably the patient themselves.

When I heard this discussion, the part of my that still harbours research ambitions thought “that is a very interesting data point.” And working in a mixed urban/rural catchment area, in a service which has experienced unit closures and admission bed centralisation, I thought of how illustrative that would be of the personal experience behind these decisions.

However, the principle that was being stated – that clinical data is that which is generated in clinical activity – seems to be one of the only ways of keeping the potential vast amount of data that could go into an EHR manageable. Recently I have been reading Helen Pearson’s “The Life Project” , a review of which will shortly enough appear. Pearson tells the story of the UK Birth Cohort Studies. Most of this story is an account of these studies surviving against the institutional odds and becoming key cornerstones of British research. Pearson explicitly tries to create a sense of civic pride about these studies, akin to that felt about the NHS and BBC. However, in late 2015 the most recent birth cohort study, the Life Study, was cancelled for sheer lack of volunteers. The reasons for this are complex, and to my mind suggest something changing in British society in general (in the 1946 study it was assumed that mothers would simply comply with the request to participate as a sort of extension of wartime duty) – but one factor was surely the amount of questions to be answered and samples to be given:

But the Life Study aims to distinguish itself, in particular by collecting detailed information on pregnancy and the first year of the children’s lives — a period that is considered crucial in shaping later development.

The scientists plan to squirrel away freezer-fulls of tissue samples, including urine, blood, faeces and pieces of placenta, as well as reams of data, ranging from parents’ income to records of their mobile-phone use and videos of the babies interacting with their parents. (from Feb 2015 article in Nature by Pearson)

All very worthy, but it seems to me that the birth cohort studies were victims of their own success. Pearson describes that, almost from the start, they were torn between a more medical outlook and a more sociological outlook. Often this tension was fruitful, but in the case of Life Study it seems to have led to a Mission Overload.

I have often felt that there is a commonality of interest between the Health IT community, the research methodology community, and the medical education community and the potential of EHRs for epidemiology research, dissemination of best evidence at point of care  and realistic “virtual patient” construction is vast. I will come back to these areas of commonality again. However, there is also a need to remember the different ways a clinician, an IT professional, an epidemiologist, an administrator, and an educationalist might look at data. The Life Study perhaps serves as a warning.