Latest / Deep Dive Ortho — Orthopaedic Surgery, AI & Digital Health Podcast. / Missing Data Bias in Total Joint Replacement Outcomes

Transcript

Missing Data Bias in Total Joint Replacement Outcomes · Deep Dive Ortho — Orthopaedic Surgery, AI & Digital Health Podcast.

• 0:00
  Imagine you've had major surgery, maybe a hip
• 0:02
  replacement. You get asked later, perhaps a year
• 0:05
  or two on, how you're doing via a questionnaire.
• 0:07
  Now this feedback, yours and thousands of others,
• 0:09
  helps gauge how successful these operations really
• 0:12
  are. Seems simple enough, right? But... What
• 0:15
  if the people who don't send back those forms,
• 0:17
  what if they aren't just, you know, randomly
• 0:19
  busy? What if they're actually a specific group
• 0:21
  and their absence completely changes the picture
• 0:24
  of success that the data shows us? That's the
• 0:26
  crucial question we're exploring today on the
• 0:28
  deep dive. We're looking closely at a particular
• 0:30
  research paper on patient -reported outcome measures,
• 0:33
  PROMs after total hip replacement, and the significant
• 0:36
  bias that can crop up when patients don't complete
• 0:38
  their follow -up. Now, these PROMs, they're really
• 0:41
  vital tools. Organizations like NICE here in
• 0:43
  the UK, or the FDA over in the US, use them to
• 0:46
  understand the real world impact of surgery beyond
• 0:49
  just, say, survival rates or basic complications.
• 0:52
  They're meant to capture the patient's own view
• 0:54
  on their quality of life, their recovery. And
• 0:56
  joining us to walk us through this paper and
• 0:58
  this, well, critical area of health care research
• 1:00
  is Professor Mo Imam. Professor Imam is a leading
• 1:04
  orthopedic surgeon and researcher, very involved
• 1:07
  in outcomes analysis, and importantly, one of
• 1:09
  the authors of the actual study we're discussing
• 1:11
  today. Welcome. Thank you. It's a pleasure to
• 1:13
  be here. So maybe let's set the scene with some
• 1:15
  headline numbers from your study. You tracked
• 1:17
  patients having total hip replacements, looked
• 1:19
  at their response rates to questionnaires over
• 1:21
  time. How did that response rate change from
• 1:25
  before the operation? Well, our initial capture
• 1:27
  rate preoperatively was actually excellent, almost
• 1:30
  99 percent. Oh, wow. OK. But yes, as time went
• 1:33
  on, we did see a steady decline at six weeks
• 1:38
  after the operation. It was down to about 90
• 1:40
  .6 percent. Right. By six months, it had dipped
• 1:42
  slightly further to 89%. Then one year out, it
• 1:46
  was 83%. And by the time we reached the two -year
• 1:48
  mark, we were down to 79 % response. 79%. So
• 1:52
  quite a drop. Yes. A considerable proportion
• 1:55
  of that initial group was, well, no longer providing
• 1:57
  feedback at that point. So doing the maths quickly,
• 2:00
  that's roughly a fifth of the original patients
• 2:02
  who aren't represented in that two -year data.
• 2:04
  Now, was that drop off just random chance? Did
• 2:07
  it happen sort of? equally across all patient
• 2:09
  types? Or did your analysis find specific things
• 2:11
  linked to who didn't respond? Our analysis was
• 2:15
  very clear on this point. The non -response was
• 2:17
  absolutely not random. Right. We found quite
• 2:20
  significant associations between certain patient
• 2:22
  characteristics and their likelihood of responding
• 2:26
  to those follow -up questionnaires. OK. And that's
• 2:28
  really the crux of it, isn't it? So tell us,
• 2:31
  who were these patients who were less likely
• 2:33
  to respond? What did your study identify? Well,
• 2:36
  we found several key factors. One was age. Age,
• 2:41
  interesting. Yes, the non -responders were significantly
• 2:43
  younger than the responders, and this is true
• 2:45
  at all the follow -up points, with a p -value
• 2:48
  less than .001, so highly significant. For instance,
• 2:54
  at the two -year point, the median age of those
• 2:55
  who responded was about 69, but for the non -responders,
• 2:59
  the median age was 65. And why might that be?
• 3:03
  Does the paper suggest reasons? Well, the paper
• 3:05
  speculates a bit. It could be linked to younger
• 3:07
  patients perhaps having busier working lives,
• 3:09
  finding it harder to make time for questionnaires,
• 3:11
  or maybe feeling less connected to the health
• 3:14
  care system once that immediate post -op phase
• 3:16
  is over. OK. Younger patients not responding.
• 3:19
  That's quite an interesting finding, maybe a
• 3:21
  bit counterintuitive. What else stood out? Another
• 3:24
  really significant factor, and this is something
• 3:26
  we highlighted as quite a novel finding in this
• 3:29
  specific context, was the association with deprivation.
• 3:33
  Deprivation. How did you measure that? We used
• 3:35
  the income deprivation affecting children index,
• 3:37
  the IDACI score. It gives a measure of socioeconomic
• 3:41
  deprivation for an area here in the UK. And we
• 3:44
  saw that patients from more deprived areas, those
• 3:47
  with higher IDCI scores, were significantly less
• 3:50
  likely to respond. Again, this was a very robust
• 3:53
  finding, p -value less than .001, and it held
• 3:57
  true across all the time points we looked at.
• 3:59
  And the reasons for that? Likely multifaceted,
• 4:01
  I'd say. It could involve things like lower health
• 4:03
  literacy, perhaps, or maybe less stable housing
• 4:06
  or social situations, or simply having other
• 4:09
  more pressing concerns in their daily lives that
• 4:12
  mean completing and returning a form just isn't
• 4:14
  a priority. So, age and socioeconomic background
• 4:17
  are factors. What about their actual health status
• 4:19
  before the surgery? Did their starting point
• 4:21
  matter? Yes, it mattered profoundly. We found
• 4:25
  that patients who were less likely to respond
• 4:27
  later on had significantly lower preoperative
• 4:30
  scores on both the Oxford hip score and the EQ5D.
• 4:33
  Right, the Oxford hip score measures pain and
• 4:36
  function specifically for the hip. Exactly. So
• 4:38
  those who started with the worst baseline, meaning
• 4:41
  more pain, more functional limitation before
• 4:43
  their surgery, were actually less likely to engage
• 4:46
  with the follow -up process afterwards. To give
• 4:48
  you an example, patients who responded to all
• 4:50
  the post -op questionnaires had a median baseline
• 4:53
  OHS of 20. But those who missed all of them,
• 4:56
  they started lower, with a median OHS of just
• 4:58
  16. So that suggests that poor health -related
• 5:02
  quality of life right at the beginning is some
• 5:04
  kind of predictor for non -response later. It
• 5:07
  certainly seems to be, yes. And maybe most tellingly,
• 5:09
  you looked at whether a patient's previous experience
• 5:12
  or their score at an earlier point influenced
• 5:14
  whether they responded next time. What did that
• 5:16
  show about why people might not respond? Yes.
• 5:18
  This finding, I think, really strengthens the
• 5:21
  argument for disengagement being linked to the
• 5:23
  outcome itself. We found a strong correlation.
• 5:26
  If a patient had lower outcome scores or reported
• 5:30
  lower satisfaction at an earlier time point,
• 5:32
  they were significantly less likely to respond
• 5:35
  to the next questionnaire down the line. Oh,
• 5:37
  okay. So, for example, patients who didn't respond
• 5:39
  at the 24 -month mark, their median Oxford HIP
• 5:42
  score back at the 12 -month follow -up was 38.
• 5:46
  Compare that to those who did respond at 24 months.
• 5:48
  Their median OHS at 12 months was 43. That's
• 5:51
  a five -point difference. Is that significant?
• 5:54
  It is, yes. It's clinically meaningful and statistically
• 5:57
  significant. It strongly implies that if a patient
• 6:00
  felt they weren't achieving the outcome they'd
• 6:02
  hoped for, or perhaps wasn't satisfied with their
• 6:05
  result or the service received, they were then
• 6:07
  less inclined to participate in future data collection.
• 6:10
  It's almost like they've voted with their feet
• 6:12
  by simply not responding. That's a really powerful
• 6:15
  insight. It suggests it's not always just about
• 6:18
  practical barriers, but maybe a psychological
• 6:20
  disengagement based on how well they feel they've
• 6:22
  done. Precisely. Were there any factors you perhaps
• 6:25
  expected to be linked to non -response but actually
• 6:28
  weren't, according to your study? Well, interestingly,
• 6:30
  in our particular cohort, we didn't find a significant
• 6:33
  link between gender or whether the patient experienced
• 6:36
  surgical or medical complications after the operation
• 6:39
  and their likelihood of responding. Really? No
• 6:42
  link to complications? Not a statistically significant
• 6:44
  one in our data set, no. And the paper does note
• 6:47
  that the finding on gender actually differs from
• 6:50
  some other studies out there. Which just highlights,
• 6:52
  I suppose, that the factors influencing non -response
• 6:54
  can vary a bit, depending on the specific patient
• 6:57
  group and the setting you're looking at. Right.
• 6:59
  Context matters. OK. So we've got a clearer picture
• 7:02
  now of who tends to be missing from this follow
• 7:04
  -up data. Younger patients, those from more deprived
• 7:08
  areas, and crucially, those who started with
• 7:10
  lower scores or had poorer outcomes or satisfaction
• 7:13
  earlier on. So let's bridge that now to the core
• 7:17
  problem this paper really tackles, the bias.
• 7:20
  If that specific group is less likely to respond,
• 7:23
  what does it fundamentally mean for the outcome
• 7:25
  data that we do see reported? Well, the fundamental
• 7:28
  consequence, the big issue, is that the reported
• 7:31
  outcomes from these data sets or registries that
• 7:33
  suffer from significant loss to follow -up are
• 7:36
  inherently biased. Biased how? Because the group
• 7:39
  that is disproportionately not responding are
• 7:42
  the very patients with the worst scores, perhaps
• 7:45
  less improvement or lower satisfaction. So, the
• 7:48
  group that does respond is therefore skewed.
• 7:51
  It's weighted towards patients who are, generally
• 7:53
  speaking, doing well and are satisfied. So, the
• 7:57
  picture of success painted by those who reply
• 7:59
  is potentially overly optimistic compared to
• 8:02
  what the entire group of patients actually experienced.
• 8:05
  Precisely. And our study provides some quite
• 8:07
  concrete evidence for this bias. We look at the
• 8:10
  actual improvement in the Oxford HIPPS score,
• 8:12
  from baseline to follow -up. At 12 months, the
• 8:15
  median improvement for patients who responded
• 8:16
  was three points higher than for those who didn't
• 8:18
  respond at that point. Three points higher. And
• 8:20
  by 24 months, that difference had actually widened
• 8:23
  slightly to four points. Four points. Now, does
• 8:25
  that difference, three or four points, sound
• 8:27
  small or is it meaningful? While it might sound
• 8:30
  small numerically, it's statistically significant
• 8:32
  and it represents a tangible skew in the reported
• 8:35
  outcome data. If you only look at the responders,
• 8:38
  you are seeing results that are, on average,
• 8:41
  notably better than what the non -responders
• 8:43
  actually experienced. Is it unique to hip replacements
• 8:50
  or PROMs data? Or is it a much wider concern
• 8:53
  in health care data and registries generally?
• 8:56
  Oh, it's a much broader concern. This phenomenon
• 8:58
  where loss to follow up isn't random, where it
• 9:01
  represents a subset with different and often
• 9:03
  less favorable outcomes. Well, that's a critical
• 9:06
  consideration that really extends far beyond
• 9:07
  just hip PROMs. It's relevant for, I'd say, virtually
• 9:10
  any study or registry data set that relies on
• 9:13
  ongoing patient participation or follow up over
• 9:15
  time. So any long term tracking? Yes. The paper
• 9:19
  touches on its importance for survival analysis
• 9:21
  in joint replacement registries, for instance,
• 9:24
  and it acknowledges that the profile of non -responders
• 9:27
  might differ depending on the procedure. For
• 9:29
  example, with hip resurfacing, you might actually
• 9:32
  lose younger, more active patients to follow
• 9:34
  up precisely because they're doing so well and
• 9:37
  have just got back to their busy lives. That's
• 9:39
  a different dynamic to what we saw in total hip
• 9:41
  replacements. But the key point remains, the
• 9:44
  missing data is rarely, if ever, random. You
• 9:47
  also made a really fascinating comparison in
• 9:50
  the paper between the UK's national PROMS data
• 9:53
  and the Swedish HIP Arthroplasty Register. What
• 9:56
  did that comparison tell you about the impact
• 9:58
  of response rates? Ah, yes. That comparison offers
• 10:01
  quite powerful sort of external support for our
• 10:04
  findings on bias. The Swedish Register, you see,
• 10:07
  consistently achieves a significantly higher
• 10:08
  response rate, typically around 90%. 90 %? That's
• 10:11
  high. It is. Compare that to the national UK
• 10:14
  PROMS data, which was around 80 % at the time
• 10:17
  we did our study. Now, what's really striking
• 10:19
  is that despite presumably performing similar
• 10:22
  surgeries with similar patients, the Swedish
• 10:25
  registry reported a lower patient satisfaction
• 10:27
  rate, about 84 % satisfied. That's compared to
• 10:30
  a much higher rate, over 95 % reporting improvement
• 10:33
  in the UK PROMS data with its lower response
• 10:36
  rate. So higher response rate, lower reported
• 10:38
  satisfaction. How does that support the bias
• 10:40
  idea? Well, our interpretation is that because
• 10:43
  the Swedish registry captures a larger proportion
• 10:45
  of the total patient population, including more
• 10:48
  of those patients with less ideal outcomes, their
• 10:51
  reported satisfaction rate is actually a more
• 10:53
  accurate reflection. It's naturally lower because
• 10:55
  it includes more of the less satisfied people.
• 10:58
  It really underscores how the response rate itself
• 11:00
  directly impacts the picture of success that
• 11:02
  gets reported. That makes sense. Given this significant
• 11:05
  bias that can creep in, what are some potential
• 11:07
  ways forward? What strategies or efforts does
• 11:10
  the paper mention to try and reduce this loss
• 11:12
  to follow up and get a more complete picture?
• 11:15
  The paper really emphasizes that substantial
• 11:17
  effort is needed first to understand and then
• 11:20
  to overcome the barriers stopping patients from
• 11:22
  following up. This could involve practical things
• 11:25
  beyond just sending out standard postal questionnaires,
• 11:28
  offering different ways to respond, maybe email
• 11:31
  or telephone calls, perhaps even app -based systems.
• 11:33
  Using technology more? Potentially, yes. And
• 11:36
  also considering things like language barriers
• 11:39
  or providing extra assistance for patients who
• 11:42
  might have lower literacy levels or perhaps less
• 11:44
  access to digital technology. It's really about
• 11:47
  making the process as easy and accessible as
• 11:49
  we can for all patient groups, particularly those
• 11:52
  we've now identified. as being less likely to
• 11:54
  respond otherwise. Right, targeting the effort.
• 11:56
  And just briefly, what were some of the limitations
• 11:58
  of your own study that you acknowledged? Important
• 12:01
  for context? Yes, of course. Like any research,
• 12:04
  there were limitations. We didn't have access
• 12:06
  to really comprehensive socio -demographic details
• 12:08
  for all patients, things like education level
• 12:10
  or specific language needs. Nor did we have detailed
• 12:14
  clinical data on comorbidities or mental health
• 12:16
  conditions, which could certainly also influence
• 12:19
  response. Okay. Also, it's worth noting a low
• 12:22
  hip score isn't always solely due to the hip
• 12:24
  surgery itself. It can be influenced by other
• 12:27
  coexisting health conditions. And while our institutional
• 12:30
  data capture rate was relatively high compared
• 12:33
  to the national average at the time, these findings
• 12:36
  would definitely benefit from being validated
• 12:38
  in other populations and different health care
• 12:40
  settings. Transparency about limitations is always
• 12:43
  good practice. Okay, let's switch gears slightly
• 12:47
  lightning round. Quick questions, sharp insights.
• 12:50
  Professor, what's the single most crucial takeaway
• 12:52
  for anyone looking at patient feedback data?
• 12:56
  Assume non -responders are different. And assume
• 12:58
  their outcomes are likely less favorable than
• 13:01
  those who did respond. Never take the data you
• 13:04
  have as the whole picture unless your follow
• 13:06
  -up is near perfect. Good point. If you could
• 13:08
  implement one change tomorrow to improve PROM's
• 13:11
  data quality based purely on this study, what
• 13:13
  would it be? I'd shift focus and importantly
• 13:16
  resources towards actively understanding why
• 13:19
  certain patients aren't responding and then putting
• 13:21
  targeted strategies in place to try and re -engage
• 13:23
  them or capture their feedback effectively. Makes
• 13:26
  sense. And finally, what's the one essential
• 13:29
  question professionals should always ask themselves
• 13:31
  when they see published outcome data that's based
• 13:34
  on patient feedback? Always ask. What was the
• 13:37
  follow -up rate? And, truthfully, was any attempt
• 13:40
  made to characterize the non -responders? Because
• 13:42
  the lower that response rate, the greater the
• 13:45
  potential for bias, and the more cautious you
• 13:47
  absolutely must be about the conclusions being
• 13:50
  drawn from it. Excellent. Very clear points.
• 13:53
  So just to recap some of the most crucial takeaways
• 13:55
  from our deep dive today for everyone listening.
• 13:58
  Firstly, Patient response rates in long -term
• 14:00
  studies like PROMs often drop off significantly,
• 14:03
  which leaves quite a substantial gap in the data
• 14:05
  we collect. Secondly, and this is critical, this
• 14:08
  non -response isn't just random. Studies, like
• 14:11
  the one we've discussed, show that the patients
• 14:13
  less likely to respond are often younger, come
• 14:15
  from more deprived areas, and significantly had
• 14:18
  lower baseline scores or experienced poorer outcomes
• 14:21
  and satisfaction earlier on. Thirdly, this leads
• 14:24
  directly to a major issue. Bias. The reported
• 14:28
  outcome data, based only on those patients who
• 14:30
  do respond, is likely painting an overly positive
• 14:33
  picture compared to the true average outcome
• 14:35
  for all patients because those less successful
• 14:37
  results are disproportionately missing. Fourthly,
• 14:40
  this isn't just an orthopedics issue. This problem
• 14:43
  of non -random loss to follow -up and the bias
• 14:45
  it creates is a vital thing to consider for anyone
• 14:48
  working with patient reported outcomes, registry
• 14:50
  data, or really any research relying on ongoing
• 14:52
  patient feedback over time. And finally, getting
• 14:54
  a genuinely accurate understanding of how well
• 14:57
  treatments are truly working requires real, dedicated
• 15:01
  effort. Effort to understand and reduce the barriers
• 15:04
  to follow -up, making sure the voices and experiences
• 15:06
  of patients who had less favorable outcomes are
• 15:09
  also included so we get a more complete and much
• 15:12
  less biased picture. That was a really essential
• 15:15
  deep dive into a critical aspect of healthcare
• 15:17
  data validity. Understanding this potential for
• 15:20
  bias and patient follow -up is just fundamental
• 15:22
  for anyone making decisions based on these kinds
• 15:24
  of metrics. And if you found this deep dive valuable,
• 15:27
  please do take a moment to rate and share the
• 15:28
  show. It really helps other professionals discover
• 15:30
  insights like these. Yes. I think this research
• 15:33
  really does underscore that the methodology of
• 15:35
  how we collect data profoundly impacts the reliability
• 15:39
  of the conclusions we can draw about clinical
• 15:41
  effectiveness. It pushes us all, I hope, to be
• 15:44
  more critical consumers of outcome data. Absolutely.
• 15:47
  So perhaps a final thought for you to consider
• 15:50
  as you look at outcome data in your own professional
• 15:52
  world. Always pause and ask yourself who might
• 15:55
  not be represented in these dumpers and what
• 15:57
  might their story and their outcome truly be?