Latest / Deep Dive Ortho — Orthopaedic Surgery, AI & Digital Health Podcast. / Noise: A Flaw in Medical Judgment

Transcript

Noise: A Flaw in Medical Judgment · Deep Dive Ortho — Orthopaedic Surgery, AI & Digital Health Podcast.

• 0:00
  Imagine this, two people, same qualifications,
• 0:03
  applying for the same job. One gets a resounding
• 0:06
  yes, the other a polite no. Or maybe two similar
• 0:10
  insurance claims, handled by different adjusters,
• 0:14
  resulting in vastly different payouts. Right,
• 0:16
  I see where you're going. And even more striking,
• 0:18
  picture two expert judges. They review nearly
• 0:21
  identical cases, but hand down sentences that
• 0:24
  differ, well, significantly. We sort of assume
• 0:28
  expertise brings consistency, don't we? But what
• 0:31
  if that inconsistency is actually the norm hiding
• 0:34
  in plain sight? A powerful thought, isn't it?
• 0:36
  This kind of variability in judgment, even among
• 0:38
  seasoned professionals, it's far more prevalent
• 0:41
  and impactful than many of us realize. Really?
• 0:44
  Yeah, we tend to focus on deliberate biases,
• 0:47
  on leaning one way or another, but often overlook
• 0:49
  the sheer Well, randomness that creeps into decisions.
• 0:53
  Absolutely, and that silent unnoticed variability
• 0:55
  is the core of our deep dive today. We're exploring
• 0:58
  a concept the authors call noise in human judgment,
• 1:02
  drawing our insights from the Amazon page for
• 1:04
  the book Noise, a flaw in human judgment, by
• 1:07
  Daniel Kahneman, Olivier Saibony, and Cassara
• 1:10
  Sunstein. A significant trio of authors there.
• 1:12
  Indeed. Our source material for this exploration
• 1:15
  is the book's product description, and importantly,
• 1:18
  the customer reviews shared right there on the
• 1:20
  page. OK. So our mission today is to really get
• 1:23
  to grips with what noise is, how it differs from
• 1:26
  the more familiar idea of bias. Which is key,
• 1:28
  I think. Definitely. And understand where this
• 1:31
  unwanted variability shows up, particularly in
• 1:34
  professional context, and maybe most importantly,
• 1:37
  why recognizing it is crucial for making better,
• 1:40
  fairer decisions. I'm here to help us navigate
• 1:42
  these ideas straight from what people are saying
• 1:44
  in those sources. Great. Let's start them by
• 1:46
  unpacking this fundamental concept. The book
• 1:49
  description boldly calls noise the last great
• 1:52
  fault in our collective decision making. That's
• 1:55
  quite a claim. It is. So how do our sources actually
• 1:58
  define noise? And what's that crucial distinction
• 2:01
  between noise and bias that one reviewer found
• 2:03
  particularly fascinating? Well, the description
• 2:06
  gives us the core definition. Noise is essentially
• 2:09
  unwanted variability in judgments. Think of it
• 2:12
  like scatter on a target. If you ask several
• 2:15
  people to estimate a value, or maybe if the same
• 2:18
  person estimates it multiple times and the answers
• 2:20
  are all over the map, that spread That inconsistency,
• 2:24
  that's noise. It's deviation around the true
• 2:27
  value, or maybe around the desired consistent
• 2:29
  outcome. OK, so if bias is like always aiming
• 2:33
  off -center but in the same direction, the noise
• 2:35
  is more like the shot scattering randomly around
• 2:38
  where you were aiming, maybe even scattered around
• 2:40
  that off -center point if bias is also present.
• 2:43
  Precisely. That's a great way to put it. Bias
• 2:45
  is a systematic error, a consistent pull or push
• 2:48
  away from accuracy. Noise is random air fluctuations
• 2:51
  that lead to different outcomes, even when the
• 2:53
  inputs are essentially the same. And as one customer
• 2:56
  review highlights, the differentiation around
• 2:58
  noise from bias was fascinating. They clearly
• 3:01
  found that distinction quite insightful. It clarifies
• 3:04
  things. It does. And it's not just that they're
• 3:06
  distinct, they can interact. The sources suggest
• 3:09
  that inconsistency and bias can actually create
• 3:11
  noise. How does that work? Well, imagine if different
• 3:14
  judges have different levels of inherent leniency.
• 3:17
  That's a form of bias, right? But the difference
• 3:21
  in their levels of bias between Judge A and Judge
• 3:23
  B introduces variability in sentencing outcomes
• 3:27
  for similar cases. And that variability is noise.
• 3:31
  So while bias might cause a consistent lean overall,
• 3:35
  noise causes this unpredictable spread. That's
• 3:38
  a critical point that even bias itself, if it's
• 3:40
  inconsistent across people or even time, contributes
• 3:43
  to this problem of noise. So where do the sources
• 3:46
  tell us this phenomenon is really prevalent?
• 3:47
  Where does it pop up? Well, sources point to
• 3:49
  several areas where subjective judgments are
• 3:51
  made, and noise is likely, well, rampant, actually.
• 3:55
  The book description and reviews specifically
• 3:57
  mentioned judicial sentencing. It's a particularly
• 3:59
  high stakes example, obviously. Hugely. One reviewer
• 4:03
  mentions being struck by the idea that judges,
• 4:05
  despite all their training and experience, can
• 4:08
  sometimes let arbitrary events affect how they
• 4:12
  sentence another human being. Arbitrary events?
• 4:15
  Like, what? Well, the reviewer implies things
• 4:17
  maybe like the outcome of a local sports game
• 4:19
  or even the weather, subtle irrelevant factors
• 4:22
  potentially contributing to variability in a
• 4:25
  decision that profoundly impacts someone's life.
• 4:27
  Wow. That example of judicial sentencing really
• 4:31
  brings the potential for unfairness into sharp
• 4:33
  focus. It's not just about minor discrepancies,
• 4:36
  is it? It's about decisions with massive consequences
• 4:38
  maybe being swayed by, well, noise. Exactly.
• 4:42
  And the sources broaden this out considerably.
• 4:44
  They state that noise appears in how companies
• 4:46
  recruit employees. Ah, hiring. I can see that.
• 4:50
  Yes. Think of different interviewers evaluating
• 4:52
  the same candidate file. perhaps quite differently.
• 4:55
  It's present in assessing insurance claims. Two
• 4:58
  identical claims could be handled with significant
• 5:01
  variability, depending on who reviews them on
• 5:03
  what day. Right. The source description kind
• 5:05
  of sums it up by saying noise effects. Indeed,
• 5:08
  anything where a judgment has to be made, which
• 5:11
  implies a vast swat of professional activities,
• 5:14
  really. So loan applications, medical diagnoses,
• 5:18
  performance reviews, project feasibility assessments,
• 5:21
  you name it. So it's definitely not confined
• 5:24
  to a few niche areas. It seems pervasive across
• 5:27
  any field requiring human evaluation or prediction.
• 5:30
  And the scale of this problem, as noted in the
• 5:33
  reviews, isn't just about one person versus another,
• 5:36
  is it? No, absolutely not. The sources highlight
• 5:39
  the multi -layered nature of this variability.
• 5:41
  One review explicitly points out that this noise
• 5:44
  exists between different individuals. So you
• 5:46
  and your colleague might judge the same report
• 5:48
  differently. It exists between different organizations.
• 5:51
  Company A might evaluate credit risk using slightly
• 5:54
  different implicit criteria than company B, leading
• 5:58
  to varied outcomes for similar applicants. Right.
• 6:01
  Different company cultures, perhaps. Perhaps.
• 6:04
  And crucially, it exists even within the same
• 6:07
  individual at different times. You might make
• 6:10
  a slightly different decision today than you
• 6:12
  would tomorrow, given the exact same circumstances
• 6:15
  influenced by transient factors you're probably
• 6:17
  unaware of. Your mood? maybe, or what you just
• 6:20
  read? That last part is perhaps the most unsettling,
• 6:24
  that even we are inconsistent with ourselves.
• 6:27
  And this leads to a crucial point highlighted
• 6:29
  in the sources, the complete unawareness of this
• 6:32
  variability. Yes, that lack of awareness is profound.
• 6:35
  The description and reviews both stress that
• 6:38
  the individuals and organizations making these
• 6:40
  noisy judgments are often completely unaware
• 6:43
  that such variability exists at all. We just
• 6:45
  don't see it. We tend to believe our judgments
• 6:46
  are consistent, based purely on the merits of
• 6:48
  the case. We're very good at constructing narratives,
• 6:51
  seeing patterns and explanations, even when the
• 6:54
  reality might just be random scatter. This silent
• 6:57
  nature of noise makes it incredibly insidious
• 7:00
  because, well, if you don't know a problem exists,
• 7:04
  you certainly can't address it. And if judgments
• 7:07
  are subject to such high degrees of unnoticed,
• 7:09
  unwanted variability, what does that mean for
• 7:11
  the outcomes? What's the impact? The sources
• 7:14
  indicate the impact is pretty significant because
• 7:17
  noise leads to inconsistencies. It results in
• 7:20
  judgments that are less accurate and, frankly,
• 7:23
  potentially unfair. Unfair. Reviews mention the
• 7:26
  need for more fairness and accuracy, implying
• 7:29
  that the presence of noise is undermining these
• 7:31
  very goals. In judicial sentencing, it could
• 7:34
  mean disparate outcomes for similar crimes. In
• 7:37
  hiring, it might mean overlooking great candidates
• 7:40
  or hiring less suitable ones just inconsistently.
• 7:43
  In insurance, it could lead to inequitable treatment
• 7:45
  of policyholders. Essentially, the consequence
• 7:48
  is that the judgment isn't truly reflecting the
• 7:50
  underlying reality or the agreed criteria. It's
• 7:53
  partly a product of random noise. This really
• 7:56
  changes how you might think about decision -making,
• 7:58
  doesn't it? It's not just about trying to avoid
• 8:01
  conscious or unconscious bias. It's also about
• 8:03
  identifying and reducing this hidden variability.
• 8:07
  Exactly. The good news is the book isn't just
• 8:09
  diagnosing the problem. It sounds like it promises
• 8:12
  solutions. Exactly. And the authors are serious
• 8:15
  figures in the field, of course. Daniel Kahneman,
• 8:19
  a Nobel Prize -winning economist, renowned for
• 8:21
  his work on judgment and decision -making, particularly
• 8:24
  thinking fast and slow. A landmark book. Absolutely.
• 8:28
  Olivier Saboni and Cassar Sunstein are also highly
• 8:31
  respected experts in their own right. The reviews
• 8:34
  mention their previous works, like Nudge, as
• 8:36
  being sort of companion pieces, perhaps. Right.
• 8:38
  Well, one reviewer importantly clarifies that
• 8:41
  Noise don't cover the same ground as thinking
• 8:44
  fast and slow. So it really is tackling a distinct
• 8:46
  area of human judgment errors. That's good to
• 8:49
  know. It's not just a repackaging of their prior
• 8:51
  work. It's something new. So what do the sources
• 8:53
  suggest the book actually offers in terms of
• 8:56
  addressing noise, practical stuff? Yes, it seems
• 8:59
  so. The description states the book shows us
• 9:02
  what we can do to make better judgments. And
• 9:05
  customer reviews really reinforce this, describing
• 9:07
  it as a practical guide to reducing noise. So
• 9:11
  the intention seems to be providing actionable
• 9:13
  methods, not just theory. And while the sources
• 9:16
  obviously don't give us the full blueprint, they
• 9:18
  do hint at the nature of these solutions, don't
• 9:21
  they? They do, yes. The reviews suggest approaches
• 9:24
  that involve introducing more structure into
• 9:27
  decision -making processes. One review talks
• 9:29
  about promoting a balance between rigid bureaucracy
• 9:32
  rules and intuition. Which is interesting. Yeah,
• 9:35
  that suggests it isn't necessarily about eliminating
• 9:37
  human judgment entirely or becoming purely robotic.
• 9:41
  Exactly. It seems more about finding ways to
• 9:43
  channel intuition within a framework that minimizes
• 9:46
  that unwanted variability, kind of guiding judgment
• 9:49
  rather than replacing it. So maybe not strict
• 9:52
  rules for every single tiny aspect of a case,
• 9:54
  but perhaps a more standardized process for evaluating
• 9:57
  the different components. Precisely. That seems
• 10:00
  to be the direction. Another review states quite
• 10:02
  clearly that structured decisions versus random
• 10:05
  assessments are better all round. Better all
• 10:08
  round. Yeah. It reinforces the idea that implementing
• 10:11
  defined steps criteria, maybe using checklists
• 10:14
  or scoring systems frameworks for making judgments,
• 10:17
  yields more consistent and reliable outcomes
• 10:19
  than relying purely on unstructured intuitive
• 10:22
  assessments, which are highly susceptible to
• 10:24
  noise. The book presumably delves into how to
• 10:28
  design and implement these structures effectively.
• 10:30
  Okay, let's bring this back directly to you,
• 10:32
  our listener. As a mid -senior professional,
• 10:34
  you are constantly making judgments and decisions.
• 10:37
  It's a core part of the job. Where might this
• 10:39
  concept of noise be lurking in your professional
• 10:43
  world? Yeah, think about the areas where your
• 10:45
  role involves subjective evaluation. Are you
• 10:47
  involved in reviewing performance appraisals?
• 10:49
  Common one. Are you assessing project proposals
• 10:52
  or budgets? evaluating vendor pitches, trying
• 10:55
  to predict market trends or sales figures. Lots
• 10:58
  of areas. Anytime you have to assess something
• 11:00
  based on information and arrive at a conclusion,
• 11:03
  your judgment is potentially subject to noise.
• 11:06
  Just ask yourself, would you and a trusted colleague
• 11:10
  grade the same piece of work identically? Or
• 11:13
  even, would you yourself arrive at precisely
• 11:15
  the same sales forecast if you did the analysis
• 11:18
  again from scratch next week? Probably not identically.
• 11:21
  Probably not. And that difference? Yeah. That's
• 11:25
  noise. And the sources highlight this powerful
• 11:28
  aha moment that seems to come with recognizing
• 11:30
  noise. One customer review really captures it,
• 11:33
  I thought. They said, everything is really so
• 11:35
  noisy, yet we hardly notice. You will be more
• 11:37
  attentive to what's being done to you and what
• 11:39
  you equally do to others unknowingly. That reviewer
• 11:42
  perfectly articulates the shift in perspective
• 11:44
  the book seems to offer, doesn't it? You move
• 11:46
  from assuming judgments are based purely on merit
• 11:48
  or clear intention to seeing the unseen forces
• 11:51
  of variability at play. You start noticing how
• 11:54
  your own judgments might fluctuate, perhaps based
• 11:57
  on mood or fatigue, and also how the judgments
• 11:59
  you receive from others Feedback evaluations
• 12:02
  could be influenced by factors unrelated to your
• 12:05
  actual performance or the quality of your work.
• 12:08
  It makes you more critical, maybe. Both of your
• 12:10
  own cognitive processes and the systems you operate
• 12:13
  within. And this aha moment, it's more than just
• 12:17
  intellectual curiosity, isn't it? It has real
• 12:19
  implications for fairness and effectiveness in
• 12:21
  the workplace. And for those in leadership positions,
• 12:24
  it sounds like this understanding is not just
• 12:26
  useful, but potentially essential. Absolutely.
• 12:30
  Multiple reviews stress this. They call the book
• 12:32
  essential reading for professionals and a must
• 12:35
  read for any leader. Why so critical for leaders?
• 12:37
  Well, because leaders are often responsible for
• 12:40
  designing and overseeing processes, ensuring
• 12:42
  consistent standards are applied, and making
• 12:44
  reliable decisions that affect their teams and
• 12:47
  the organization's overall direction. If their
• 12:50
  own judgments, or the judgments within their
• 12:52
  teams, are full of unnoticed noise, well, it
• 12:56
  undermines trust, it can lead to frustration
• 12:58
  among staff, and ultimately results in suboptimal
• 13:01
  outcomes. Right. Addressing noise is presented
• 13:04
  in the reviews almost as a guide to more fairness
• 13:07
  and accuracy. And those are surely central goals
• 13:10
  for any effective leader. This exploration of
• 13:14
  noise, this hidden, unwanted variability in human
• 13:16
  judgment, truly underscores how much we might
• 13:18
  be leaving to chance or random factors in areas
• 13:21
  we assume are governed by logic and consistent
• 13:23
  evaluation. It's a subtle but powerful challenge
• 13:26
  to our confidence in purely subjective decision
• 13:29
  -making. It really is. It requires a certain
• 13:31
  humility, I think, about the reliability of our
• 13:33
  own minds and a critical look at the systems
• 13:34
  we use to make important calls. Are they designed
• 13:37
  to minimize this scatter? So here's a final thought
• 13:39
  to leave you with, building on everything we've
• 13:41
  discussed today from the sources. If individual
• 13:44
  judgments are susceptible to noise, how much
• 13:47
  is that variability potentially being compounded
• 13:50
  within complex organizational processes and systems?
• 13:53
  Interesting question. Think about a multi -stage
• 13:56
  approval process maybe, or a yearly performance
• 13:59
  review cycle that involves input from multiple
• 14:01
  people across different times. If every step
• 14:04
  has some inherent noise, how much silent noise
• 14:08
  might be accumulating and interacting to influence
• 14:10
  the final outcome. Accumulative effect. Exactly.
• 14:13
  And if this is happening largely unnoticed, as
• 14:16
  the sources suggest, how would you even begin
• 14:18
  to measure or mitigate that systemic noise within
• 14:21
  your own team or department? It feels like a
• 14:23
  significant challenge. It is. But perhaps, as
• 14:26
  you say, recognizing it, acknowledging it could
• 14:28
  exist, is the crucial first step. A fascinating
• 14:31
  question to ponder as you look at the decisions
• 14:33
  and processes around you. Indeed. Well, that
• 14:36
  concludes our deep dive into the concept of noise,
• 14:40
  drawing insights from the Amazon page for the
• 14:42
  book Noise. We've uncovered what noise is, how
• 14:45
  it differs from bias, where the sources say it
• 14:47
  typically appears, and why understanding it is
• 14:50
  so vital for professionals seeking greater fairness
• 14:52
  and accuracy and judgment. Hopefully it's given
• 14:55
  you a fresh perspective. We certainly hope it
• 14:57
  has, and perhaps prompted you to start listening
• 14:59
  for the silent hum of noise in your own professional
• 15:02
  environment.