A07: Advertising Failure

The story of Doctor Kim A. Adcock’s approach to solving a problem in the radiology department at Kaiser Permanente in Denver reads like script background for one of those “procedural” TV shows such as CSI. We know who died (far too many) and we know who did it (doctors, sort of) but we’re not sure how to handle the evidence to make sure nobody gets killed next time.

Procedures that seemed reasonable to Kaiser in 1995 because they “had always been done that way,” turned out to be entirely unreasonable, with deadly consequences. And a solution that seemed impossible because of fear, turned out to be the best and most logical of solutions, and has saved countless lives.

I read this story when it first appeared in 2002 and have cherished its insights ever since. Now ten years later, I had to go find it to share it with this class. Since reading it, and other stories like it, I cannot look at statistics of any kind without wondering what they really mean. If the crime rate goes down, does that mean there is less crime? Maybe not. It might mean fewer people are reporting crimes.

For example, in New Orleans since Katrina, distrust of the police runs so high most citizens in some neighborhoods would rather suffer crime in silence than involve the police. The very first thought that came to my mind listening to that story was, “I’ll bet the crime rate has gone down in those neighborhoods” and not because there’s less crime. The mayor though, and the chief of police, can trumpet those statistics as if they’re doing a better job in those same neighborhoods.

But I digress. Your assignment for TUE FEB 14 is to read “Mammogram Team Learns From Its Errors,” and contribute a comment to an ongoing discussion of the counterintuitivities (I’m going to keep using this word until the rest of the world adopts it) it contains.

I don’t need an organized essay from any one of you, but I do need a contribution from everyone in the form of a comment to this post you’re reading now. I’m setting up the assignment this way to encourage you to read the entire comment string and reply to your classmates, not repeat what they’ve already said.

You may make your own original observations, of course, when you see an opportunity to point out something new. Or you may reply directly to a classmate’s observation with rebuttal or clarification. And you may contribute as well or as often as you like for a better grade. The minimum for a passing grade is one substantial comment.

ASSIGNMENT SPECIFICS

  • Follow the link from the sidebar or this link here to the article.
  • Read the article looking for evidence of counterintuitivity. At what points in the story do people think or act contrary to what their intuition told them? How do you explain their feelings or thoughts? At what point in human culture does common knowledge change so that we develop different intuitions?
  • Read the comments to this post that precede yours.
  • If you have something new to say, add your new insight to the conversation in a comment of your own.
  • If your insight is not unique, respond instead to someone else’s comment with refutation or additional support. Never repeat. Never merely agree.

GRADE DETAILS

  • DUE THU FEB 28 before class.
  • Customary late penalties. (0-24 hours 10%) (24-48 hours 20%) (48+ hours, 0 grade)
  • Quizzes and Exercises category (10%)
Advertisements

About davidbdale

Inventor of and sole practitioner of 299-word Very Short Novels. www.davidbdale.wordpress.com
This entry was posted in A07: Advertising Failure, David Hodges, Professor Post. Bookmark the permalink.

44 Responses to A07: Advertising Failure

  1. lebano55 says:

    My initial reaction to this article was that I was only somewhat satisfied with the results of Dr. Adcock’s efforts to improve the accuracy of mammograms. This segment in particular was a specific cause of concern for me: “Here in Denver, Dr. Adcock has winnowed his team down to a few specialists. By contrast, most of the 20,000 doctors in the United States reading breast X-rays are generalists with limited training and practice in mammography. Many lack the skill needed to do so effectively, yet neither they nor their patients have the tools to find out who is good and who is not.” That’s great news for the Denver area, but what does that leave the rest of the country, or the world? Breast cancer is documented in my family, so any significant margin for error leaves me with the opinion that there’s a need for a higher standard in this field.

    • davidbdale says:

      I couldn’t agree with you more, Steve. But maybe there’s cause for hope. The story appeared in 2002. Since then, if the program has shown positive results, maybe it’s been widely adopted. It’s certainly worth a bit of research to find out. (Might make a good paper too.)

    • smithk53 says:

      I agree with you. This past December my grandmother found out that she has had breast cancer since 2010 and since her first mammogram in 2010 she had one in 2011 and then with her third mammogram the doctors finally saw the tumor in 2012. Her and my grandfather live in Orbisonia, PA in the absolute middle of nowhere where the closest hospital is about an hour away. This is such a small local hospital and two of my cousins work there as nurses. This hospital is deffinitly not equip with specialty doctors and certainly not equip with the skills or manpower to keep statistics of thier work.

      • davidbdale says:

        What’s the remedy for the women of Orbisonia, Kirsten? I have other questions as well, if you’re willing to answer. How is it now clear she had cancer in 2010? And why was she having mammograms every year? That can’t be customary for everyone.

        • smithk53 says:

          Since my grandparents come from such a secluded area they live simpler lives than most and like most older people today they listen to what the doctors tell them all of the time. So when my grandmother was told she had to come back every year for a mammogram she did. I think that the reason why she kept getting yearly mammograms was because she had breast cancer in the past and that was her doctor’s way of making sure it didn’t come back. The doctors just didn’t see the tumor in 2010 when they looked at the films. The doctors looked at all of her films and saw that the tumor was present since 2010.

          • davidbdale says:

            That’s a perfect explanation. Thanks. Maybe it also means the chance to compare that whole series of films will make the team at Orbisonia General better able to spot early-developing tumors in the future.

  2. adkins70 says:

    “Even Dr. Adcock is wary of having the government police doctors’ performance. ‘I could see that being counterproductive,’ he says.”

    It would seem intuitive to enforce stricter federal regulations on doctors for mammography screening accuracy because it would force doctors to improve their statistics. But, Adcock is valid in saying the measure could prove counterproductive, with the possibilities of: giving biopsies to far more women than necessary and a decrease in doctors willing to perform mammograms. What would make a great step forward is the tri-annual tests given to Adcock’s team being applied nationally. That would ensure that the hospitals and doctors were aware of their accuracy without excess pressure from the federal government, allowing issues to be handled in-house.

    • davidbdale says:

      What you’re suggesting is so logical and inherently good it’s hard to imagine it would ever be achieved, Brent, but yes, of course, the best possible procedure would be for radiology departments to be so exemplary that they didn’t require oversight at all. Imagine the cost savings alone of not having to constantly monitor one’s own effectiveness to satisfy an outside agency with a national agenda.

  3. briannewaters3 says:

    I think the part that I questioned most was how long it took for even just one hospital to be creating statistics for mammography reading accuracy. When I watch a Phillies game, they can show me Ryan Howard’s batting average against right-handed pitchers during Saturday night games in May. Whenever I see these ridiculous and unnecessary stats I am forced to believe that statisticians are also working hard on important stats like maybe… how many tumors are being accurately located through mammograms. You would think statistics would have been found on this medical topic because it is so black and white; did they pick up on the tumor or not. I understand why statistics are not created on all different procedures because there are so many factors that play into each operation such as patients’ medical history. While tumors do come in all shapes and sizes, the “experts” who are reading them should be able to identify all types, especially if that is their one job. I definitely commend Kaiser and Dr. Adcock for fixing a problem that most would not have even have noticed and for also confidently sharing their findings with the world. I believe that this practice of creating statistics for accurate mammogram readings should be spread nationwide. Doctors are supposed to save lives so take the PR risk and maybe save some lives in the process.

    • davidbdale says:

      Thanks, Brianne. I share your astonishment that nobody was checking the accuracy of readings before Adcock’s plan. Your point about the binary nature of the essential statistic is very well made. At bottom, only two situations are in play: there is a tumor or there isn’t, and two variables: the reader got it right or got it wrong. When lives hang in the balance, it’s not unreasonable to assume somebody was tracking and counting those occurrences.

    • smithk53 says:

      Brianne I agree with you on the fact that, like in major league sports, doctors and hospitals should have stats of their own, however I don’t think that mammogram readings are as black and white as you suggest they are. It’s scary to think that a doctor could miss something as serious as cancer when, after all, they were trained to recognize cancer spots on mammograms, but in the same respect in any kind of work that deals with humans there is always human error no matter how good a person is at thier job. I would assume the reason why doctors dont want thier stats calculated is because that would show them and others how good or bad they are at their job and after eight plus years of rigorous schooling who would want to be told they aren’t good at what they do. What a waste of time and money that would be! But that is no excuse for incompetence and that is why I also commend Kaiser and Dr. Adcock for raising the standard of medicine by making at the very least radiologists accountable.

      • davidbdale says:

        You’re quite right that tumors aren’t easy to spot (and would be invisible to untrained eyes). I think we’re all sympathetic to the difficulty of getting the results right, and to the importance of doing so. It’s true that there either is one or there isn’t one, but that by no means it’s easy to tell the difference (particularly at the early stages of development).

    • kovnat77 says:

      I thoroughly agree with the need to redirect the focus of statisticians everywhere onto slightly more important things than just an athletes batting average, or superfluous “fun facts” used for societies enjoyment, or distraction. when you have such brain power, or potential to use a skill for good, it seems that most often times we find ourselves manipulating those people into using this skill for entertainment or otherwise. An example of this could be those highly precocious children that are absolutely brilliant, and instead of parents encouraging them to focus their brain powers on something more productive and conducive to their intelligence, we see them on television on shows like ” are you smarter than a fifth grader” and other things like that trying to win money that their parents will most likely either steal from them, or funnel into their own accounts anyway.
      With that realization in mind, does this mean that getting those intelligent minds together, and focused on furthering man kind in a positive way is impossible? To be able to shine light on those statistics that could help to improve the mammogram analyzing field would be a truly forward thinking move.

      • davidbdale says:

        I’m not sure there’s a cadre of freelance statisticians roving the landscape in search of contracts, Sammy, but if there is such a herd, they would surely be better used improving health care than working on game shows. Hear, hear!

    • primav01 says:

      Brianne, when I read your comparison between this situation and the fact that statisticians have statistics so readily available to them and so easy to get, I really connected this in the same way. When I read the article I was thinking the same thing but in terms of watching the New York Islanders hockey team. The announcers are constantly fed with statistics that are important, and even many that are irrelevant and really not needed, but they are available to the announcers so easily. Those who are supposed to be identifying the tumors, who, like you said are experts, they should be getting statistics and information just as easy or even faster than a common sports announcer does. I believe this to be kind of upsetting that such important information and statistics on things like tumors on a human beings body do not get the sort of urgency that a hockey, football, basketball, or baseball announcer has right at their fingertips.

  4. justinbaker2007 says:

    “But today, Dr. Adcock’s team is missing one-third fewer cancers and has achieved what experts say is nearly as high a level of accuracy as mammography can offer.”

    While there are always two sides to everything, I agree with what Dr. Adcock did. At the end of the day, what he did helped save more lives. Everyone keeps “score” no matter what they are doing, and I am shocked that Doctors were not keeping score before Dr. Adcock did. Even when I was a kid playing basketball with my friend, he and I would agree to not keep score, but in the backs of our minds we would know the score. Doctors had to have known how well they were doing. Dr. Adcock was the only one to publicize the score and cut off the Doctors that were not doing well enough. While Doctors are not perfect, we go to them to help save our lives and to hear a statistic such as “…one doctor had missed 10 cancers in the space of 18 months”, it is scary to think about how many Doctors are really not at the standard we, as patients, want them to be at.

    • davidbdale says:

      Baker, I completely agree that it’s stunning to hear that statistics were not public knowledge before Adcock. I want to ask you, though, how the radiologists before he started to keep records would “have known how well they were doing,” as you say? Did they get feedback when cancers they had missed showed up on films in other hospitals a few years later? I don’t think they did. Did they get feedback when tumors they spotted turned out to be actual tumors? Maybe they did, but this only served to make them feel like experts because they had correctly identified tumors. Not facing their misses and learning what they looked like was a critical failing of the earlier system.

      • justinbaker2007 says:

        That’s a very good point. Doctors should be notified if they missed something if it showed up later at a different hospital. It not only shows which Doctors should not be Doctors, but it also helps patients pick the right Doctors and Hospitals to go to.

  5. kaileewhiting says:

    “Don’t put too much faith in a doctor being board certified in radiology. Many doctors passed before the late 1980’s, when mammography was added to the exam.”

    I read this article yesterday afternoon before heading out to go to my next class, hoping I could reflect on what was said later that day. And even 24 hours later, this quote stuck out to me like a sore thumb. There are doctors out there, practicing radiology and reading mammograms, who have never been tested on reading a mammogram. Now I don’t think that makes a whole lot of sense, or makes someone like myself, with a long family history of breast cancer, or anyone else in my boat feel the least bit comfortable. It is incredibly scary to believe there are doctors out there who may not know what they are doing. If the world was perfect, I say that we retest all of the radiologists in the field. But that would probably cost millions of dollars to create such an initiative. Or maybe, this is a complete shot in the dark, but with all the fundraising and charity work there is revolving around breast cancer, how about they redirect some of the funding to the education and training of radiologists? Just an idea.

    • davidbdale says:

      Your comment makes me laugh and cry at the same time, Kailee. Most likely I’m just too emotional, but it does seem both ludicrous and horrifying that films are being read by clinicians not certified to read them. The most frightening aspect of trusting our well-being to well-meaning human beings is that they always seem to answer to bureaucracies with very suspect agendas. I hate to say categorically that capitalism sucks, but when profit is your primary motive, everybody looks like money.

  6. Rickc1030 says:

    Dr. Adcock’s method of comparing scores to find the weakness in his department of doctors is genius. It is unfortunate for those doctor’s who lost their job, but then again, one of those doctors had 10 chances in the course of 18 months. That many mistakes are unacceptable in a job where people’s lives are on the line. How can you allow someone to work in the mammogram department when some radiologist have never even been tested on reading them?! That’s like giving a wrecking ball to a construction worker who’s been trained only with a forklift! The article states that now that Adcock has purged his mammogram department, they are missing one third fewer cancers. There needs to be a more central re-focus on how we’re teaching the radiologist of the future so that steps like Adcock’s score board method wouldn’t needed to be taken again.

    ”Every mammography program in the country should be doing something like this,” says Dr. Robert A. Smith, the American Cancer Society’s screening chief.

    • davidbdale says:

      As I wrote to Steve Lebano yesterday, “I couldn’t agree with you more. But maybe there’s cause for hope. The story appeared in 2002. Since then, if the program has shown positive results, maybe it’s been widely adopted. It’s certainly worth a bit of research to find out. (Might make a good paper too.)” I wonder if anybody will take me up on this idea. I’d love to have someone else do the research on how much the practice has improved since 2002.

  7. anthonymatias97 says:

    The system that Dr. Adcock has come up with is brilliant. People use statistics for just about anything from how well kids do in school to the field goal percentages of players in the NBA. There can be statistics on every little thing so when I came upon,

    “Mammography, he felt, was well-suited to a statistical approach. Unlike, say, hip surgery, with its many gradations of success — is it the ability to walk, or run, with or without a limp? — the equation in mammography is fairly straightforward. The radiologist concludes that a woman appears to have cancer or not,”

    while reading this article shocked me. This man who was smart enough to apply statistics to something as complex as mammography, couldn’t grasp the fact that statistical evidence could be used in another medical field such as hip surgery. I don’t know if he lost sight of applying it to other medical fields or if he just had tunnel vision on his own field because his system is doing wonders for mammography. But why couldn’t it be done forthings like the PSA test for mens prostate cancer? This system is able to tell the patients of the world which doctors are getting the right results and which doctors can’t provide them with the accurate readings. It would be able to potentially save many lives and detect cancer earlier so patients can get the treatment right away.

    I thought this was a counter-intuitive statement because he managed to incorporate stats into his field and just shot down the idea of adding statistical evidence to another field. Maybe he said this to make the system for mammography seem unique. I can’t speak for him in that regard but if systems like this were applied to other medical fields the world of medicine might be a safer place because we would be able to located the doctors who would be able to make the most accurate readings.

    • davidbdale says:

      I see your point, Tony. Is it possible you’re being a little unfair to Dr. Adcock? First of all, we have a paraphrase from the author, not Adcock’s own comments, about hip surgery for example.

      And even if he did make the comparison, it was something he thought even before he started to develop his own technique, just an impression that something with a binary “tumor/no tumor” pair would be easier to quantify than data that required judgements on a scale like “slightly improved/moderately improved/improved/substantially improved/perfectly healed.”

      I don’t mean to overprotect him any more than you mean to over-criticize him, I’m sure. It’s perfectly reasonable to ask: why the blind spot here?

  8. tbrody92 says:

    It’s truly astonishing how Dr. Adcock’s system wasn’t implemented before 2002. While the system causes many doctors to lose their jobs, it certainly is more effective at locating cancerous masses and saving lives: the ultimate goal of medicine and health technology, which, unfortunately, I believe many doctors will ignore, and here’s why:

    I did a little research into how necessary this program would be today, and the results may be somewhat shocking. Mammograms can now be shot in 3-D as of 2012, which may (even now) be causing a further de-skilling of the doctors who practice radiology despite having little to no knowledge in that field. The United States has put an emphasis on quantity over quality since the turn of the century and Industrialization. The current economic state of the US has made the emphasis even more bold. If the doctor looks at the mammogram film and sees no cancer, he/she will likely make their judgment then and there; there are more mammograms to be done, which ultimately means better business. Though I do believe that doctors want to cure their patients (after all, it’s their job), the time and effort that go into Dr. Adcock’s system does not blend with the industrialized society we live in today, and therefore, I doubt that the doctors taking the mammograms are always fully trained to do such a thing.

  9. kmbuttari says:

    Dr. Adcock’s method is genius. He uses what Dr. Ken Heilbrunn has described as the “shame method”. Instead of telling his doctors “Oh don’t worry, everyone makes mistakes” he essentially says “You screwed up. This should be better. Fix it” which works significantly better. It’s as if doctors had just been accepting that they wouldn’t be too accurate even if there was a large amount of room for improvement. Dr. Adcock was able to very easily and effectively show them that they could and should be doing significantly better, and it worked. Hopefully this method is being used in more hospitals to further increase the over all effectiveness of a mammogram.

    • davidbdale says:

      That may be unnecessarily tough on the doctors, Kevin. If there’s no procedure in place to let radiologists know they’ve missed a tumor or ordered a biopsy for one that didn’t exist, they’re victims of the system far more than they are callous check-collectors “accepting that they wouldn’t be too accurate.” I don’t know for sure which is true; do you?

  10. kovnat77 says:

    One question that this article left with me, was that if you read a massive amount of mammograms, lets say 2,000, does this make you better than a person who has only met the states requirements of 480, but they spent an extraordinarily long time making sure each image was read correctly?

    “Over the next few years, several more radiologists were fired or resigned in the face of concerns about their interpretive skill. Then Dr. Adcock spotted an even trickier problem. Nearly half the original 20 radiologists were reading far fewer mammograms than the others. They met the federal minimum of 480 a year, but with the others reading as many as 14,000, Dr. Adcock agreed with experts who say the government minimum is far too low.”

    In this respect, Dr. Adcock then fired the ones who had not read enough scans to give them a precise set of information illustrating their judgement levels. So they were penalized for not having read enough, regardless of whether or not they were doing accurate readings.

    It has been said that ” if you spend 10,000 hours doing anything, you can become an expert.” It makes me wonder if Adcock believes highly in that statement and whether or not that is something that he has based his strict regimen at the Denver hospital around.

    • davidbdale says:

      I love your question, Sammy, but I’m certainly not qualified to answer it. I don’t think you can assume that those who read only 480 “spent more time” on their readings to insure accuracy. In fact, I’m not sure you can assume they spent more time at all. Maybe they merely tried to work part time or split their duties among radiology readings and other job functions. Anyway, the explanation I remember is that 480 isn’t a large enough sample to get reliable statistics that the radiologists are accurate. If they can’t be verified good, his reasoning went, for the safety of the patients, they must be assumed bad. Maybe the better choice would have been to divide the work so that they read enough to be verified accurate?

      • kovnat77 says:

        I completely agree, he should have given each “unqualified” reader the opportunity to garner enough information to show their statistical ratings, in that way, he possibly could have a stronger team than he was aware of.

  11. clarkn92 says:

    I found this article very interesting, yet heart breaking. Dr. Adcock’s method for eliminating mistakes made by doctors may be harsh, but is rather effective. I hope that Dr. Adcock’s method is something that other medical professors would want to follow to eliminate said mistakes. It is a shame to see how many mistakes his staff has made in the past. Cancer is a serious issue, and can lead to death if not treated. Mammograms are something that women take very seriously in order to ensure they do not suffer from Breast Cancer. Although Breast Cancer has the highest recovery rate, it still can lead to death if it is not caught at an early stage. It is hard for me to know that these women who are left untreated could die due to misdiagnosis. The bottom line is that these doctors should know that their mistakes affect the patients tremendously. One mistake could mean life or death.

    • davidbdale says:

      You sound extremely sympathetic, Nicole, and I’m sure you are to the patients. I suspect you are to the doctors too, who would love to do perfect work. Most likely they feel prouder now that they’ve reduced their mistakes. The job is not for sissies. The stakes are high. And caring professionals have to be willing to put their egos aside and suffer the scrutiny of quality control.

    • jpassalacqua says:

      The doctors who seem to be keeping their jobs seem to be doing the bare minimum of work required by the state. They have a higher statistic of cancers caught, but they are treating significantly less patients. If the doctor who did procedures on 5000 women had a lower statistic than the doctor who had dont only 500 procedures, we need to also look at how many cancers were found total. A doctor can go through hundreds of cases without finding a single patient with cancer. If none of the women are found to have cancer, that may be true. The more patients checked, the harder it is to get a 100% accuracy. More cases means more chances to lower your statistic. Doctors could be fired for no good reason. Maybe the only doctors should be the ones with a high statistic, as well as a high amount of patients.

      • davidbdale says:

        I’m having a hard time understanding what you mean by statistic, Joe. You don’t make clear whether you mean a larger number of cancers found per 100 (a higher percentage, true or false), for example; or a greater accuracy of cancers correctly discovered (true positives and true negatives). Also, you’re making mistakes of number and quantity. Fewer patients, not less patients, because you can count them. Also a high number of patients, not a high amount of patients, for the same reason. Beyond that, I think your logic is sound, but I can’t be sure because of the ambiguity of your statistic language.

  12. adamtwths says:

    This is an interesting controversy, but the patients health should come before all other matters. So the hospital has to do whatever has to be done to make sure that these women are safe and are getting adequate treatment. If Dr. Adcock’s system is working and helping women stay healthy then his system should be used. Although doctors are loosing jobs in the long run women’s lives are being saved, and that is the number 1 goal. Patients have the right to choose the best doctor. They shouldn’t be left in the dark when it comes to their health and well being. Adcock’s method gives doctors the incentive to work hard and strive to be the best at their job. The medical community is very competitive, but this is nesscary because people deserve to know that their health is in the most capable hands. Overall Adcock’s method should continue and should be used in all hospitals, it is very affective and saves lives.

    • davidbdale says:

      All very reasonable, Adam. Watch out for errors of simple grammar and punctuation, please. I’ll be tightening the requirements for passing grades in the second half of the semester.
      * the patients’ health (plural and possessive for the health of multiple patients)
      * losing jobs
      * very effective

  13. cdisarcina says:

    The doctors should take responsibility for their actions. If they fail to locate a tumor after reading a mammogram they need to own up to it. It’s understandable that doctors make mistakes, but when that mistake can endanger lives, it should not happen. They’ve obviously had the training and the experience to read mammograms, there is no excuse. Perhaps if they took more time with each patients files and do more than just a glance over the x-rays, the percentage error would be lower. I think that there should be no leniency, if a doctor makes a mistake, they’re fired. It’s a person’s life at stake, not just a misread x-ray.

    • davidbdale says:

      Damn, Chris. I’m glad you’re not my boss. People do die in hospitals, you know. Lucky for me, grammar rarely kills (although it did land this guy in jail).

      • billykluge says:

        Christopher’s approach of firing doctors for their failed mammogram readings is exactly what every hospital across the country needs but very few hospitals check their doctor’s accuracy. Doctors are humans and may make mistakes from time to time but when a doctor misses ten women with cancer in eighteen months that person is clearly not doing their job right. Luckily Dr. Adcock was able to fire this doctor due to his use of statistics to track doctors performances. Tracking doctors is happening in very few hospitals which means they may be missing countless cancers with no consequence. Just because someone has a doctorate degree does not mean they are always right.

  14. davidbdale says:

    Again, clearing 0s is the quickest way to boost a poor grade.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s