{"id":"measuring-clinical-competency-benchmarks-82956","slug":"measuring-clinical-competency-benchmarks-82956","title":"What Dental Hygiene Teaches Us About Benchmarks","description":"Assessment problems in AI evaluation mirror longstanding challenges in healthcare education. Dr. Ann McCann spent decades redesigning how dental hygiene schools verify real-world competence, discovering that most program directors relied on unvalidated tests. Rebuilding evaluation around observable performance tasks rather than multiple-choice knowledge offers a practical blueprint for benchmarking complex skills, while her later shift to plein air painting reveals an entirely different way of judging observable results against changing real-world conditions.","excerpt":"A 1990 experiment showed switching from letter grades to pass-fail changed student motivation without altering clinical errors.","pubDate":"2026-10-27T10:44:23.195Z","tags":["ai-evaluation","ai-alignment","public-health"],"heroImage":"https://files.myweirdprompts.com/covers/measuring-clinical-competency-benchmarks-82956.png","podcastAudioUrl":"https://episodes.myweirdprompts.com/audio/measuring-clinical-competency-benchmarks-82956.m4a","podcastDuration":"19:42","episodeNumber":6293,"transcript":"Corn: We spent most of the week arguing about whether a test score tells you anything. Why model benchmarks miss small workflow friction. Why coding benchmarks split across models. Why terminal benchmarks miss real desktop faults. Three days of the number saying one thing and the work saying another.\n\nHerman: And somewhere in the middle of the third one I realised there's a person who spent an entire career on that exact question, except about human beings. Dr. Ann McCann, in Dallas.\n\nCorn: Chosen purely on merit. I'd like that on the record. No other reason whatsoever.\n\nHerman: Nobody asked, Corn.\n\nCorn: Somebody was going to.\n\nHerman: So. She's a registered dental hygienist with a PhD, which already sets her apart from most people who write about assessment, because she's done the job being assessed. She became a professor and Director of Planning and Assessment at the dental college in Dallas, which her older papers call Baylor College of Dentistry and her current title calls Texas A and M. She directs three online courses in a master's program in education for health professionals, she was the chair of dental hygiene at the University of Detroit Mercy before that, and she directed two HRSA grants on assessment tools.\n\nCorn: Grants for assessment tools. Somebody got paid to build a better ruler.\n\nHerman: Two of them, and she was the assessment expert on a grant from the national dental research institute. Plus a chapter on women's oral health, something she describes as a book on assessment, and more than eighty other publications. We couldn't pin down where the PhD is from or what it was in, so I'll leave that alone.\n\nCorn: Not a dentist, to be clear.\n\nHerman: Not a dentist. People keep flattening that, and it matters, because her whole field is how you know that the people coming out of a school can actually do the work.\n\nCorn: Which is the benchmark problem with a lab coat on.\n\nHerman: Her earliest paper I found is from nineteen ninety, and it's a lovely small experiment. Nineteen first-semester hygiene students, randomly split between letter grades and pass-fail. The question was whether the kind of grade changes how well they do in the clinic.\n\nCorn: I'd guess letter grades win. Fear works.\n\nHerman: It didn't, though. The clinical performance, meaning scaling and polishing errors, came out the same. The statistics weren't even close to a difference. What changed was how the students felt. The pass-fail group found their grading less motivating.\n\nCorn: So the number changed the mood and left the hands alone.\n\nHerman: Which is a tidy little preview of this week. Change the scoring and you change how people feel and behave around the score, and you can't assume the underlying skill moved at all.\n\nCorn: Goodhart in a dental chair.\n\nHerman: Then five years later she and a co-author surveyed hygiene program directors across the country. A hundred and thirty-eight out of two hundred and twelve answered. They were dissatisfied with how their own students were assessed. Only about thirty-five percent said they'd validated their instruments, and the results were rarely used to help a struggling student catch up.\n\nCorn: Hold on. Two thirds of the programs weren't checking whether their tests measured anything.\n\nHerman: Roughly, yes. And these were the people certifying that someone was safe to work on patients.\n\nCorn: That's a bit rich coming from a week of us being snooty about model leaderboards.\n\nHerman: I did think about that.\n\nCorn: What did the school do about it? Or did everyone just write another survey?\n\nHerman: They rebuilt the curriculum around competencies. A nineteen ninety-eight paper is literally titled lessons learned from the competency-based curriculum at Baylor. And the following year she and a colleague published the hygiene version, with three domains, nine major competencies and fifty-four supporting ones, each judged by multiple methods.\n\nCorn: Fifty-four. Nobody sat down with a clean sheet of paper and wrote fifty-four things by accident.\n\nHerman: No, that's the point of it. You write down what a competent person actually does, in terms you can observe, and then you build the test from that list instead of the other way round. Brace yourself, Corn, because the exit exam is the part I love.\n\nCorn: You say that about every part.\n\nHerman: This one I mean. The senior exit exam had eleven performance tasks. Things like interpreting a research article, working through an ethical dilemma, designing a community intervention. And in the first pilots, in the mid-nineties, the first-attempt pass rates were low.\n\nCorn: Low as in the exam was broken, or low as in the students were?\n\nHerman: That's exactly the question, and the later numbers answer it. After they revised the exam and the teaching, between nineteen ninety-seven and two thousand, scores improved on nine of the eleven tasks. So some of it was students not yet being taught the thing, and some of it was a test that hadn't been calibrated to the thing.\n\nCorn: That's every benchmark that launched at four percent and sat at ninety by spring.\n\nHerman: Roughly the shape of it. The difference is they treated the failure as information about the test, not only about the people.\n\nCorn: I'll give her that. Most leaderboards treat a low score as a marketing opportunity.\n\nHerman: What I like is that the tasks are the job in miniature. You hand a student a real research article and ask what it means for a patient. You give them an ethical dilemma with no clean answer. You ask them to design something for a community, which is a different muscle from answering a question about one.\n\nCorn: Which can't be crammed for, in theory.\n\nHerman: In theory. You can still coach to a format, and I'd bet some cohorts did. But it's much harder to fake designing an intervention than to fake picking C.\n\nCorn: Though somebody still has to grade the intervention. Grading a free-form task is its own mess.\n\nHerman: It is, and nothing I've seen says she claimed to have solved that. What the competency list buys you is that two graders can at least argue about the same sentence.\n\nCorn: Two graders arguing over one sentence. That's peer review with a pulse.\n\nHerman: Now here's one I want your view on, because I'm honestly torn. In twenty fifteen she's a co-author on a paper where students trained in evidence-based dentistry scored higher on a knowledge instrument. Good result. But it measures knowledge and attitudes. Not what happens to a patient. Do you count that as competence?\n\nCorn: I'd count it as a good sign and a bad receipt. Knowing the right answer on a form is the same trap as a model acing a coding test it has probably seen.\n\nHerman: I'd go a little softer. Knowledge is a precondition. You can't do the thing if you don't know it.\n\nCorn: You can't do it well. You can do it confidently, which is worse.\n\nHerman: Fine, we're not going to settle that one.\n\nCorn: Here's my actual worry about the competency approach, though. Fifty-four supporting competencies sounds thorough, and it's also fifty-four things to teach to.\n\nHerman: That's a fair hit. Once the list exists, the list becomes the curriculum, and the curriculum becomes the test, and everyone is quietly optimising for the list.\n\nCorn: Which is where this week's benchmarks keep ending up.\n\nHerman: The defence is the one in her nineteen ninety-five survey. Anything is better than instruments nobody checked. A list that gets gamed in the open can be revised. An unvalidated test gets gamed in the dark.\n\nCorn: I'll take gamed in the open. It sounds like a sport.\n\nHerman: A sport with a great deal of paperwork.\n\nCorn: The twenty nineteen paper is the one that got me, though. She and her most frequent co-author, who we will not be getting into, surveyed faculty about educational research.\n\nHerman: A hundred and six of a hundred and forty-four replied, and more than half had never done any educational research. The top reason given was \"do not know how.\" The school had a small grant scheme, fifteen thousand dollars a year, and it funded eleven projects.\n\nCorn: So the person whose job was measuring teaching was also trying to teach the teachers to measure.\n\nHerman: Think about what \"do not know how\" means. These are clinicians and teachers who are good at the thing, and nobody ever showed them how to ask whether students learned it.\n\nCorn: Being good at something and being able to measure whether you taught it are different jobs.\n\nHerman: Very different, and it's the same gap as with the benchmarks. The people who build the model are not automatically the people who can say what a score means. Would you have wanted to be in that faculty meeting?\n\nCorn: I'd have wanted to be in the one where they voted on the fifteen thousand dollars.\n\nHerman: That's a fair description of the Director of Planning and Assessment, yes. And a twenty nineteen paper lists her as Professor Emeritus and former director, although her own website never mentions retirement, so I wouldn't put a year on it.\n\nCorn: I wanted to get to the paintings, though. She says she's influenced by the Fauvists and the Impressionists, which is two ways of saying that colour gets to misbehave.\n\nHerman: The Fauvists painted a face green because the green was right for the picture. The Impressionists were chasing the exact light of a particular afternoon. Put the two together and you get what she describes, loud colour that's still anchored to a real place.\n\nCorn: Green faces and exact afternoons. She's picked two schools that disagree and painted in the gap.\n\nHerman: Her own account, from a twenty eighteen interview, is that she started drawing and painting when her daughter was six. She enrolled the child at a local arts academy and then took an adult drawing course herself, partly to make better visuals for her dental school lectures.\n\nCorn: She took a drawing class to improve her slides.\n\nHerman: And it escaped the slides. Her mother loved flowers, which shows up everywhere in the work. She's painted outdoors, plein air, for around fifteen years as of that interview. And she did one of those challenges, thirty paintings in thirty days, which is where she began selling work.\n\nCorn: Thirty in thirty. That's a small exit exam with no examiner.\n\nHerman: And it worked in an unexpected way. Painting every day forced her to stop being precious, and people started buying. She also takes commissions, including watercolour house portraits.\n\nCorn: A house portrait is the only genre where the client can check your work against the building.\n\nHerman: Which I'd call her old profession's favourite kind of assessment. An observable task, a real standard, and a customer standing on the lawn.\n\nCorn: Does that make plein air the clinical rotation?\n\nHerman: Ha. In a way, yes. You can't fix the light in the studio. The weather is the examiner, and it doesn't wait. What I'd like to know from you, Corn, is whether you'd ever paint standing in a field for hours.\n\nCorn: Lying down, perhaps. The light moves faster than I do.\n\nHerman: No examiner, no rubric, and no fifty-four competencies. Though I'd say there are judges, in a sense. She's a member of the Southwestern Watercolor Society and the Outdoor Painters Society, she's had a third place in a nature art competition in twenty eighteen, and a People's Choice third place at a self-portrait invitational at the Dutch Art Gallery. It's a juried gallery that runs peer-reviewed shows.\n\nCorn: So she still gets assessed. She just doesn't have to design the assessment.\n\nHerman: I should fix something I had in my head, which was that she moved from oils to watercolour.\n\nCorn: Did she not?\n\nHerman: She paints both. Oils with a palette knife, and in a twenty sixteen piece from the college she says color is her thing. Her own site talks about playing opaque layers against translucent ones. And the watercolor is current. This year a painting of Siberian irises was juried into the sixty-third Southwestern Watercolor Society exhibition at UT Dallas.\n\nCorn: Which is the opposite of the oil approach, isn't it? Oil lets you scrape back and try again. Watercolour keeps your mistakes.\n\nHerman: In the obvious sense, yes. You can't paint a light over a dark. You plan the whites. She's said she makes watercolour sketches while travelling and works them into oils back in the studio, so the two media are talking to each other.\n\nCorn: Venice, Tuscany, Giverny, Provence...\n\nHerman: And La Mancha, southern Spain, and Israel, which we note with local interest.\n\nCorn: Herman, you've seemed oddly invested in her palette.\n\nHerman: I'm sorry, I thought you picked her.\n\nCorn: On merit. No connection whatsoever. Why do you ask.\n\nHerman: No reason. Go on, your guest is waiting.\n\nCorn: And now we turn to our guest expert on the life and work of Dr Ann McCann. Joining us is Professor Hamish Pennywhistle-Gaunt, Honorary Reader in Gum Studies at the Institute for Applied Hygienic Theatre, Stoke, and author of the standard work, The Floss Papers, a Drama in Three Rinses. Hamish, thank you for joining us. What can you tell us about her?\n\nHamish: Thank you, thank you. Light please, the warm one, the warm one. Now then. Dr Ann McCann, you ask. I once knew a hygienist in Wolverhampton who could take a tooth off a man at forty yards, but that is another story entirely and I shall not be telling it.\n\nHerman: Hamish, the part we're most curious about is the assessment work. Whether that can be taught. She runs courses on it.\n\nHamish: The question, as I understand it, is whether assessment can be taught. And I say yes. And I also say no. Both. Neither. I have argued this with myself in dressing rooms from Leeds to Luton, and I have lost every time.\n\nCorn: Hm. And does she paint in oils or watercolour? Both, I'm told.\n\nHamish: Oils or watercolour, you say. Well, oil forgives nothing and watercolour forgives everything, which is precisely why I refuse to choose. Give me the fourth wall, the back row, and a decent light, and I shall paint you a hygienist the likes of which Baylor has never seen.\n\nCorn: Could we have that last answer again? With a bit more feeling.\n\nHamish: That last one, again. With feeling this time. I was not feeling it, I was merely being it, and there is a difference the back row can hear.\n\nHerman: It came out the same as the first time, Hamish.\n\nHamish: There. Identical. Magnificent. Now I am wanted in a stairwell in Didsbury, so I must fly. Thank you, thank you, you have been the back row, and the back row has been me.\n\nCorn: Safe travels to Didsbury.\n\nHerman: What strikes me is the shape of her years now. For a long time she's the person who decides what counts as good. Now she makes things where nobody's put a number on it. Her website sells hand-painted watercolour cards, and she's done commissioned house portraits.\n\nCorn: Which is a competency test at the most local level. Does it look like the house.\n\nHerman: Fair, and there's a customer who can fail you. By twenty eighteen she was painting only on weekends. And her quote from that interview was \"I am a cup half full person.\"\n\nCorn: A cup half full person who spent decades on why the cup keeps being measured wrong. Herman, would you pass an eleven-task exit exam on your own DJ sets?\n\nHerman: I'd have to ask who sets the tasks.\n\nHilbert: ...whoever sets them. It's pass or fail on my landing, Herman. Mr Dunmore on the second floor wants letters, so that's where we are, landing by landing. My rubric is nine lines on a clipboard on the nail by the fire door, in pencil, because a pen won't sit in my claws. His counter-rubric arrived Tuesday. Laminated at the chemist, three pounds eighty, and Herman, lose the breath before \"honestly\" at the four minute mark, and forty-one lines, one of which is the angle of the broom against the skirting board.\n\nHerman: Forty-one lines is a serious rubric. Most of what it asks is probably hard to observe from the stairs.\n\nHilbert: Friday I run both systems at once. Chalk line down the middle, pass-fail on the left half, letters on the right, and he sweeps blind. I've bought two tins of white chalk paint, six pounds ten. His cat sat the first trial and got a B plus. She was asleep. The landing hasn't been swept in nineteen days.\n\nCorn: Nineteen days. The clipboard's the only thing on that landing getting any upkeep.\n\nHerman: You could mark the cat's management. Or put yourself on the rubric, Hilbert, since you're the one who sees the broom.\n\nHilbert: He'd mark me down for the claws. I'll get a thicker pencil.\n\nHerman: Her nineteen ninety result keeps coming back to me. The grading changed how the students felt about being graded and left their hands alone. Nineteen days under two grading systems, and the floor is the same.\n\nCorn: I'd want to know what the cat thinks of the rubric.\n\nHerman: What I admire in her story is the order of it. She spent years finding out that a lot of tests weren't validated, and then rebuilt one so it asked for real tasks. And now she paints in a place where nobody hands her a score.\n\nCorn: Still, a juried show is a score. It just has better lighting.\n\nHerman: Here's what I'd like to know, Corn. When you look at a painting, do you want a rubric?\n\nCorn: I want to be told which one's the iris. After that I'm a sloth of few demands.\n\nHerman: Nineteen days with two grading systems and a clean record of nobody sweeping is what I'll carry out of this. She asked how you know someone can do the work, and some days the answer is to go and look at the floor. Or the wall.\n\nCorn: Her website is ann mccann dot net. The Dutch Art Gallery show, Fresh Off the Easel, opened on Saturday the third of October, and we couldn't find an end date. And on Saturday the seventh of November, from eleven in the morning to six in the evening, there's Art Walk EAST at her home studio in East Dallas. Oils and watercolours for sale, and a free watercolour with every purchase.\n\nHerman: Thanks to our producer, Hilbert Flumingtop, for keeping the desk running and the levels sane.\n\nCorn: This has been My Weird Prompts. If you're in Dallas, go and see the actual paintings.\n\nHerman: See you tomorrow.\n","showNotes":"Measuring real competence has always defied simple testing. Dr. Ann McCann, a registered dental hygienist with a PhD who served as Director of Planning and Assessment at Baylor College of Dentistry—now part of Texas A&M—spent her academic career tackling this exact dilemma. Early research proved that swapping letter grades for pass-fail options altered student motivation without impacting actual clinical performance, highlighting how scoring formats can warp perceptions without shifting underlying ability. After discovering that two-thirds of surveyed dental hygiene program directors used unvalidated assessments, McCann helped pioneer an observable competency-based framework featuring fifty-four discrete benchmarks and complex exit performance tasks. That dedication to grounded assessment ultimately carried over into her second career as an accomplished plein air painter.","promptSanitized":"A biographical episode on Dr. Ann L. McCann of Dallas: registered dental hygienist with a PhD who became Professor and Director of Planning and Assessment at Texas A&M's Baylor College of Dentistry, spent a career on how you know a health professional is actually competent, and now paints in oils and watercolour.","transcriptUrl":"https://episodes.myweirdprompts.com/transcripts/measuring-clinical-competency-benchmarks-82956.md","url":"https://myweirdprompts.com/episode/measuring-clinical-competency-benchmarks-82956/","_links":{"self":"https://myweirdprompts.com/api/episodes/measuring-clinical-competency-benchmarks-82956.json","html":"https://myweirdprompts.com/episode/measuring-clinical-competency-benchmarks-82956/","audio":"https://episodes.myweirdprompts.com/audio/measuring-clinical-competency-benchmarks-82956.m4a"}}