Operational Truth: Why Trustworthy AI Cannot Be Benchmarked

Presented by Shahid Shah, Founder of Intellectual Frontiers LLC, at the IEEE Baltimore Technical Colloquium, University of Maryland, Baltimore County, on November 7, 2026, in the AI Meets Physical Reality track.

Trustworthy AI requires evidence from operation, not just benchmark scores. This presentation connects NIST AI RMF and TEVV, ISO/IEC 42001, the EU AI Act, FDA guidance, ISO 21448 and NERC to practical post-deployment monitoring, human oversight and measurable evidence of safe use.

Updated presentation: 16 slides plus one backup slide.

Transcript

1. T R AC K : A I M E E T S P H YS I C A L R E A L I T Y Operational Truth Why trustworthy AI cannot be benchmarked Shahid Shah Founder, Intellectual Frontiers LLC IEEE Baltimore Technical Colloquium 2026 · 7 November 2026 · University of Maryland, Baltimore County
2. STA RT H E R E Your AI system’s best month may be the month people stopped checking it TEMPE, ARIZONA · 18 MARCH 2018 The NTSB named Uber’s lack of any countermeasure for automation An Uber test vehicle driving itself struck and killed a pedestrian. The complacency as part of what caused the safety operator, who was there to catch the system’s mistakes, was crash. looking down for most of the trip. Source: NTSB Highway Accident Report HAR-19/03, adopted 19 November 2019. FRAMEWORK NIST AI RMF, MANAGE 4.1: post-deployment monitoring plans should cover override. Track how often people use the override, not only that one exists. DO THIS Pull your AI system’s override rate this week. If it is close to zero, find out why before you call it a success. IEEE Baltimore Technical Colloquium 2026 · 7 November 2026 · UMBC 2
3. W H O W R I T E S T H E T E ST A vendor’s validation tells you how the model did on the vendor’s data 0.76–0.83 AUC for the Epic Sepsis Model, as reported by its developer 0.63 AUC in an independent study at Michigan Medicine. It missed 67% of patients with sepsis and alerted on 18% of all patients. Source: Wong et al., external validation of the Epic Sepsis Model, JAMA Internal Medicine, 2021. FRAMEWORK NIST AI RMF, MAP: document the context of use before you measure. Run TEVV yourself, on your own data, instead of relying on the developer’s results. DO THIS Before the vendor demo, write down on your own data what result means continue and what means stop. IEEE Baltimore Technical Colloquium 2026 · 7 November 2026 · UMBC 3
4. THE CLAIM Trust is not a property of the model. It is a property of the system in operation. That system is the model plus the people using it, the data arriving today and what everyone is rewarded for. FRAMEWORK FDA, Health Canada and MHRA Good Machine Learning Practice, principle 7: judge the performance of the human-AI team, not the model alone. DO THIS Name the decision your AI informs, and test that decision where it is made. IEEE Baltimore Technical Colloquium 2026 · 7 November 2026 · UMBC 4
5. T H E F R A M E WO R K S YO U A L R E A DY H AV E Each framework answers a different question, so use each one for its own question Framework What it is Use it to NIST AI RMF 1.0 and GenAI Profile Voluntary, any sector Organize the risk work: Govern, Map, Measure, Manage NIST TEVV Voluntary NIST program Produce evidence: test, evaluate, verify, validate ISO/IEC 42001:2023 Certifiable standard Run AI governance as an audited cycle EU AI Act Law, high-risk AI in the EU Set up oversight (Art. 14) and post-market monitoring (Art. 72) FDA GMLP and PCCP guidance US medical devices Judge the human-AI team, monitor, plan changes ahead ISO 21448 (SOTIF) Standard, road vehicles Find hazards from performance limits with no fault NERC PRC-004 and PRC-005 Mandatory, bulk power system Copy its periodic testing and misoperation review DO THIS Start with what your sector requires, organize the rest under the AI RMF, and use TEVV for the methods. IEEE Baltimore Technical Colloquium 2026 · 7 November 2026 · UMBC 5
6. AN OLD FINDING Engineers described this problem with automation in 1983 Bainbridge, “Ironies of Automation,” Automatica, 1983 Parasuraman and Riley, Human Factors, 1997 Automation takes over the routine work and leaves people People over-rely on automation that is usually right, and the rare, hard cases, with less practice and less attention ignore automation that raises too many false alarms. than before. What’s new with AI is that its wrong answers read as fluently as its right ones. FRAMEWORK EU AI Act, Article 14: people who oversee high-risk AI must be able to stay aware of automation bias. Forty years after Bainbridge, it’s in law. DO THIS Expect your reviewers’ skill to fade. Give them practice cases with known answers on a schedule. IEEE Baltimore Technical Colloquium 2026 · 7 November 2026 · UMBC 6
7. H OW SYST E M S D R I F T A deployed system drifts when the world, the people or the incentives change The world The people The incentives Inputs move away from the validation Reviewers come to depend on it, Teams start working to the number the data, some inputs go missing, and accept fluent answers on sight and lose system reports, and the work reshapes adversaries find what the system practice at the hard cases. itself around the tool. misreads. FRAMEWORK ISO 21448 (SOTIF) covers hazards from performance limits with no fault present. EU AI Act, Article 72: monitor a high-risk system for its whole lifetime. DO THIS Give each of the three a named owner. Most teams watch only the first. IEEE Baltimore Technical Colloquium 2026 · 7 November 2026 · UMBC 7
8. B E L I E VA B L E CO M P E T E N C E The better the system looks, the less anyone checks it Each week without a visible failure adds trust and removes a reason to look. A benchmark can’t see the decline, because the benchmark’s data never changed. Dashed line: drift begins. Illustrative shape, not data. Trust people place in it How often it holds up in the field FRAMEWORK NIST AI RMF, MEASURE 2.4: monitor the system’s behavior in production. Count the reviewer as part of the system you monitor. DO THIS Mix cases with known answers into the queue, and track whether reviewers still catch the wrong ones. IEEE Baltimore Technical Colloquium 2026 · 7 November 2026 · UMBC 8
9. T H E B L I N D S P OT You only learn outcomes for the cases your system lets through The AI flags a case People investigate it The outcome is recorded The AI scores it routine Nobody looks The outcome is never learned If you judge the system on its own records, its misses can’t show up in the numbers. FRAMEWORK FDA Good Machine Learning Practice, principle 10: monitor deployed models for drift. That takes outcomes for the cases the model waved through, too. DO THIS Keep a small random sample handled without the AI, so you learn outcomes for both kinds of case. IEEE Baltimore Technical Colloquium 2026 · 7 November 2026 · UMBC 9
10. S AY I N G “ I D O N ’ T K N OW ” A system that never says “I don’t know” should fail your acceptance test ANSWERS EVERY CASE HANDS BACK WHAT IT CAN’T JUDGE An input it has never seen gets an answer that looks Uncertain cases go to a person, marked. How often just like the others. Your reviewers can’t tell which that happens becomes a number you can watch. ones to check. FRAMEWORK NIST AI RMF, MEASURE 2.6: show the system can fail safely, especially beyond its knowledge limits. Handing the case back is how it fails safely. DO THIS Put it in the contract: the system flags inputs unlike its validation data, and you test that it does. IEEE Baltimore Technical Colloquium 2026 · 7 November 2026 · UMBC 10
11. YO U A L R E A DY D O T H I S You test protection relays on a schedule; your AI deserves the same discipline What the grid already requires The same rule for an AI system A documented maintenance program with maximum test intervals (NERC PRC-005) Re-validation against fresh real outcomes, on a set schedule Every misoperation identified and corrected (NERC PRC-004) Every wrong answer that reached a decision reviewed, with its cause recorded Monitoring extends a test interval only if its alarms reach someone A dashboard counts only if someone is paged when it fires FRAMEWORK ISO/IEC 42001 sets up the same cycle for AI: monitoring, internal audit, corrective action. NERC shows what that cycle looks like when it is enforced. DO THIS Put every AI system that informs operations into the same periodic testing and misoperation review. IEEE Baltimore Technical Colloquium 2026 · 7 November 2026 · UMBC 11
12. O P E R AT I O N A L ACC E P TA N C E T E ST I N G ( OAT ) In OAT, a named person decides after comparing answers with real outcomes Observe Compare Detect Decide Log inputs, outputs and Check a sample of Watch for drift in the Continue, narrow the what people did with each answer. answers against what really happened. world, the people and the incentives. scope, or stop. Record who and why. Then the loop runs again, for as long as the system informs decisions. FRAMEWORK NIST TEVV supplies the methods for Compare. FDA’s predetermined change control plans show Decide done in advance: allowed changes and their tests are written before deployment. DO THIS Write down what result means continue, narrow or stop before you see any data. IEEE Baltimore Technical Colloquium 2026 · 7 November 2026 · UMBC 12
13. E X T E N D I N G N I ST PA ST G O - L I V E OAT carries each NIST AI RMF function past go-live Function What the AI RMF asks What OAT adds while the system runs Govern Accountability and a culture of risk management A named person who can stop the system, and a record of each decision Map The context the system is used in The specific decision the AI informs, written down Measure Testing, evaluation, verification and validation (TEVV) Reliance signals: overrides, time to accept, abstentions, spotcheck error Manage Treating risk, including post-deployment monitoring Continue, narrow and stop conditions set before the data comes in FRAMEWORK NIST AI 100-1 (AI RMF 1.0, 2023) and AI 600-1 (Generative AI Profile, 2024) are voluntary. They say what to manage; OAT is one way to do it in operation. DO THIS Open your AI RMF profile and add one post-deployment measure under each function. IEEE Baltimore Technical Colloquium 2026 · 7 November 2026 · UMBC 13
14. W H E N YO U D O N ’ T N E E D T H I S A benchmark is enough when feedback is fast and errors are cheap FAST FEEDBACK · CHEAP ERRORS FAST FEEDBACK · COSTLY ERRORS Benchmark and routine monitoring. Search ranking, spam filtering. Alarms that page a person. Fraud holds, network intrusion blocks. SLOW FEEDBACK · CHEAP ERRORS SLOW FEEDBACK · COSTLY ERRORS Scheduled spot checks. Document tagging, routine summaries. Full OAT. Sepsis alerts, autonomous driving, load forecasts. FRAMEWORK The EU AI Act draws a similar line: post-market monitoring is required for high-risk systems, not for every AI system. DO THIS Place each of your AI systems in one box. Spend your OAT effort on the bottom right. IEEE Baltimore Technical Colloquium 2026 · 7 November 2026 · UMBC 14
15. D O T H I S O N M O N DAY Four things you can start without a budget request 1 Pull the override rate and the time to accept. Treat a number near zero as a question. 2 Write your continue, narrow and stop conditions before the next vendor demo. 3 Route a small random sample around the AI, and mix known-answer cases into the queue. 4 Add your AI to the periodic testing and misoperation review your critical equipment already gets. SOURCES NIST AI RMF (Map; Measure 2.4, 2.6; Manage 4.1) and TEVV · EU AI Act Articles 14 and 72 · FDA Good Machine Learning Practice 7 and 10, PCCP guidance · ISO/IEC 42001 · ISO 21448 · NERC PRC-004, PRC-005 IEEE Baltimore Technical Colloquium 2026 · 7 November 2026 · UMBC 15
16. Questions Which of your AI systems sits in the slow-feedback, costly-error box, and who checks it today? Framework Speaker operationaltruth.org shahidshah.com IEEE Baltimore Technical Colloquium 2026 · 7 November 2026 · UMBC 16
17. W H AT OAT M E A S U R E S Measure how people rely on the system, not only how accurate it is Signal What it tells you Override rate How often people reject or edit the output. Near zero can mean over-trust. Time to accept How long someone looks before acting on an answer. Out-of-range inputs The share of inputs unlike the data the system was validated on. Abstention rate How often the system says it does not know. Outcome lag How long until you learn whether an answer was right. Spot-check error The error rate on a random sample that experts review by hand. IEEE Baltimore Technical Colloquium 2026 · 7 November 2026 · UMBC Backup