Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The architecture Factory

Avatar for William El Kaim William El Kaim PRO
October 11, 2026
46

The architecture Factory

Avatar for William El Kaim

William El Kaim PRO

October 11, 2026

Transcript

  1. EA CODEX · AN INTRODUCTION The architecture factory The factory

    turns enterprise intent into architecture that runs and checks itself. William El Kaim Author of AI-Augmented Enterprise Architecture eacodex.ai https://eacodex.ai/book/
  2. THE TALK IN ONE SLIDE Enterprise architecture must transform itself

    with an architecture factory to govern delivery that agents now carry out SITUATION COMPLICATION QUESTION AI agents now write code and act on enterprise data across the delivery chain. Architecture still lives in documents that agents cannot read and pipelines cannot check. How does architecture keep control when delivery runs at machine speed? THE ANSWER IN SIX MESSAGES A Delivery now runs in agent loops that documents cannot govern B The EA Codex makes architecture typed and executable C A use case and a tool in production show the approach at work D Realizing the factory is platform engineering, done with craftsmanship E On the road ahead, the method matures and the architecture function changes F Proven practices make the factory's agents reliable SLIDES 3 AND 4 Loops need a goal, a checker and limits, and those have to come from architecture. SLIDES 5 TO 9 It makes the Continuous Architecture principles executable, in nested loops. The SAP API policy change fits in one pull request, and Compass already prepares reviews. SLIDES 10 AND 11 SLIDES 12 TO 17 A team builds it in layers, and the Cockpit prototype and other approaches show the way. SLIDES 18 TO 20 v2 shrinks the core, architects take new roles, and the work starts with one domain. SLIDES 21 TO 24 They cover the layers around the model, the harness, the permission ladder and evals. EA CODEX · eacodex.ai 2
  3. A · WHY ARCHITECTURE MUST CHANGE A B C D

    E F Architecture written as documents cannot govern delivery that now runs on agents WHAT MOST EA FUNCTIONS HOLD TODAY Principles, standards, reference architectures and roadmaps are written as Word files, wiki pages and slide decks. Documents keep the content but lose the logic that connects it: a principle is stored without its check, a standard without its conformance test. PRINCIPLES 01 The principles were last revised three years ago and are rarely cited in design. AI agents now write code and act on enterprise data. Every ambiguity left in an architecture document becomes a defect downstream, at a speed no review board can follow. REFERENCE ARCHITECTURES “The enterprise has the right vocabulary and the wrong artifacts.” Reference architectures are PDFs whose authors have since left. 03 STANDARDS 02 The standards catalog is a wiki page that has drifted from what teams deploy. ROADMAPS 04 Roadmaps are decks that never connect to delivery. AI-AUGMENTED ENTERPRISE ARCHITECTURE, CHAPTER 6 EA CODEX · eacodex.ai 3
  4. A · WHY ARCHITECTURE MUST CHANGE A B C D

    E F Stop polishing prompts and build loops, whose goal, checker and limits come from architecture PROMPT an instruction CONTEXT → a working view WHY A BETTER PROMPT IS NOT ENOUGH “A better prompt can improve one response. A better harness improves every run.” HARNESS → an operating environment → LOOP local correction → GRAPH coordination THE FOUR PARTS OF A LOOP 01 A goal with a clear done 02 “The failing test passes and nothing else broke.” A checker outside the model Tests, compilers, schemas or evals say pass or fail. MORI, HARNESS ENGINEERING A prompt asks once and hopes. A loop repeats think, act, observe and decide until a checker outside the model says the work is done, or a limit stops it. WHAT ARCHITECTURE BRINGS EA CODEX · 03 A stop rule Stop on success, at the turn limit, or when two attempts repeat. 04 A budget Turns and money, both capped before the loop starts. Specifications define the done, fitness functions and evals act as checkers, and policies set the stop rules. eacodex.ai Sources: Yarchi, “The 5 levels of AI agents” (loop engineering) · Mori, “Harness Engineering” 4
  5. B · THE ANSWER: AN ARCHITECTURE FACTORY A B C

    D E F The EA Codex turns architecture into typed, linked knowledge that people and agents share EnterpriseIntent BusinessCapability TransformationRoadmap ArchitecturePrinciple TechnologyStandard ReferenceArchitecture DecisionRecord ArchitectureSpecification VariabilitySpec AgentContract EvaluationSuite DataContract 01 Kinds 02 Knowledge graph 03 Variability 04 Executable rules 12 of the Codex kinds, all declared under apiVersion ea.codex/v1 THE LOOP runs in Git EA CODEX INTENT · eacodex.ai → DECISION → SPECIFICATION Every artifact has a typed schema, so two architects produce comparable objects. Objects name what they depend on. A changed principle reveals everything it affects. A product line fixes what all solutions share and governs what is allowed to vary. Validation criteria become policy code that checks every change in the pipeline. → EXECUTION → FEEDBACK ↺ 5
  6. B · THE ANSWER: AN ARCHITECTURE FACTORY A B C

    D E F Like a driver, the architecture factory needs a map, the rules of the road and live navigation What can exist? What is allowed? Where are we, and how do we get there? THE ATLAS THE RULES OF THE ROAD GPS NAVIGATION The Atlas defines the typed kinds, their schemas and the relationships allowed between them. It gives architects a stable map of the architecture model. The rules define permitted and prohibited moves. In architecture they express policies, constraints, decision rights and validation conditions. Navigation locates the organization on the live landscape. Its route is a process graph, Brief, Map, Act and Double-check, with a gate at each turn. The map changes only when someone changes the model. A rule applies wherever its governed context matches. Feedback from each run of the loop recalculates the route. IN THE CODEX IN THE CODEX IN THE CODEX Kinds, schemas and the Atlas Principles, standards and fitness functions The BMAD process graph and the loop KNOWLEDGE GRAPH The graph connects the Atlas, the rules and the live landscape. All three read the same identities and relationships, so navigation can explain why a route is valid. EA CODEX · eacodex.ai Conceptual analogy · Sources in the speaker notes 6
  7. B · THE ANSWER: AN ARCHITECTURE FACTORY A B C

    D E F Enterprise knowledge enters the factory as typed objects, Markdown and live sources TYPED OBJECTS · IN THE CODEX TODAY MARKDOWN KNOWLEDGE · PLANNED IN V2 LIVE SOURCES · REACHED THROUGH MCP YAML kinds that controls can check Text that people and agents both read Systems of record stay where they are What qualifies What it holds What stays outside A concept becomes a kind only if a control acts on it or no other tool can hold it. Glossaries, domain notes for finance or HR, skills and agent guides such as AGENTS.md. Capabilities, processes, risks and KPIs stay in the EA repository and their own tools. Business knowledge How v2 will type it How the Codex points to them A business object carries identity, lifecycle and cross-attribute rules that policy code enforces. A CodexPack kind, with glossary, skill, decision and publication profiles. By identifier, such as eatool:bc.pv.operations or risk:RSK-2026-014. Example Today How agents reach them BO-ADVERSE-EVENT and its rule SR-AE-002 on case status. Skills and agent guides already sit in the repository as plain Markdown files. Model Context Protocol (MCP) servers expose each source to agents and assistants. OPEN EXAMPLE Paul Stean’s EA-as-Code stores the IT landscape as JSON in Git, with Markdown rules checked on each pull request. GOVERNED ACCESS EA CODEX · github.com/paulstean/EA-as-Code Every MCP call passes a gateway, and a tool access policy lists what each agent may reach (principle PRIN-AI-008). eacodex.ai Source: ea.codex/v1.1 repository · inclusion-criteria.md, relationships.md, v2/design-intent.md 7
  8. B · THE ANSWER: AN ARCHITECTURE FACTORY A B C

    D E F The factory runs three nested cycles, each closed by its own check ARCHITECTURE LOOP Weeks to quarters INTENT → DECISION → SPECIFICATION → EXECUTION → FEEDBACK ENTERPRISE KNOWLEDGE SOURCES Closed by evidence that changes the next intent EA repository, documents, data, code and tickets ↓ Each specification is delivered through delivery loops DELIVERY LOOP FRAME Days to weeks → MODEL → BUILD → VERIFY ACCESS MCP servers behind a governed gateway Closed by a release gate. The EA Codex uses BMAD. ↓ Each build step runs agent loops FORM AGENT LOOP Seconds to minutes THINK → ACT → OBSERVE → DECIDE Closed by a checker, a stop rule and a budget Each outer loop gives the inner one its goal, and each inner loop returns evidence to the outer one. THE PRINCIPLE EA CODEX Typed YAML in the Codex today, Markdown packs planned in v2 · eacodex.ai Sources: AI-Augmented Enterprise Architecture, figure 16.9 · Yarchi, “The 5 levels of AI agents” 8
  9. B · THE ANSWER: AN ARCHITECTURE FACTORY A B C

    D E F The factory makes the six Continuous Architecture principles executable CONTINUOUS ARCHITECTURE PRINCIPLE HOW THE FACTORY APPLIES IT IN THE CODEX 01 Architect products → The Codex and its factory are run as a product, with an owner, a backlog 02 Focus on quality attributes → Principles and standards carry validation criteria, and fitness functions 03 Delay design decisions until they are → A decision is recorded when its trigger arrives, with a sunset date, and 04 Architect for change with the power of → Each change is one reviewable pull request, and the factory grows one 05 Architect for build, test, deploy and → Policy code runs in the delivery pipeline, and evidence from operations 06 Organize teams after the system design → Every object names its owners, and the architecture function works as a needed small operate EA CODEX · eacodex.ai Releases v1.0, v1.1, v2 and versioned releases. check them on every change. variability envelopes keep options open. ArchitecturePrinciple, FitnessFunction DecisionRecord, VariabilitySpecification One pull request per change loop at a time. RegoPackage, EvidenceRecord feeds the next intent. platform team. Owners metadata, DesignAuthorityBody Source: Murat Erder and Pierre Pureur, Continuous Architecture, 2015 · ea.codex/v1.1 9
  10. C · FROM USE CASE TO PRACTICE A B C

    D E F Use case: a vendor policy change across the whole estate is absorbed in one pull request On 22 April 2026, SAP restricted access to its systems to published APIs (application programming interfaces) and SAP-endorsed mechanisms. Every integration and every AI agent reading SAP data came into scope. 01 DECISION 02 DEC-SAP-API-001 It records the trigger and the rule every SAP-bound flow must follow. PRINCIPLE INT-002 → SAP-approved data access stays stable when the policy changes again. 03 STANDARD 04 STD-SAP-INT-001 → It classifies seven integration patterns, from preferred to prohibited. FITNESS FUNCTIONS FF-SAP-001 / 002 → Policy code in the pipeline rejects any non-compliant change. 05 SPEC CHANGE SPEC-PV-001 → The ACME Pharma triage agent now declares a published SAP API. THE USUAL RESPONSE THE CODEX RESPONSE A working group builds a spreadsheet inventory and grants exceptions with no end date. Four to eight weeks later, half the inventory is out of date. The five changes are reviewed together in one pull request. They merge as one, leave one audit trail, and can be reverted as one if the decision proves wrong. EA CODEX · eacodex.ai Source: AI-Augmented Enterprise Architecture, chapter 14 10
  11. C · FROM USE CASE TO PRACTICE A B C

    D E F At Pernod Ricard, Compass already runs part of the factory for architecture reviews WHAT COMPASS ALREADY DOES 01 Codify architecture knowledge Principles, patterns and controls are encoded, and deviations from the standards library are flagged. 02 Codify business domain knowledge For each new demand, project documents and Excel questionnaires pre-fill the assessment with the business context. 03 Close the loop Compass produces the decision paper that the review board validates. The draft decision is created in LeanIX and then finalized there. 04 Automate the processes Architecture diagrams and board documents are generated, and Compass is run as a product that evolves with the practice. AI ASSISTS, PEOPLE DECIDE EA CODEX · eacodex.ai • • • Principles are scored from 0 to 5 with an AI estimate, the application tier is estimated, and a checklist prepares the document for the review board. COMPASS Principles, patterns and controls are encoded and evaluated systematically. Architecture diagrams are generated and publishing them directly to LeanIX is planned. Architects review every suggestion and draft, and the review board owns the decision. Source: Compass introduction, Pernod Ricard Enterprise Architecture, content baseline 30 September 2026 11
  12. D · REALIZING THE FACTORY A B C D E

    F The factory must be built, with craftsmanship and a dedicated team, like any new platform WHAT CAN BE BOUGHT WHAT HAS TO BE BUILT Commodity layers that every enterprise can share Your knowledge, your rules and your evidence Language models, agent runtimes and coding assistants Your principles, standards and decisions as typed kinds The EA repository for the landscape, such as SAP LeanIX The mappings between those kinds and your real landscape Connectors, pipelines and observability tooling Fitness functions and policies that encode your rules Generic eval and policy engines Your agent harness and eval cases drawn from your failures HOW TO BUILD IT Run it as a platform Staff a dedicated team Build with craftsmanship Grow it in layers A product owner, a backlog and versioned releases of the Codex. Architects work alongside platform, data and eval engineers. Every change goes through review, tests and a pull request. Start small and add a layer when a real failure justifies it. EA CODEX · eacodex.ai 12
  13. D · REALIZING THE FACTORY A B C D E

    F The factory runs on a knowledge graph that feeds governed architecture workflows The graph joins live landscape facts to Codex kinds and rules, and governed workflows use it as context. KNOWLEDGE GRAPH The graph connects current landscape facts to every governed Codex object. It exposes dependencies, impact paths, and accumulated evidence. ENTERPRISE KNOWLEDGE SOURCES ENTERPRISE ARCHITECTURE CODEX The EA repository, documents, data, code and tickets, reached through MCP servers. Knowledge without a system of record is planned as Markdown packs in v2. The Codex defines the typed kinds that architecture governs. Its rules define required relationships and constraints. They also define decision rights and validation conditions. GOVERNED ARCHITECTURE WORKFLOWS Architecture process automation A portfolio or delivery event starts the workflow. 01 EA CODEX · 02 Agents read graph context and perform the governed task. eacodex.ai Kind and rule validation 03 Controls record evidence and refresh the graph. 01 A contributor proposes a kind or rule. 02 The workflow validates the schema and its references. 03 The design authority approves and publishes the version. 13
  14. D · REALIZING THE FACTORY A B C D E

    F EACodex Cockpit: Typed contracts carry every step of the loop, from intent to feedback Concept WHERE IT ACTS 01 INTENT → DECISION → SPECIFICATION → PURPOSE → Covered Partly covered Not covered FEEDBACK CONTINUOUS ARCHITECTURE LOOP The Enterprise Architecture Codex turns architecture into typed, versioned objects. People and software agents read the same definitions. 02 EXECUTION 01 Intent states the desired outcome and constraints. INFORMATION MODEL A public schema defines each kind and its validation contract. Explicit references connect the objects into a portable semantic graph. 05 03 GOVERNANCE Every downstream object remains traceable to the enterprise intent. Decision records make ownership and trade-offs explicit. 04 Feedback changes the next intent and specification. EXECUTION Execution produces evidence against the specification and its controls. That evidence informs the next decision and specification. 04 Execution produces change and evidence. Typed contracts connect every step. 02 Decision records the accountable choice and trade-offs. 03 Specification defines the executable contract and controls. The diagram summarizes the documented operating model. Sources are in the speaker notes. EA CODEX · eacodex.ai 14
  15. D · REALIZING THE FACTORY A B C D E

    F The EACodex Cockpit prototype shows what a crafted factory workbench could look like 01 One knowledge graph It shows Codex artifacts and the IT landscape in a single view. 02 BMAD as a Kanban Work moves through Brief, Map, Act and Double-check. 03 Governance you can run Fitness functions check artifacts for conformance. 04 A council of ten AI agents A simulated design authority approves or escalates. 05 Agents under contract Agents iterate inside the envelope the Codex defines. FROM PROTOTYPE TO PLATFORM A running factory needs the platform work described earlier. EA CODEX · eacodex.ai Prototype · Source: eacodex.ai/cockpit 15
  16. D · REALIZING THE FACTORY A B C D E

    F Ergane Iasa's design workspace, built on its body of knowledge (BTABoK) WHERE IT ACTS 01 INTENT → DECISION → SPECIFICATION → PURPOSE 03 GOVERNANCE Assistants propose and the architect signs the decision. Reviews check the rails that each concern carries. 04 STATUS, OCTOBER 2026 The libraries and the MCP server are live. Ergane is a Premium product still being built; repository sync and coding agents are underway, and drift review is planned. Covered Partly covered Not covered FEEDBACK 01 The architect frames the concern and the capability. KNOWLEDGE MODEL The BTABoK provides canvas, viewpoint and pattern libraries. Canvases carry schemas, and each viewpoint lists the concerns that define done. → ARCHITECTURE AND DELIVERY LOOPS Ergane is Iasa's design workspace: boards, canvases, decisions and models that a team edits together. Assistants reach the BTABoK through a Model Context Protocol (MCP) server. 02 EXECUTION In production 05 Evidence and drift come back to the architect. 04 Developer and agent build it and prove it. 02 Architects sign. Engineers verify. The agent proposes options and their consequences. 03 The architect interrogates and signs the decision. The diagram summarizes the documented operating model. Sources are in the speaker notes. EA CODEX · eacodex.ai 16
  17. D · REALIZING THE FACTORY A B C D E

    F ArcKit A repository-based governance harness for architecture delivery WHERE IT ACTS 01 INTENT → DECISION → SPECIFICATION → EXECUTION PURPOSE 03 GOVERNANCE Hooks gate unsafe tool calls. Observers add provenance and traceability to the generated artifacts. 04 01 Planning defines principles and the project structure. DELIVERY METHOD Commands and templates structure the work across United Kingdom Government Digital Service delivery phases. Architects review and refine each draft. REPOSITORY MODEL Version control stores the files and their review history. Teams add delivery evidence to the repository rather than to a central portfolio system. Covered Partly covered Not covered FEEDBACK GOVERNED DELIVERY LOOP ArcKit operates around an artificial intelligence coding assistant. It produces reviewable architecture artifacts inside the project repository. 02 → In production 05 Live verifies traceability and operational readiness. 04 Beta reviews designs and prepares delivery. Repository evidence informs the next cycle. 02 Discovery identifies stakeholders and risks before the business case. 03 Alpha develops requirements and evaluates options. The diagram summarizes the documented operating model. Sources are in the speaker notes. EA CODEX · eacodex.ai 17
  18. E · THE ROAD AHEAD A B C D E

    F Lessons from v1 lead to a smaller core in v2, built around one visible chain → INTENT POLICY → DECISION → SPECIFICATION LESSONS FROM V1 01 FEEDBACK Intent PolicyConstraint ArchitecturePrinciple TechnologyStandard DecisionRecord Specification Control EvidenceRecord FeedbackSignal BusinessObject CodexPack Governance bodies are architecture too V2 · LAYER 2 · TYPED PROFILES UNDER EACH PARENT Specification A written bar stops metamodel creep Three inclusion tests keep capabilities, processes and KPIs in their own tools. Control Too many peer kinds hide the argument CodexPack Readers met 27 kinds before they could see a chain of seven concepts. EA CODEX · eacodex.ai agent, data, sovereignty and product-line contracts (13 profiles) fitness function, policy implementation, scenario pack (5) EvidenceRecord 05 → EVIDENCE Every kind needs the same envelope The design authority and its delegation tiers became kinds in v1.1. 04 → V2 · LAYER 1 · A CONCEPTUAL CORE OF ABOUT A DOZEN KINDS Four kinds kept content outside metadata and spec until v1.1 normalized them. 03 CONTROL Unchecked strings break the chain v1.1 typed the policies, fitness functions and evidence that v1.0 held as plain text. 02 → conformance result, approval record, runtime trace (6) glossary, skill, decision and publication packs (6) Not yet scheduled. A migration script will convert v1 files to v2. Source: ea.codex repository · CHANGELOG.md (v1.0.0 and v1.1.0, 28 April 2026) and docs/v2/design-intent.md 18
  19. E · THE ROAD AHEAD A B C D E

    F Building the factory changes the skills and the operating model of the architecture function FROM TO Role of the architect Reviews decisions one by one Main artifact Documents, wiki pages and slide decks → Typed, versioned Codex objects kept in Review Architecture board meetings → Pull requests with evidence, prepared by Git SKILLS TO GROW 01 Writing specifications and schemas 02 Working in Git with pull requests 03 Policy as code and fitness functions 04 Designing evals and reading the numbers 05 Designing harnesses for agents 06 Running the Codex as a product an AI council Compliance Team EA CODEX → Designs the policies, constraints and controls that govern execution Periodic audits and exception lists → Fitness functions that check every Architects working alone → Architects with platform, data and eval change engineers · eacodex.ai 19
  20. E · THE ROAD AHEAD A B C D E

    F Codify the knowledge first, then close the loop and automate the processes 01 STEP 1 STEP 2 Codify architecture knowledge Write principles, standards, decisions and reference architectures as typed Codex kinds. 02 STEP 3 Codify business domain knowledge → Capture the capabilities, business objects and rules of one domain, such as finance or HR. 03 STEP 4 Close the loop → Connect intent, decision, specification, execution and feedback, with fitness functions and evals as checkers. 04 Automate the processes → Automate the architecture processes the loop exposes, starting with reviews and conformance checks. DELIVERABLE DELIVERABLE DELIVERABLE DELIVERABLE A versioned architecture Codex Domain kinds linked to the architecture A running loop with a release gate Automated, governed workflows START SMALL EA CODEX · Begin with one business domain and one loop, then grow the factory in layers. eacodex.ai 20
  21. F · BEST PRACTICES A B C D E F

    Production agents depend on five layers around the model, and most of them are architecture work Picture the model as a new employee. Each layer answers one question about the work around him. LEVEL 1 LEVEL 2 LEVEL 3 LEVEL 4 LEVEL 5 Context Loop Jev Harness Evals What is on his desk Checklist or goal The front desk sorting the mail His office, his keys, his reviewer The same test every month What does the model see? Who decides the next step? Which cheap decisions can a classifier take? What can it reach, and who checks it? How do we know it improved? Put stable instructions first, keep large content on disk, and restate the goal at the end. A loop needs a clear done, an external checker, a stop rule and a budget. Use a workflow when you can write the steps. A fast classifier (Jev from TypeSafe) answers narrow questions and sends unsure items to the main model. Containment, guide files, tests and linters. People approve 93% of requests, so remove dangerous actions. Turn real failures into cases with one behavioral check each, verify the judge, and sample live traffic. FIRST MOVE FIRST MOVE FIRST MOVE FIRST MOVE FIRST MOVE Freeze the system prompt. Set a turn and cost limit. Label 200 examples of the commonest decision. Run the agent as a user that cannot delete. Write the last five failures as test cases. EA CODEX · eacodex.ai Source: Yarchi (@undefinedKi), “The 5 levels of AI agents”, X, 28 September 2026 21
  22. F · BEST PRACTICES A B C D E F

    Reliable agents come from the harness: contracts, policies and evidence that live outside the model 01 The model is not the agent 02 Turn every request into a contract 03 Give the agent a map of the project 04 Put a gateway in front of the tools 05 Move memory out of the conversation 06 Make evidence the gate to completion 07 Separate the builder from the verifier The harness turns the model's reasoning into controlled execution. Define the objective, constraints and a measurable “done when” first. Context is an attention budget, so load detail only when needed. Each tool has a contract, and the harness authorizes every call. Facts, decisions, state and lessons live outside the transcript. Tests and queries prove “done”; models judge what code cannot. An independent verifier tries to break the result. EA CODEX · eacodex.ai 08 Move permissions out of the model 09 Stop retrying blindly 10 Turn repeated instructions into checks 11 Record the run 12 Give every run a receipt 13 Make every failure improve the harness 14 Start with the smallest useful harness Risky actions climb a policy ladder, from automatic to prohibited. Classify each failure, then repair, escalate or stop within limits. Recurring rules move from the prompt into automated controls. Traces of context, tools and approvals let you repair a single step. Each run reports what changed, what was verified and what was not. Fix the system that allowed the failure, and every later run gains. Add layers only when the failure surface justifies them. Source: Mori (@0xjmori), “Harness Engineering: How to Build AI Agents That Actually Work”, X, 27 September 2026 22
  23. F · BEST PRACTICES A B C D E F

    Put people where judgment matters, and let policy handle the rest THE PERMISSION LADDER LOW RISK Read, search, inspect Automatic 93% of approval requests are accepted, so an approval prompt protects little. Remove the dangerous actions instead. FOUR ROLES FOR PEOPLE REVERSIBLE Edit the workspace, run tests, create drafts Automatic, with a trace Approval required 02 Approve at the gates that matter In the Cockpit prototype, level 1 changes pass alone and levels 3 and 4 go to a council. 03 IRREVERSIBLE OR SENSITIVE Delete data, rotate credentials Set the intent and the rules Architects write the policies and controls the factory enforces. EXTERNAL EFFECT Send, deploy, purchase 01 Calibrate the judges People label samples, so that evals measure something real. Hard gate or prohibited 04 Turn failures into rules Each failure a person reviews becomes a new test, rule or permission. EA CODEX · eacodex.ai Sources: Mori, “Harness Engineering” (permission ladder) · Yarchi, “The 5 levels of AI agents” (93%) · eacodex.ai/cockpit 23
  24. F · BEST PRACTICES A B C D E F

    Only evals with honest numbers can prove that an AI system actually works A language model does not crash when it fails: it returns a fluent, confident answer that may be wrong. An eval is a repeatable test that says, with a number you can trust, whether the system does what you want and whether a change made it better or worse. → COLLECT cases with gold answers RUN → the system on each 01 Collect gold cases first 04 Ask the judge yes or no 07 Report honest numbers Start with 20 to 50 real failures, each with the right answer written down. Use one question per failure type, with context and quoted evidence. Every rate gets an interval, and versions are compared on the same cases. EA CODEX · eacodex.ai GRADE code, judge or human → COMPARE number with error bar 02 Read outputs before grading 05 Test the judge itself 08 Run agents several times Error analysis on a hundred outputs gives the failure list to test. Compare it with human labels and count the bad outputs it actually catches. Report success on all runs, cost per success and reliability separately. 16 / 26 Humans found 26 bad replies out of 60. The AI judge flagged only 10, so 16 wrong answers would have reached customers. 03 Use code for definite answers 06 Show the judge the evidence 09 Keep one grader hidden Rules check labels, formats and required facts, but cannot see meaning. Tool calls and final state reveal success, where the chat only shows tone. Any grader used to train or select the system will eventually be gamed. Source: Sergii Makarevych (@sermakarevich), “Evals: how to know whether an AI system actually works”, X, 3 October 2026 24