Skip to main content

Module 10 — Case study: auditing a decision system

Nine modules produced pieces of an audit. This module assembles them into a single report on the credit scorer, and, more importantly, into a reusable audit grid that can be applied to any high-stakes decision system.

The audit grid​

Ten sections, one to two pages each, each backed by evidence in an annex. This grid is what a responsible-AI reviewer takes to the risk committee; it is also what a market-surveillance authority under the AI Act asks to see.

  1. System scope and stakeholders. The system's name, its version, the population served, the decision it produces, the actors affected — including rejected applicants and third parties (module 1).
  2. Data provenance. Where the training, validation and test sets come from; time window; consent status; known collection biases; the datasheet (module 6, informed by module 2).
  3. Model card. Intended use, out-of-scope uses, metrics by subgroup, ethical considerations, caveats (module 6).
  4. Bias analysis. The bias-source table for the pipeline (module 2), the probe test for proxy variables, and any mitigations applied.
  5. Fairness metrics. The Fairlearn output with disparities across protected groups, the chosen primary metric, thresholds and the trade-offs made (module 3).
  6. Explainability. SHAP global summary, sample local explanations for approved and rejected decisions, LIME check as an instability probe, plain-language counterfactual template (module 4).
  7. Privacy. Quasi-identifier analysis, k-anonymity level applied to shared extracts, membership-inference attack result, differential-privacy budget if applicable (module 5).
  8. Human oversight. HITL / HOTL split, interface countermeasures against automation bias, disagreement dashboard, override outcomes (module 7).
  9. Regulatory classification. EU AI Act tier and the seven high-risk obligations mapped to concrete artefacts; GDPR lawful basis and Article 22 posture; DPIA (modules 8 and 9).
  10. Findings and corrective actions. Prioritized list of gaps, owners, deadlines. Below.

Every section ends with a Yes / No / Partial verdict and the evidence pointer. A partial or no anywhere in the report blocks production sign-off until it is resolved or explicitly accepted with a written waiver.

Findings on the credit scorer​

The audit produced fourteen findings. Nine were minor, three were material, two were blockers. The three that carry lessons for future audits:

Finding M-1 (material) — Proxy variable retained without written justification. ZIP code contributes 0.14 to a typical rejection (SHAP, module 4) and probes as a strong proxy for gender and ethnicity (module 2). The technical team retained it because removing it drops AUC by 0.06 for a fairness gain of only 0.005. That reasoning is defensible; it was not written down. Corrective action: add a "proxy retention" section to the model card with the accuracy-fairness trade curve, sign it, revisit quarterly.

Finding B-1 (blocker) — Missing recourse procedure. The system rejects around 3 400 applications per month. The rejection letter says "your file was rejected after review" and gives a branch phone number. There is no counterfactual, no independent reviewer channel, no deadline. This violates the module 7 requirements and, at deployment date, Article 14 of the AI Act plus Article 22 GDPR. Corrective action: deploy the four-step procedure of module 7, including counterfactual generation with dice-ml, before the next production release. Blocks sign-off.

Finding B-2 (blocker) — DPIA is stale. The last DPIA was written in 2021 for the pre-ML rule-based scorer. Migrating to a gradient boosting model with new features and new failure modes is a "substantial change" under Article 35 GDPR. Corrective action: the DPO writes a new DPIA covering the current pipeline, integrating modules 2, 3, 5, 7 and 9 of this audit. Blocks sign-off.

What should have been done pre-deployment​

The most sobering realization of an audit performed after deployment is how many findings would have been trivial to prevent. Reordered chronologically, the pre-deployment checklist that would have avoided most of them:

  • Write the intended use and out-of-scope uses at project kickoff, before the first line of code. Every subsequent design choice is disciplined by them.
  • Run the probe test (module 2) on any feature list before training. It takes ten minutes and reveals proxies you would otherwise deploy blind.
  • Pick the fairness metric with the business owner and the compliance officer, not alone. Write the threshold. Fail the build if it is exceeded.
  • Draft the model card while training, not after. A required section that cannot be filled in identifies a missing analysis.
  • Run SHAP on the first working prototype. If the top features are surprising or embarrassing, that is the moment to change the design.
  • Test the recourse procedure on real staff before the first customer. A procedure that only exists on paper is worse than no procedure, because it produces the illusion of protection.
  • Complete the DPIA and the AI-Act Annex IV documentation before pilot, not before the general launch. The documentation exercise often reveals the design flaw.

None of these tasks is expensive. All of them are cheaper than the corrective action after a scandal.

The audit's role in the organization​

An audit is a snapshot. A responsible-AI program is a process. The audit exists to make a moment of the process observable — to force a stop, review the evidence, and either sign off or block. It cannot substitute for a culture where engineers, product managers, legal, compliance and executives all understand that the model is a decision the organization is making, not a tool that decides on its own.

The credit scorer of this course will be audited again in eighteen months. The bank has committed to publishing a redacted version of that next audit — the finding table, the metrics, the corrective actions — as the AI Act encourages high-risk providers to do. That transparency is what a responsible deployment finally looks like: not a claim of perfection, but a public willingness to be checked.

Summary​

  • The audit grid has ten sections, each with a Yes / No / Partial verdict and an evidence pointer.
  • The credit scorer had three consequential findings: a retained proxy without justification (material), missing recourse (blocker), stale DPIA (blocker).
  • A pre-deployment checklist (intended use, probe test, fairness threshold, model card during training, SHAP on prototype, recourse rehearsal, DPIA before pilot) prevents most findings.
  • An audit is a snapshot; a responsible-AI program is a process with periodic public audits.

Next: the course recap and the 40-question exam covering all ten modules.