Skip to main content

Module 7 — Human oversight and recourse

An audit that measures fairness, explains individual decisions and documents the model is still incomplete without a working procedure for the moment the model gets it wrong. Every credit model rejects applicants who would have repaid, and approves applicants who default. That is not a bug of the specific model, it is a property of any classifier that is not perfect. What distinguishes a responsible deployment from an irresponsible one is what happens next: whether a human sees the decision, whether the applicant can contest it, and whether the system learns from the contested cases.

Two flavors of human oversight​

Human in the loop (HITL) means a human validates every decision before it is applied. The model produces a recommendation; a loan officer reviews the file and approves or rejects. The model is a decision-support tool, not a decision-making one. HITL is heavy — a hundred decisions per officer per day is a limit — and it is what regulators require for genuinely high-stakes contexts.

Human on the loop (HOTL) means the model decides autonomously, and humans monitor aggregate behavior — fairness metrics, calibration, drift — and intervene when the aggregate crosses a threshold. HOTL scales; a single team can oversee millions of decisions. It works well when individual decisions are low-stakes or easily reversible, and dangerously when they are not.

The credit scorer belongs to a middle ground. High-value loans (above 20 000 euros or below the confidence threshold of 0.3 to 0.7 on the model's output) go through a loan officer under HITL. Low-value, high-confidence decisions are auto-approved or auto-rejected under HOTL, with a mandatory sampling of one percent of them re-examined by a human every week. Both channels feed the fairness dashboard.

The trap: automation bias​

Placing a human in the loop is not enough. Automation bias is the well-documented tendency of humans to defer to an automated recommendation even when they have information that contradicts it. Aviation studies from the 1990s established the pattern; medical decision-support and financial risk systems have replicated it. The loan officer who is told "80 % probability of default" checks the file with less rigor than an officer who had to form the estimate themselves. They approve or reject as the machine suggests, and the "human in the loop" is now a rubber stamp.

Two counter-measures reduce automation bias. First, do not show the model's recommendation until the officer has recorded their own preliminary assessment. Second, make disagreement observable: the interface shows the officer their disagreement rate with the model, and the fraction of their overrides that were validated by outcomes. An officer who has overridden the model twenty times in a month, with twelve overrides producing good loans, is doing the job. An officer who has agreed with the model on every one of their 400 decisions is not doing the job — they are the extension arm of the model.

The right to an explanation​

Several jurisdictions grant applicants a right to an explanation for automated decisions with significant effects — GDPR Article 22 in the EU (module 9), similar provisions in Brazil's LGPD, in California's CCPA amendments, and in several US state credit laws. The right is not fully specified in most statutes; what has emerged in practice is a two-part explanation:

  • Why was I rejected? The top three or four features that pushed the applicant over the threshold, in plain language, with an indicative weight. This is exactly the SHAP waterfall of module 4, rewritten for a non-expert reader.
  • What would need to change? A counterfactual explanation: the smallest set of changes to the applicant's inputs that would flip the decision. "If your existing debt were below 15 000 euros instead of 22 000 and your employment length above three years instead of one, this application would have been approved." The dice-ml library and the alibi library both generate counterfactuals for tabular classifiers.

A rejection notification that says "our model rejected your application" and stops there is legally and morally insufficient. A notification that gives the top features and a realistic counterfactual is defensible.

A working recourse procedure​

An applicant who disputes the decision must have an accessible path. What "accessible" means matters: a phone number that connects to a queue is not a procedure. The audit prescribes the following, all of which end up in the model card and the customer-facing FAQ.

Step one: request for reconsideration. The applicant writes what the model got wrong. The bank commits to a fifteen-day answer window. The reconsideration is handled by an officer who did not process the original file and who does not see the model's score until after their own assessment.

Step two: independent review. If the applicant is still refused, they can request review by a second officer, who is presented with the file, the two previous assessments, and any documents the applicant has submitted since. This step happens with a thirty-day window.

Step three: correction of the underlying record. If either review finds that an input feature was factually wrong (income mis-recorded, employment status stale), the record is corrected in the source system and a note is attached to the model's training-data audit log.

Step four: aggregate learning. Every contested case, whether the outcome changes or not, is added to a "hard cases" set that the model team reviews quarterly. Systematic patterns — many contested rejections in one region, one occupation category — trigger a targeted audit.

The design of the interface matters​

Small choices in the loan officer's interface have large effects on the aggregate outcome. Showing the model's confidence as a red-yellow-green bar changes officers' judgments differently from showing a probability. Highlighting features that pushed toward rejection primes the officer to see risk everywhere. The audit requires that the interface be tested with the target users before deployment, that the tests be re-run after every interface change, and that the outcome — average time per decision, agreement rate with the model, subgroup differences in override patterns — be logged.

Summary​

  • Human in the loop validates each decision, human on the loop monitors aggregates; the credit scorer combines both by decision value and confidence band.
  • Automation bias turns HITL into a rubber stamp; countermeasures include hiding the model's suggestion until the human commits, and displaying disagreement rates.
  • The right to explanation requires a top-features report and a counterfactual ("what would need to change") for every rejected applicant.
  • A working recourse procedure has independent reviewers, deadlines, source-record correction, and an aggregate feedback loop into future audits.

Next module: the EU AI Act's risk classification, the obligations for high-risk systems, and where a credit scorer sits under this framework.