Assess the Credit Decision, Not Just the AI Model

A lender uses a machine-learning model to score requests for credit limit increases. The model reads account history, repayment behavior, stated income, and other permitted inputs, then sends a risk score to a policy engine, software that applies business rules and thresholds. Some customers receive an increase. Others enter manual review or receive a denial.

The model description might be thorough. It could name the algorithm family, training data period, validation method, accuracy measures, and prohibited attributes excluded from training. None of that, by itself, documents the decision a customer experiences.

The consequential system includes the application form, data checks, model score, policy threshold, manual-review queue, denial notice, customer-service script, and correction process. A valid AI impact assessment has to follow that chain. Otherwise, the team may test the model while missing a stale income field, an overloaded review queue, or a generic denial notice that prevents a customer from correcting bad data.

That distinction matters under existing rules, not merely proposed AI policy. The US Equal Credit Opportunity Act and Regulation B require adverse-action notices to provide “the specific reasons for the action taken,” or explain the applicant’s right to obtain them. The Consumer Financial Protection Bureau has said that creditors cannot use a complex algorithm as a reason for failing to provide specific and accurate explanations.

An impact assessment does not establish compliance, and this walkthrough is not legal advice. It gives product, risk, engineering, and operations teams a decision record that counsel and auditors can interrogate.

Start with the outcome and work backward

Give the assessment a decision name, not a model name. “Credit limit model assessment” is too broad. “Decision to approve, manually review, or deny an existing customer’s requested credit limit increase” states the action and its possible outcomes.

Then draw the decision boundary. For this workflow, it begins when the customer submits the request and ends only after the outcome is delivered, its reasons are recorded, and any challenge reaches a defined resolution. A model output is an intermediate event. It does not become the customer’s outcome until the policy engine interprets it.

Write down the sequence in operational terms: the application captures data; validation checks whether required fields are present; the model produces a score; rules compare that score with approval and review thresholds; a fraud flag may override the route; reviewers handle borderline cases; the notification service generates the result. Name the owner of each step and the system that logs it.

This exercise often finds governance problems before statistical testing begins. If the notification service receives only a final score and no reason codes, the lender may be unable to produce a specific explanation. If reviewers can override a denial but the system records neither the original result nor the override rationale, later testing cannot separate model behavior from human intervention.

Document people through their exposure to the decision

“Affected users” is not a sufficient entry. Describe who encounters the decision, what information the system has about them, and how the outcome changes their options.

For the credit limit workflow, direct subjects are existing customers requesting more credit. People may encounter different error patterns because their account histories are thin, their income changes seasonally, shared accounts obscure who made payments, or source data uses inconsistent names and addresses. Protected-class analysis belongs in the assessment where lawful and appropriate, but demographic categories should not replace these operational exposure paths.

Indirectly affected people may include authorized users or household members who rely on the account. Staff also matter. Reviewers can face a queue shaped by the automated threshold, while customer-service workers may have to explain outcomes without access to the variables that drove them.

Connect each population to a plausible harm. An incorrect denial can restrict access to credit and trigger repeated applications elsewhere. An incorrect approval can expose a customer to unaffordable borrowing. A delayed manual review can make a nominally available appeal useless for a time-sensitive expense.

A vague reason code can block correction even when the underlying account data is wrong.

Do not label all of these “bias risk.” That phrase hides the mechanism. State whether the suspected failure comes from incomplete data, a proxy relationship, threshold selection, feedback from earlier decisions, reviewer behavior, or the notice-generation layer. Different mechanisms require different controls.

Use a decision record that can be tested

A practical assessment can fit into one maintained record if every claim points to evidence. Use these fields:

1. Decision and owner: Name the outcome, accountable business owner, system operator, and approval authority.
2. Decision boundary: Record the starting event, possible routes, final outcomes, and systems involved.
3. Purpose and alternative: State why automation is used and what would happen without it, such as universal manual review or a fixed ruleset.
4. Affected people: Describe direct and indirect populations, including cases with limited or unreliable data.
5. Harm pathways: Connect each failure to an outcome, rather than listing broad risk labels.
6. Controls and fallback: Name thresholds, data checks, human-review rules, shutdown conditions, and the nonautomated route.
7. Notice and appeal: Preserve the reasons delivered, submission channel, response owner, resolution standard, and expected handling period.
8. Evidence and monitoring: Link tests, logs, review samples, complaint records, override analysis, and unresolved limitations.
9. Change triggers: Define which modifications require review or a separate assessment.

The alternative matters because automation has a real tradeoff. Universal manual review consumes staff time and creates a queue; a fully automated route responds faster but can propagate a bad data field at scale. Sending more applications to review may reduce automatic denials, yet the control fails if staffing cannot keep pace and customers wait without a usable status update.

Evidence should match the claim being made. A validation report can show how the model performed on held-out data, meaning records not used to train it. It cannot prove that denial notices contain accurate reasons. For that claim, sample completed decisions, trace each notice back through the policy rule and model reason codes, then check whether a reviewer could reproduce the stated cause from the stored record.

The same discipline applies to appeals. A help-center page proves that a channel exists. It does not show that customers can submit corrections, that staff can change the outcome, or that successful challenges feed back into data-quality work. Keep records of appeal grounds, source-data corrections, response times, outcomes, and repeated failure patterns.

Restrict access and retention because these records contain sensitive financial information.

Make the appeal path part of the system

The fallback for the credit limit decision should be defined before launch. A customer who disputes an income value or account record needs a route to human review, and the reviewer needs authority to correct data, rerun the appropriate step, or issue a different decision with a recorded rationale.

Automation bias, the tendency to defer to a system’s recommendation, can turn that review into theater. Test the appeal path with cases where the automated result is deliberately wrong because an input is stale or mismatched. If reviewers cannot identify the error without seeing a preferred answer first, change the interface or review order.

Appeals also reveal where the decision boundary was drawn too narrowly. A lender might conclude that the model is performing as validated while complaints show that an upstream income field is not refreshing. The model can be technically unchanged and still participate in a newly harmful decision process.

Decide when a new use needs its own assessment

Reusing the same model does not automatically justify reusing the assessment. The deciding issue is whether the decision, affected population, harm pathway, control structure, or appeal route has materially changed.

Suppose the lender takes the limit-increase score and uses it to prioritize debt-collection contacts. That is a new assessment. The model may be identical, but the action, people’s exposure, operational staff, likely harms, notices, and challenge mechanism differ.

A separate assessment is also warranted when the system moves from advising a reviewer to issuing outcomes automatically; when a new product reaches people with materially different data patterns; when an external data source changes what can be inferred; or when the organization removes the review route that previously contained errors. These changes alter the decision architecture.

A routine model retraining may stay within the existing assessment if the purpose, inputs, population, thresholds, outcome routes, and controls remain within documented limits. It still needs change review and fresh evidence. Set those limits in advance, including the performance or disparity conditions that pause deployment, because deciding after a metric moves invites an owner to redefine acceptable risk around the result.

Frameworks can support this work without setting the scope for you. The NIST AI Risk Management Framework is voluntary guidance that organizes governance around functions such as mapping, measuring, and managing risk. It is useful for structuring evidence, but an organization still has to identify the concrete decision and applicable enforceable requirements. A completed framework worksheet does not replace the adverse-action notice, decision log, or functioning correction route in the credit limit workflow.

Questions people ask

Does every model need its own AI impact assessment?

No. Scope the assessment around a consequential decision or workflow. Several models may support one decision, while one model reused for credit approval and collection prioritization may require separate assessments because the outcomes, affected people, harms, and appeal paths differ.

Can a model card replace an AI impact assessment?

A model card can document intended use, training data, evaluation results, and limitations. It usually does not cover policy thresholds, human overrides, notification systems, operational delays, or customer appeals, all of which shape the final decision and need separate evidence.

What evidence should an auditor be able to inspect?

An auditor should be able to trace sampled outcomes from submitted inputs through model output, business rules, overrides, final notice, and any appeal. The record should also link claimed controls to test results, monitoring reports, complaint patterns, and named owners.

When should an existing assessment be reopened?

Reopen it when data, thresholds, automation level, affected population, outcome, or fallback changes beyond documented limits. Create a separate assessment when the same system begins supporting a materially different decision, even if its model code and performance measures remain unchanged.