A HIPAA Agreement Does Not Prove an AI Summary Is Accurate

Consider one concrete workflow: a clinician records a visit, an AI system turns the audio or transcript into a draft after-visit summary, and the summary is released through the patient portal. Before deployment, the health system signs a business associate agreement, or BAA, with the vendor.

That signed document matters. It does not show that the summary preserved the clinician’s instructions, distinguished a stopped medication from a current one, or kept uncertainty attached to a possible diagnosis. A vendor can meet its contractual duties for safeguarding protected health information, or PHI, while its model produces an unsafe sentence.

The distinction is easy to lose during procurement because “HIPAA compliant” often appears beside security questionnaires, encryption claims and a willingness to sign a BAA. HIPAA does not create a general accuracy certification for software, and a BAA is not a product approval. It allocates specific responsibilities for health data shared between regulated parties.

The contract answers a privacy question

The HIPAA Privacy Rule permits a covered entity, such as a health system, to disclose PHI to a business associate after obtaining “satisfactory assurance that the business associate will appropriately safeguard the information,” under 45 CFR § 164.502(e)(1)(i). The required contract must also “establish the permitted and required uses and disclosures” of that information under 45 CFR § 164.504(e)(2)(ii)(A).

A BAA typically addresses what the vendor may do with PHI, which safeguards it must apply, how it reports certain incidents, how subcontractors are bound, and what happens to the data when the relationship ends. The Security Rule separately requires satisfactory assurances that a business associate will appropriately safeguard electronic PHI.

Those are enforceable privacy and security obligations, not a promise that generated clinical text is correct. The contract may say the vendor can use a transcript to create the after-visit summary and cannot reuse the transcript outside the agreed purpose. Unless another provision says otherwise, it may remain silent about how often the model omits a medication change, invents a follow-up interval or assigns a symptom to the wrong speaker.

There is no contradiction. The vendor could encrypt the transcript in transit, restrict employee access, preserve audit logs and delete customer data on schedule, yet still run a model that converts “do not restart the medication” into “restart the medication.” In that failure, the PHI may have remained protected throughout. The clinical meaning did not.

A procurement team should therefore ask for two different records. The first is evidence that the data arrangement satisfies the organization’s privacy and security requirements, including an executed BAA where HIPAA requires one. The second is a clinical safety file tied to the exact summary workflow. One cannot substitute for the other.

Accuracy belongs to the intended workflow

An AI summary does not have one context-free accuracy rate. Performance depends on the input, the output format and what happens next.

For the after-visit workflow, the input might be a raw recording, a speaker-labeled transcript or structured notes from the electronic health record. Each route creates different failure modes. Noisy audio can corrupt drug names before the language model sees them. A transcript can misattribute a family member’s medical history to the patient.

Structured data can conflict with what the clinician said during the visit, leaving the model to reconcile sources without a reliable rule.

The output also matters. A draft visible only to the clinician carries a different exposure from text sent automatically to the patient. A concise patient summary can omit qualifiers that would remain visible in a full note, while a longer summary can add plausible connective language unsupported by the source.

Testing should reproduce the intended path rather than scoring isolated model responses. The evaluation set needs representative encounters and known difficult cases, with authorized clinical reviewers comparing each summary against its source. Reviewers should classify errors that affect the workflow, including medication status, dose or frequency, allergy information, follow-up instructions, negation, speaker attribution and unsupported additions. A generic similarity score can reward fluent overlap while missing the sentence that changes care.

Acceptance criteria should be written before the pilot produces favorable examples. For high-consequence fields, an organization may decide that any unsupported instruction blocks automatic release, even if the rest of the summary reads well. Lower-consequence wording problems may be editable. The threshold is a governance choice tied to patient exposure, not a number the vendor can define alone.

Local testing costs staff time. Clinicians must review source material, adjudicate disagreements and document why an error matters; the work becomes more expensive when specialties use different abbreviations or when the deployment includes multiple languages. Skipping that work does not remove the cost. It moves the cost into production, where corrections, patient calls and incident review are harder to contain.

Human review needs a real control surface

Returning to the portal workflow, “human in the loop” is meaningful only if the clinician can inspect the evidence before release. A checkbox at the bottom of a polished summary is weak control when the reviewer cannot jump to the relevant transcript passage, see which facts came from the medical record, or identify text generated without direct support.

The interface should place the draft beside its source and make edits attributable to a reviewer. High-risk content can receive targeted checks, such as requiring confirmation of medication changes and follow-up timing before the release button becomes available. If the source is incomplete or the model fails, the fallback should be the existing manual documentation path rather than a silent retry that produces different wording.

That design adds time to each encounter. It may still save effort if the draft is usually useful, but the organization should measure total review time and correction burden instead of counting only the seconds required to generate text. Automation bias, the tendency to accept machine output because it looks finished, can erase the safety value of nominal review.

The same principle applies after launch. Logs should identify the model configuration, prompt or template, source records used, generated output, reviewer edits and release status, subject to the same access controls that protect other PHI. When a vendor changes the model behind an application programming interface, or API, previously collected results may no longer describe the deployed system. Change notice and regression testing therefore belong in the product arrangement, even though HIPAA does not supply them as an accuracy regime.

Procurement needs a second set of commitments

The BAA should remain focused on HIPAA responsibilities. Product and service terms can address the separate operational questions: the intended use, support for safety investigations, notice before material model changes, access to useful logs, retention of evaluation artifacts and the customer’s ability to suspend automated release.

A claim that the tool is “for administrative use” deserves comparison with the real workflow. If patients receive the output and use it to follow medication or follow-up instructions, the practical exposure is not resolved by a label in sales material. Product classification under Food and Drug Administration rules can also depend on intended use and functionality; an executed BAA neither establishes nor eliminates medical-device obligations. Organizations need qualified review for their circumstances.

NIST’s AI Risk Management Framework offers voluntary guidance for mapping, measuring and managing AI risks. It can help structure an evaluation, but it is not a HIPAA accuracy requirement and does not become enforceable merely because a vendor cites it. A customer can incorporate specific controls into a contract or internal policy. That contractual step, not the framework’s publication, creates the operative commitment between the parties.

For the after-visit summary, a defensible release gate is concrete: the BAA and security review cover the data relationship; representative clinical testing covers the model in the intended workflow; the interface gives an accountable reviewer access to the source; and monitoring can connect a reported problem to the generated text, configuration and approval event. If one of those records is missing, a “HIPAA-ready” badge does not fill the gap.

This analysis is an explainer, not professional, medical or legal advice.

Questions people ask

Does signing a

BAA make an AI medical-summary vendor HIPAA compliant?

A BAA documents required assurances and allocates duties for permitted uses, safeguards and other handling of PHI. It is not a government certification of the vendor’s entire operation, and it says nothing about summary accuracy unless separate contract language creates that commitment.

Can a

HIPAA-compliant AI summary still contain a dangerous error?

Yes. Privacy safeguards can work while the model drops a negation, confuses speakers or adds an unsupported instruction. Clinical testing and a review interface must catch those failures before the summary reaches the patient or enters another care workflow.

What should a health system test before releasing AI summaries?

Test the complete local workflow with representative source material, including difficult audio, conflicting records and high-consequence instructions. Authorized clinical reviewers should compare outputs with sources, record meaningful error types, measure correction burden and confirm that the manual fallback works when the system fails.

Should

AI-generated summaries be sent to patients automatically?

A BAA does not answer that policy question. Automatic release needs evidence that the system meets predefined safety criteria for the exact patient-facing use, plus monitoring and a reliable way to stop release after a model change or detected failure; otherwise, require review before publication.