Before Signing an AI BAA, Trace Where Patient Data Goes

Take one concrete workflow: a clinician opens a generative AI assistant, pastes a patient’s progress note and laboratory results, then asks it to draft discharge instructions. The clinician reviews the draft before copying it into the electronic health record.

That looks like one request. Technically, it may create several copies of protected health information, or PHI: the prompt in browser memory, an application database entry, a model-provider request, a safety log, an uploaded file in object storage, and a support record if someone investigates a failed response. A business associate agreement, or BAA, sets contractual duties for handling PHI. It does not, by itself, show which of those copies exist.

The hospital’s job before approval is to reconcile two records. One is the contract, including the BAA and service terms. The other is an evidenced data-flow diagram showing where the discharge-summary request travels, who can access it, how long each copy remains, and what happens when deletion fails.

This is an operational checklist, not legal advice. Privacy, security, clinical, procurement, and legal teams should apply it to the hospital’s obligations and the proposed use.

Start with the BAA’s actual coverage

The HIPAA Privacy Rule requires a BAA to establish the business associate’s permitted and required uses and disclosures of PHI. Under 45 CFR 164.504(e), the contract must provide that the business associate will “not use or further disclose the information other than as permitted or required by the contract or as required by law.” It must also address safeguards, reporting, subcontractors, government access to compliance records, and return or destruction of PHI where feasible when the contract ends.

Those are meaningful requirements. They still leave implementation questions open.

For the discharge workflow, mark every service covered by the agreement: the hospital-facing application, the company operating it, any affiliated model provider, and each subcontractor that creates, receives, maintains, or transmits PHI on the vendor’s behalf. Confirm that the product name and deployment mode in the order form match the system clinicians will use. A BAA attached to an enterprise product does not necessarily cover a consumer account, an experimental feature, or a plug-in enabled later.

Read the permitted-use language beside the vendor’s main service terms, privacy notice, product settings, and model-improvement policy. If one document says customer content may improve services while another says PHI is used only to deliver the contracted service, require the vendor to resolve the conflict in signed terms. A settings toggle labeled “do not train” is weaker evidence than a contractual restriction paired with a technical control and a testable log.

Trace the discharge request one hop at a time

Ask the vendor to demonstrate the data path for the actual product configuration, rather than supplying a generic cloud architecture diagram. Begin when the clinician submits the note. Follow the prompt through authentication, the application layer, content filtering, model inference, output storage, monitoring, and support systems, then follow deletion through the same path.

The diagram should identify data categories at each hop. Prompt text and model output are obvious. Uploaded documents, extracted text, embeddings, feedback labels, account identifiers, network addresses, and error traces may also contain PHI. An embedding is a numeric representation used to retrieve related text; it can remain linked to a patient record even when the original document is no longer visible in the interface.

Return to the temporary file. If the clinician uploads a PDF rather than pasting text, establish where the original file lands, whether optical character recognition creates another copy, and whether a malware scanner or document-conversion service receives it. “Temporary” is not a retention period. The vendor should state the deletion trigger, the expected deletion window, what remains in backups, and whether administrators can restore the file after the user deletes it.

Ask for evidence from a controlled test. Submit synthetic patient information, delete the conversation and file, then have the vendor show the corresponding audit events across the application, storage service, and downstream systems. Synthetic data cannot prove that production isolation is correct, but the test reveals whether deletion is an engineered workflow or only a user-interface action.

Separate model training from every other reuse

The phrase “we do not train on your data” is too narrow for approval. Training usually means adjusting model parameters using examples, yet vendors may retain prompts for abuse detection, manual review, product analytics, evaluation, debugging, or the creation of labeled datasets without calling those activities training.

For the discharge request, require a written answer covering the prompt, uploaded note, generated draft, clinician edits, ratings, and support tickets. The answer should distinguish model training from evaluation and service monitoring, name the default behavior, identify any opt-in or opt-out control, and state whether the restriction applies to the hospital-facing vendor as well as its model provider.

Also verify enforcement. Useful evidence includes configuration exports, administrative screenshots, policy identifiers, access-control records, and contract language that prevents the vendor from changing the setting without notice. If staff can copy production prompts into an internal testing tool, the hospital needs controls around that path too. A policy prohibition without access restrictions or reviewable logs depends on every employee remembering the rule during an incident.

De-identification deserves its own line. HIPAA recognizes specific routes for de-identifying health information, but removing names from a prompt does not automatically satisfy them, particularly when dates, rare diagnoses, locations, or free text remain. If the vendor proposes using de-identified customer data, require the method, the party making the determination, the permitted purpose, and controls against relinking records. Do not treat “anonymized” as a technical specification.

Name every subprocessor that can touch PHI

HIPAA requires a business associate to ensure that subcontractors handling PHI agree to the same restrictions and conditions that apply to the business associate. That contractual chain matters, but the hospital still needs to know who is in it.

For this workflow, the list may include the model host, cloud infrastructure provider, document parser, content-moderation service, logging platform, customer-support system, and backup provider. Some vendors will say a service cannot view customer content because it only supplies infrastructure. Record that distinction, but verify whether the service stores encrypted data, manages encryption keys, receives plaintext for processing, or exposes content to support personnel under an escalation procedure.

Require the subprocessor’s function, processing location, PHI categories, retention behavior, and contractual role. Establish how the hospital receives notice of additions or replacements, how much review time it gets, and what remedy exists if it objects. A public web page that changes without an alert gives procurement a record, not a control.

Geography belongs in the same review. HIPAA does not create a general rule that all PHI must remain in the United States, but state law, hospital policy, payer commitments, or other contracts may impose tighter limits. The vendor should identify primary processing regions, backup regions, remote support locations, and whether requests can fail over elsewhere.

Test access, logs, and incident handling

The HIPAA Security Rule requires covered entities and business associates to implement technical safeguards, including access control, audit controls, integrity protections, authentication, and transmission security. A compliance statement does not establish that the AI product exposes enough evidence for the hospital to investigate the discharge request.

Confirm how users authenticate, whether the product supports the hospital’s identity provider, and how access ends when a worker changes roles or leaves. Review privileged access separately. Vendor engineers and support staff should receive narrowly scoped access, with approval, expiration, and logging that the hospital can examine after an incident.

Then inspect the audit trail. It should connect the clinician’s action to a timestamp, account, patient or encounter reference where appropriate, model request, file event, administrative change, and deletion event without placing unnecessary PHI into the log itself. The tradeoff is real: sparse logs make investigations difficult, while verbose logs create another sensitive data store. The vendor should document what it records, who can search it, and how long it retains entries.

Run failure cases against the same workflow. Cancel an upload. Trigger a model timeout. Delete a conversation while processing is underway.

Ask support to investigate a synthetic request. Each test should reveal whether data lands in retry queues, dead-letter queues, screenshots, or ticket attachments, and whether the normal retention rule still applies. A dead-letter queue stores failed processing jobs for inspection or retry; it can preserve payloads long after the clinician sees an error.

Set a release gate the hospital can enforce

Approval should end with written conditions tied to the discharge workflow. Record which fields clinicians may submit, whether file upload is enabled, which model configuration is approved, where output may be copied, and what human review is required before clinical use. If the vendor cannot show deletion of uploaded files, disable uploads and permit only a constrained text field, or keep the product out of PHI workflows.

Assign evidence owners and renewal triggers. The security team can own architecture and access testing, privacy can own permitted uses and retention, clinical leadership can own review requirements, and procurement can track subprocessor notices and product changes. Reassess when the vendor changes its model host, enables memory or retrieval, adds plug-ins, alters retention, or moves processing regions. The signed BAA remains important.

The release decision rests on whether the real data path matches it.

Questions people ask

Does signing a BAA make an AI product HIPAA compliant?

No single contract makes a product or workflow compliant. A BAA assigns required duties between covered entities and business associates, while the hospital and vendor must still implement appropriate safeguards, limit uses and disclosures, configure the service correctly, and operate it according to those terms.

Can a vendor keep prompts if it says they are not used for training?

Possibly, depending on the signed terms and applicable requirements. A no-training statement may still allow retention for security monitoring, debugging, evaluation, support, or legal obligations, so the hospital should obtain separate answers for each purpose, storage location, access path, and deletion schedule.

Must every

AI subprocessor sign a BAA directly with the hospital?

Not necessarily. HIPAA generally requires the business associate to impose applicable restrictions and conditions on subcontractors that handle PHI, creating a contractual chain. The hospital should still identify those subprocessors, understand their functions, and verify that its agreement addresses changes, incidents, return or destruction, and downstream handling.

What should block the discharge-summary pilot from launching?

Block PHI use if the vendor cannot identify where prompts and files persist, establish permitted uses, account for subprocessors, show access and deletion evidence, or support an adequate audit trail. The hospital can test with synthetic data while gaps are resolved, but synthetic testing should not become informal permission to add real patient information.