AI Search APIs Add Citations. You Still Need to Audit Them.

A useful evaluation starts with one ordinary request: “What changed in US restrictions on advanced-chip exports during 2024?” It asks for a dated change, sits across government documents and specialist reporting, and creates several ways for a citation system to look convincing while getting the evidence wrong.

The newly practical part is integration. A developer can send that question to a citation-enabled search API and receive generated prose plus linked sources, often without separately operating a crawler, search index, ranking system, and answer model. That compresses a substantial retrieval stack into one call.

It does not make the source set trustworthy by default. In hands-on inspection, the important unit was not the blue citation marker beside a sentence. It was the chain underneath it: the claim, the cited URL, the page that URL resolved to, the passage on that page, and the original publisher behind any republished copy.

That chain should become the evaluation record.

Save more than the displayed answer

For the chip-export query, store the complete API response before rendering it. The record should include the prompt, answer text, citation metadata, retrieval or publication dates when supplied, request time, model or endpoint identifier, and every returned URL before and after redirects.

This adds storage and some engineering work, but it makes failures reproducible. A user-facing citation can change destination after a publisher updates its redirects; a result can disappear from the next run; an API may revise its answer while retaining similar prose. Without the raw response, the team cannot tell whether the model changed, the web changed, or the application dropped metadata during rendering.

Resolve each URL with the same network conditions your production service uses. Follow redirects, record the final status code, and retain the page title plus a short relevant extract where licensing permits. A status of 200 means the server returned a page, not that the cited article still exists. Some sites serve a generic error page with a successful status, a failure commonly called a soft 404.

The chip-export row now has an auditable shape: one answer, several claims, several citations, and a saved page state for each link. That is enough to start scoring what the interface hides.

Count publishers, not citation markers

Source diversity is the first trap. Five citation markers can lead to five domains carrying substantially the same wire report, corporate announcement, or rewritten article. Syndication, meaning publication of the same underlying report by multiple outlets, inflates the visible source count without adding independent evidence.

Normalize the source set by ownership and origin. Strip tracking parameters from URLs, map regional subdomains to the same publisher, compare headlines and opening paragraphs for copied text, and label the earliest identifiable source. If three pages reproduce one report, count one reporting origin and three accessible copies. Both numbers matter, but they answer different questions.

For a starter evaluation, score diversity from 0 to 2. Give 0 when every material claim depends on one origin, 1 when multiple origins appear but one dominates, and 2 when independent sources support the answer and at least one is a primary document where the query calls for it. A primary document is the original record, such as a government rule, court filing, company release, or research paper.

The chip-export prompt should push the system toward government publications because the question asks what rules changed. News coverage can explain impact and industry response, yet it should not substitute for the text or official summary of the rule. A result set composed entirely of commentary may read well while leaving the central factual claim one editorial rewrite away from its source.

Paywalls create the opposite distortion. Retrieval systems often favor pages they can fetch and parse, which can exclude original reporting behind subscription controls while elevating accessible rewrites. Do not automatically penalize an API for failing to quote text it cannot access. Do record whether the returned set appears to route around a named or discoverable original report, because the answer may otherwise present derivative coverage as independent confirmation.

Test whether each page supports its claim

Entailment is the next check: whether the cited material logically supports the claim attached to it. A page can be authoritative, current, and topically relevant without proving the sentence the model wrote.

Split the answer into atomic claims, each containing one assertion that can be checked independently. “The rule expanded controls and took effect in a particular month” contains at least two claims. If the citation documents the scope change but gives a different effective date, a single marker beside the sentence conceals a partial failure.

Use a three-level score. A 2 means the cited passage directly supports the claim. A 1 means it supports only part of the claim or requires a reasonable inference. A 0 means the page is irrelevant, contradictory, or too vague to establish the assertion.

Save the supporting passage beside the score; otherwise reviewers will repeatedly search long pages and make inconsistent judgments.

Citation placement deserves a separate flag. Some APIs attach several links to a paragraph rather than mapping one source to one claim, so a reviewer cannot tell which page supposedly supports a number, date, or causal statement. Mark the claim unsupported when no returned source establishes it, even if another citation in the paragraph discusses the same broad topic.

This is where the chip-export row earns its place in the set. Rules can be announced, published, amended, and enforced on different dates. A generated answer may combine those events into one smooth sentence, while its citation supports only the announcement. The page is real.

The date may be real. The relationship between them is still wrong.

Make freshness a property of the claim

Freshness is not the age of the newest link. It is whether the evidence is current enough for the claim being made.

Add an “as of” date to every time-sensitive prompt, then record the newest relevant source and whether a later primary document supersedes it. The 2024 chip-export question has a closed date range, which makes it useful for regression testing: later commentary may improve context, but it should not silently introduce events outside the requested period.

Stable facts need a different rule. An older standards document can remain the best evidence for a technical definition, while yesterday’s blog post merely repeats it. Score freshness as 0 when the answer relies on evidence that was outdated by the prompt’s cutoff, 1 when the timing is unclear, and 2 when the source is current for that specific claim. Do not reward recency on its own.

Dead-link testing belongs in the same scheduled run. Check every citation at evaluation time and again after a fixed interval, following redirects and detecting login walls, removed pages, domain parking, and soft 404s. A product can pass at launch and decay later because citations are web dependencies, not permanent footnotes.

Build a small set that exposes retrieval bias

A 40-prompt starter set is enough to establish a repeatable review habit without pretending to be a universal benchmark. Allocate 10 prompts to recent changes, 10 to topics where original reporting commonly sits behind a paywall, 10 to questions that should resolve to primary documents, and 10 to stable facts that should resist recency bias.

Write expected source characteristics rather than one approved answer. For the chip-export row, that might require at least one relevant government source, separation of publication and effective dates, and no claim that depends solely on duplicated copies of one article. This keeps the test usable when pages change while preserving the behavior that matters.

Run the set against every endpoint or configuration under consideration. Report claim-level entailment, independent source origins, freshness failures, and link failures separately; one blended score lets a system with many reachable but weak citations cancel out a smaller number of serious unsupported claims.

The cost is mostly review time. Automated checks can resolve URLs, cluster near-duplicate pages, compare dates, and highlight sentences without citation markers. A person still needs to judge whether a passage proves a claim, especially where the answer compresses legal language, attributes causation, or combines multiple dates. For low-stakes discovery, sample that review.

For a product presenting current policy, health, financial, or compliance information, inspect every high-impact claim before treating the output as publishable.

The fallback is straightforward. If a response fails the source rules, show search results instead of synthesized prose, narrow the answer to supported claims, or label the result as unverified and send it to review. Citation-enabled APIs make the first draft newly cheap to build. The source audit determines whether that draft is fit to display.

Questions people ask

Does a citation mean an AI answer is fact-checked?

No. It means the system returned a link associated with some part of the answer. The page may discuss the topic without supporting the exact claim, and several citations may trace back to one syndicated report rather than independent evidence.

How many prompts should an AI search evaluation include?

A 40-prompt set is a practical starting point for one product workflow, provided it covers recent events, paywalled-origin reporting, primary documents, and stable facts. Expand it from production failures rather than adding broad trivia that does not resemble user requests.

Can dead-link checking be automated?

Most of it can. A scheduled job can follow redirects, log status codes, detect repeated error text, and flag pages that now require authentication, but a successful response still needs occasional human inspection because servers can return generic or irrelevant pages with a 200 status.

Should an app hide answers that fail citation checks?

For a high-impact claim, hiding or narrowing the answer is safer than displaying unsupported prose with authoritative-looking markers. Lower-stakes products can fall back to a list of links, disclose that the answer is unverified, or route a sample to human review.