Coding Agents Can Migrate One Dependency, With Guardrails

The concrete case is deliberately small: a Node.js application with one direct dependency selected for an update, a committed lockfile, an existing test suite, and a clean Git branch. Each coding agent gets the same repository and the same job. Update that dependency, make only required compatibility changes, run the project’s checks, and leave a patch that another engineer can review.

This is a better agent test than asking for a new feature. The desired change already has a boundary, while the repository supplies observable evidence about whether the agent respected it. The dependency declaration should move. The lockfile, which records the exact resolved package graph, may move with it.

Application code should change only if the new release requires an API adjustment. Everything else needs an explanation.

The evaluation described here is a reproducible hands-on fixture rather than a product ranking. It is designed to compare multiple agents under identical conditions without turning a passing test command into a safety claim. That distinction matters because an agent can finish the visible task while weakening the project around it.

Give the agent a contract, not a general request

“Update the dependency and fix anything that breaks” grants too much discretion. It lets the agent rewrite tests, alter compiler settings, replace package-manager configuration, or clean up unrelated code, any of which can make the final test run green without preserving the application’s original constraints.

A bounded instruction should name the dependency, the permitted package manager, and the commands that define the project’s normal checks. It should prohibit toolchain upgrades, broad formatting, unrelated refactors, and changes that suppress warnings or disable validation. The agent may inspect release notes and source files, but it should report uncertainty rather than editing around a failure it cannot explain.

The repository also needs a known starting state. Run its tests before handing it over, record the runtime and package-manager versions, and confirm that Git shows no local modifications. Otherwise, a preexisting failure or a newly generated lockfile can be misattributed to the agent.

That clean branch is the anchor for the whole evaluation. After every run, the reviewer returns to the same evidence: Git status, the full diff, the resolved dependency graph, and the command log. Agent narration is useful for orientation, but it is not proof.

Separate the agent’s work from the judge

A credible evaluation does not allow the agent to define its own success. The agent can install packages and run tests inside its workspace, but an external harness, meaning a script controlled by the evaluator, should rerun the checks after the agent stops.

The harness starts from the submitted patch in a fresh environment, installs dependencies using the committed lockfile, then runs the project’s test, build, lint, and type-check commands where those checks already exist. A small application may not use every category. The evaluator should preserve the repository’s real workflow rather than adding decorative checks that no maintainer normally runs.

The fresh install catches a common class of false confidence. An agent’s working directory may contain cached packages, generated files, or undeclared local state that lets tests pass during the run, while a reviewer or deployment system receives only the patch. Reconstructing the application from version-controlled files tests the artifact that would enter review.

Keep the judge outside the writable repository when possible. If an agent can edit the evaluation script, its configuration, or the expected outputs, then the test measures whether the agent can satisfy a movable target. The same rule applies to network access and secrets: provide only the registry and documentation access required for the migration, and do not place deployment credentials in the environment.

A green test run can conceal the wrong patch

Tests answer a narrow question: the behaviors they exercise still produce expected results. They do not establish that the agent preserved untested behavior, retained project policy, or changed only the files needed for the migration.

Configuration drift is especially easy to miss. An agent encountering a type error may loosen a compiler option. A lint failure can disappear after a rule is disabled. A build problem may be routed around by changing a target environment.

In each case, the suite can pass because the standard moved, not because the dependency migration succeeded.

Lockfiles require similar scrutiny. Updating one direct dependency can legitimately change transitive dependencies, the packages pulled in beneath it, but a different package-manager version may rewrite large portions of the file or resolve a broader graph than intended. The reviewer should compare the old and new dependency trees, check that the selected package moved as requested, and identify every unrelated addition or removal.

Then inspect file scope. Generated artifacts, editor settings, cache files, reformatted source, changed snapshots, and package-manager metadata all create review cost. Some may be required, but the agent must connect each one to the dependency update. “Tests pass” does not explain why an editor configuration appeared in the patch.

This is where the clean branch earns its keep. A short migration should usually produce a diff whose purpose can be understood file by file. If the patch sprawls, rejecting it and rerunning with tighter permissions may cost less than reconstructing the agent’s reasoning from a long transcript.

Score evidence instead of confidence

The useful output is not an agent’s declaration that the task is complete. It is a patch plus an evidence bundle: the files changed, commands run, command results, dependency tree difference, and a concise account of any compatibility edits.

A practical rubric can judge four dimensions without inventing a single composite score. First, correctness: the requested dependency is updated and the application still builds and behaves as covered by the checks. Second, scope: every changed file is necessary or explicitly justified. Third, policy preservation: compiler, lint, test, runtime, and package-manager settings remain intact unless the migration genuinely requires a reviewed change.

Fourth, reproducibility: the evaluator can install and verify the patch from a clean checkout.

Keep those dimensions separate. An agent that produces correct code with an unexplained configuration edit should not be averaged into a pass, because the configuration change is the finding. The same applies to an agent that writes a neat summary but leaves a lockfile that cannot reproduce the tested installation.

Manual inspection still belongs in the loop. The reviewer should read changed application code, check release notes for known migration requirements, and perform one targeted smoke test around the dependency’s role in the application. A smoke test is a brief end-to-end check of a critical path. It covers the gap between a broad automated suite and the exact behavior most likely to have changed.

Orchestration determines the safety boundary

The model is only one component. A supervisor should create an isolated worktree, enforce the command and network policy, capture tool calls, stop runaway retries, and return control before any branch is pushed or pull request is opened. That supervisor is orchestration: software that coordinates the agent’s tools, state, limits, and handoff.

Parallel runs can be useful because different agents may expose different migration requirements, but they also multiply model charges, package downloads, test time, and human review. For a small dependency update, several full runs may cost more attention than the patch warrants. A sensible first deployment uses one agent, one independent verification pass, and human approval; comparison runs belong in evaluation or in unusually consequential migrations.

Wall-clock time is often dominated by dependency installation and the project’s own checks rather than text generation. Spending also depends on how much repository context the model reads and how often it retries failed commands. Record both, along with the reviewer’s minutes, because a cheap autonomous run that creates a large ambiguous diff merely transfers the work downstream.

The safe fallback is plain. If the clean-environment checks fail, the dependency graph moves beyond the approved scope, or a configuration change lacks a release-note-backed reason, do not ask the same run to keep improvising indefinitely. Preserve its log, discard the worktree, and hand the narrow failure to an engineer or restart from the clean branch with a more specific instruction.

A coding agent can therefore own the mechanical middle of this job: inspect, edit, install, test, and prepare. It should not own the acceptance standard. The final artifact remains the small patch on the clean branch, accompanied by enough external evidence that a reviewer can reject it without trusting the agent’s account.

Questions people ask

Are passing tests enough to approve an agent’s dependency update?

No. Tests only cover encoded behaviors, and an agent can make them pass by changing configuration, snapshots, or test code. Approval should also require a clean-environment install, a dependency-tree comparison, full diff review, and confirmation that every changed file belongs to the migration.

Should the agent be allowed to edit tests?

Only when the dependency intentionally changes behavior and the reviewer can trace the new expectation to documentation or an approved requirement. Otherwise, test edits remove an independent constraint. Keep the original suite available outside the agent’s workspace so the submitted patch can be checked against it.

Is it safer to run several coding agents on the same update?

Several runs can reveal divergent approaches and hidden assumptions, but they multiply execution cost and review work. For routine updates, one bounded run with an external verification harness is usually easier to assess. Parallel agents make more sense while evaluating tools or handling a migration with competing compatibility strategies.

What should a coding agent submit for human review?

It should submit a minimal patch, the exact commands it ran, their results, and a short explanation of compatibility changes and unresolved uncertainty. The review package should also expose the dependency graph difference and any generated files, rather than asking the reviewer to infer them from a success message.