Give an AI Python and It Gains a Calculator and an Attack Surface

Consider one deployment test: an employee uploads `invoices.csv`, containing 10,000 invoice rows, and asks an AI assistant to calculate the net amount, identify duplicate invoice numbers, and return a CSV containing the exceptions.

A model working alone can describe the right method. It may even produce a convincing total if some rows fit inside its context window, the bounded amount of text it can process at once. Yet its answer remains generated text. There is no guarantee that it parsed every row, applied the same rule to each value, or preserved enough evidence to reproduce the result.

Give the same model a sandboxed Python environment and the workflow changes. The model writes code, the sandbox runs it against the uploaded file, and the model reads structured output before answering. That documented architectural change makes whole-file calculations, schema checks, chart generation, and file conversion practical, but it also turns a text interface into a system that can consume memory, open files, launch processes, and potentially contact other machines.

The `invoices.csv` test is therefore about more than answer quality. It is the point where model evaluation becomes infrastructure review.

The model-only answer is cheap but difficult to verify

Without execution, the assistant might suggest loading the CSV, grouping rows by invoice number, summing a numeric column, and separating duplicates. That is useful when the user needs instructions. It is weaker when the user needs the resulting exception file.

Large language models generate tokens, meaning fragments of text selected from statistical patterns. They are good at writing a Python expression such as `df["amount"].sum()`, but generating that expression is different from running it over 10,000 rows. Asking the model to perform the arithmetic in prose consumes context, hides intermediate operations, and can produce an answer whose formatting looks more certain than its calculation.

A tool-enabled response adds several observable steps. The service stores the upload, gives the model a file reference, accepts generated code, executes that code in an isolated environment, and returns standard output, errors, or created files. The model can then repair a malformed column name or report that the amount field contains text values instead of silently forcing an answer.

That retry loop costs more than a model-only response. It adds at least one tool call, sandbox startup time, compute usage, and often another model turn after execution. A failed script can add further turns. The gain is narrower but important: the total and duplicate set can come from deterministic code, while the executed script and input hash can be retained for review.

For the invoice workflow, execution earns its cost because the requested deliverable is a computed file. It would be wasteful for drafting an email about the invoices, where no calculation or file transformation is required.

A sandbox must be a disposable room, not a shared server

A sandbox is an isolated computing environment with restricted access to the host system. Isolation is the control that lets the model run untrusted code without inheriting the application server’s permissions.

Start each `invoices.csv` job in a fresh container or virtual machine and destroy it when the run ends. The input should appear on a read-only mount, while a separate empty working directory receives generated files. Do not mount a user’s home folder, a shared upload directory, source code, cloud credentials, or the host’s container socket.

A random job identifier should separate concurrent runs, and storage quotas should stop one job from filling the underlying disk.

The process should run as an unprivileged user with no route to host administration interfaces. Container isolation alone is not a complete security boundary if the runtime is old, misconfigured, or allowed privileged access, so the execution layer also needs patched images, reduced operating-system capabilities, and system-call filtering where the platform supports it.

For an initial invoice calculator, a defensible starting envelope might allow 60 seconds of wall-clock time, less CPU time than that, 512 MB of memory, a small process limit, and a storage quota sized for the uploaded file plus its result. Those numbers are an example configuration, not universal thresholds. A video-processing tool needs a different envelope; a CSV aggregation that regularly exceeds it probably needs a purpose-built data service rather than a more permissive general sandbox.

Timeouts need a visible fallback. If the Python job crosses its limit, kill the full process tree, retain the exit reason, and tell the model that execution timed out. Letting the model claim a partial result after termination defeats the purpose of running the calculation.

Deny the network before debating an allowlist

The invoice job does not need internet access. Its network policy should therefore deny all inbound and outbound connections, including access to private address ranges, cloud metadata endpoints, and internal service names.

This matters even when the uploaded CSV looks harmless. A cell can contain instructions aimed at the model, while generated code can mistakenly or deliberately attempt to send invoice data to an external endpoint. Network denial breaks that route without requiring the system to judge whether every instruction is malicious.

Some execution jobs genuinely need remote data. In that case, route outbound traffic through a controlled proxy and allow exact destinations, methods, and, where practical, URL paths. Resolve domain names through the proxy, block redirects to unapproved hosts, cap response sizes, set connection deadlines, and log the destination plus byte count. A broad rule allowing all HTTPS traffic is network access with better branding, not a narrow control.

Credentials require an even tighter design. Do not place reusable API keys in environment variables that arbitrary Python or shell code can read. A brokered tool, which performs one approved operation outside the sandbox, can fetch a specific record without exposing the underlying credential. That makes authenticated data access practical while keeping general code execution credential-free.

Packages and shells widen the permission set

Python’s standard library already reads files, starts subprocesses, and opens network sockets, although operating-system controls can block the latter two. Third-party packages add supply-chain risk and make runs harder to reproduce if versions change between jobs.

For `invoices.csv`, build a small image containing the approved parsing library and pin each dependency to a reviewed version. Disable package managers during the run. If the model asks to install a missing package, return a structured denial rather than briefly opening internet access, because installation can execute package code and can pull a different artifact tomorrow.

Shell access deserves a separate decision. Python is enough to parse the file, calculate totals, and write the exception report. A shell adds command chaining, process management, and access to every binary in the image; those capabilities are useful for compiling software or converting media, but they add little to this case. Leave the shell unavailable, remove unnecessary executables, and expose a narrower Python runner.

This is also where product scope should win over model ambition. If users only need five approved spreadsheet operations, typed functions for those operations are easier to secure and audit than arbitrary code. General execution becomes worth the added surface when requests vary enough that predefined functions cannot cover them.

Bound what comes back to the model

Execution output can become an attack and a cost problem. A script can print the entire 10,000-row invoice file, emit an endless error stream, or create a large archive that consumes storage and model context.

Set separate limits for standard output, standard error, individual artifacts, total artifacts, and file count. The model usually needs a compact summary such as row count, column names, validation failures, aggregate values, and the path of the generated exception file. It does not need every row copied into the prompt.

Created files should stay outside the chat renderer until the service checks their size and type. Serve downloads with safe content-disposition headers, scan formats covered by the organization’s malware controls, and avoid rendering generated HTML or SVG inline. For the invoice run, return a CSV download and a textual summary; if the file exceeds the artifact limit, stop and ask the user to narrow the request rather than silently truncating financial rows.

The run log is part of the answer

A model message saying “calculated with Python” is not an audit trail. Record the input file hash, generated code, sandbox image identifier, installed package versions, resource limits, start and stop times, exit status, truncated stdout and stderr, artifact hashes, model identifier, and any approval event. Keep access to those logs narrower than access to the chat because code and errors may reproduce sensitive input.

Return to the concrete test. A reviewer should be able to establish that `invoices.csv` was mounted read-only, that the job had no network route, that the approved image parsed all 10,000 rows, and that the downloaded exception file corresponds to the recorded artifact hash. The model’s prose is then an explanation layered over evidence, not the only evidence available.

That standard also clarifies the adoption decision. Enable sandboxed execution for workflows where users need calculations or transformed files and where the organization can operate the isolation, limits, patching, and logs. Keep model-only responses for explanation and drafting. Use fixed functions instead of arbitrary code when the action set is small.

Questions people ask

Is sandboxed

Python more accurate than a model-only answer?

For arithmetic over a complete file, executed Python can produce a reproducible result from explicit code, while a model-only response may infer or approximate the calculation in text. Python does not fix bad column definitions, missing rows, or incorrect business rules, so the service should expose validation errors and retain the script.

Should an AI code sandbox have internet access?

Not for the `invoices.csv` workflow. Deny network access by default; when a task requires remote data, use a proxy with exact destination rules, response limits, deadlines, and logs, or replace open networking with a brokered function that performs one approved request without exposing credentials.

How long should model-generated code be allowed to run?

Set a workload-specific wall-clock, CPU, memory, process, storage, and output budget. A 60-second wall-clock limit and 512 MB of memory can be an initial test envelope for CSV work, but teams should tune from observed jobs and move consistently heavier tasks to dedicated services.

Is a container enough to run AI-generated code safely?

No single boundary is enough. A container should be disposable, unprivileged, patched, stripped of host mounts and credentials, constrained by operating-system controls, and placed behind a deny-by-default network policy. The service also needs package restrictions, output handling, termination of the full process tree, and logs that connect each artifact to its run.