Running agents in containers

The quickest way to get started is to start with Getting started: Codex.

Then see Tools Reference for details and examples for using refresh.py and launch.py.

The rest of this page has details and additional context on containers.

What’s a container?

A container is a lightweight, isolated collection of software. You build a container image, which includes an operating system and whatever software you need, and then run that image as a container.

Containers add setup overhead, but they provide strong isolation from the rest of the system.

Why containers for running agents?

Without a container, agent tools generally have read access to the entire host filesystem. If you have PII or sensitive information anywhere on the system, it could potentially be exposed to agents.

Codex and Claude Code both support sandboxing. Pi does not. The configuration of sandboxes can be finicky, and varies by harness. In practice, using a sandbox still requires vigilant monitoring of the model’s requests and careful management of allow/deny lists in respective agents’ config files to avoid exposing private information. With no standardized config format, it’s tricky to maintain this.

The primary challenge is that these tools usually need access to standard binaries like git and ls, which are outside the working directory and therefore if we completely restricted all read access, these binaries couldn’t be used.

The solution is to run them in a container. Running inside a container uses the principle of least privilege, narrowing exposure to the mounted workspace and the small set of config paths (here, this is done by launch.py). Tools such as git and ls are installed inside the container, so the agent does not need to read them from the host.

In practice, containers reduce the blast radius of problems caused by agents, at the expense of additional complexity. The entire goal of this repo is to make that additional complexity as easy to deal with as possible.

Because the container already provides isolation, Codex’s own sandbox is disabled inside the container with --sandbox danger-full-access. The container boundary replaces the built-in sandbox rather than layering on top of it.

Another solution would be to run in a full virtual machine (VM) but this ends up being a lot more overhead than a container, and doesn’t work on a remote system like NIH’s Biowulf.

Warning

These containers do NOT lock down network access – they are primarily intended to lock down inappropriate filesystem access.

Podman, Docker, Singularity?

Docker is a popular container runtime. However, it has a restrictive license and may require a paid license for use at NIH.

Podman is a drop-in replacement for Docker with a more permissive license. Unlike Docker, Podman does not need to run containers as root. Install Podman Desktop to use it.

Singularity is a different container runtime. A Singularity container can be built from a Docker or Podman container. It also does not need to run as root. It is already available on NIH HPC; see NIH-specific Biowulf’s Singularity page.

How are the images created?

This repo uses GitHub Actions to automatically build images on each change to the code and tests those images (to the extent that it can, without real credentials to test models). The workflow also runs on a daily schedule, so the images are rebuilt regularly even when the code hasn’t changed. This picks up new releases of Codex, Claude Code, and Pi, as well as upstream package and base-image updates.

Note

You should NOT update the harnesses yourself, since as soon as you exit the container you’ll lose those changes.

The main workflow builds a Podman container using the Dockerfile as the specification (which, among other things, includes installation of Codex, Claude Code, and Pi). It saves this as a Docker Archive tarball, which is then passed to Singularity to convert it into the Singularity Image Format (SIF).

When this happens on code in the main branch, both images are pushed to the GitHub Container Registry (GHCR).

GitHub Actions publishes the Podman image to GHCR with these tags:

• sha-<git sha> for every build

• claude-<version>, codex-<version>, pi-<version> pinning each harness to a specific version

• claude-latest, codex-latest, pi-latest pointing at the newest build for each harness

• latest (the newest overall build) on main

The Singularity (SIF) image is published to the -sif GHCR package with the same set of tags.

latest tags

Because the images are rebuilt daily, the overall latest tag changes every day even when nothing you care about has changed. To avoid forcing a fresh image pull every new day you run launch.py, the workflow keeps the per-harness *-latest tags stable:

• A <harness>-<version> tag is only pushed the first time that version is seen. If it already exists in the registry, it is left untouched.

• <harness>-latest points to the <harness>-<version> tag of the built harness version (in practice, this is always the latest), so its digest only moves when that harness’s version actually changes.

This is why launch.py defaults to the per-harness *-latest tag for the harness you run (e.g. codex-latest for codex): you only re-pull when your harness has a new version, not every day.

There is a trade-off here. If there’s a new claude version, but codex didn’t change, the image with the new claude version gets other system libraries updated as well. But the still-the-same codex version does not. So the codex version would be the latest for codex, but might have “stale” system libraries.

In practice, these harnesses update very frequently (it’s uncommon to go a week without an update!), so the system libraries are not expected to get stale. You can always pass --tag latest to launch.py to always use the newest overall image instead.

The GitHub Actions container workflow builds linux/amd64 images only. Podman automatically handles this different architecture on macOS. There is a performance hit, but the limiting factor of these harnesses is model calling, not CPU usage.

The GitHub Actions workfow first builds and tests the Podman image, then derives the version tags by running the built container and reading claude --version, codex --version, and pi --version. The Singularity phase then converts that same tested Podman image into a SIF artifact.

The workflow also sets org.opencontainers.image.source to the GitHub repository URL so the GHCR package stays linked to the repository and inherits its permissions and visibility.

Running containers without launch.py

You can use the containers outside the context of launch.py like this to get a bash shell, from which you can start one of the agents:

podman run --rm -it ghcr.io/nichd-bspc/llm

singularity exec ghcr.io/nichd-bspc/llm-sif bash

Warning

This will not mount the credentials properly.

Singularity will automatically mount your entire home directory unless you use --no-home and will expose all env vars unless you also use --cleanenv.

Consider using the output of launch.py --dry-run shell as a starting point for composing your own commands, since that shows all of the mounts and environment variable exports needed.

Terminology

Throughout these docs we use the terms local, remote, host, and native.

• Local: the machine where you are logging in with a web browser, for example a laptop or desktop

• Remote: a host in a data center, such as Biowulf, without that browser-based flow available

• Host: the system running Podman/Docker/Singularity

• Native: running an agent tool directly on the host rather than inside a container

For example:

Machine	Running	Native?	Local?	Host
Mac laptop	Codex in Podman container	containerized	local	macOS
Biowulf	Codex in Singularity container	containerized	remote	Linux
Mac laptop	Codex installed on macOS	native	local	N/A
Biowulf	Codex installed on Linux	native	remote	N/A

Login model

This section explains why refresh.py exists.

In browser-based single sign-on flows like those used here, the browser must be able to redirect to a specific localhost port that a tool is listening on in order for the tool to detect that login was successful and then save a local file to persist that information. For example:

• For Codex, codex login opens a browser to https://auth.openai.com/log-in, then waits for a localhost redirect to a specific port and when it receives it, saves credentials to ~/.codex/auth.json

• For Claude, aws sso login opens a browser to the configured page (e.g., NIH-specific https://nih.awsapps.com/start), then waits for a localhost redirect to a specific port and when it receives it, saves credentials under ~/.aws/sso

This does not work cleanly inside a container. The container does not have a GUI (and therefore no browser). If you paste the login URL into a browser running on the host, the browser redirects to the host’s localhost rather than the container’s localhost. So the callback never reaches Codex inside the isolated container… and it waits indefinitely.

The same issue exists on remote systems. If you run codex login on a remote system, it helpfully prints a URL to visit. If you paste that into a local browser and log in, the website redirects to your local machine. But Codex is still listening inside the container on the remote machine. The redirect never reaches the remote, let alone inside the container on the remote, so login cannot complete there either.

Port forwarding and tunneling can work around this, but it gets awkward. Copying the relevant credential files is simpler.

refresh.py automates this. Locally, it is just running codex login and aws sso login (but only if you’re not already logged in). It knows what files need to be transported to the remote host (see Configuration and credential files for these) and takes care of the rsync commands for that as well.

For AWS/Bedrock specifically, refresh.py does not copy the ~/.aws/sso token cache to the remote. Instead it creates ~/.aws/credentials.json with current short-lived role credentials on the host and syncs ~/.aws/config and ~/.aws/credentials.json (the llm-export profile and its credentials file). See How Bedrock credentials reach the container for how that mechanism works.

Because the remote runs entirely off these exported credentials and cannot refresh them independently, they expire on the usual STS schedule (typically ~1 hour) and you must re-run refresh.py to push fresh ones; see the warning in Setting up AWS Single Sign-On.

Note that this mechanism of copying credentials to other systems is also one of the approaches suggested in the Codex auth documentation.

Refreshing credentials without stopping container

You must refresh credentials outside the container (see Login model) but you don’t need to stop the container to do this. See Credentials expired or missing for troubleshooting expired or missing credentials. For example, Claude Code running in a container may not be able to connect due to credentials expiring, but as soon as you use refresh.py and the credentials on the host are updated, Claude Code can see them since they are mounted into the container. While Claude Code does retry attempts, if it has been a while between old credentials expiring and new ones being available then you might need to re-send your latest prompt, or just send “continue” as the next prompt.

Mounts and config

The goal of a container is to isolate it from the rest of the system. But in order to be useful, we need to allow some parts of the system into the container. For example, we need to provide credentials to an agent running inside a container, and we typically want to add the current working directory inside the model so that we can work on the files there.

We can mount files from the host into the container when first starting the container by giving a source location on the host and an intended destination path inside the container. The launch.py script does this automatically for the working directory and the crendential files, and allows you to specify additional paths if needed with --mount.

The host’s home directory is not mounted. Even though Singularity mounts it by default, this setup specifically disables that behavior to reduce exposure.

Only the credentials and config needed for each tool is mounted – unless you call launch.py with shell which will mount them all.

If you regularly mount the same extra paths, set LLM_DEVCONTAINER_MOUNTS to a shell-style list of mount specs using the same format as --mount. For example:

export LLM_DEVCONTAINER_MOUNTS="$HOME/data $HOME/.gitconfig:/home/devuser/.gitconfig:ro"

The user inside the container is called devuser, and the home directory is created at /home/devuser inside the container image.

Conda envs only work on Linux

The container is linux/x86_64 architecture. If the host matches that architecture (like NIH’s Biowulf) you can mount directories into the container using --path-prepend or --conda-env. This is a convenient way to provide development tools, like everything inside a conda environment, inside the container without needing to change the image.

launch.py \
  --conda-env ~/miniconda3/envs/env-to-use \
  --mount ~/data/examples \
  codex

However, if you mount binaries from a macOS ARM64 host to the Linux AMD64 container, they will not run inside the container because of the architecture mismatch. The primary limitation is that the macOS filesystem is not case-sensitive. So common conda packages, like ncurses, that rely on case-senstive filenames, will not install.

This case-insensitivity is on the host macOS side. A container running on a macOS host is still affected by this. So even running conda inside a container to build a linux/amd64 env will fail. Typically tools depending on ncurses – of which there are many – will fail because ncurses itself requires case sensitivity.

There is a workaround, but it’s not straightforward…

Workaround for using conda on macOS

One workaround is to not use a container at all. This is risky since it exposes the rest of the machine. You may want to carefully construct a config file in the local directory (.codex or .claude) with appropriate permissions to lock down the sandbox.

Another workaround is to create a case-sensitive volume on macOS and get it into the running container.

Warning

This workaround is not straightforward, but it’s mostly a one-time setup and it does work.

1. Make a case-sensitive volume (one-time setup).

First, create a new volume on macOS that is case sensitive:

• Open Disk Utility (/Applications/Utilities/Disk Utility)

• Select your APFS container in the sidebar

• Click + (Add Volume)

• Set the format to APFS (Case-sensitive)

• Name it (e.g., devel) and click Add

• The new volume mounts at /Volumes/devel

2. Re-initialize podman (every restart of Podman).

Next, we need to tell Podman about that new volume, and get it mounted in the podman machine. The podman machine is the VM used by Podman to emulate Linux – it’s sort of the parent container of the normal containers we use. We do this by re-making the podman machine, including its default mounts but also our new one (you’ll need to change your volume name to the one you created above):

# Re-initialize the podman machine with our new mount. Needs to happen every
# time Podman Desktop is restarted.
podman machine stop
podman machine rm --force
podman machine init \
  --volume /Users:/Users \
  --volume /private:/private \
  --volume /Volumes/devel:/Volumes/devel
podman machine start

This needs to be re-run every time Podman Desktop restarts, because Podman Desktop uses default mounts that do not include our custom one. There does not appear to be a config file we can change to make this more permanent.

3. Install conda on the case-sensitive volume (one-time setup).

Here is how to create and mount the right directories to be able to install a version of conda to that case-sensitive directory, using the Linux container. This effectively creates a linux/amd64-usable conda installation we can use to create linux/amd64 environments. Note that this method makes separate cache and conda dirs to mount into the container’s home directory to avoid contaminating the host’s directories.

This command:

• creates directories to be used by conda and mounts them

• mounts the case-sensitive volume

• runs the container using shell, and effectively sending an in-line bash script with bash -c ... that performs the Miniforge installation from the Miniforge docs.

• exits the container when done

# install miniforge on case-sensitive volume, using container
mkdir -p ~/.devcontainer/.{cache,conda}
launch.py \
  --mount ~/.devcontainer/.conda:/home/devuser/.conda \
  --mount ~/.devcontainer/.cache:/home/devuser/.cache \
  --mount /Volumes/devel \
  shell \
  bash -c ' \
  curl -fSsL "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" \
  > /Volumes/devel/miniforge.sh && \
  bash /Volumes/devel/miniforge.sh -p /Volumes/devel/miniforge -u -b -s '

4. Create environments (as needed).

Now that conda is installed, for conda env creation and mainentance you can use the following:

# launch a shell for conda maintenance
launch.py \
  --mount ~/.devcontainer/.conda:/home/devuser/.conda \
  --mount ~/.devcontainer/.cache:/home/devuser/.cache \
  --mount /Volumes/devel \
  --path-prepend /Volumes/devel/miniforge/bin \
  shell

If you’re working in a project directory on the regular macOS filesystem and want a case-sensitive env you can let an agent use:

# single command for creating an env
launch.py \
  --mount ~/.devcontainer/.conda:/home/devuser/.conda \
  --mount ~/.devcontainer/.cache:/home/devuser/.cache \
  --mount /Volumes/devel \
  --path-prepend /Volumes/devel/miniforge/bin \
  shell conda create -p /Volumes/devel/proj-env --file requirements.txt

5. Using environments (routine usage).

Now whenever you run an agent, provide that path to --conda-env. In this example for testing, we’re running codex exec to just run that prompt and then exit, but running the agent as normal will allow it to use that conda env.

# routine usage
launch.py \
  --conda-env /Volumes/devel/proj-env \
  codex exec "what version of pandas do I have installed, and what directory is it in?"