Build Cache Warming

When Pilum builds several services in parallel, it first fills the shared build cache once. This phase is called warming. It runs before the first build step, takes a few seconds when the cache is already warm, and saves most of the build time when it is cold.

Why warming exists

Go, Cargo, npm, pip and most other toolchains keep a shared cache that parallel builds can use safely. Sharing is not the same as deduplicating, though. If ten Go services start building at the same moment on an empty cache, each build misses on the same shared packages and compiles them itself. The cache ends up correct, but the shared work was done ten times.

A warm run compiles or downloads the shared dependencies once. The builds that follow find them in the cache and only compile their own code.

Measured on a real monorepo

Torch has 19 Go services built in parallel. On a 10-core Mac with Pilum’s default sizing (5 workers, GOMAXPROCS=2 per build) and a cold Go cache:

	Wall time
Without warming	490s
With warming	69s (60s warm + 9s builds)

With a warm cache, the warm phase takes about a second, so there is little reason to turn it off.

How the warm phase works

• Runs once, before any build. It runs after the Docker preflight check and before step 1.
• All languages at once. Each distinct warm command is a group. All groups run concurrently, and the first build step starts when every group has finished.
• Only when a build step will run. If no build-tagged step runs for a service, for example under --only-tags=deploy, that service is not warmed. pilum push and deploy-only runs skip the phase entirely.
• Uses the build’s environment. The warm command gets the same environment as the service’s first build step: build.env_vars plus the step’s own env. A cache filled for one GOOS/GOARCH misses for another, so the environment has to match.
• No retries. The timeout is the longest build-step timeout among the group’s services.
• Best effort. If a warm command fails, Pilum marks the group failed and prints Warming failed; continuing with cold builds. The pipeline keeps going and the exit code is unaffected. If something is really broken, the builds will report it.

Output during a deploy:

━━━ Warming build caches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ✓ go (19 services, auto)    60.1s
  ✓ pnpm (4 services, auto)   11.4s

Automatic groups are labelled auto. Groups from build.warm are labelled with the language (or the command’s first word) and the warm_dir, if one is set.

Automatic warming

If a service doesn’t set build.warm, Pilum looks for a toolchain it recognizes and plans the warm command itself. Pilum only does this when it can derive the command without guessing, and only when the command leaves the project tree untouched or does something the build would do first anyway.

Toolchain	Detected by	Workspace root	Warm command	Touches the project?
Go	language: go, or go.mod in the service dir	nearest go.work, else nearest go.mod	go build <flags> ./svc-a ./svc-b ...	No. Multi-package builds discard output; a single package adds -o /dev/null
pnpm	language: node/bun, or package.json in the service dir; then pnpm-lock.yaml	nearest directory with a lockfile	pnpm fetch --store-dir <pnpm store path>	No. Runs in an isolated copy
Bun	bun.lock / bun.lockb	same	bun install --frozen-lockfile --ignore-scripts	No. Isolated copy
Yarn 2+	yarn.lock plus .yarnrc.yml (or __metadata: in the lockfile)	same	yarn install --immutable --mode=skip-build	No. Isolated copy
Yarn 1	yarn.lock	same	yarn install --frozen-lockfile --ignore-scripts --non-interactive	No. Isolated copy
npm	package-lock.json	same	npm ci --ignore-scripts --no-audit --no-fund	No. Isolated copy
uv	language: python, or pyproject.toml/requirements.txt/setup.py in the service dir; then uv.lock	nearest uv.lock, poetry.lock or requirements.txt	uv sync --frozen --no-install-workspace --quiet	No. Isolated copy
pip	requirements.txt (no uv.lock)	same	pip download --quiet -r requirements.txt -d .pilum-download (pip, pip3 or python3 -m pip)	No. Isolated copy
Cargo	language: rust, or Cargo.toml in the service dir	nearest Cargo.lock	cargo fetch --locked [--target <triple>]	No. Download only; --locked fails rather than rewrite Cargo.lock
Gradle	language: java/kotlin, or a build.gradle(.kts)/settings.gradle(.kts)	nearest settings file, else nearest build file	./gradlew (or gradle) -q --no-configuration-cache --init-script <tmp> pilumResolveAll	Writes .gradle/ and build/, which the build writes first anyway. Best effort
Maven	language: java/kotlin, or pom.xml	outermost reactor pom.xml that lists the service as a module	./mvnw (or mvn) -B -q dependency:go-offline	Marker files under target/. Best effort
.NET	language: dotnet, or a *.csproj/*.fsproj/*.vbproj	nearest *.sln/*.slnx, else nearest project	dotnet restore <solution>, or one dotnet restore <dir> per service when there isn’t exactly one solution	Writes obj/, which every build does first
Composer	language: php, or composer.json; needs Composer 2.5+	nearest composer.lock	composer install --download-only --no-scripts --no-plugins --no-interaction --quiet	No. Primes the cache only

These are recognized but not warmed automatically, because they have no download-only mode. Set build.warm if you want them warmed:

• Poetry (poetry.lock)
• Bundler (Gemfile.lock). bundle cache writes vendor/cache into the project.

C and C++ have no toolchain entry and are never warmed automatically.

Some details by toolchain:

• Go passes only the build.flags that change compiled packages: asmflags, gcflags, mod, race, tags, trimpath. Link-only flags such as ldflags don’t affect the cache, so they’re left out. Services with different compile flags warm separately.
• Cargo fetches per target. Services with different build.flags.target values warm separately.
• pnpm is passed the store path that pnpm store path reports for the project. The isolated copy may sit on another filesystem (tmpfs on Linux CI), and pnpm keeps one store per filesystem.
• Gradle and Maven prefer the project wrapper (gradlew, mvnw) over an installed tool. If a Java project has both Gradle and Maven files, it is warmed as Gradle.
• Missing tools are skipped. If the package manager isn’t on PATH, that group isn’t warmed and its builds run as usual.

Detection rules

A service is warmed automatically only when all of these hold:

1. It has a host build. build.cmd is set. A Dockerfile-only service (no build.cmd) has no host build to speed up.
2. A build step will run for it in this invocation.
3. The toolchain resolves.
   • If build.language is set, it picks the candidate toolchains. Aliases are normalized: golang becomes go; nodejs, javascript, js, typescript and ts become node; csharp, c#, fsharp, f# and .net become dotnet; c++ and cxx become cpp; rb becomes ruby.
   • Without a language, a toolchain counts only if its manifest is in the service directory itself. A Node app inside a Go monorepo has a go.mod above it, and that says nothing about how the app builds.
   • An unknown language resolves to nothing, so nothing is warmed by guesswork.
4. The workspace root is inside the repository. The search for go.work, lockfiles and solutions walks up from the service directory but stops at the git repository that holds your working directory, or at the working directory itself outside a repo. A lockfile in an unrelated parent directory is never used.
5. At least two services share the group. Services are grouped by toolchain, workspace root, build environment and variant (Go compile flags, Cargo target). A group of one is skipped: warming a lone service is the same work its build would do, just earlier.

Isolation for Node and Python

npm, Yarn and Bun can’t download without installing into the project, pnpm fetch writes node_modules/.pnpm, and uv and pip create a .venv or download directory. So Pilum runs these commands in a temporary copy of the files the install reads, then deletes the copy. Only the global cache keeps the result.

Toolchain	Files copied (relative to the workspace root)
Node (all package managers)	every lockfile; .npmrc, pnpm-workspace.yaml, .pnpmfile.cjs, .yarnrc, .yarnrc.yml, bunfig.toml; every package.json (workspace members included); everything under .yarn/releases, .yarn/plugins, patches, .yarn/patches
uv	uv.lock, uv.toml, .python-version, every pyproject.toml
pip	every requirements*.txt and constraints*.txt, so -r and -c includes resolve

Files that don’t exist are skipped. Directories such as node_modules, .git, dist, build, .venv and vendor are not searched.

Explicit warming with build.warm

Set build.warm to run your own command instead of the automatic one:

services/api/pilum.yaml

name: api
type: gcp-cloud-run
project: my-project
region: us-central1
registry_name: services
template: go

build:
  language: go
  cmd: go build -o ./dist/app .
  warm: go build ./...
  warm_dir: services        # optional; default is where pilum runs
  env_vars:
    CGO_ENABLED: "0"
    GOOS: linux
    GOARCH: amd64

How explicit warm commands behave:

• One run per distinct command. Services with the same warm, warm_dir and build environment share a single run. Give every service in a module the same warm and they warm once.
• No minimum group size, and no build.cmd requirement. An explicit command always runs if the service has a build step in this invocation.
• warm_dir is relative to the directory you run pilum from, normally the project root. Without it the command runs there.
• The command runs through the shell, like build.cmd, with the service’s build environment.
• It runs in the project. Unlike automatic Node/Python warming, an explicit command is not isolated. If it writes node_modules or .venv, that’s your choice.

Use the same env_vars and build tags the build uses, so the cache entries the warm run creates are the ones the build looks up.

Turning warming off

Scope	How
One service	build.warm: false in its pilum.yaml. This disables both automatic and explicit warming for it.
One run	pilum deploy --no-warm (also on build, publish, push, dry-run)

build:
  language: node
  cmd: npm run build
  warm: false

Previewing with dry-run

pilum dry-run shows the warm phase and the exact command for each group:

pilum dry-run --tag=v1.2.0

━━━ Warming build caches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ◌ go (2 services, auto) warm build cache
      → go build ./services/api ./services/worker
  ◌ pnpm (3 services, auto) warm build cache
      → (in a copy of pnpm-lock.yaml, bun.lock, ..., apps/web/package.json from .: pnpm fetch --store-dir /Users/me/Library/pnpm/store/v10)

━━━ Step 1/5: build binary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

The file list for isolated groups is shortened here. Explicit commands with a warm_dir are shown as (cd <dir> && <cmd>). With --json, each group appears in steps as an entry with "step": "warm build cache" and the group’s services joined by commas.

Parallelism

Pilum sizes the worker pool separately for each step, based on what the step does. It prints the plan before the step runs:

● Using 5 workers (host: 10 CPUs ÷ 2 per build, 32768MB RAM → 24 by memory, 19 tasks; GOMAXPROCS=2)

CPU-bound steps are steps tagged build, plus untagged steps, which are treated as CPU-bound to be safe.

• Cores per build is the largest build.resources.cpu among the services in the run. The default is 2, because go build, BuildKit and bundlers are all internally parallel.

• Workers = cores ÷ cores per build.

• Memory cap: workers are also limited to 75% of RAM ÷ the largest per-build memory. Per-build memory is build.resources.memory (MB), or a default for the language:

  Language	Default build memory
  python	512 MB
  go, c, php, ruby (and unknown)	1024 MB
  node, bun	1536 MB
  rust, cpp, dotnet	2048 MB
  java, kotlin	3072 MB

• GOMAXPROCS: each build gets cores ÷ workers. Pilum doesn’t override a GOMAXPROCS that is already set in the step env, in build.env_vars, or in your shell.

• Docker steps: if a step runs docker, Pilum sizes it from the Docker daemon’s CPUs and memory instead of the host’s. On Docker Desktop that means the VM, which is usually smaller than the machine.

Network-bound steps are steps tagged only push, deploy or execute. They get one worker per task, up to 16.

--max-workers N overrides the count for every step. For CPU-bound steps, GOMAXPROCS becomes cores ÷ N.

Use build.resources when a service is heavier than its language default:

build:
  language: go
  cmd: go build -o ./dist/app .
  resources:
    cpu: 4        # cores one build uses
    memory: 3072  # MB one build needs

Warm commands run outside this pool: every group runs at once, and Pilum doesn’t set GOMAXPROCS for them.

Warming in CI

Warming matters most on cold runners, where every CI job starts with an empty cache. There are two ways to deal with that, and they combine well:

1. Restore the language cache before Pilum runs. On a restored cache the warm phase finishes in about a second and the builds are fast.
2. Let warming handle cache misses. When the cache key changes (say go.sum changed), the warm phase compiles the shared dependencies once instead of once per service.

GitHub Actions example for Go:

- uses: actions/setup-go@v5
  with:
    go-version-file: go.mod
    cache: true                 # restores GOCACHE and the module cache, keyed on go.sum
    cache-dependency-path: "**/go.sum"

- uses: SID-Technologies/pilum-action@v1
  with:
    command: deploy
    tag: ${{ github.sha }}

For Node, actions/setup-node with cache: pnpm (or npm/yarn) does the same for the package manager’s store. See Environments & CI for a complete workflow.

Tips:

• Keep GOOS/GOARCH and other build env identical between CI runs, or the restored cache won’t match what the build looks up.
• Warming only speeds up host builds (build.cmd). Dependencies compiled inside a Dockerfile use Docker’s layer cache instead.

Next Steps

• Build Configuration — build.cmd, warm, warm_dir, resources
• Multi-Service Monorepo — waves, selection and failure handling
• Environments & CI — running Pilum in GitHub Actions
• CLI Commands — --no-warm, --max-workers and other flags