Skip to content
Pilum

Multi-Service Monorepo

Pilum is built for monorepos with many services. This guide covers project layout, service selection, dependency ordering, failure handling, parallelism and build cache warming.

my-project/
├── .gitignore                # also read by Pilum for discovery
├── .pilumignore              # exclude examples, fixtures, etc.
├── .pilum.yml                # optional project config (webhooks, compose)
├── go.mod                    # or go.work, package.json + lockfile, ...
├── _templates/               # Dockerfiles, shared by services
│   └── go
├── services/
│   ├── api-gateway/
│   │   ├── pilum.yaml        # template: go
│   │   └── main.go
│   ├── user-service/
│   │   ├── pilum.yaml
│   │   └── main.go
│   └── worker/
│       ├── pilum.yaml
│       └── main.go
├── shared/                   # shared libraries (no pilum.yaml)
│   ├── auth/
│   └── database/
└── tools/
    └── my-cli/
        └── pilum.yaml        # type: homebrew

Dockerfiles live in _templates/ at the project root, not in each service directory. A service picks one with template: <file>. Pilum runs docker build from the project root with -f ./_templates/<template> and passes the service’s directory as --build-arg SERVICE_NAME=<path>, so one Dockerfile serves every service in that language. pilum init scaffolds _templates/<language> for you.

services/api-gateway/pilum.yaml
name: api-gateway
type: gcp-cloud-run
project: my-project
region: us-central1
registry_name: services
template: go          # → ./_templates/go


build:
  language: go
  cmd: go build -o ./dist/app .
  env_vars:
    CGO_ENABLED: "0"
    GOOS: linux
    GOARCH: amd64

Always set type:, because it picks the recipe. With only provider: gcp, Pilum falls back to the first GCP recipe, which builds and pushes but never deploys.

Service Discovery

Pilum finds every pilum.yaml under the current directory, up to 4 levels deep, skipping paths matched by .gitignore and .pilumignore:

Terminal window
$ pilum list
Found 4 services:


  • api-gateway
      Provider: gcp
      Project:  my-project
      Region:   us-central1
      Service:


  • user-service
      Provider: gcp
      Project:  my-project
      Region:   us-central1
      Service:


  • worker
      ...

Both flat (api/pilum.yaml) and nested (services/workers/email/pilum.yaml) layouts work, as long as each pilum.yaml is within the depth limit. Service names must be unique across the repo.

Selecting Services

Every pipeline command (deploy, build, publish, push, dry-run) runs on all services unless you narrow it:

Terminal window
# By name: space- or comma-separated
pilum deploy api-gateway worker --tag=v1.0.0
pilum deploy api-gateway,worker --tag=v1.0.0


# By glob (quote it so the shell doesn't expand it)
pilum deploy 'api-*' --tag=v1.0.0


# By provider
pilum deploy --provider=gcp --tag=v1.0.0


# Only services with git changes, plus their dependents
pilum deploy --only-changed --tag=v1.0.0
pilum deploy --only-changed --since=v0.9.0 --tag=v1.0.0

Filters are applied in order: names, then --provider, then --only-changed. A name that matches nothing prints a warning with a suggestion. If you named services and none matched, the command fails.

Only changed services

--only-changed keeps services with changed files in their own directory since --since, plus uncommitted and untracked files. The default --since is the default branch (main or master). Then it adds every service that depends on one of those, transitively:

Terminal window
$ pilum deploy --only-changed --tag=v1.0.0 --debug
DEBUG: Service database has direct changes
DEBUG: Service api included (depends on changed service)
● Found 2 service(s) with changes (1 direct, 1 via dependencies)

A change in shared/, outside every service directory, doesn’t mark any service as changed. If services must redeploy when shared code changes, deploy them explicitly. For the right --since value in CI, see Environments & CI.

Service Dependencies

Use depends_on to order services:

services/api/pilum.yaml
name: api
type: gcp-cloud-run
project: my-project
region: us-central1
registry_name: services
template: go
build:
  language: go
  cmd: go build -o ./dist/app .
depends_on:
  - database-migrations
services/worker/pilum.yaml
name: worker
# ...
depends_on:
  - api
  - database-migrations

Pilum builds a dependency graph and groups services into waves by depth. Services in a wave run in parallel, and waves run one after another. Waves apply to every step tagged build, deploy or execute. Push steps (publish to registry) always run flat in parallel, because pushing an image doesn’t depend on another service.

━━━ Step 4/5: deploy to cloud run ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Wave 1/3:
  ✓ database-migrations   3.2s
  Wave 2/3:
  ✓ api                   2.9s
  Wave 3/3:
  ✓ worker                3.1s

Waves are used only when at least one selected service declares depends_on. --no-deps turns them off and runs every step flat.

If you deploy a subset whose dependencies aren’t selected, those dependencies are not built or updated. The summary says so:

  ⚠ 1 dependency NOT built or updated (not part of this run):
     worker → api
     Include them explicitly if you expected a fresh build.

Failure handling

Failures are handled at two levels:

  1. Within a step, a failure skips only that service’s dependents in later waves. Independent services in the same step run to completion:

    ━━━ Step 1/5: build binary ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
      Wave 1/2:
      ✗ database-migrations
      ✓ billing
      Wave 2/2:
      ○ api      (dependency database-migrations failed)
      ○ worker   (dependency database-migrations failed)

  2. Across steps, once any step has failed, Pilum skips all remaining steps for every service. Only steps marked always_run in the recipe still run, such as the npm recipe’s manifest restore. In the example above, billing built successfully but is not pushed or deployed in this run.

So a monorepo deploy is all-or-nothing per step: if one service’s build fails, nothing reaches the deploy step. The command exits non-zero and reports the error from the failed step.

Circular dependencies

pilum check does not detect cycles. At deploy time Pilum prints:

✗ Dependency order could not be resolved: circular dependency detected involving: [api worker]
✗ Deploying in declaration order — dependent services may run before what they depend on.

Any step that uses waves then fails with the same cycle error. Run pilum dry-run after changing depends_on to catch this before a real deploy.

Parallel Execution

Pilum sizes the worker pool per step:

  • Build steps are limited by CPU and memory. Workers = cores ÷ cores per build (default 2), capped by 75% of RAM ÷ the heaviest language’s build memory. Each build gets GOMAXPROCS = cores ÷ workers. Docker build steps are sized from the Docker daemon (on Docker Desktop, the VM).
  • Push, deploy and execute steps mostly wait on the network and get one worker per service, up to 16.
Terminal window
pilum deploy --tag=v1.0.0                  # automatic sizing
pilum deploy --max-workers=2 --tag=v1.0.0  # at most 2 at a time, every step
pilum deploy --max-workers=1 --tag=v1.0.0  # one at a time

Set build.resources.cpu and build.resources.memory on services that are heavier than their language default. Details are in Build Cache Warming → Parallelism.

Build Cache Warming

When many services share dependencies, parallel builds on a cold cache each compile the same packages. Before the first build step, Pilum runs one warm command per toolchain and workspace, for example go build over all the Go services’ packages, or pnpm fetch in a temporary copy of the lockfile. The builds then only compile their own code.

━━━ Warming build caches ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  ✓ go (19 services, auto)    60.1s

On Torch’s 19 Go services, this cut a cold-cache build from 490s to 69s. Warming is automatic for Go, Node, Python, Rust, JVM, .NET and PHP services that have a build.cmd, when two or more share a workspace. It is skipped when no build step runs. Turn it off with --no-warm or build.warm: false. See Build Cache Warming.

Mixed Providers

A single monorepo can deploy to different providers. Each service declares its own type:

services/api/pilum.yaml
name: api
type: gcp-cloud-run
project: my-gcp-project
region: us-central1
registry_name: services
template: go
build:
  language: go
  cmd: go build -o ./dist/app .
services/lambda-handler/pilum.yaml
name: lambda-handler      # also the CloudFormation stack name
type: aws-lambda
project: my-project
region: us-east-1
# Runtime, memory and timeout live in the SAM template (template.yaml) in this directory
tools/cli/pilum.yaml
name: my-cli
type: homebrew
description: My command-line tool
license: Apache-2.0
homebrew:
  project_url: https://github.com/my-org/my-cli
  tap_url: https://github.com/my-org/homebrew-tap
  token_env: GH_TOKEN
build:
  language: go
  version_var: github.com/my-org/my-cli/cmd.version

Deploy them all together:

Terminal window
pilum deploy --tag=v1.0.0

Pilum runs step 1 of every service’s recipe, then step 2, and so on. A service whose recipe has fewer steps has nothing to do in the later ones.

Multi-Region Services

A service can deploy to several regions:

services/global-api/pilum.yaml
name: global-api
type: gcp-cloud-run
project: my-project
registry_name: services
template: go
regions:
  - us-central1
  - europe-west1
  - asia-east1
build:
  language: go
  cmd: go build -o ./dist/app .

Pilum expands this into one instance per region:

Terminal window
$ pilum list
Found 3 services:


  • global-api (us-central1)
      Provider: gcp
      Project:  my-project
      Service:


  • global-api (europe-west1)
      ...
Terminal window
# All regions
pilum deploy global-api --tag=v1.0.0


# One region
pilum deploy "global-api (us-central1)" --tag=v1.0.0

Excluding Services

Use .pilumignore to exclude directories:

.pilumignore
# Development only
examples
fixtures
test-*


# Archived services
archived/
*-deprecated

.gitignore is read too (disable that with --no-gitignore). See Ignore Files for the full syntax.

Naming Conventions

Consistent names make globs and filters useful:

PatternExampleSelect with
{domain}-{type}user-api, user-worker'user-*'
{type}-{name}api-gateway, worker-email'worker-*'

Don’t put environments in names (prod-api, staging-api). Use one service with an environments block and --env instead. See Environments & CI.

CI/CD Tips

# Push to main: deploy what changed since the previous push
- uses: actions/checkout@v4
  with:
    fetch-depth: 0
- uses: SID-Technologies/pilum-action@v1
  with:
    command: deploy
    tag: ${{ github.sha }}
    flags: --only-changed --since=${{ github.event.before }}
# Pull requests: validate and preview
- run: pilum check
- run: pilum dry-run --only-changed --since=origin/${{ github.base_ref }} --tag=pr-${{ github.event.pull_request.number }}

Restore the language cache (for example actions/setup-go with caching on) so the warm phase is near-instant on CI runners. A complete workflow is in Environments & CI.

Next Steps