GCP Managed Instance Groups

The gcp-mig-container recipe ships new container versions to a managed instance group (MIG) that already exists. Each release builds and pushes the image, creates a new instance template for that release, and rolls the group onto it without dropping capacity.

Pilum does not create the MIG, load balancer, health check, firewall rules or Cloud NAT. Set those up once with Terraform, a script or the console; Pilum only ships new versions.

Use this recipe for workloads that need a VM rather than Cloud Run: a fixed outbound IP through Cloud NAT, long-lived connections, host networking, or a proxy that sits inside a VPC.

Prerequisites

1. Google Cloud SDK installed and authenticated
2. Docker installed and authenticated to Artifact Registry
3. An existing MIG whose instance template runs Container-Optimized OS with a startup script that meets the metadata contract below
4. Permission to create instance templates and update the MIG, plus permission to act as the VMs’ service account

# Authenticate
gcloud auth login
gcloud auth configure-docker us-central1-docker.pkg.dev

# Confirm the group exists and see which template it runs
gcloud compute instance-groups managed describe egress-proxy-mig \
  --zone us-central1-a --project my-gcp-project

How the Container Runs

GCP retired the container startup agent (gce-container-declaration, used by gcloud compute instance-templates create-with-container). GCP stopped creating VMs from those templates on July 31, 2026. Pilum refuses any template that still contains gce-container-declaration metadata.

Instead, the template runs the container the way Google now recommends: a Container-Optimized OS VM whose startup-script reads its configuration from instance metadata and calls docker run. On each release, Pilum copies the MIG’s current template and changes only the metadata it manages:

Metadata key	Set by Pilum	Holds
container-image (or mig.image_metadata_key)	Every release	The image to run. The key must already exist in the template.
container-env	When env_vars or mig.env_vars are set	KEY=VALUE lines
container-secrets	When secrets are set	NAME=projects/<p>/secrets/<s>/versions/<v> lines. References only, never secret values.

Everything else in the template, including the startup script itself, is copied unchanged.

A startup script that meets the contract:

#!/bin/bash
# Runs on every boot. Pilum updates the metadata it reads in each release's template.
set -euo pipefail
md() { curl -sf -H 'Metadata-Flavor: Google' "http://metadata.google.internal/computeMetadata/v1/instance/$1"; }

export HOME=/home/app && mkdir -p "$HOME"   # COS: the root filesystem is read-only
IMAGE=$(md attributes/container-image)
ENV_FILE=$HOME/container.env
umask 077
{ md attributes/container-env || true; echo; } > "$ENV_FILE"

# Secrets are read from Secret Manager with the VM's service account,
# which needs roles/secretmanager.secretAccessor on them.
SECRETS=$(md attributes/container-secrets || true)
if [ -n "$SECRETS" ]; then
  TOKEN=$(md service-accounts/default/token | tr -d '\n ' | sed -E 's/.*"access_token":"([^"]+)".*/\1/')
  while IFS='=' read -r name ref; do
    [ -n "$name" ] || continue
    value=$(curl -sf -H "Authorization: Bearer $TOKEN" "https://secretmanager.googleapis.com/v1/$ref:access" \
      | tr -d '\n ' | sed -E 's/.*"data":"([^"]+)".*/\1/' | base64 -d)
    printf '%s=%s\n' "$name" "$value" >> "$ENV_FILE"
  done <<< "$SECRETS"
fi

docker-credential-gcr configure-docker --registries="${IMAGE%%/*}"
docker rm -f app 2>/dev/null || true
docker run -d --name app --restart=always --network host --env-file "$ENV_FILE" "$IMAGE"

Env values and secrets must be single-line, since they pass through an env file. Pilum rejects env values that contain a newline.

Service Configuration

name: egress-proxy
type: gcp-mig-container
project: my-gcp-project
region: us-central1
image: us-central1-docker.pkg.dev/my-gcp-project/my-repo/egress-proxy  # no tag; it comes from --tag
template: egress-proxy.dockerfile

build:
  language: go
  cmd: "CGO_ENABLED=0 go build -o ./dist/app ."

env_vars:                         # -> container-env
  PORT: "3128"
secrets:                          # -> container-secrets: name, name:version, or a full path
  UPSTREAM_TOKEN: upstream-token:latest

mig:
  name: egress-proxy-mig
  zone: us-central1-a             # or region: for a regional MIG
  template_base: egress-proxy-tpl # each release creates <base>-<tag>

  # Optional. Unset keeps what the current template has.
  machine_type: e2-small
  subnet: proxy-subnet            # in the MIG's region, or a full resource path for Shared VPC
  external_ip: false              # outbound traffic then goes through Cloud NAT
  network_tags: [egress-proxy]    # firewall rules target these
  service_account: [email protected]

  # Rollout. Defaults shown.
  max_surge: 1                    # 3 for regional MIGs
  max_unavailable: 0
  wait_timeout: 600               # seconds; the deploy fails if the group isn't stable by then

The Dockerfile is _templates/egress-proxy.dockerfile at the project root, as for the other container recipes. See Build Configuration.

Required Fields

Field	Description
name	Pilum service name
project	GCP project ID
region	GCP region of the image’s Artifact Registry repository
image	Image repository without a tag, e.g. us-central1-docker.pkg.dev/proj/repo/app. The tag comes from --tag.
template	Dockerfile template name, relative to _templates/
mig.name	Name of the existing managed instance group
mig.zone or mig.region	mig.zone for a zonal MIG, mig.region for a regional one. Set exactly one.
mig.template_base	Instance template name prefix. Each release creates <template_base>-<tag>.

Optional Fields

MIG Configuration

VM settings left unset are copied from the MIG’s current template, so a group set up by Terraform or a script keeps its configuration.

Field	Type	Default	Description
mig.image_metadata_key	string	container-image	Metadata key the startup script reads the image from
mig.machine_type	string	-	Machine type for new VMs, e.g. e2-small
mig.network	string	-	VPC network name or resource path
mig.subnet	string	-	Subnetwork name (resolved in the MIG’s region) or resource path, e.g. for Shared VPC
mig.external_ip	bool	-	false removes the VMs’ external IP (egress then needs Cloud NAT); true adds an ephemeral one
mig.network_tags	list	-	Network tags, replacing the template’s. Firewall rules target these.
mig.service_account	string	-	Service account email the VMs run as
mig.env_vars	map	-	Container env vars, merged with top-level env_vars (top-level wins)
mig.template_region	string	-	Create regional templates in this region instead of global ones. Must match the scope of the MIG’s current template.
mig.max_surge	int or percent	1 zonal, 3 regional	Extra instances created during the rollout, e.g. 1 or "20%"
mig.max_unavailable	int or percent	0	Instances allowed to be unavailable during the rollout
mig.wait_timeout	int	600	Seconds to wait for the group to become stable before failing the deploy

A regional MIG’s fixed max_surge must be 0 or at least the number of zones it spans. The default of 3 covers the usual three-zone distribution.

With max_unavailable: 0, an old instance is removed only after its replacement is up, so serving capacity never drops during a rollout.

There is no per-instance static IP, because MIG instances are replaced on every rollout. For a fixed outbound IP, reserve one on the Cloud NAT gateway.

Template Naming

Each release creates an instance template named <template_base>-<tag>:

• The tag is lowercased, and characters GCP rejects (dots, underscores, +) become -. --tag v1.4.2 gives egress-proxy-tpl-v1-4-2.
• The name must be at most 63 characters and start with a letter. Pilum never truncates: two tags that differ only after the cut would map to the same template. Shorten mig.template_base or the tag instead.
• Each release needs its own tag. An empty tag or latest is rejected before anything is built, because reusing a template name would make the rollout a no-op. The tag comes from --tag, or from version: in pilum.yaml when --tag is not given.
• Re-running a deploy with the same tag is safe. If the template already exists with the same image, it is reused. If the MIG already runs it (for example, after a wait timeout), nothing is created and the rolling update is a no-op. If it exists with a different image, the deploy fails and asks for a new tag.

GCP’s CLI cannot copy an instance template, so Pilum copies it itself: it reads the current template, applies the changes above, and inserts the new one through the Compute Engine API using your gcloud credentials.

Env Vars and Secrets

secrets and env_vars must be maps (KEY: value). A list of {name, value} entries is ignored.

env_vars:
  PORT: "3128"
  LOG_LEVEL: info
secrets:
  UPSTREAM_TOKEN: upstream-token                                   # latest version
  DB_PASSWORD: db-password:3                                       # version 3
  API_KEY: projects/other-project/secrets/api-key/versions/latest  # full path

A secret reference may be name, name:version or a full projects/.../secrets/... path. Pilum stores it in container-secrets as a full version resource path; the VM’s service account reads the value at boot.

Pilum only writes container-env and container-secrets when the service sets them. Removing every env var from pilum.yaml leaves the last container-env value in the template. Delete it from the template by hand if you need it gone.

Note

env_vars are also passed to docker build through --build-arg, as for the other container recipes. Don’t put values there that shouldn’t end up in image build history.

Regional Templates

By default Pilum creates global instance templates. If your MIG runs a regional template, set mig.template_region to that region:

mig:
  name: egress-proxy-mig
  region: us-central1
  template_base: egress-proxy-tpl
  template_region: us-central1

The scope must match the MIG’s current template. If it doesn’t, the deploy fails with a message telling you to set mig.template_region to match.

Deployment Steps

1. build binary — Runs build.cmd in the service directory (skipped when cmd is empty)
2. build docker image — Builds _templates/<template> from the project root, tagged <image>:<tag>
3. publish to registry — Pushes <image>:<tag>
4. create instance template — Copies the MIG’s current template as <template_base>-<tag> with the new image, env and secrets metadata, and any VM overrides
5. start rolling update — gcloud compute instance-groups managed rolling-action start-update onto the new template
6. wait for stable — gcloud compute instance-groups managed wait-until --stable; fails the deploy if the group isn’t stable within mig.wait_timeout

The wait for stable step has a 900-second step timeout. Keep mig.wait_timeout below that, or the step is killed before gcloud reports the timeout.

Example Commands

# Deploy a release
pilum deploy egress-proxy --tag v1.4.2

# Preview every command without running anything
pilum dry-run egress-proxy --tag v1.4.2

# Build and push only (no template, no rollout)
pilum publish egress-proxy --tag v1.4.2

# Check which template the group runs
pilum status egress-proxy

# Roll back to the previous template
pilum rollback egress-proxy

Generated Commands

For the configuration above, pilum dry-run egress-proxy --tag v1.4.2 shows the rollout commands:

gcloud compute instance-groups managed rolling-action start-update egress-proxy-mig \
  --version template=egress-proxy-tpl-v1-4-2 \
  --max-surge 1 \
  --max-unavailable 0 \
  --zone us-central1-a \
  --project my-gcp-project

gcloud compute instance-groups managed wait-until egress-proxy-mig \
  --stable \
  --timeout 600 \
  --zone us-central1-a \
  --project my-gcp-project

For a regional MIG, --zone becomes --region. With mig.template_region set, the template is referenced as projects/<project>/regions/<region>/instanceTemplates/<name>.

Rollback

pilum rollback egress-proxy --dry-run   # Show what would change
pilum rollback egress-proxy             # Asks for confirmation
pilum rollback egress-proxy --to v1.4.1 # A tag, or a full template name
pilum rollback egress-proxy --yes       # No prompt (required in CI)

Rollback rolls the group onto the newest template in the <template_base> family that is older than the one it currently runs, using the same surge settings as a deploy, then waits until the group is stable. Running it again steps back one more template. --to accepts a tag (turned into <template_base>-<tag>) or a template name in the family.

The previous template is read from GCP, not from local history, so rollback works from any machine. Rollback fails if the MIG’s current template is not in the template_base family; pass --to in that case. See Rollback and Status.

Status and Logs

pilum status describes the MIG and reports:

• Ready when the group is stable and every instance runs the target template, otherwise Updating
• The template the group runs and how many instances are stable, e.g. template egress-proxy-tpl-v1-4-2, 3/3 instances stable

pilum logs is not supported for gcp-mig-container yet. Use Cloud Logging for the MIG’s instances:

gcloud logging read 'resource.type="gce_instance"' --project my-gcp-project --limit 50

Troubleshooting

latest Tag Rejected

✗ service 'egress-proxy' validation failed: gcp-mig-container needs a unique --tag per release (got "latest")

Pass a unique tag, such as a version or git SHA:

pilum deploy egress-proxy --tag "$(git rev-parse --short HEAD)"

Template Uses the Retired Container Agent

template uses gce-container-declaration, the retired container startup agent

The MIG’s current template was created with create-with-container. Replace it with a Container-Optimized OS template whose startup script reads the container-image metadata (see How the Container Runs), roll the group onto it once, then deploy with Pilum.

Missing Image Metadata

template has no container-image metadata; its startup script must read the image from it (set mig.image_metadata_key if it uses another key)

The image key must already exist in the template, which proves its startup script reads it. Add container-image metadata to the template, or set mig.image_metadata_key to the key your script uses.

Template Already Exists With Another Image

template egress-proxy-tpl-v1-4-2 already exists and runs ..., not ...; use a new --tag

The tag was used for a different image. Deploy with a new tag.

Template Scope Mismatch

instance group egress-proxy-mig runs egress-proxy-tpl-v1-4-1 (regional in us-central1), but new templates would be global; set mig.template_region to match

Set mig.template_region to the region of the MIG’s current template, or remove it if the current template is global.

Group Has Several Versions

instance group has 2 versions (canary in progress?); finish or cancel it first

A canary or partial rollout is in progress. Finish or cancel it in GCP, then deploy again.

Template Name Too Long

instance template name "..." is 68 characters; GCP allows 63

Shorten mig.template_base or use a shorter tag.

Group Not Stable in Time

The wait for stable step fails when instances don’t become healthy within mig.wait_timeout. Check the instances’ serial console or Cloud Logging for startup-script errors, and confirm the health check passes. Re-running the deploy with the same tag resumes: the template is reused and the wait runs again.

Next Steps

• Rollback and Status — Roll back releases and check what is running
• GCP Cloud Run — Serverless alternative for HTTP services
• Service Configuration — Full pilum.yaml reference
• CLI Commands — All commands and flags