GCP Cloud Run Jobs

Deploy batch workloads, migrations and other run-to-completion tasks with Google Cloud Run Jobs. The gcp-cloud-run-job recipe builds and pushes an image the same way as GCP Cloud Run, then creates or updates the job and, optionally, runs it.

Prerequisites

1. Google Cloud SDK installed and authenticated
2. Docker installed and running
3. A GCP project with the Cloud Run and Artifact Registry APIs enabled
4. An Artifact Registry Docker repository in the job’s region

# Install gcloud
brew install google-cloud-sdk

# Authenticate
gcloud auth login
gcloud auth configure-docker us-central1-docker.pkg.dev

# Create the Artifact Registry repository (once per region)
gcloud artifacts repositories create my-repo \
  --repository-format=docker \
  --location=us-central1 \
  --project=my-project

Service Configuration

name: db-migrations
type: gcp-cloud-run-job
project: my-project
region: us-central1
registry_name: my-repo
template: go            # Dockerfile at _templates/go in the project root

build:
  language: go
  cmd: "CGO_ENABLED=0 go build -o ./dist/migrate ./cmd/migrate"

job:
  timeout: "300s"
  memory: 1Gi
  cpu: "1"
  max_retries: 2
  execute_on_deploy: true
  service_account: [email protected]
  cloudsql_instances:
    - my-project:us-central1:main-db

secrets:
  DATABASE_URL: database-url:latest

depends_on:
  - my-api

type: gcp-cloud-run-job is required. Without it, a provider: gcp service resolves to gcp-artifact-registry-image and is never deployed.

Required Fields

Field	Description
name	Job name. Also the image name, unless image_name is set.
type	gcp-cloud-run-job
project	GCP project ID
region	GCP region, e.g. us-central1
registry_name	Artifact Registry repository name. A bare name such as my-repo, not a full host or path.
template	Dockerfile name under _templates/ in the project root

The image is built and named exactly as for Cloud Run services: <region>-docker.pkg.dev/<project>/<registry_name>/<name>:<tag>. See The Dockerfile and Image Naming.

Optional Fields

Job Configuration

Field	Type	Default	Description
job.timeout	string	600s	--task-timeout: maximum run time per task, e.g. 300s, 10m. Always sent.
job.max_retries	int	-	--max-retries: retries per failed task
job.parallelism	int	-	--parallelism: maximum concurrent tasks
job.task_count	int	-	--tasks: total number of tasks
job.cpu	string	-	--cpu per task
job.memory	string	-	--memory per task, e.g. 512Mi, 2Gi
job.service_account	string	-	--service-account: identity the job runs as
job.cloudsql_instances	list	-	--set-cloudsql-instances, entries as project:region:instance
job.execute_on_deploy	bool	false	Run the job (and wait for it) after it is deployed

Apart from job.timeout, a key that is not set sends no flag, so the job keeps its current value or Cloud Run’s default. The other defaults listed in the recipe file are what pilum init writes; they are not applied at deploy time.

job.cloudsql_instances uses --set-cloudsql-instances, so the list replaces the job’s current connections.

Secrets and Environment Variables

secrets and env_vars are maps, exactly as for Cloud Run services:

env_vars:
  LOG_LEVEL: info
  BATCH_SIZE: "500"     # quote numbers; unquoted values are dropped

secrets:
  DATABASE_URL: database-url:latest
  API_KEY: projects/my-project/secrets/api-key/versions/3

• A secret value can be name:version or projects/<project>/secrets/<name>/versions/<version>. The full path is converted to name:version.
• Pilum always states the full set. Each deploy sends either --set-env-vars or --clear-env-vars, and either --set-secrets or --clear-secrets.
• A list-form secrets: block is ignored, which clears the job’s secrets on the next deploy.

Cloud Run Jobs vs Cloud Run Services

Feature	Cloud Run Service	Cloud Run Job
Runs	Continuously (HTTP server)	To completion (batch)
Triggered by	HTTP requests	Manual, scheduler, or on deploy
Scaling	0-N based on traffic	Fixed task count
Use case	APIs, web apps	Migrations, ETL, data processing

Deployment Steps

The gcp-cloud-run-job recipe runs:

1. build binary [build] — Runs build.cmd in the service directory. Skipped when build.cmd is empty.
2. build docker image [build] — docker build from the project root with -f ./_templates/<template>.
3. publish to registry [push] — docker push of the versioned tag.
4. deploy job [deploy] — gcloud run jobs deploy (creates or updates the job).
5. execute job [execute] — gcloud run jobs execute --wait. Only when job.execute_on_deploy: true; otherwise the step does nothing.

Dependency Ordering

Steps tagged build, deploy and execute respect depends_on. When a job depends on a service, Pilum deploys the service in an earlier wave:

migrations/pilum.yaml

name: db-migrations
type: gcp-cloud-run-job
depends_on:
  - database-proxy  # deployed before the migration job

Pilum groups services into waves by dependency depth and runs each wave in turn. If a service in one wave fails, its dependents in later waves are skipped. --no-deps turns waves off.

Execute on Deploy

When job.execute_on_deploy is true, the execute step runs the job and blocks until it finishes:

gcloud run jobs execute db-migrations --region us-central1 --wait --project my-project

A failed execution fails the step, and so the deploy. The step’s timeout is 600 seconds; a job that runs longer than that is reported as failed by Pilum even if it later succeeds in Cloud Run.

When execute_on_deploy is false (the default), the execute step is a no-op. There is no CLI flag that forces an execution; --only-tags=execute still does nothing unless execute_on_deploy is true. To run the job by hand:

gcloud run jobs execute db-migrations --region us-central1 --wait

Caution

pilum publish skips only deploy-tagged steps, so it does run the execute-tagged step. With execute_on_deploy: true, pilum publish builds and pushes the new image and then executes the job that is already deployed, which still runs the previous image. Use pilum publish --exclude-tags=deploy,execute to build and push without running anything.

Example Commands

# Build, push, deploy and (if execute_on_deploy) run the job
pilum deploy db-migrations --tag=v1.0.0

# Deploy without executing, even with execute_on_deploy: true
pilum deploy db-migrations --exclude-tags=execute --tag=v1.0.0

# Update the job definition only (the image for this tag must already exist)
pilum deploy db-migrations --only-tags=deploy --tag=v1.0.0

# Preview commands
pilum dry-run db-migrations --tag=v1.0.0

Generated Deploy Command

For the configuration above, pilum dry-run --tag=v1.0.0 shows (wrapped for reading):

gcloud run jobs deploy db-migrations \
  --image us-central1-docker.pkg.dev/my-project/my-repo/db-migrations:v1.0.0 \
  --region us-central1 \
  --task-timeout 300s \
  --max-retries=2 \
  --cpu 1 \
  --memory 1Gi \
  --service-account [email protected] \
  --set-cloudsql-instances my-project:us-central1:main-db \
  --clear-env-vars \
  --set-secrets DATABASE_URL=database-url:latest \
  --project my-project

gcloud run jobs execute db-migrations --region us-central1 --wait --project my-project

Rollback, Status and Logs

# Point the job back at the image of the previous execution
pilum rollback db-migrations

# Or at a specific image tag
pilum rollback db-migrations --to v0.9.0

# Show the job's state
pilum status db-migrations

# List recent executions
pilum logs db-migrations

• Jobs have no revisions or traffic. pilum rollback reads the job’s executions from Cloud Run, finds the most recent one that ran a different image, and runs gcloud run jobs update db-migrations --image <that image>. Only the image changes; the job keeps its current configuration.
• If no earlier execution ran a different image, pass --to <tag>.
• Rollback does not execute the job.
• pilum logs for a job runs gcloud run jobs executions list. For log lines, use Cloud Logging (see below).

See Rollback and Status.

Troubleshooting

Job fails to execute

Step 5: execute job
  db-migrations  ✗ - execution failed

Check the execution and its logs:

pilum logs db-migrations
gcloud logging read "resource.type=cloud_run_job AND resource.labels.job_name=db-migrations" --limit 50

Timeout exceeded

If tasks hit the task timeout, raise job.timeout:

job:
  timeout: "1800s"  # 30 minutes

The execute step itself times out after 600 seconds. For longer jobs, set execute_on_deploy: false and trigger the job outside the deploy.

Execute step skipped

The execute step only runs when execute_on_deploy is set:

job:
  execute_on_deploy: true

Or execute the job directly with gcloud run jobs execute db-migrations --region us-central1 --wait.

Next Steps

• GCP Cloud Run — Deploy HTTP services
• Rollback and Status — Roll back, check status, read logs
• Service Configuration — Full pilum.yaml reference
• CLI Commands — All commands and flags