Google Cloud Run Jobs & Scheduler

See how to use Google CLI commands to configure Google Cloud Run Jobs and Cloud Scheduler Jobs to execute a Python script on a specified interval from any Google region. The Python script does not use or require an HTTP request.

source: ChatGPT

This is Part Five of a multiple article series on “Building a Data Platform on GCP”. Part One “Building a Data Platform on GCP” defined the functional requirements, and detailed how to install the required software. Part Two “GCP Infrastructure & Authentication” explained how to use Google application default credentials (ADC) to authenticate a user-managed service account. Part Three “Google Cloud Pub/Sub Messaging” showed how to use a Python script to generate and subscribe to the Google Pub/Sub Messaging service. Part Four “Containerization using Docker” covered how to build a local Docker image for a Python script, run it locally, and then push it to Google Artifact Registry (repository).

Google Cloud Run Jobs

Unlike a Cloud Run service, which listens for and serves requests, Cloud Run Jobs only runs its tasks and exits when finished. After you create or update a job, you can execute the job as a one-off, **on a schedule**, or as part of a workflow. You can structure a job as a single task or as multiple, independent tasks (up to 10,000 tasks) that can be executed in parallel. Each task runs one container instance and can be configured to retry in case of failure.

QuickStart — gcp_part5.bat

Everything that follows in this article can be executed in a Windows batch file named “gcp_part5.bat” included in the Github repository. If you execute the batch file “gcp_part5.bat” from within a Windows command window with the current working directory set to the Python virtual environment folder named “gcp”, all that follows will be done for you automatically. Make sure to edit the local variables defined in this file before you run it!

Alternatively, you can execute each of the commands in that follow in a Windows command prompt window manually. The benefit of the details that follow is each command is explained in context.

Deploy Artifact Registry Image to Cloud Run Job

From the prior article “Containerization using Docker”, a Docker image that exists in the Google Artifact Registry will be run on a Google virtual machine (VM) as a Cloud Run Job.

Configure Roles & Enable APIs for Cloud Run Jobs & Scheduler

Before creating a Cloud Run Job, make sure the user-managed service account has the required roles for Cloud Run Jobs and the Scheduler APIs. The role “roles/cloudscheduler.admin” was the least permissive role I could find for Cloud Scheduler.

# Add the role run.invoker for Cloud Run
# gcloud projects add-iam-policy-binding PROJECT_ID --member=serviceAccount:some-name@some-project.iam.gserviceaccount.com --role=roles/run.invoker
$ gcloud projects add-iam-policy-binding data-platform-v0-0 --member=serviceAccount:svc-act-pubsub@data-platform-v0-0.iam.gserviceaccount.com --role=roles/run.invoker

# Add the role roles/cloudscheduler.admin for Cloud Scheduler
# gcloud projects add-iam-policy-binding PROJECT_ID --member=serviceAccount:some-name@some-project.iam.gserviceaccount.com --role=roles/cloudscheduler.admin
$ gcloud projects add-iam-policy-binding data-platform-v0-0 --member=serviceAccount:svc-act-pubsub@data-platform-v0-0.iam.gserviceaccount.com --role=roles/cloudscheduler.admin

# List the roles assigned to the user-manged service account
gcloud projects get-iam-policy PROJECT_ID --flatten="bindings[].members" --format="table(bindings.role)" --filter="bindings.members:serviceAccount:some-name@some-project.iam.gserviceaccount.com"
gcloud projects get-iam-policy data-platform-v0-0 --flatten="bindings[].members" --format="table(bindings.role)" --filter="bindings.members:serviceAccount:svc-act-pubsub@data-platform-v0-0.iam.gserviceaccount.com"

Enable the Cloud Run and Google Scheduler APIs.

# Enable the Cloud Run API
$ gcloud services enable run.googleapis.com

# Enable the Google Scheduler API
$ gcloud services enable cloudscheduler.googleapis.com

Use the Google CLI to make sure that the Google Cloud Run API is enabled.

# List the enabled services
$ gcloud services list

Update service account impersonation.

# gcloud auth application-default login --impersonate-service-account SERVICE_ACCT_EMAIL
$ gcloud auth application-default login --impersonate-service-account svc-act-pubsub@data-platform-v0-0.iam.gserviceaccount.com

Find Docker Image in Google Artifact Registry

Find the Docker image for the Python publishing script in the Google Artifact Registry (repository) so it can be used to create a Cloud Run Job.

List all Docker images in a repository (gcloud help):

# List the Docker images in a repository and include the tag
# gcloud artifacts docker images list LOCATION-docker.pkg.dev/PROJECT/REPOSITORY --include-tags
$ gcloud artifacts docker images list us-east4-docker.pkg.dev/data-platform-v0-0/repo-data-platform --include-tags

The output from the above command will look something like this:

IMAGE                                                                                DIGEST
  TAGS    CREATE_TIME          UPDATE_TIME          SIZE
us-east4-docker.pkg.dev/data-platform-v0-0/repo-data-platform/data-platform-pub  sha256:9bbe2f2d3184e45b22b8e8793ab1efbbec88dd5bf131f502bc2d1bacdde0b874  latest  2024-09-10T22:16:01  2024-09-10T22:16:01  110491769

Use the tag “us-east4-docker.pkg.dev/data-platform-v1–5/repo-data-platform/data-platform-pub” to reference this image in subsequent gcloud commands that need the IMAGE_URL.

Create a Cloud Run Job

Use the Google CLI to create a Google Run Job from the Docker image that resides in the Google Artifact Registry. You can use “gcloud run jobs create” or “gcloud run jobs deploy”.

The argument “ — set-env-vars=”. This sets a Cloud Run environment variable with key ”GCP_RUN_JOBS_REGION” and value the Google Run Jobs location/region location such as “us-east4". It is used by the Python publisher script to assign that region value to the “source=” for the data package.

In this way, the same Python script / Docker image / GCP repository container can be used for Google Run Jobs executed in various regions.

Additionally, environment variables for the PROJECT_ID, TOPIC_ID, SUBSCRIPTION_ID, DATASET_ID, and TABLE_ID can be optionally provided to set those values in the Python scripts. Otherwise, it is necessary to edit those hard coded values in the scripts.

# Create a Google Cloud Run job from the Docker image
# gcloud run jobs create JOB_NAME --image=IMAGE_URL --region=REGION
$ gcloud run jobs create data-platform-pub-run-job --image=us-east4-docker.pkg.dev/data-platform-v0-0/repo-data-platform/data-platform-pub:latest --region=us-east4 --set-env-vars="GCP_RUN_JOBS_REGION=us-east4" --set-env-vars="GCP_RUN_JOBS_REGION=us-east4" --set-env-vars="GCP_PROJECT_ID=data-platform-v0-0" --set-env-vars="GCP_TOPIC_ID=streaming_data_packet_topic"

# Alternatively, use the deploy command to create/update Cloud Run Job from a Docker image
# gcloud run jobs deploy JOB_NAME --image=IMAGE_URL --region=REGION
$ gcloud run jobs deploy data-platform-pub-run-job --image=us-east4-docker.pkg.dev/data-platform-v0-0/repo-data-platform/data-platform-pub:latest --region=us-east4 --set-env-vars="GCP_RUN_JOBS_REGION=us-east4" --set-env-vars="GCP_RUN_JOBS_REGION=us-east4" --set-env-vars="GCP_PROJECT_ID=data-platform-v0-0" --set-env-vars="GCP_TOPIC_ID=streaming_data_packet_topic"

List the Cloud Run Jobs:

# List the jobs (by name)
$ gcloud run jobs list

# List the jobs by JOB URI
$ gcloud run jobs list --uri

The online documentation for gcloud Cloud Run Job can be found here.

Execute a Created Cloud Run Job

First, run the Python script “gcp_data_platform_sub.py” locally so it will that you can see when a message is published by the Python script “gcp_data_platform_pub.py” running in Cloud Run Job.

Use the Google CLI to execute the Google Run Job by name.

# Execute a Google Cloud Run Job
# gcloud run jobs execute my-job --region=REGION
$ gcloud run jobs execute data-platform-pub-run-job --region=us-east4

The output will look something like what is shown below. You can click on the URL provided to see the logs.

C:\Users\[username]\AppData\Local\Google\Cloud SDK>gcloud run jobs execute data-platform-pub-run-job --region=us-east4
OK Creating execution... Done.
  OK Provisioning resources...
Done.
Execution [data-platform-pub-run-job-kd5tt] has successfully started running.

View details about this execution by running:
gcloud run jobs executions describe data-platform-pub-run-job-kd5tt

Or visit https://console.cloud.google.com/run/jobs/executions/details/us-east4/data-platform-pub-run-job-kd5tt/tasks?project=012345678901

Note that output provided a job execution name that is the job name plus the suffix of “-kd5tt” added to uniquely reference the execution.

You can see more information about that specific job execution using this Google CLI command:

# gcloud run jobs executions describe JOB-EXECUTION-NAME --region=REGION
$ gcloud run jobs executions describe data-platform-pub-run-job-kd5tt --region=us-east4

The above output shows the service account for that execution as:

Service account: svc-act-pubsub@data-platform-v0-0.iam.gserviceaccount.com

Additionally, if you inspect the Cloud Run Job in the Google Console, you will see that the identity used by the job is “svc-act-pubsub” under the security tab. The creator of the job is listed as the user “username@gmail.com”.

You can also see the Cloud Run job that just ran by visiting the Google Cloud Run Console, click on the job listed, select an Execution ID, and then click on the “LOGS” tab. The output from the Python script execution will be shown, and it will include the message number.

Create & Execute a Cloud Run Job

You can use the gcloud command “run jobs deploy” to create and run a job immediately rather than separately executing “run jobs create” and/or “run jobs deploy” to execute the job immediately after it has been created.

You can use the argument “ — execute-now” with the “run jobs create” and/or “run jobs deploy” gcloud commands to execute the job immediately after it has been created.

$ gcloud run jobs create data-platform-pub-run-job --image=us-east4-docker.pkg.dev/data-platform-v0-0/repo-data-platform/data-platform-pub:latest --region=us-east4 --service-account=svc-act-pubsub@data-platform-v0-0.iam.gserviceaccount.com --execute-now

$ gcloud run jobs deploy data-platform-pub-run-job --image=us-east4-docker.pkg.dev/data-platform-v0-0/repo-data-platform/data-platform-pub:latest --region=us-east4 --service-account=svc-act-pubsub@data-platform-v0-0.iam.gserviceaccount.com --execute-now

Cloud Run Job Locations

Data will be streamed by the Python publishing script running in Google Run Jobs from the following Tier 1 pricing locations:

• us-east-4
• asia-east1
• europe-west1

Delete a Cloud Run Job

Below is how to use the Google CLI to delete the Cloud Run Job. BUT don’t do this yet! Keep the Cloud Run Job and continue to the next step to trigger execution of the Cloud Run Job with Google Scheduler Jobs.

# gcloud run jobs delete job-name --region=us-east4
$ gcloud run jobs delete data-platform-pub-run-job --region=us-east4

Schedule a Cloud Run Job

I spent many hours figuring out how to create a Google Scheduler job with the Google CLI (gcloud) to run a Cloud Run Job on a regular interval. Mistakes and missing information in the Google CLI documentation caused most of my problems. Follow my detailed instructions below for success.

When you read the Google Scheduler Jobs CLI documentation on how to create a Scheduler Job, you will see that the “gcloud scheduler jobs create” requires the argument “COMMAND” and it has the three options of:

• app engine
• http
• pubsub

For my application, I wanted to simply run a Python script configured as a Cloud Run Job on a regular interval. The correct “COMMAND” for this instance is “http”. That choice implies that it will “trigger an action via HTTP”. In our case, that action will be to for Google Scheduler to call the Google Cloud Run Job via its URI. Get the Google Cloud Run Job URI by executing the command:

# List the jobs by JOB URI
$ gcloud run jobs list --uri

Create a Schedule Job to execute the Cloud Run Job using its URI as follows:

# Create a Schedule Job to execut a Cloud Run Job using its URI
# gcloud scheduler jobs create COMMAND SCHEDULER_JOB_NAME --location=SCHEDULER_REGION --schedule="SCHEDULE" --uri="https://CLOUD_RUN_REGION-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/PROJECT-ID/jobs/JOB-NAME:run" --oauth-service-account-email SERVICE_ACCOUNT_EMAIL
$ gcloud scheduler jobs create http data-platform-pub-run-job --location=us-east4 --schedule="*/2 * * * *" --uri="https://us-east4-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/data-platform-v0-0/jobs/data-platform-pub-run-job:run" --oauth-service-account-email "svc-act-pubsub@data-platform-v0-0.iam.gserviceaccount.com"

The arguments for the “gcloud scheduler jobs create” are:

• COMMAND: choices are “app engine”, “http”, “pubsub”. Choose “http”.
• SCHEDULER_JOB_NAME: a name for the scheduler job that will be created.
• — location : the region you want the scheduler to execute from. Probably best to use the same value for the argument “ — region=REGION” used for the “gcloud cloud run jobs create” command earlier, BUT make sure that region is listed as available to Scheduler Jobs by using the CLI command “gcloud scheduler locations list”. Note that the documentation doesn’t indicate this is a required argument, doesn’t even mention it as a create command argument, but it is required.
• — schedule: See cron job format and time zone. The format “/2 *” will run the job every 2 minutes of every hour. See https://crontab.guru/.
• — uri: the URI for the cloud run job obtained by executing the CLI command “gcloud run jobs list — uri”. The Google CLI documentation for format of this is incorrect.
• oauth-service-account-email: This can be the user-managed service account that you created.

You should be able to see your Scheduler Job in the Google Console for Scheduler Jobs, BUT I found it can take several minutes for it to appear (refresh the browser tab, the refresh link doesn’t work). Alternatively, you can see if Cloud Run job was executed by visiting the Google Cloud Run Console, click on the job listed, select an Execution ID, and then click on the “LOGS” tab. The output from the Python script execution will be shown, and it will include the message number.

Use the Google CLI command below to list the Scheduler Jobs. The field “lastAttemptTime” shows the last time the schedule was triggered.

# List Scheduler Jobs
# gcloud scheduler jobs list --location=REGION
$ gcloud scheduler jobs list --location=us-east4

Run the Python script “gcp_data_platform_sub.py” in a separate window locally, and then execute the Google CLI command below to run the Scheduler Job you created:

# Run a Scheduler Job
# gcloud scheduler jobs run SCHEDULER_JOB_NAME --location=SCHEDULER_REGION
$ gcloud scheduler jobs run data-platform-pub-run-job --location=us-east4

You should see the Pub/Sub message created by the Google Run Jobs (running the Python pub script) received by the subscription Python script “gcp_data_platform_sub.py” running locally.

After you have confirmed the Scheduler Job has run successfully, make sure to at least pause it so you don’t use up your Google free credits.

# gcloud scheduler jobs pause SCHEDULER_JOB_NAME --location=SCHEDULER_REGION
$ gcloud scheduler jobs pause data-platform-pub-run-job --location=us-east4

You can also see Google Scheduler Jobs using the console, HOWEVER as of September 2024 the “Refresh” link on the Google Scheduler Jobs console page doesn’t work. You must refresh the browser tab.

The Google Cloud Logs Explorer is a great online tool for reviewing Cloud Run Jobs and Scheduler Jobs to see if they are executing properly.

You can also use the Google Cloud CLI to view the logs:

# View the Google Logs
$ gcloud logging logs list

# View a particular log from the list output above with:
$ gcloud logging read NAME

The Google Cloud CLI commands below are helpful for managing Scheduler Jobs.

# Get details about a Scheduler Job
# gcloud scheduler jobs describe SCHEDULER_JOB_NAME --location=SCHEDULER_REGION
$ gcloud scheduler jobs describe data-platform-pub-run-job --location=us-east4

# lists the locations where Cloud Scheduler is available
$ gcloud scheduler locations list 

$ gcloud scheduler jobs pause SCHEDULER_JOB_NAME --location=SCHEDULER_REGION

$ gcloud scheduler jobs resume SCHEDULER_JOB_NAME --location=SCHEDULER_REGION

$ gcloud scheduler jobs delete SCHEDULER_JOB_NAME --location=SCHEDULER_REGION

Pub/Sub Subscription By Cloud Run Jobs

Previously, Cloud Run Jobs and Cloud Scheduler Jobs was configured to execute the publishing Python script “gcp_data_platform_pub.py”, and the subscription Python script “gcp_data_platform_sub.py” was run locally in order to receive (Pull) and ack (acknowledge) the Pub/Sub messages. We will now configure a Google Cloud Run Job for the Python publisher script, and a Cloud Scheduler Job to execute that Cloud Run Job.

If you look at the subscription detail, you will see that the message will be retained for 604800s or 7 days (the minimum is 10 minutes, max is 7 days), and the expiration for the subscription is 31 days or 2678400 seconds (min is 1 day, max is unlimited). Note also that if you retain unacknowledged messages in a subscription for more than 24 hours, you will incur additional charges. The message acknowledge deadline (ackDeadlineSeconds) is the default of 60 seconds (min 10 sec, max 600 sec). Dead letter topic options also exist.

By default, the retry policy for a subscription is set to use Retry immediately, causing Pub/Sub to resend the message when the acknowledgment deadline expires. An option Retry after exponential backoff delay exists where you can set the maximum and minimum backoff values

# Get details about the subscription
# gcloud pubsub subscriptions describe SUBSCRIPTION
$ gcloud pubsub subscriptions describe streaming_data_packet_subscription

# OUTPUT:
ackDeadlineSeconds: 60
enableExactlyOnceDelivery: true
expirationPolicy:
  ttl: 2678400s
messageRetentionDuration: 604800s
name: projects/data-platform-v0-0/subscriptions/streaming_data_packet_subscription
pushConfig: {}
state: ACTIVE
topic: projects/data-platform-v0-0/topics/streaming_data_packet_topic

The Python subscription script acknowledges a message immediately, and then processes the data. But if the subscriber script isn’t running, then a message won’t get acknowledged immediately. Therefore, to avoid message publish retry, the subscription pull should occur on an interval substantially less than the publishing interval.

Data processing consists of decoding the message, and then pushing the clean data to Google storage. This subscription processing time should be measured and analyzed to insure that the publisher doesn’t send a message on an interval shorter than the subscription processing time.

Considering all of the information above, I chose to configure the publisher to execute every 2 minutes, and the subscriber every 1 minute. This works well with the schedule cron job argument since it only accepts integer values for the minutes.

Build a Python Subscription Docker Image Locally

We will follow most of the steps from the prior article “Containerization using Docker”, but this time a Docker image for the Python subscriber script will be created as “data-platform-sub”.

You need a “Dockerfile” in order to build a Docker image. Use a text editor to edit the file named “Dockerfile” (no filename extension) in the folder where your Python virtual environment (venv) folder and the Python scripts reside. Only the last line of that file that already exists needs to be edited, specifically “CMD [“gcp_dataplatformsub.py”]”

# syntax=docker/dockerfile:1

# slim version of Python 3.12 to minimize the size of the container and make it as lightweight as possible
FROM python:3.12-slim

# Set the working directory inside the container
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Optimize pip
ENV PIP_DEFAULT_TIMEOUT=100 \
    # Allow statements and log messages to immediately appear
    PYTHONUNBUFFERED=1 \
    # disable a pip version check to reduce run-time & log-spam
    PIP_DISABLE_PIP_VERSION_CHECK=1 \
    # cache is useless in docker image, so disable to reduce image size
    PIP_NO_CACHE_DIR=1

# Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# set the default executable for a container 
ENTRYPOINT [ "python" ]

# Define the command to run your Python script
CMD ["gcp_data_platform_sub.py"]

Open up a Windows CMD window and navigate to the folder where your Python virtual environment (venv) folder and the Python scripts reside. Execute the Docker CLI command below to build a Docker image locally with the image name “data-platform-sub”.

# Build the Docker image with the name "data-platform-sub"
# docker build -t IMAGE .
$ docker build -t data-platform-sub .

Push Docker Image To Google Artifact Registry

All required roles for the user-managed service agent to work with Google Artifacts (repository) have been assigned previously, and the required Google APIs enabled.

Recall from the prior article that a repository named “repo-data-platform” already exists. Verify that by executing the following Google CLI commands:

# gcloud artifacts repositories list --limit=5
$ gcloud artifacts repositories list --limit=5 --location=us-east4

# gcloud artifacts repositories describe <REPOSITORY> --location=LOCATION
$ gcloud artifacts repositories describe repo-data-platform --location=us-east4

All subsequent references to the “Docker CLI” mean the use of a Windows command prompt (CMD window) with the current folder the same as the Python script folder and virtual environment.

In the Docker CLI, tag the Docker image you created previously, and then push it to the Google Artifact Registry into a repository by running the following in the Docker CLI.

# Tag the local Docker image "data-platform-sub"
# docker tag SOURCE-IMAGE LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE:TAG
$ docker tag data-platform-sub us-east4-docker.pkg.dev/data-platform-v0-0/repo-data-platform/data-platform-sub

# Push the tagged image named "data-platform-sub" to Artifact Registry in the repository named "repo-data-platform"
# docker push LOCATION-docker.pkg.dev/PROJECT-ID/REPOSITORY/IMAGE
$ docker push us-east4-docker.pkg.dev/data-platform-v0-0/repo-data-platform/data-platform-sub

List all files in a repository by the Docker image name, because you will need the image name later for Google Cloud Run Jobs.

# gcloud artifacts docker images list LOCATION-docker.pkg.dev/PROJECT/REPOSITORY --include-tags
$ gcloud artifacts docker images list us-east4-docker.pkg.dev/data-platform-v0-0/repo-data-platform --include-tags

If you don’t see anything listed from the prior command, wait a few minutes.

The Docker image now resides in the Google Artifact Registry under the repository named “repo-data-platform” in the location “us-east4”. In the next section, Google Cloud Run Jobs will be configured with that Docker image.

Create Cloud Run Job

Configure a Cloud Run Job for the Docker image named “us-east4-docker.pkg.dev/data-platform-v1–5/repo-data-platform/data-platform-sub” in the Google Artifact Registry repository “repo-data-platform”. Recall that roles for Cloud Run Job have already been previously assigned to the user-managed service agent, and the Google API activated.

Use the Google CLI to create a Google Run Job.

# Create a Google Cloud Run job from the Docker image
# gcloud run jobs create JOB_NAME --image=IMAGE_URL --region=REGION
$ gcloud run jobs create data-platform-sub-run-job --image=us-east4-docker.pkg.dev/data-platform-v0-0/repo-data-platform/data-platform-sub:latest --region=us-east4

List the Cloud Run Jobs and get the URI for the Cloud Run Job you just created.

# List the jobs by JOB URI
$ gcloud run jobs list --uri

OUTPUT:
https://us-east4-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/608676951033/jobs/data-platform-sub-run-job

Create & Run Cloud Scheduler Job

Create a Cloud Scheduler Job to run the Cloud Run Job on an interval of every 1 minute. Recall that roles for Cloud Scheduler Jobs has already been previously assigned to the user-managed service agent, and the Google API activated.

# Create a Schedule Job to execute a Cloud Run Job using its URI
# gcloud scheduler jobs create COMMAND SCHEDULER_JOB_NAME --location=SCHEDULER_REGION --schedule="SCHEDULE" --uri="https://CLOUD_RUN_REGION-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/PROJECT-ID/jobs/JOB-NAME:run" --oauth-service-account-email SERVICE_ACCOUNT_EMAIL
$ gcloud scheduler jobs create http data-platform-sub-run-job --location=us-east4 --schedule="*/1 * * * *" --uri="https://us-east4-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/data-platform-v0-0/jobs/data-platform-sub-run-job:run" --oauth-service-account-email "svc-act-pubsub@data-platform-v0-0.iam.gserviceaccount.com"

Use the Google CLI command below to list the Scheduler Jobs. The field “lastAttemptTime” shows the last time the schedule was triggered.

# List Scheduler Jobs
# gcloud scheduler jobs list --location=REGION
$ gcloud scheduler jobs list --location=us-east4

The Cloud Scheduler Job “data-platform-sub-run-job” should be paused. Resume that job:

# Resume a Cloud Scheduler Job
# gcloud scheduler jobs resume SCHEDULER_JOB_NAME --location=SCHEDULER_REGION
$ gcloud scheduler jobs resume data-platform-sub-run-job --location=us-east4

# List Scheduler Jobs
# gcloud scheduler jobs list --location=REGION
$ gcloud scheduler jobs list --location=us-east4

Go to the Google Cloud Run Jobs console and review the jobs listed and their status. If both Cloud Run Jobs are running successfully, then pause them to avoid using up Google Free Tier credits or charges.

# List Scheduler Jobs
# gcloud scheduler jobs list --location=REGION
$ gcloud scheduler jobs list --location=us-east4

# gcloud scheduler jobs pause SCHEDULER_JOB_NAME --location=SCHEDULER_REGION
$ gcloud scheduler jobs pause data-platform-pub-run-job --location=us-east4
$ gcloud scheduler jobs pause data-platform-sub-run-job --location=us-east4

Review Google Cloud Billing

Go to the Google Cloud Billing console and review your billing status.

Conclusion

You now know how to use Google Cloud Run Jobs to run a Python script that doesn’t require an HTTP request, and to trigger execution of that Cloud Run Job on a regular interval using its URI called by Cloud Scheduler Jobs.

Related Stories By The Author

[embed]Persistent Storage For Your Cloud Run Python App How to configure a Python script executed by Google Cloud Run to write & read to a Cloud Storage Bucket.medium.com

[embed]Python Web App in Google Cloud Run Create a web app from a Python script using Flask that runs in Google Cloud Run Service with persistent storage access.medium.com

[embed]Deploy Your Custom MCP AI Tool to Cloud Run See my complete working example of how to create a Model Context Protocol (MCP) compliant AI tool and then deploy it to…medium.com

Related Links

Github repository

Python Cloud Client Libraries

Google IAM predefined roles

…VM running Linux in Asia. I followed the ‘Create a Linux VM instance in Compute Engine’ guide.

Google Cloud Run Jobs gcloud documentation

Cloud Run predefined roles

Cloud Scheduler cron job format and time zone.

[embed]Configure service identity | Cloud Run Documentation | Google Cloud A Cloud Run job has a service identity that is used as the authenticated account for accessing Google Cloud APIs from…cloud.google.com

Configure Cloud Run environment variables

[embed]Cloud Run — Microservices Microservices are a set of functionalities mounted on the same application.medium.com