Managing Secrets in Cloud Composer Workflows

Managing secrets securely in Cloud Composer (Google Cloud's managed Apache Airflow service) is critical for maintaining data confidentiality and integrity in your workflows. Below is a comprehensive guide on how to manage secrets in Cloud Composer workflows:

🔐 Managing Secrets in Cloud Composer Workflows

Overview

Cloud Composer workflows often require credentials, API keys, or other sensitive information to connect to databases, cloud services, or third-party APIs. Hardcoding these secrets in your DAGs is insecure and should be avoided. Instead, Google Cloud offers several secure methods for handling secrets:

✅ Recommended Methods for Secret Management

1. Using Secret Manager

Google Secret Manager is the preferred method for storing and retrieving secrets in Cloud Composer.

Steps:

Create a Secret:

bash

Copy

Edit

gcloud secrets create my-db-password --replication-policy="automatic"

gcloud secrets versions add my-db-password --data-file="password.txt"

Grant Access to the Composer Environment:

Identify the Composer’s service account.

Grant it permission to access the secret:

bash

Copy

Edit

gcloud secrets add-iam-policy-binding my-db-password \

--member="serviceAccount:<your-composer-sa>" \

--role="roles/secretmanager.secretAccessor"

Access the Secret in a DAG:

In your Python DAG file:

python

Copy

Edit

from google.cloud import secretmanager

def get_secret(secret_id):

client = secretmanager.SecretManagerServiceClient()

name = f"projects/<project-id>/secrets/{secret_id}/versions/latest"

response = client.access_secret_version(request={"name": name})

return response.payload.data.decode("UTF-8")

Use the function inside your tasks:

python

Copy

Edit

db_password = get_secret("my-db-password")

2. Using Environment Variables

Environment variables can be defined in the Composer environment configuration. While this is a quick method, it is less secure and harder to audit.

Steps:

Set environment variables:

bash

Copy

Edit

gcloud composer environments update my-environment \

--update-env-variables=DB_PASSWORD=yourpassword

Access them in your DAG:

python

Copy

Edit

import os

db_password = os.environ.get("DB_PASSWORD")

⚠️ Avoid putting secrets directly in Airflow Variables or Connections if not encrypted.

3. Using Airflow Connections (with Secret Backend)

You can use Airflow's secrets backend to integrate with Secret Manager.

Steps:

Enable the secrets backend in airflow.cfg via Composer overrides:

yaml

Copy

Edit

[secrets]

backend = airflow.providers.google.cloud.secrets.secret_manager.CloudSecretManagerBackend

backend_kwargs = {"connections_prefix": "airflow-connections", "variables_prefix": "airflow-variables"}

Store a secret in Secret Manager:

bash

Copy

Edit

gcloud secrets create airflow-connections-my_conn_id --replication-policy="automatic"

In DAG, use:

python

Copy

Edit

from airflow.hooks.base import BaseHook

conn = BaseHook.get_connection("my_conn_id")

🧩 Best Practices

Rotate secrets regularly.

Avoid printing secrets in logs.

Use least privilege when granting access to secrets.

Audit secret access via Cloud Audit Logs.

Use Composer 2+ for improved security and secret management features.

🧠 Summary Table

Method Security Level Ease of Use Recommended For

Secret Manager (direct) ✅ High ⚠️ Medium Most secure workflows

Secret Manager (backend) ✅ High ✅ Easy Seamless Airflow usage

Environment Variables ⚠️ Low-Medium ✅ Easy Temporary/testing use

Airflow Variables ❌ Low ✅ Easy Not recommended for secrets

Learn Google Cloud Data Engineering Course

Cloud Composer - Cross-Service Integration

Creating Version-Controlled File Systems in Cloud Storage

Cloud Storage as a Staging Area for Enterprise ETL Pipelines

Visit Our Quality Thought Training in Hyderabad

Get Directions

July 17, 2025

Thursday, July 17, 2025

Managing Secrets in Cloud Composer Workflows