Building a Fan-out Architecture with Cloud Pub/Sub and Cloud Functions

Building a fan-out architecture using Google Cloud Pub/Sub and Cloud Functions is a scalable way to distribute data to multiple consumers. This pattern is often used when you need to send the same message to multiple processing components, such as sending a single user event to different microservices for analytics, logging, and notifications.

Here’s a step-by-step guide to building a fan-out architecture with Pub/Sub and Cloud Functions:

🚧 Overview of the Architecture

Producer publishes a message to a Pub/Sub topic.

Multiple Cloud Functions are subscribed to the same topic (or to different topics that receive the same message).

Each Cloud Function processes the message independently.

✅ Step-by-Step Implementation

1. Create a Pub/Sub Topic

bash

Copy

Edit

gcloud pubsub topics create user-events

2. Create Cloud Functions that Subscribe to the Topic

Each function acts as a subscriber and performs a different task.

bash

Copy

Edit

gcloud functions deploy logUserEvent \

  --runtime=nodejs18 \

  --trigger-topic=user-events \

  --entry-point=logUserEvent \

  --region=us-central1

In your index.js:

js

Copy

Edit

exports.logUserEvent = (message, context) => {

  const data = JSON.parse(Buffer.from(message.data, 'base64').toString());

  console.log(`Logging event for user: ${data.userId}`);

};

Repeat for each processing task, changing the function name and logic.

Examples:

notifyUser

updateAnalytics

storeInBigQuery

All of them will be triggered by the same Pub/Sub topic, creating a fan-out effect.

🌀 Alternative: Fan-Out with Multiple Topics

If you want more control (e.g., decoupling subscribers), you can:

Use a primary topic.

Attach a Cloud Function to the primary topic that republishes the message to multiple secondary topics.

Subscribe different functions to those secondary topics.

Advantage: Easier to manage individual flows and permissions.

🔐 Permissions

Ensure that the Cloud Functions have the pubsub.subscriber role and the Pub/Sub service account has permission to invoke the functions.

bash

Copy

Edit

gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \

  --member="serviceAccount:YOUR_FUNCTION_SERVICE_ACCOUNT" \

  --role="roles/pubsub.subscriber"

🔄 Monitoring and Logging

Use Cloud Logging to monitor function logs.

Use Cloud Monitoring to track function invocations, errors, and latencies.

✅ Benefits of Fan-Out with Pub/Sub + Cloud Functions

Scalability: Cloud Functions scale automatically.

Loose coupling: Each subscriber is independent.

Flexibility: Easily add or remove subscribers.

Resilience: One function’s failure doesn’t affect others.

Learn Google Cloud Data Engineering Course

Read More

Message Sharding and Load Balancing with Cloud Pub/Sub

Cloud Pub/Sub - Design Patterns & Enterprise Messaging

Visit Our Quality Thought Training in Hyderabad

Get Directions