Cloud Pub/Sub (GCP)

Image source: Google Cloud Documentation

1. Overview

Cloud Pub/Sub is a fully managed, global, serverless messaging service that enables asynchronous, event-driven communication between services. It decouples publishers (services that send messages) from subscribers (services that receive and process messages).

Key Concept: Pub/Sub provides at-least-once delivery. Applications must be idempotent to handle potential duplicate messages.

2. Core Concepts

3. Subscription Types

3.1. Pull Subscription

• The subscriber initiates requests to fetch messages from Pub/Sub
• Best for: High-throughput batch jobs, worker fleets, or when subscriber is behind a firewall
• Subscriber controls the rate of message consumption

3.2. Push Subscription

• Pub/Sub sends HTTP POST requests to a predefined endpoint (webhook)
• Best for: Serverless environments (Cloud Run, Cloud Functions)
• Endpoint must be publicly accessible or use roles/run.invoker with authenticated requests

3.3. BigQuery Subscription

• Messages are written directly to a BigQuery table
• No subscriber code required
• Ideal for analytics pipelines

3.4. Cloud Storage Subscription

• Messages are written directly to Cloud Storage as objects
• Useful for archiving message streams

4. Message Lifecycle

4.1. Delivery Guarantees

Exam Tip: If a question mentions duplicate message handling, the answer is to make your application idempotent.

4.2. Acknowledgement

• Message Retention: Unacknowledged messages stored for up to 7 days
• Retry Policy: Configurable number of delivery attempts before sending to Dead Letter Topic

5. Advanced Features

5.1. Dead Letter Topics

• Messages that fail delivery after maximum retry attempts are sent here
• Allows for investigation and manual reprocessing
• Requires a separate topic and subscription

5.2. Message Ordering

• Enable Ordering Key on subscription to deliver messages in publish order
• All messages with the same ordering key are delivered in FIFO order
• Requires single-region topic

5.3. Message Filtering

• Subscriptions can filter messages by attributes
• Reduces cost by avoiding unnecessary message delivery
• Filter is applied at Pub/Sub level before delivery

gcloud pubsub subscriptions create high-value-orders \
    --topic=orders \
    --message-filter='attributes.type="order" AND attributes.amount > 100'

5.4. Replay (Seek)

• Rewind subscription to a specific timestamp or snapshot
• Useful for disaster recovery or reprocessing historical events

5.5. Fan-out

Fan‑out in Pub/Sub means a single published message is delivered to multiple independent subscribers. Each subscription receives its own copy of the message, allowing multiple services to react to the same event without coupling. Adding more subscribers does not affect the publisher.

• One topic can have multiple subscriptions
• Each subscription receives a copy of every message
• Enables parallel processing by different consumers

In Pub/Sub, each subscription can define its own filter. A message is delivered to a subscription only if it matches that filter. This allows selective fan‑out without creating multiple topics.

Image source: Own work (Mermaid diagram).

5.6. Schema Registry

• Define message structure using Avro or Protocol Buffers
• Ensures data quality and validation

6. IAM Roles

Exam Tip: Use the principle of least privilege - grant only publisher or subscriber roles, not admin.

7. Configuration Commands

7.1. Create Topic

gcloud pubsub topics create TOPIC_NAME

7.2. Create Pull Subscription

gcloud pubsub subscriptions create SUB_NAME \
    --topic=TOPIC_NAME

7.3. Create Push Subscription

gcloud pubsub subscriptions create SUB_NAME \
    --topic=TOPIC_NAME \
    --push-endpoint=https://example.com/webhook

7.4. Publish Message

gcloud pubsub topics publish TOPIC_NAME --message="Hello World"

7.5. Pull Messages

gcloud pubsub subscriptions pull SUB_NAME --auto-ack

7.6. Configure Dead Letter Topic

gcloud pubsub subscriptions update SUB_NAME \
    --dead-letter-topic=DEAD_LETTER_TOPIC \
    --max-delivery-attempts=5

7.7. Enable Message Ordering

gcloud pubsub topics update TOPIC_NAME --enable-message-ordering

8. Java Code Example

Add a dpendency into the pom.xml:

<dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>spring-cloud-gcp-starter-pubsub</artifactId>
</dependency>

Using Pub/Sub Template in Java:

@Service
@RequiredArgsConstructor
public class MessagingService {

    private final PubSubTemplate pubSubTemplate;

    public void sendWithHeaders(String topic, String payload) {
        var headers = Map.of(
            "origin", "spring-boot-app",
            "priority", "high",
            "correlation-id", "uuid-1234");

        pubSubTemplate.publish(topic, payload, headers)
            .addCallback(
                result -> System.out.println("Message sent successfully! ID: " + result),
                ex -> System.err.println("Failed to send: " + ex.getMessage()));
    }
}

9. Comparison with Alternatives

10. Exam Prep Summary

10.1. Key Points to Remember

1. Global Service: Topics and subscriptions are global resources (not regional)
2. At-least-once Delivery: Applications must be idempotent
3. Serverless: Automatically scales, no capacity planning needed
4. Fan-out: Multiple subscriptions = multiple copies of each message
5. Ordering: Use ordering keys for FIFO delivery
6. Ack Deadline: Default is 10 seconds, configurable up to 600 seconds
7. Retention: Messages stored for up to 7 days
8. Dead Letter Topics: For failed messages after max retries

10.2. When to Choose Pub/Sub

• Decoupling microservices for independent scaling
• Buffering traffic spikes (IoT, analytics)
• Event-driven architectures
• Asynchronous communication between services

10.3. Common Exam Traps

11. External Links

• Pub/Sub - The Cloud Girl