Reliable Webhook Delivery Platform
A dedicated platform converts internal events into reliable signed webhooks with per-tenant rate limiting, exponential-backoff retries, and dead-lettering. Cloud Tasks and Cloud Run on GCP isolate slow customers and guarantee at-least-once delivery.
Overview
SaaS products notify customers of events by calling webhooks: HTTP callbacks to customer-provided URLs. Doing this reliably is harder than it looks. Endpoints go down, respond slowly, or change; deliveries must be retried, signed, deduplicated, and rate-limited per tenant. This reference architecture is a dedicated webhook delivery platform that turns internal events into dependable outbound notifications.
Use it whenever your product emits events that external systems consume, and you need delivery guarantees, signing, and a self-service way for customers to manage subscriptions.
Components
- API Gateway: lets customers register endpoints, secrets, and event subscriptions.
- Pub/Sub: ingests internal domain events to be fanned out as webhooks.
- Cloud Tasks: schedules per-endpoint delivery attempts with controlled rate and retry backoff.
- Cloud Run: stateless delivery workers that sign and POST payloads.
- Firestore: stores subscriptions, secrets, and delivery state.
- Cloud SQL: holds the delivery log and per-tenant analytics.
- Cloud Monitoring: tracks delivery success rates and dead-letter volume.
Data Flow
Internal events land on Pub/Sub. A dispatcher matches each event to subscribed endpoints and enqueues a Cloud Task per endpoint. A Cloud Run worker pulls a task, signs the payload with the tenant's secret, and POSTs it. On a non-2xx response or timeout, the task is retried with exponential backoff up to a limit, after which it moves to a dead-letter store for inspection or manual replay.
Scaling and Resilience
Cloud Tasks decouples generation from delivery and applies per-queue rate limits so one slow customer cannot starve others. Cloud Run scales delivery workers to zero when idle and up under load. Circuit breaking pauses delivery to consistently failing endpoints. Idempotency keys and a delivery ID let customers safely deduplicate. Dead-letter handling preserves undeliverable events for replay.
Security
Every payload is HMAC-signed with a per-endpoint secret so receivers can verify authenticity. The platform enforces HTTPS-only endpoints and validates URLs against SSRF risks. API Gateway authenticates subscription management calls. Secrets are stored encrypted, and delivery logs redact sensitive fields. Per-tenant isolation prevents cross-customer data exposure.
Trade-offs and Alternatives
Guaranteed delivery means at-least-once semantics, so receivers must be idempotent. Building this platform is real engineering; small products may start with a managed webhook service or a simple queue plus worker. Polling APIs are an alternative when receivers cannot host an endpoint. The dedicated platform pays off once webhook volume, tenant count, and reliability expectations grow.