Skip to content

Welcome to Danube Messaging

🌊 Danube Messaging is a lightweight, cloud‑native messaging platform built in Rust. It delivers sub‑second dispatch with cloud economics by combining a Write‑Ahead Log (WAL) with object storage, so you get low‑latency pub/sub and durable streaming—on one broker.

Danube enables one or many producers publish to topics, and multiple consumers receive messages via named subscriptions. Choose Non‑Reliable (best‑effort pub/sub) or Reliable (at‑least‑once streaming) per topic to match your workload.

For design details, see the Architecture.

Try Danube in minutes

Docker Compose Quickstart: Use the provided Docker Compose setup with MinIO and ETCD.

Danube capabilities

🏗️ Cluster & Broker Characteristics

  • Stateless brokers: Metadata in ETCD and data in WAL/Object Storage
  • Horizontal scaling: Add brokers in seconds; partitions rebalance automatically
  • Leader election & HA: Automatic failover and coordination via ETCD
  • Rolling upgrades: Restart or replace brokers with minimal disruption
  • Multi-tenancy: Isolated namespaces with policy controls
  • Security-ready: TLS/mTLS support in Admin and data paths

đź“‹ Schema Registry

  • Centralized schema management: Single source of truth for message schemas across all topics
  • Schema versioning: Automatic version tracking with compatibility enforcement
  • Multiple formats: Bytes, String, Number, JSON Schema, Avro, Protobuf
  • Validation & governance: Prevent invalid messages and ensure data quality

Cloud-Native by Design

Danube's architecture separates compute from storage, enabling:

🌩️ Write-Ahead Log + Cloud Persistence

  • Sub-millisecond producer acknowledgments via local WAL
  • Asynchronous background uploads to S3/GCS/Azure object storage
  • Automatic failover with shared cloud state
  • Infinite retention without local disk constraints

⚡ Performance & Scalability

  • Hot path optimization: Messages served from in-memory WAL cache
  • Stream per subscription: WAL + cloud storage from selected offset
  • Multi-cloud support: AWS S3, Google Cloud Storage, Azure Blob, MinIO

Core Concepts

Learn the fundamental concepts that power Danube messaging:

Topics - Named channels for message streams

  • Non‑partitioned: served by a single broker
  • Partitioned: split across brokers for scale and HA

Subscriptions - Named configurations for message delivery

  • Exclusive, Shared, Failover patterns for queueing and fan‑out

Dispatch Strategies - Message delivery guarantees

  • Non‑Reliable: in‑memory, best‑effort delivery, lowest latency
  • Reliable: WAL + Cloud persistence with acknowledgments and replay

Danube Stream Messages - Message structure

Messaging Patterns

Architecture Deep Dives

Explore how Danube works under the hood:

System Overview - Complete architecture diagram and component interaction

Persistence (WAL + Cloud) - Two-tier storage architecture

  • Write‑Ahead Log on local disk for fast durable writes
  • Background uploads to object storage for durability and replay at cloud cost
  • Seamless handoff from historical replay to live tail

Schema Registry - Centralized schema management

  • Schema versioning and compatibility checking
  • Support for JSON Schema, Avro, and Protobuf
  • Data validation and governance

Internal Services - Service discovery and coordination

Integrations

Danube Connect - Plug-and-play connector ecosystem

  • Source connectors: Import data from MQTT, HTTP webhooks, databases, Kafka , etc.
  • Sink connectors: Export to Delta Lake, ClickHouse, vector databases, APIs, etc.
  • Pure Rust framework with automatic retries, metrics, and health checks

Learn more: Architecture | Build Source Connector | Build Sink Connector

Crates in the workspace

Repository: https://github.com/danube-messaging/danube

CLIs and client libraries:

Client libraries

Contributions for other languages (Python, Java, etc.) are welcome.