Skip to content

Welcome to Danube Messaging

🌊 Danube Messaging is a lightweight, cloud‑native messaging platform built in Rust. It delivers sub‑second dispatch with cloud economics by combining a Write‑Ahead Log (WAL) with object storage, so you get low‑latency pub/sub and durable streaming—on one broker.

Danube enables one or many producers publish to topics, and multiple consumers receive messages via named subscriptions. Choose Non‑Reliable (best‑effort pub/sub) or Reliable (at‑least‑once streaming) per topic to match your workload.

For design details, see the Architecture.

Try Danube in minutes

Docker Compose Quickstart: Use the provided Docker Compose setup with MinIO and ETCD.

Danube architecture

🏗️ Cluster & Broker Characteristics

  • Stateless brokers: Metadata in ETCD and data in WAL/Object Storage
  • Horizontal scaling: Add brokers in seconds with zero-downtime expansion
  • Intelligent load balancing: Automatic topic placement and rebalancing across brokers
  • Rolling upgrades: Restart or replace brokers with minimal disruption
  • Security-ready: TLS/mTLS support in Admin and data paths
  • Leader election & HA: Automatic failover and coordination via ETCD
  • Multi-tenancy: Isolated namespaces with policy controls

🌩️ Write-Ahead Log + Cloud Persistence

  • Cloud-Native by Design - Danube's architecture separates compute from storage
  • Multi-cloud support: AWS S3, Google Cloud Storage, Azure Blob, MinIO
  • Hot path optimization: Messages served from in-memory WAL cache
  • Stream per subscription: WAL + cloud storage from selected offset
  • Asynchronous background uploads to S3/GCS/Azure object storage
  • Infinite retention without local disk constraints

🎯 Intelligent Load Management

  • Automated rebalancing: Detects cluster imbalances and redistributes topics automatically
  • Smart topic assignment: Places new topics on least-loaded brokers using configurable strategies
  • Resource monitoring: Tracks CPU, memory, throughput, and backlog per broker in real-time
  • Configurable policies: Conservative, balanced, or aggressive rebalancing based on workload
  • Graceful topic migration: Moves topics between brokers without downtime

Core Capabilities

📨 Message Delivery

  • Topics: Partitioned and non-partitioned with automatic load balancing
  • Reliable Dispatch: At-least-once delivery with configurable storage backends
  • Non-Reliable Dispatch: High-throughput, low-latency for real-time scenarios

🔄 Subscription Models

  • Exclusive: Single consumer per subscription
  • Shared: Load-balanced message distribution across consumers
  • Failover: Automatic consumer failover with ordered delivery

📋 Schema Registry

  • Centralized schema management: Single source of truth for message schemas across all topics
  • Schema versioning: Automatic version tracking with compatibility enforcement
  • Multiple formats: Bytes, String, Number, JSON Schema, Avro, Protobuf
  • Validation & governance: Prevent invalid messages and ensure data quality

🤖 AI-Powered Administration

Danube features the AI-native messaging platform administration through the Model Context Protocol (MCP):

  • Natural language cluster management: Manage your cluster by talking to AI assistants (Claude, Cursor, Windsurf)
  • 32 intelligent tools: Full cluster operations accessible via AI - topics, schemas, brokers, diagnostics, metrics
  • Automated troubleshooting: AI-guided workflows for consumer lag analysis, health checks, and performance optimization
  • Multiple interfaces: CLI commands, Web UI, or AI conversation - your choice

Example: Ask Claude "What's the cluster balance?" or "Create a partitioned topic for analytics" and watch it happen.

Architecture Deep Dives

Explore how Danube works under the hood:

System Overview - Complete architecture diagram and component interaction

Load Manager & Rebalancing - Smart topic assignment and automatic rebalancing

Persistence (WAL + Cloud) - Two-tier storage architecture

  • Write‑Ahead Log on local disk for fast durable writes
  • Background uploads to object storage for durability and replay at cloud cost
  • Seamless handoff from historical replay to live tail

Schema Registry - Centralized schema management

  • Schema versioning and compatibility checking
  • Support for JSON Schema, Avro, and Protobuf
  • Data validation and governance

Internal Services - Service discovery and coordination

Integrations

Danube Connect - Plug-and-play connector ecosystem

  • Source connectors: Import data from MQTT, HTTP webhooks, databases, Kafka , etc.
  • Sink connectors: Export to Delta Lake, ClickHouse, vector databases, APIs, etc.
  • Pure Rust framework with automatic retries, metrics, and health checks

Learn more: Architecture | Build Source Connector | Build Sink Connector

Crates in the workspace

Repository: https://github.com/danube-messaging/danube

CLIs and client libraries:

Client libraries

Contributions for other languages (Python, Java, etc.) are welcome.