connect

Redpanda Connect is a data pipeline tool that moves and transforms data between different systems. If you have data arriving from one place, say a message queue or a cloud storage bucket, and you need to clean it up, reformat it, or enrich it before sending it somewhere else, this tool handles that flow. It describes itself as a stream processor: it takes in a continuous stream of messages, applies whatever transformations you configure, and passes the results to an output.

Configuration is done in a single YAML file. You specify an input (where data comes from), a list of processing steps (what to do to each message), and an output (where the result goes). The tool includes a built-in mapping language called Bloblang for writing transformations, which lets you reshape and compute values from each message without writing custom code in a programming language.

The list of supported sources and destinations is long: AWS services like S3, SQS, Kinesis, and DynamoDB; Azure and GCP storage and messaging services; Kafka; Redis; RabbitMQ; MQTT; MongoDB; PostgreSQL and MySQL; Elasticsearch; and more. This breadth means you can often connect two systems you already use without writing a custom integration from scratch.

Reliability is a stated design goal. Redpanda Connect processes messages using an in-process transaction model that does not require disk-persisted state, so it can guarantee at-least-once delivery even if the process crashes. It also exposes health check endpoints and emits metrics compatible with Prometheus and Statsd, making it straightforward to monitor in a production environment.

It can run as a static binary on Linux or Mac, via Homebrew, or as a Docker image. Custom plugins can be written in Go for cases where the built-in processors do not cover a specific need. Full documentation is available on the Redpanda documentation site.