Fluent Plugin for Amazon Kinesis

This plugin lets you send log and event data from Fluentd directly to Amazon Kinesis, AWS's managed service for streaming data. Think of it as a bridge that captures events flowing through Fluentd and deposits them into Kinesis, where they can be processed, analyzed, or stored. The plugin acts as an output destination—you configure it to tell Fluentd "when you see events with this tag, send them to Kinesis."

The plugin actually provides three different ways to send data, depending on your needs. The most common is kinesis_streams, which sends individual events to Kinesis Data Streams. There's also kinesis_firehose for sending to Kinesis Data Firehose (useful if you want to automatically load data into S3, Redshift, or other destinations). Finally, kinesis_streams_aggregated packs multiple events together before sending them, which can be more efficient if you're handling high volumes of small events. You install it as a Ruby gem and configure it in your Fluentd config file by specifying your AWS region, stream name, and how you want to authenticate.

The plugin handles the tricky parts of talking to AWS for you. It manages authentication in multiple ways—it can use AWS credentials from your EC2 instance's built-in role (the easiest option if you're on AWS), from environment variables, from local credential files, or by assuming a different AWS role. It automatically batches events together to send efficiently, retries failed requests with smart backoff logic, and lets you compress data or customize how records are formatted before sending. You can also adjust performance settings like how often to flush data and how many worker threads to use.

Who uses this? DevOps teams running Fluentd on servers or containers who want to collect logs or metrics into Kinesis for real-time processing, analytics, or long-term storage on AWS. It's commonly paired with tools that read from Kinesis—like Lambda functions, Kinesis Analytics, or other data pipelines—to build end-to-end logging and monitoring systems.