Skip to content
v2.278.0

AWS Lambda Event Sources ​

An event source mapping is an AWS Lambda resource that reads from an event source and invokes a Lambda function. You can use event source mappings to process items from a stream or queue in services that don't invoke Lambda functions directly. Lambda provides event source mappings for the following services. Read more about lambda event sources here.

This module includes classes that allow using various AWS services as event sources for AWS Lambda via the high-level lambda.addEventSource(source) API.

NOTE: In most cases, it is also possible to use the resource APIs to invoke an AWS Lambda function. This library provides a uniform API for all Lambda event sources regardless of the underlying mechanism they use.

The following code sets up a lambda function with an SQS queue event source -

ts
import { SqsEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';

declare const fn: lambda.Function;
const queue = new sqs.Queue(this, 'MyQueue');
const eventSource = new SqsEventSource(queue);
fn.addEventSource(eventSource);

const eventSourceId = eventSource.eventSourceMappingId;
const eventSourceMappingArn = eventSource.eventSourceMappingArn;
python
from aws_cdk.aws_lambda_event_sources import SqsEventSource

# fn: lambda.Function

queue = sqs.Queue(self, "MyQueue")
event_source = SqsEventSource(queue)
fn.add_event_source(event_source)

event_source_id = event_source.event_source_mapping_id
event_source_mapping_arn = event_source.event_source_mapping_arn
java
import software.amazon.awscdk.services.lambda.eventsources.SqsEventSource;

Function fn;

Queue queue = new Queue(this, "MyQueue");
SqsEventSource eventSource = new SqsEventSource(queue);
fn.addEventSource(eventSource);

String eventSourceId = eventSource.getEventSourceMappingId();
String eventSourceMappingArn = eventSource.getEventSourceMappingArn();
csharp
using Amazon.CDK.AWS.Lambda.EventSources;

Function fn;

var queue = new Queue(this, "MyQueue");
var eventSource = new SqsEventSource(queue);
fn.AddEventSource(eventSource);

var eventSourceId = eventSource.EventSourceMappingId;
var eventSourceMappingArn = eventSource.EventSourceMappingArn;
go
import "github.com/aws/aws-cdk-go/awscdk"

var fn Function

queue := sqs.NewQueue(this, jsii.String("MyQueue"))
eventSource := awscdk.NewSqsEventSource(queue)
fn.AddEventSource(eventSource)

eventSourceId := eventSource.eventSourceMappingId
eventSourceMappingArn := eventSource.eventSourceMappingArn

The eventSourceId property contains the event source id. This will be a token that will resolve to the final value at the time of deployment.

The eventSourceMappingArn property contains the event source mapping ARN. This will be a token that will resolve to the final value at the time of deployment.

SQS ​

Amazon Simple Queue Service (Amazon SQS) allows you to build asynchronous workflows. For more information about Amazon SQS, see Amazon Simple Queue Service. You can configure AWS Lambda to poll for these messages as they arrive and then pass the event to a Lambda function invocation. To view a sample event, see Amazon SQS Event.

To set up Amazon Simple Queue Service as an event source for AWS Lambda, you first create or update an Amazon SQS queue and select custom values for the queue parameters. The following parameters will impact Amazon SQS's polling behavior:

  • visibilityTimeout: May impact the period between retries.
  • batchSize: Determines how many records are buffered before invoking your lambda function.
  • maxBatchingWindow: The maximum amount of time to gather records before invoking the lambda. This increases the likelihood of a full batch at the cost of delayed processing.
  • maxConcurrency: The maximum concurrency setting limits the number of concurrent instances of the function that an Amazon SQS event source can invoke.
  • provisionedPollerConfig: Controls the number of pollers that can be provisioned to read from the queue. Valid ranges: minimumPollers 2–200 (default: 2), maximumPollers 2–2000 (default: 200).
  • enabled: If the SQS event source mapping should be enabled. The default is true.
ts
import { SqsEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';

const queue = new sqs.Queue(this, 'MyQueue', {
  visibilityTimeout: Duration.seconds(30),      // default,
});
declare const fn: lambda.Function;

fn.addEventSource(new SqsEventSource(queue, {
  batchSize: 10, // default
  maxBatchingWindow: Duration.minutes(5),
  reportBatchItemFailures: true, // default to false
}));
python
from aws_cdk.aws_lambda_event_sources import SqsEventSource
# fn: lambda.Function


queue = sqs.Queue(self, "MyQueue",
    visibility_timeout=Duration.seconds(30)
)

fn.add_event_source(SqsEventSource(queue,
    batch_size=10,  # default
    max_batching_window=Duration.minutes(5),
    report_batch_item_failures=True
))
java
import software.amazon.awscdk.services.lambda.eventsources.SqsEventSource;
Function fn;


Queue queue = Queue.Builder.create(this, "MyQueue")
        .visibilityTimeout(Duration.seconds(30))
        .build();

fn.addEventSource(SqsEventSource.Builder.create(queue)
        .batchSize(10) // default
        .maxBatchingWindow(Duration.minutes(5))
        .reportBatchItemFailures(true)
        .build());
csharp
using Amazon.CDK.AWS.Lambda.EventSources;
Function fn;


var queue = new Queue(this, "MyQueue", new QueueProps {
    VisibilityTimeout = Duration.Seconds(30)
});

fn.AddEventSource(new SqsEventSource(queue, new SqsEventSourceProps {
    BatchSize = 10,  // default
    MaxBatchingWindow = Duration.Minutes(5),
    ReportBatchItemFailures = true
}));
go
import "github.com/aws/aws-cdk-go/awscdk"
var fn Function


queue := sqs.NewQueue(this, jsii.String("MyQueue"), &QueueProps{
	VisibilityTimeout: awscdk.Duration_Seconds(jsii.Number(30)),
})

fn.AddEventSource(awscdk.NewSqsEventSource(queue, &SqsEventSourceProps{
	BatchSize: jsii.Number(10),
	 // default
	MaxBatchingWindow: awscdk.Duration_Minutes(jsii.Number(5)),
	ReportBatchItemFailures: jsii.Boolean(true),
}))

S3 ​

You can write Lambda functions to process S3 bucket events, such as the object-created or object-deleted events. For example, when a user uploads a photo to a bucket, you might want Amazon S3 to invoke your Lambda function so that it reads the image and creates a thumbnail for the photo.

You can use the bucket notification configuration feature in Amazon S3 to configure the event source mapping, identifying the bucket events that you want Amazon S3 to publish and which Lambda function to invoke.

ts
import * as s3 from 'aws-cdk-lib/aws-s3';
import { S3EventSource } from 'aws-cdk-lib/aws-lambda-event-sources';

const bucket = new s3.Bucket(this, 'mybucket');
declare const fn: lambda.Function;

fn.addEventSource(new S3EventSource(bucket, {
  events: [ s3.EventType.OBJECT_CREATED, s3.EventType.OBJECT_REMOVED ],
  filters: [ { prefix: 'subdir/' } ], // optional
}));
python
import aws_cdk.aws_s3 as s3
from aws_cdk.aws_lambda_event_sources import S3EventSource
# fn: lambda.Function


bucket = s3.Bucket(self, "mybucket")

fn.add_event_source(S3EventSource(bucket,
    events=[s3.EventType.OBJECT_CREATED, s3.EventType.OBJECT_REMOVED],
    filters=[s3.NotificationKeyFilter(prefix="subdir/")]
))
java
import software.amazon.awscdk.services.s3.*;
import software.amazon.awscdk.services.lambda.eventsources.S3EventSource;
Function fn;


Bucket bucket = new Bucket(this, "mybucket");

fn.addEventSource(S3EventSource.Builder.create(bucket)
        .events(List.of(EventType.OBJECT_CREATED, EventType.OBJECT_REMOVED))
        .filters(List.of(NotificationKeyFilter.builder().prefix("subdir/").build()))
        .build());
csharp
using Amazon.CDK.AWS.S3;
using Amazon.CDK.AWS.Lambda.EventSources;
Function fn;


var bucket = new Bucket(this, "mybucket");

fn.AddEventSource(new S3EventSource(bucket, new S3EventSourceProps {
    Events = new [] { EventType.OBJECT_CREATED, EventType.OBJECT_REMOVED },
    Filters = new [] { new NotificationKeyFilter { Prefix = "subdir/" } }
}));
go
import "github.com/aws/aws-cdk-go/awscdk"
import "github.com/aws/aws-cdk-go/awscdk"
var fn Function


bucket := s3.NewBucket(this, jsii.String("mybucket"))

fn.AddEventSource(awscdk.NewS3EventSource(bucket, &S3EventSourceProps{
	Events: []EventType{
		s3.EventType_OBJECT_CREATED,
		s3.EventType_OBJECT_REMOVED,
	},
	Filters: []NotificationKeyFilter{
		&NotificationKeyFilter{
			Prefix: jsii.String("subdir/"),
		},
	},
}))

In the example above, S3EventSource is accepting Bucket type as parameter. However, Functions like from_bucket_name and from_bucket_arn will return IBucket and is not compliant with S3EventSource. If this is the case, please consider using S3EventSourceV2 instead, this class accepts IBucket.

ts
import * as s3 from 'aws-cdk-lib/aws-s3';
import { S3EventSourceV2 } from 'aws-cdk-lib/aws-lambda-event-sources';

const bucket = s3.Bucket.fromBucketName(this, 'Bucket', 'amzn-s3-demo-bucket');
declare const fn: lambda.Function;

fn.addEventSource(new S3EventSourceV2(bucket, {
  events: [ s3.EventType.OBJECT_CREATED, s3.EventType.OBJECT_REMOVED ],
  filters: [ { prefix: 'subdir/' } ], // optional
}));
python
import aws_cdk.aws_s3 as s3
from aws_cdk.aws_lambda_event_sources import S3EventSourceV2
# fn: lambda.Function


bucket = s3.Bucket.from_bucket_name(self, "Bucket", "amzn-s3-demo-bucket")

fn.add_event_source(S3EventSourceV2(bucket,
    events=[s3.EventType.OBJECT_CREATED, s3.EventType.OBJECT_REMOVED],
    filters=[s3.NotificationKeyFilter(prefix="subdir/")]
))
java
import software.amazon.awscdk.services.s3.*;
import software.amazon.awscdk.services.lambda.eventsources.S3EventSourceV2;
Function fn;


IBucket bucket = Bucket.fromBucketName(this, "Bucket", "amzn-s3-demo-bucket");

fn.addEventSource(S3EventSourceV2.Builder.create(bucket)
        .events(List.of(EventType.OBJECT_CREATED, EventType.OBJECT_REMOVED))
        .filters(List.of(NotificationKeyFilter.builder().prefix("subdir/").build()))
        .build());
csharp
using Amazon.CDK.AWS.S3;
using Amazon.CDK.AWS.Lambda.EventSources;
Function fn;


var bucket = Bucket.FromBucketName(this, "Bucket", "amzn-s3-demo-bucket");

fn.AddEventSource(new S3EventSourceV2(bucket, new S3EventSourceProps {
    Events = new [] { EventType.OBJECT_CREATED, EventType.OBJECT_REMOVED },
    Filters = new [] { new NotificationKeyFilter { Prefix = "subdir/" } }
}));
go
import "github.com/aws/aws-cdk-go/awscdk"
import "github.com/aws/aws-cdk-go/awscdk"
var fn Function


bucket := s3.Bucket_FromBucketName(this, jsii.String("Bucket"), jsii.String("amzn-s3-demo-bucket"))

fn.AddEventSource(awscdk.NewS3EventSourceV2(bucket, &S3EventSourceProps{
	Events: []EventType{
		s3.EventType_OBJECT_CREATED,
		s3.EventType_OBJECT_REMOVED,
	},
	Filters: []NotificationKeyFilter{
		&NotificationKeyFilter{
			Prefix: jsii.String("subdir/"),
		},
	},
}))

SNS ​

You can write Lambda functions to process Amazon Simple Notification Service notifications. When a message is published to an Amazon SNS topic, the service can invoke your Lambda function by passing the message payload as a parameter. Your Lambda function code can then process the event, for example publish the message to other Amazon SNS topics, or send the message to other AWS services.

This also enables you to trigger a Lambda function in response to Amazon CloudWatch alarms and other AWS services that use Amazon SNS.

For an example event, see Appendix: Message and JSON Formats and Amazon SNS Sample Event. For an example use case, see Using AWS Lambda with Amazon SNS from Different Accounts.

ts
import * as sns from 'aws-cdk-lib/aws-sns';
import { SnsEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';

declare const topic: sns.Topic;
const deadLetterQueue = new sqs.Queue(this, 'deadLetterQueue');

declare const fn: lambda.Function;
fn.addEventSource(new SnsEventSource(topic, {
  filterPolicy: { },
  deadLetterQueue: deadLetterQueue,
}));
python
import aws_cdk.aws_sns as sns
from aws_cdk.aws_lambda_event_sources import SnsEventSource

# topic: sns.Topic

# fn: lambda.Function

dead_letter_queue = sqs.Queue(self, "deadLetterQueue")
fn.add_event_source(SnsEventSource(topic,
    filter_policy={},
    dead_letter_queue=dead_letter_queue
))
java
import software.amazon.awscdk.services.sns.*;
import software.amazon.awscdk.services.lambda.eventsources.SnsEventSource;

Topic topic;

Function fn;

Queue deadLetterQueue = new Queue(this, "deadLetterQueue");
fn.addEventSource(SnsEventSource.Builder.create(topic)
        .filterPolicy(Map.of())
        .deadLetterQueue(deadLetterQueue)
        .build());
csharp
using Amazon.CDK.AWS.SNS;
using Amazon.CDK.AWS.Lambda.EventSources;

Topic topic;

Function fn;

var deadLetterQueue = new Queue(this, "deadLetterQueue");
fn.AddEventSource(new SnsEventSource(topic, new SnsEventSourceProps {
    FilterPolicy = new Dictionary<string, object> { },
    DeadLetterQueue = deadLetterQueue
}));
go
import sns "github.com/aws/aws-cdk-go/awscdk"
import "github.com/aws/aws-cdk-go/awscdk"

var topic Topic

var fn Function

deadLetterQueue := sqs.NewQueue(this, jsii.String("deadLetterQueue"))
fn.AddEventSource(awscdk.NewSnsEventSource(topic, &SnsEventSourceProps{
	FilterPolicy: map[string]interface{}{
	},
	DeadLetterQueue: deadLetterQueue,
}))

When a user calls the SNS Publish API on a topic that your Lambda function is subscribed to, Amazon SNS will call Lambda to invoke your function asynchronously. Lambda will then return a delivery status. If there was an error calling Lambda, Amazon SNS will retry invoking the Lambda function up to three times. After three tries, if Amazon SNS still could not successfully invoke the Lambda function, then Amazon SNS will send a delivery status failure message to CloudWatch.

DynamoDB Streams ​

You can write Lambda functions to process change events from a DynamoDB Table. An event is emitted to a DynamoDB stream (if configured) whenever a write (Put, Delete, Update) operation is performed against the table. See Using AWS Lambda with Amazon DynamoDB for more information about configuring Lambda function event sources with DynamoDB.

To process events with a Lambda function, first create or update a DynamoDB table and enable a stream specification. Then, create a DynamoEventSource and add it to your Lambda function. The following parameters will impact Amazon DynamoDB's polling behavior:

  • batchSize: Determines how many records are buffered before invoking your lambda function - could impact your function's memory usage (if too high) and ability to keep up with incoming data velocity (if too low).
  • bisectBatchOnError: If a batch encounters an error, this will cause the batch to be split in two and have each new smaller batch retried, allowing the records in error to be isolated.
  • reportBatchItemFailures: Allow functions to return partially successful responses for a batch of records.
  • maxBatchingWindow: The maximum amount of time to gather records before invoking the lambda. This increases the likelihood of a full batch at the cost of delayed processing.
  • maxRecordAge: The maximum age of a record that will be sent to the function for processing. Records that exceed the max age will be treated as failures.
  • onFailure: In the event a record fails after all retries or if the record age has exceeded the configured value, the record will be sent to S3 bucket, SQS queue or SNS topic that is specified here
  • parallelizationFactor: The number of batches to concurrently process on each shard.
  • retryAttempts: The maximum number of times a record should be retried in the event of failure.
  • startingPosition: Will determine where to being consumption, either at the most recent ('LATEST') record or the oldest record ('TRIM_HORIZON'). 'TRIM_HORIZON' will ensure you process all available data, while 'LATEST' will ignore all records that arrived prior to attaching the event source.
  • tumblingWindow: The duration in seconds of a processing window when using streams.
  • enabled: If the DynamoDB Streams event source mapping should be enabled. The default is true.
  • filters: Filters to apply before sending a change event from a DynamoDB table to a Lambda function. Events that are filtered out are not sent to the Lambda function.
ts
import * as dynamodb from 'aws-cdk-lib/aws-dynamodb';
import { DynamoEventSource, SqsDlq } from 'aws-cdk-lib/aws-lambda-event-sources';

declare const table: dynamodb.Table;

const deadLetterQueue = new sqs.Queue(this, 'deadLetterQueue');

declare const fn: lambda.Function;
fn.addEventSource(new DynamoEventSource(table, {
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
  batchSize: 5,
  bisectBatchOnError: true,
  onFailure: new SqsDlq(deadLetterQueue),
  retryAttempts: 10,
}));
python
import aws_cdk.aws_dynamodb as dynamodb
from aws_cdk.aws_lambda_event_sources import DynamoEventSource, SqsDlq

# table: dynamodb.Table

# fn: lambda.Function


dead_letter_queue = sqs.Queue(self, "deadLetterQueue")
fn.add_event_source(DynamoEventSource(table,
    starting_position=lambda_.StartingPosition.TRIM_HORIZON,
    batch_size=5,
    bisect_batch_on_error=True,
    on_failure=SqsDlq(dead_letter_queue),
    retry_attempts=10
))
java
import software.amazon.awscdk.services.dynamodb.*;
import software.amazon.awscdk.services.lambda.eventsources.DynamoEventSource;
import software.amazon.awscdk.services.lambda.eventsources.SqsDlq;

Table table;

Function fn;


Queue deadLetterQueue = new Queue(this, "deadLetterQueue");
fn.addEventSource(DynamoEventSource.Builder.create(table)
        .startingPosition(StartingPosition.TRIM_HORIZON)
        .batchSize(5)
        .bisectBatchOnError(true)
        .onFailure(new SqsDlq(deadLetterQueue))
        .retryAttempts(10)
        .build());
csharp
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.Lambda.EventSources;

Table table;

Function fn;


var deadLetterQueue = new Queue(this, "deadLetterQueue");
fn.AddEventSource(new DynamoEventSource(table, new DynamoEventSourceProps {
    StartingPosition = StartingPosition.TRIM_HORIZON,
    BatchSize = 5,
    BisectBatchOnError = true,
    OnFailure = new SqsDlq(deadLetterQueue),
    RetryAttempts = 10
}));
go
import dynamodb "github.com/aws/aws-cdk-go/awscdk"
import "github.com/aws/aws-cdk-go/awscdk"

var table Table

var fn Function


deadLetterQueue := sqs.NewQueue(this, jsii.String("deadLetterQueue"))
fn.AddEventSource(awscdk.NewDynamoEventSource(table, &DynamoEventSourceProps{
	StartingPosition: lambda.StartingPosition_TRIM_HORIZON,
	BatchSize: jsii.Number(5),
	BisectBatchOnError: jsii.Boolean(true),
	OnFailure: awscdk.NewSqsDlq(deadLetterQueue),
	RetryAttempts: jsii.Number(10),
}))

The following code sets up a Lambda function with a DynamoDB event source. A filter is applied to only send DynamoDB events to the Lambda function when the id column is a boolean that equals true.

ts
import * as dynamodb from 'aws-cdk-lib/aws-dynamodb';
import { DynamoEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';

declare const table: dynamodb.Table;

declare const fn: lambda.Function;
fn.addEventSource(new DynamoEventSource(table, {
  startingPosition: lambda.StartingPosition.LATEST,
  filters: [
    lambda.FilterCriteria.filter({
      eventName: lambda.FilterRule.isEqual('INSERT'),
      dynamodb: {
        NewImage: {
          id: { BOOL: lambda.FilterRule.isEqual(true) },
        },
      },
    }),
  ],
}));
python
import aws_cdk.aws_dynamodb as dynamodb
from aws_cdk.aws_lambda_event_sources import DynamoEventSource

# table: dynamodb.Table

# fn: lambda.Function

fn.add_event_source(DynamoEventSource(table,
    starting_position=lambda_.StartingPosition.LATEST,
    filters=[
        lambda_.FilterCriteria.filter({
            "event_name": lambda_.FilterRule.is_equal("INSERT"),
            "dynamodb": {
                "NewImage": {
                    "id": {"BOOL": lambda_.FilterRule.is_equal(True)}
                }
            }
        })
    ]
))
java
import software.amazon.awscdk.services.dynamodb.*;
import software.amazon.awscdk.services.lambda.eventsources.DynamoEventSource;

Table table;

Function fn;

fn.addEventSource(DynamoEventSource.Builder.create(table)
        .startingPosition(StartingPosition.LATEST)
        .filters(List.of(FilterCriteria.filter(Map.of(
                "eventName", FilterRule.isEqual("INSERT"),
                "dynamodb", Map.of(
                        "NewImage", Map.of(
                                "id", Map.of("BOOL", FilterRule.isEqual(true))))))))
        .build());
csharp
using Amazon.CDK.AWS.DynamoDB;
using Amazon.CDK.AWS.Lambda.EventSources;

Table table;

Function fn;

fn.AddEventSource(new DynamoEventSource(table, new DynamoEventSourceProps {
    StartingPosition = StartingPosition.LATEST,
    Filters = new [] { FilterCriteria.Filter(new Dictionary<string, object> {
        { "eventName", FilterRule.IsEqual("INSERT") },
        { "dynamodb", new Dictionary<string, IDictionary<string, IDictionary<string, object>>> {
            { "NewImage", new Struct {
                Id = new Struct { BOOL = FilterRule.IsEqual(true) }
            } }
        } }
    }) }
}));
go
import dynamodb "github.com/aws/aws-cdk-go/awscdk"
import "github.com/aws/aws-cdk-go/awscdk"

var table Table

var fn Function

fn.AddEventSource(awscdk.NewDynamoEventSource(table, &DynamoEventSourceProps{
	StartingPosition: lambda.StartingPosition_LATEST,
	Filters: []map[string]interface{}{
		lambda.FilterCriteria_Filter(map[string]interface{}{
			"eventName": lambda.FilterRule_isEqual(jsii.String("INSERT")),
			"dynamodb": map[string]map[string]map[string]interface{}{
				"NewImage": map[string]map[string]interface{}{
					"id": map[string]interface{}{
						"BOOL": lambda.FilterRule_isEqual(jsii.Boolean(true)),
					},
				},
			},
		}),
	},
}))

Kinesis ​

You can write Lambda functions to process streaming data in Amazon Kinesis Streams. For more information about Amazon Kinesis, see Amazon Kinesis Service. To learn more about configuring Lambda function event sources with kinesis and view a sample event, see Amazon Kinesis Event.

To set up Amazon Kinesis as an event source for AWS Lambda, you first create or update an Amazon Kinesis stream and select custom values for the event source parameters. The following parameters will impact Amazon Kinesis's polling behavior:

  • batchSize: Determines how many records are buffered before invoking your lambda function - could impact your function's memory usage (if too high) and ability to keep up with incoming data velocity (if too low).
  • bisectBatchOnError: If a batch encounters an error, this will cause the batch to be split in two and have each new smaller batch retried, allowing the records in error to be isolated.
  • reportBatchItemFailures: Allow functions to return partially successful responses for a batch of records.
  • maxBatchingWindow: The maximum amount of time to gather records before invoking the lambda. This increases the likelihood of a full batch at the cost of possibly delaying processing.
  • maxRecordAge: The maximum age of a record that will be sent to the function for processing. Records that exceed the max age will be treated as failures.
  • onFailure: In the event a record fails and consumes all retries, the record will be sent to S3 bucket, SQS queue or SNS topic that is specified here
  • parallelizationFactor: The number of batches to concurrently process on each shard.
  • retryAttempts: The maximum number of times a record should be retried in the event of failure.
  • startingPosition: Will determine where to begin consumption. 'LATEST' will start at the most recent record and ignore all records that arrived prior to attaching the event source, 'TRIM_HORIZON' will start at the oldest record and ensure you process all available data, while 'AT_TIMESTAMP' will start reading records from a specified time stamp.
  • startingPositionTimestamp: The time stamp from which to start reading. Used in conjunction with startingPosition when set to 'AT_TIMESTAMP'.
  • tumblingWindow: The duration in seconds of a processing window when using streams.
  • enabled: If the event source mapping should be enabled. The default is true.
ts
import * as kinesis from 'aws-cdk-lib/aws-kinesis';
import { KinesisEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';

const stream = new kinesis.Stream(this, 'MyStream');

declare const myFunction: lambda.Function;
myFunction.addEventSource(new KinesisEventSource(stream, {
  batchSize: 100, // default
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
}));
python
import aws_cdk.aws_kinesis as kinesis
from aws_cdk.aws_lambda_event_sources import KinesisEventSource

# my_function: lambda.Function


stream = kinesis.Stream(self, "MyStream")
my_function.add_event_source(KinesisEventSource(stream,
    batch_size=100,  # default
    starting_position=lambda_.StartingPosition.TRIM_HORIZON
))
java
import software.amazon.awscdk.services.kinesis.*;
import software.amazon.awscdk.services.lambda.eventsources.KinesisEventSource;

Function myFunction;


Stream stream = new Stream(this, "MyStream");
myFunction.addEventSource(KinesisEventSource.Builder.create(stream)
        .batchSize(100) // default
        .startingPosition(StartingPosition.TRIM_HORIZON)
        .build());
csharp
using Amazon.CDK.AWS.Kinesis;
using Amazon.CDK.AWS.Lambda.EventSources;

Function myFunction;


var stream = new Stream(this, "MyStream");
myFunction.AddEventSource(new KinesisEventSource(stream, new KinesisEventSourceProps {
    BatchSize = 100,  // default
    StartingPosition = StartingPosition.TRIM_HORIZON
}));
go
import kinesis "github.com/aws/aws-cdk-go/awscdk"
import "github.com/aws/aws-cdk-go/awscdk"

var myFunction Function


stream := kinesis.NewStream(this, jsii.String("MyStream"))
myFunction.AddEventSource(awscdk.NewKinesisEventSource(stream, &KinesisEventSourceProps{
	BatchSize: jsii.Number(100),
	 // default
	StartingPosition: lambda.StartingPosition_TRIM_HORIZON,
}))

To use a dedicated-throughput consumer with enhanced fan-out

ts
import * as kinesis from 'aws-cdk-lib/aws-kinesis';
import { KinesisConsumerEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';

const stream = new kinesis.Stream(this, 'MyStream');
const streamConsumer = new kinesis.StreamConsumer(this, 'MyStreamConsumer', {
  stream,
  streamConsumerName: 'MyStreamConsumer',
});

declare const myFunction: lambda.Function;
myFunction.addEventSource(new KinesisConsumerEventSource(streamConsumer, {
  batchSize: 100, // default
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
}));
python
import aws_cdk.aws_kinesis as kinesis
from aws_cdk.aws_lambda_event_sources import KinesisConsumerEventSource

# my_function: lambda.Function


stream = kinesis.Stream(self, "MyStream")
stream_consumer = kinesis.StreamConsumer(self, "MyStreamConsumer",
    stream=stream,
    stream_consumer_name="MyStreamConsumer"
)
my_function.add_event_source(KinesisConsumerEventSource(stream_consumer,
    batch_size=100,  # default
    starting_position=lambda_.StartingPosition.TRIM_HORIZON
))
java
import software.amazon.awscdk.services.kinesis.*;
import software.amazon.awscdk.services.lambda.eventsources.KinesisConsumerEventSource;

Function myFunction;


Stream stream = new Stream(this, "MyStream");
StreamConsumer streamConsumer = StreamConsumer.Builder.create(this, "MyStreamConsumer")
        .stream(stream)
        .streamConsumerName("MyStreamConsumer")
        .build();
myFunction.addEventSource(KinesisConsumerEventSource.Builder.create(streamConsumer)
        .batchSize(100) // default
        .startingPosition(StartingPosition.TRIM_HORIZON)
        .build());
csharp
using Amazon.CDK.AWS.Kinesis;
using Amazon.CDK.AWS.Lambda.EventSources;

Function myFunction;


var stream = new Stream(this, "MyStream");
var streamConsumer = new StreamConsumer(this, "MyStreamConsumer", new StreamConsumerProps {
    Stream = stream,
    StreamConsumerName = "MyStreamConsumer"
});
myFunction.AddEventSource(new KinesisConsumerEventSource(streamConsumer, new KinesisEventSourceProps {
    BatchSize = 100,  // default
    StartingPosition = StartingPosition.TRIM_HORIZON
}));
go
import "github.com/aws/aws-cdk-go/awscdk"
import "github.com/aws/aws-cdk-go/awscdk"

var myFunction Function


stream := kinesis.NewStream(this, jsii.String("MyStream"))
streamConsumer := kinesis.NewStreamConsumer(this, jsii.String("MyStreamConsumer"), &StreamConsumerProps{
	Stream: Stream,
	StreamConsumerName: jsii.String("MyStreamConsumer"),
})
myFunction.AddEventSource(awscdk.NewKinesisConsumerEventSource(streamConsumer, &KinesisEventSourceProps{
	BatchSize: jsii.Number(100),
	 // default
	StartingPosition: lambda.StartingPosition_TRIM_HORIZON,
}))

Kafka ​

You can write Lambda functions to process data either from Amazon MSK or a self-managed Kafka cluster. The following parameters will impact to the polling behavior:

  • startingPosition: Will determine where to begin consumption. 'LATEST' will start at the most recent record and ignore all records that arrived prior to attaching the event source, 'TRIM_HORIZON' will start at the oldest record and ensure you process all available data, while 'AT_TIMESTAMP' will start reading records from a specified time stamp.
  • startingPositionTimestamp: The time stamp from which to start reading. Used in conjunction with startingPosition when set to 'AT_TIMESTAMP'.
  • batchSize: Determines how many records are buffered before invoking your lambda function - could impact your function's memory usage (if too high) and ability to keep up with incoming data velocity (if too low).
  • maxBatchingWindow: The maximum amount of time to gather records before invoking the lambda. This increases the likelihood of a full batch at the cost of possibly delaying processing.
  • onFailure: In the event a record fails and consumes all retries, the record will be sent to SQS queue or SNS topic that is specified here
  • enabled: If the Kafka event source mapping should be enabled. The default is true.
  • bisectBatchOnError: If a batch encounters an error, this will cause the batch to be split in two and have each new smaller batch retried, allowing the records in error to be isolated. Available in provisioned mode only.
  • reportBatchItemFailures: Allow functions to return partially successful responses for a batch of records. Available in provisioned mode only.
  • retryAttempts: The maximum number of times a record should be retried in the event of failure. Available in provisioned mode only.
  • maxRecordAge: The maximum age of a record that will be sent to the function for processing. Records that exceed the max age will be treated as failures. Available in provisioned mode only.
  • provisionedPollerConfig: Controls the number of pollers that can be provisioned to read from the event source. Valid ranges: minimumPollers 1–200 (default: 1), maximumPollers 1–2000 (default: 200).

The following code sets up Amazon MSK as an event source for a lambda function. Credentials will need to be configured to access the MSK cluster, as described in Username/Password authentication.

ts
import { Secret } from 'aws-cdk-lib/aws-secretsmanager';
import { ManagedKafkaEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';

// Your MSK cluster arn
const clusterArn = 'arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4';

// The Kafka topic you want to subscribe to
const topic = 'some-cool-topic';

// The secret that allows access to your MSK cluster
const secret = new Secret(this, 'Secret', { secretName: 'AmazonMSK_KafkaSecret' });

declare const myFunction: lambda.Function;
myFunction.addEventSource(new ManagedKafkaEventSource({
  clusterArn,
  topic: topic,
  secret: secret,
  batchSize: 100, // default
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
  bisectBatchOnError: true,
  reportBatchItemFailures: true,
  retryAttempts: 3,
  maxRecordAge: Duration.hours(24),
  provisionedPollerConfig: {
    minimumPollers: 1,
    maximumPollers: 3,
  },
}));
python
from aws_cdk.aws_lambda_event_sources import ProvisionedPollerConfig
from aws_cdk.aws_secretsmanager import Secret
from aws_cdk.aws_lambda_event_sources import ManagedKafkaEventSource

# my_function: lambda.Function


# Your MSK cluster arn
cluster_arn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4"

# The Kafka topic you want to subscribe to
topic = "some-cool-topic"

# The secret that allows access to your MSK cluster
secret = Secret(self, "Secret", secret_name="AmazonMSK_KafkaSecret")
my_function.add_event_source(ManagedKafkaEventSource(
    cluster_arn=cluster_arn,
    topic=topic,
    secret=secret,
    batch_size=100,  # default
    starting_position=lambda_.StartingPosition.TRIM_HORIZON,
    bisect_batch_on_error=True,
    report_batch_item_failures=True,
    retry_attempts=3,
    max_record_age=Duration.hours(24),
    provisioned_poller_config=ProvisionedPollerConfig(
        minimum_pollers=1,
        maximum_pollers=3
    )
))
java
import software.amazon.awscdk.services.secretsmanager.Secret;
import software.amazon.awscdk.services.lambda.eventsources.ManagedKafkaEventSource;

Function myFunction;


// Your MSK cluster arn
String clusterArn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4";

// The Kafka topic you want to subscribe to
String topic = "some-cool-topic";

// The secret that allows access to your MSK cluster
Secret secret = Secret.Builder.create(this, "Secret").secretName("AmazonMSK_KafkaSecret").build();
myFunction.addEventSource(ManagedKafkaEventSource.Builder.create()
        .clusterArn(clusterArn)
        .topic(topic)
        .secret(secret)
        .batchSize(100) // default
        .startingPosition(StartingPosition.TRIM_HORIZON)
        .bisectBatchOnError(true)
        .reportBatchItemFailures(true)
        .retryAttempts(3)
        .maxRecordAge(Duration.hours(24))
        .provisionedPollerConfig(ProvisionedPollerConfig.builder()
                .minimumPollers(1)
                .maximumPollers(3)
                .build())
        .build());
csharp
using Amazon.CDK.AWS.SecretsManager;
using Amazon.CDK.AWS.Lambda.EventSources;

Function myFunction;


// Your MSK cluster arn
var clusterArn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4";

// The Kafka topic you want to subscribe to
var topic = "some-cool-topic";

// The secret that allows access to your MSK cluster
var secret = new Secret(this, "Secret", new SecretProps { SecretName = "AmazonMSK_KafkaSecret" });
myFunction.AddEventSource(new ManagedKafkaEventSource(new ManagedKafkaEventSourceProps {
    ClusterArn = clusterArn,
    Topic = topic,
    Secret = secret,
    BatchSize = 100,  // default
    StartingPosition = StartingPosition.TRIM_HORIZON,
    BisectBatchOnError = true,
    ReportBatchItemFailures = true,
    RetryAttempts = 3,
    MaxRecordAge = Duration.Hours(24),
    ProvisionedPollerConfig = new ProvisionedPollerConfig {
        MinimumPollers = 1,
        MaximumPollers = 3
    }
}));
go
import "github.com/aws/aws-cdk-go/awscdk"
import "github.com/aws/aws-cdk-go/awscdk"

var myFunction Function


// Your MSK cluster arn
clusterArn := "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4"

// The Kafka topic you want to subscribe to
topic := "some-cool-topic"

// The secret that allows access to your MSK cluster
secret := awscdk.NewSecret(this, jsii.String("Secret"), &SecretProps{
	SecretName: jsii.String("AmazonMSK_KafkaSecret"),
})
myFunction.AddEventSource(awscdk.NewManagedKafkaEventSource(&ManagedKafkaEventSourceProps{
	ClusterArn: jsii.String(ClusterArn),
	Topic: topic,
	Secret: secret,
	BatchSize: jsii.Number(100),
	 // default
	StartingPosition: lambda.StartingPosition_TRIM_HORIZON,
	BisectBatchOnError: jsii.Boolean(true),
	ReportBatchItemFailures: jsii.Boolean(true),
	RetryAttempts: jsii.Number(3),
	MaxRecordAge: awscdk.Duration_Hours(jsii.Number(24)),
	ProvisionedPollerConfig: &ProvisionedPollerConfig{
		MinimumPollers: jsii.Number(1),
		MaximumPollers: jsii.Number(3),
	},
}))

The following code sets up a self managed Kafka cluster as an event source. Username and password based authentication will need to be set up as described in Managing access and permissions.

ts
import { Secret } from 'aws-cdk-lib/aws-secretsmanager';
import { SelfManagedKafkaEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';

// The list of Kafka brokers
const bootstrapServers = ['kafka-broker:9092'];

// The Kafka topic you want to subscribe to
const topic = 'some-cool-topic';

// The secret that allows access to your self hosted Kafka cluster
declare const secret: Secret;

// (Optional) The consumer group id to use when connecting to the Kafka broker. If omitted the UUID of the event source mapping will be used.
const consumerGroupId = "my-consumer-group-id";

declare const myFunction: lambda.Function;
myFunction.addEventSource(new SelfManagedKafkaEventSource({
  bootstrapServers: bootstrapServers,
  topic: topic,
  consumerGroupId: consumerGroupId,
  secret: secret,
  batchSize: 100, // default
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
  bisectBatchOnError: true,
  reportBatchItemFailures: true,
  retryAttempts: 3,
  maxRecordAge: Duration.hours(24),
  provisionedPollerConfig: {
    minimumPollers: 1,
    maximumPollers: 3,
  },
}));
python
from aws_cdk.aws_lambda_event_sources import ProvisionedPollerConfig
from aws_cdk.aws_secretsmanager import Secret
from aws_cdk.aws_lambda_event_sources import SelfManagedKafkaEventSource

# The secret that allows access to your self hosted Kafka cluster
# secret: Secret

# my_function: lambda.Function


# The list of Kafka brokers
bootstrap_servers = ["kafka-broker:9092"]

# The Kafka topic you want to subscribe to
topic = "some-cool-topic"

# (Optional) The consumer group id to use when connecting to the Kafka broker. If omitted the UUID of the event source mapping will be used.
consumer_group_id = "my-consumer-group-id"
my_function.add_event_source(SelfManagedKafkaEventSource(
    bootstrap_servers=bootstrap_servers,
    topic=topic,
    consumer_group_id=consumer_group_id,
    secret=secret,
    batch_size=100,  # default
    starting_position=lambda_.StartingPosition.TRIM_HORIZON,
    bisect_batch_on_error=True,
    report_batch_item_failures=True,
    retry_attempts=3,
    max_record_age=Duration.hours(24),
    provisioned_poller_config=ProvisionedPollerConfig(
        minimum_pollers=1,
        maximum_pollers=3
    )
))
java
import software.amazon.awscdk.services.secretsmanager.Secret;
import software.amazon.awscdk.services.lambda.eventsources.SelfManagedKafkaEventSource;

// The secret that allows access to your self hosted Kafka cluster
Secret secret;

Function myFunction;


// The list of Kafka brokers
String[] bootstrapServers = List.of("kafka-broker:9092");

// The Kafka topic you want to subscribe to
String topic = "some-cool-topic";

// (Optional) The consumer group id to use when connecting to the Kafka broker. If omitted the UUID of the event source mapping will be used.
String consumerGroupId = "my-consumer-group-id";
myFunction.addEventSource(SelfManagedKafkaEventSource.Builder.create()
        .bootstrapServers(bootstrapServers)
        .topic(topic)
        .consumerGroupId(consumerGroupId)
        .secret(secret)
        .batchSize(100) // default
        .startingPosition(StartingPosition.TRIM_HORIZON)
        .bisectBatchOnError(true)
        .reportBatchItemFailures(true)
        .retryAttempts(3)
        .maxRecordAge(Duration.hours(24))
        .provisionedPollerConfig(ProvisionedPollerConfig.builder()
                .minimumPollers(1)
                .maximumPollers(3)
                .build())
        .build());
csharp
using Amazon.CDK.AWS.SecretsManager;
using Amazon.CDK.AWS.Lambda.EventSources;

// The secret that allows access to your self hosted Kafka cluster
Secret secret;

Function myFunction;


// The list of Kafka brokers
var bootstrapServers = new [] { "kafka-broker:9092" };

// The Kafka topic you want to subscribe to
var topic = "some-cool-topic";

// (Optional) The consumer group id to use when connecting to the Kafka broker. If omitted the UUID of the event source mapping will be used.
var consumerGroupId = "my-consumer-group-id";
myFunction.AddEventSource(new SelfManagedKafkaEventSource(new SelfManagedKafkaEventSourceProps {
    BootstrapServers = bootstrapServers,
    Topic = topic,
    ConsumerGroupId = consumerGroupId,
    Secret = secret,
    BatchSize = 100,  // default
    StartingPosition = StartingPosition.TRIM_HORIZON,
    BisectBatchOnError = true,
    ReportBatchItemFailures = true,
    RetryAttempts = 3,
    MaxRecordAge = Duration.Hours(24),
    ProvisionedPollerConfig = new ProvisionedPollerConfig {
        MinimumPollers = 1,
        MaximumPollers = 3
    }
}));
go
import "github.com/aws/aws-cdk-go/awscdk"
import "github.com/aws/aws-cdk-go/awscdk"

// The secret that allows access to your self hosted Kafka cluster
var secret Secret

var myFunction Function


// The list of Kafka brokers
bootstrapServers := []*string{
	"kafka-broker:9092",
}

// The Kafka topic you want to subscribe to
topic := "some-cool-topic"

// (Optional) The consumer group id to use when connecting to the Kafka broker. If omitted the UUID of the event source mapping will be used.
consumerGroupId := "my-consumer-group-id"
myFunction.AddEventSource(awscdk.NewSelfManagedKafkaEventSource(&SelfManagedKafkaEventSourceProps{
	BootstrapServers: bootstrapServers,
	Topic: topic,
	ConsumerGroupId: consumerGroupId,
	Secret: secret,
	BatchSize: jsii.Number(100),
	 // default
	StartingPosition: lambda.StartingPosition_TRIM_HORIZON,
	BisectBatchOnError: jsii.Boolean(true),
	ReportBatchItemFailures: jsii.Boolean(true),
	RetryAttempts: jsii.Number(3),
	MaxRecordAge: awscdk.Duration_Hours(jsii.Number(24)),
	ProvisionedPollerConfig: &ProvisionedPollerConfig{
		MinimumPollers: jsii.Number(1),
		MaximumPollers: jsii.Number(3),
	},
}))

If your self managed Kafka cluster is only reachable via VPC also configure vpc vpcSubnets and securityGroup.

You can specify event filtering for managed and self managed Kafka clusters using the filters property:

ts
import { ManagedKafkaEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';

// Your MSK cluster arn
const clusterArn = 'arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4';

// The Kafka topic you want to subscribe to
const topic = 'some-cool-topic';

declare const myFunction: lambda.Function;
myFunction.addEventSource(new ManagedKafkaEventSource({
  clusterArn,
  topic,
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
  filters: [
    lambda.FilterCriteria.filter({
      stringEquals: lambda.FilterRule.isEqual('test'),
    }),
  ],
}));
python
from aws_cdk.aws_lambda_event_sources import ManagedKafkaEventSource

# my_function: lambda.Function


# Your MSK cluster arn
cluster_arn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4"

# The Kafka topic you want to subscribe to
topic = "some-cool-topic"
my_function.add_event_source(ManagedKafkaEventSource(
    cluster_arn=cluster_arn,
    topic=topic,
    starting_position=lambda_.StartingPosition.TRIM_HORIZON,
    filters=[
        lambda_.FilterCriteria.filter({
            "string_equals": lambda_.FilterRule.is_equal("test")
        })
    ]
))
java
import software.amazon.awscdk.services.lambda.eventsources.ManagedKafkaEventSource;

Function myFunction;


// Your MSK cluster arn
String clusterArn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4";

// The Kafka topic you want to subscribe to
String topic = "some-cool-topic";
myFunction.addEventSource(ManagedKafkaEventSource.Builder.create()
        .clusterArn(clusterArn)
        .topic(topic)
        .startingPosition(StartingPosition.TRIM_HORIZON)
        .filters(List.of(FilterCriteria.filter(Map.of(
                "stringEquals", FilterRule.isEqual("test")))))
        .build());
csharp
using Amazon.CDK.AWS.Lambda.EventSources;

Function myFunction;


// Your MSK cluster arn
var clusterArn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4";

// The Kafka topic you want to subscribe to
var topic = "some-cool-topic";
myFunction.AddEventSource(new ManagedKafkaEventSource(new ManagedKafkaEventSourceProps {
    ClusterArn = clusterArn,
    Topic = topic,
    StartingPosition = StartingPosition.TRIM_HORIZON,
    Filters = new [] { FilterCriteria.Filter(new Dictionary<string, object> {
        { "stringEquals", FilterRule.IsEqual("test") }
    }) }
}));
go
import "github.com/aws/aws-cdk-go/awscdk"

var myFunction Function


// Your MSK cluster arn
clusterArn := "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4"

// The Kafka topic you want to subscribe to
topic := "some-cool-topic"
myFunction.AddEventSource(awscdk.NewManagedKafkaEventSource(&ManagedKafkaEventSourceProps{
	ClusterArn: jsii.String(ClusterArn),
	Topic: jsii.String(Topic),
	StartingPosition: lambda.StartingPosition_TRIM_HORIZON,
	Filters: []map[string]interface{}{
		lambda.FilterCriteria_Filter(map[string]interface{}{
			"stringEquals": lambda.FilterRule_isEqual(jsii.String("test")),
		}),
	},
}))

By default, Lambda will encrypt Filter Criteria using AWS managed keys. But if you want to use a self managed KMS key to encrypt the filters, You can specify the self managed key using the filterEncryption property.

ts
import { ManagedKafkaEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';
import { Key } from 'aws-cdk-lib/aws-kms';

// Your MSK cluster arn
const clusterArn = 'arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4';

// The Kafka topic you want to subscribe to
const topic = 'some-cool-topic';

// Your self managed KMS key
const myKey = Key.fromKeyArn(
  this,
  'SourceBucketEncryptionKey',
  'arn:aws:kms:us-east-1:123456789012:key/<key-id>',
);

declare const myFunction: lambda.Function;
myFunction.addEventSource(new ManagedKafkaEventSource({
  clusterArn,
  topic,
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
  filters: [
    lambda.FilterCriteria.filter({
      stringEquals: lambda.FilterRule.isEqual('test'),
    }),
  ],
  filterEncryption: myKey,
}));
python
from aws_cdk.aws_lambda_event_sources import ManagedKafkaEventSource
from aws_cdk.aws_kms import Key

# my_function: lambda.Function


# Your MSK cluster arn
cluster_arn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4"

# The Kafka topic you want to subscribe to
topic = "some-cool-topic"

# Your self managed KMS key
my_key = Key.from_key_arn(self, "SourceBucketEncryptionKey", "arn:aws:kms:us-east-1:123456789012:key/<key-id>")
my_function.add_event_source(ManagedKafkaEventSource(
    cluster_arn=cluster_arn,
    topic=topic,
    starting_position=lambda_.StartingPosition.TRIM_HORIZON,
    filters=[
        lambda_.FilterCriteria.filter({
            "string_equals": lambda_.FilterRule.is_equal("test")
        })
    ],
    filter_encryption=my_key
))
java
import software.amazon.awscdk.services.lambda.eventsources.ManagedKafkaEventSource;
import software.amazon.awscdk.services.kms.Key;

Function myFunction;


// Your MSK cluster arn
String clusterArn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4";

// The Kafka topic you want to subscribe to
String topic = "some-cool-topic";

// Your self managed KMS key
IKey myKey = Key.fromKeyArn(this, "SourceBucketEncryptionKey", "arn:aws:kms:us-east-1:123456789012:key/<key-id>");
myFunction.addEventSource(ManagedKafkaEventSource.Builder.create()
        .clusterArn(clusterArn)
        .topic(topic)
        .startingPosition(StartingPosition.TRIM_HORIZON)
        .filters(List.of(FilterCriteria.filter(Map.of(
                "stringEquals", FilterRule.isEqual("test")))))
        .filterEncryption(myKey)
        .build());
csharp
using Amazon.CDK.AWS.Lambda.EventSources;
using Amazon.CDK.AWS.KMS;

Function myFunction;


// Your MSK cluster arn
var clusterArn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4";

// The Kafka topic you want to subscribe to
var topic = "some-cool-topic";

// Your self managed KMS key
var myKey = Key.FromKeyArn(this, "SourceBucketEncryptionKey", "arn:aws:kms:us-east-1:123456789012:key/<key-id>");
myFunction.AddEventSource(new ManagedKafkaEventSource(new ManagedKafkaEventSourceProps {
    ClusterArn = clusterArn,
    Topic = topic,
    StartingPosition = StartingPosition.TRIM_HORIZON,
    Filters = new [] { FilterCriteria.Filter(new Dictionary<string, object> {
        { "stringEquals", FilterRule.IsEqual("test") }
    }) },
    FilterEncryption = myKey
}));
go
import "github.com/aws/aws-cdk-go/awscdk"
import "github.com/aws/aws-cdk-go/awscdk"

var myFunction Function


// Your MSK cluster arn
clusterArn := "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4"

// The Kafka topic you want to subscribe to
topic := "some-cool-topic"

// Your self managed KMS key
myKey := awscdk.Key_FromKeyArn(this, jsii.String("SourceBucketEncryptionKey"), jsii.String("arn:aws:kms:us-east-1:123456789012:key/<key-id>"))
myFunction.AddEventSource(awscdk.NewManagedKafkaEventSource(&ManagedKafkaEventSourceProps{
	ClusterArn: jsii.String(ClusterArn),
	Topic: jsii.String(Topic),
	StartingPosition: lambda.StartingPosition_TRIM_HORIZON,
	Filters: []map[string]interface{}{
		lambda.FilterCriteria_Filter(map[string]interface{}{
			"stringEquals": lambda.FilterRule_isEqual(jsii.String("test")),
		}),
	},
	FilterEncryption: myKey,
}))

Consumption mode (Kafka queues) ​

By default, Lambda reads a Kafka topic as a consumer group: each partition is read by one poller, in order (lambda.KafkaConsumptionMode.STREAM). If your self-managed cluster runs Kafka 4.2.0 or later, you can instead read the topic as a share group (lambda.KafkaConsumptionMode.QUEUE). In queue mode, several pollers can read the same partition in parallel, which removes head-of-line blocking at the cost of ordering.

ts
import { Secret } from 'aws-cdk-lib/aws-secretsmanager';
import { SelfManagedKafkaEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';

declare const myFunction: lambda.Function;
declare const secret: Secret;

myFunction.addEventSource(new SelfManagedKafkaEventSource({
  bootstrapServers: ['kafka-broker-1:9092', 'kafka-broker-2:9092'],
  topic: 'orders',
  secret,
  consumerGroupId: 'orders-queue',
  consumptionMode: lambda.KafkaConsumptionMode.QUEUE,
}));
python
from aws_cdk.aws_secretsmanager import Secret
from aws_cdk.aws_lambda_event_sources import SelfManagedKafkaEventSource

# my_function: lambda.Function
# secret: Secret


my_function.add_event_source(SelfManagedKafkaEventSource(
    bootstrap_servers=["kafka-broker-1:9092", "kafka-broker-2:9092"],
    topic="orders",
    secret=secret,
    consumer_group_id="orders-queue",
    consumption_mode=lambda_.KafkaConsumptionMode.QUEUE
))
java
import software.amazon.awscdk.services.secretsmanager.Secret;
import software.amazon.awscdk.services.lambda.eventsources.SelfManagedKafkaEventSource;

Function myFunction;
Secret secret;


myFunction.addEventSource(SelfManagedKafkaEventSource.Builder.create()
        .bootstrapServers(List.of("kafka-broker-1:9092", "kafka-broker-2:9092"))
        .topic("orders")
        .secret(secret)
        .consumerGroupId("orders-queue")
        .consumptionMode(KafkaConsumptionMode.QUEUE)
        .build());
csharp
using Amazon.CDK.AWS.SecretsManager;
using Amazon.CDK.AWS.Lambda.EventSources;

Function myFunction;
Secret secret;


myFunction.AddEventSource(new SelfManagedKafkaEventSource(new SelfManagedKafkaEventSourceProps {
    BootstrapServers = new [] { "kafka-broker-1:9092", "kafka-broker-2:9092" },
    Topic = "orders",
    Secret = secret,
    ConsumerGroupId = "orders-queue",
    ConsumptionMode = KafkaConsumptionMode.QUEUE
}));
go
import "github.com/aws/aws-cdk-go/awscdk"
import "github.com/aws/aws-cdk-go/awscdk"

var myFunction Function
var secret Secret


myFunction.AddEventSource(awscdk.NewSelfManagedKafkaEventSource(&SelfManagedKafkaEventSourceProps{
	BootstrapServers: []*string{
		jsii.String("kafka-broker-1:9092"),
		jsii.String("kafka-broker-2:9092"),
	},
	Topic: jsii.String("orders"),
	Secret: Secret,
	ConsumerGroupId: jsii.String("orders-queue"),
	ConsumptionMode: lambda.KafkaConsumptionMode_QUEUE(),
}))

Queue mode changes how some options behave:

  • startingPosition and startingPositionTimestamp aren't supported. Leave them unset.
  • maxBatchingWindow isn't supported. Leave it unset.
  • retryAttempts accepts 0 to 10000, and defaults to 0. -1 (infinite retries) isn't supported. As in stream mode, retryAttempts is only available in provisioned mode (provisionedPollerConfig).
  • The consumption mode can't be changed after the event source mapping is created. To switch modes, create a new event source (for example, with a different consumerGroupId) and remove the old one.
  • consumerGroupId is the name of the share group. Kafka doesn't allow a consumer group and a share group with the same name, so if the ID already exists on the broker as a consumer group, Lambda disables the mapping. Use a new ID, or omit it to let Lambda generate one.

Other options work the same as in stream mode. Queue mode isn't available for ManagedKafkaEventSource (Amazon MSK).

Failure Destinations ​

You can specify failure destinations for records that fail processing. Kafka event sources support Kafka Topic Destinations, S3 Bucket Destinations, SQS Queue and SNS topic:

Kafka Topic Destination ​

For Kafka event sources, you can send failed records to another Kafka topic using KafkaDlq:

ts
import { ManagedKafkaEventSource, KafkaDlq } from 'aws-cdk-lib/aws-lambda-event-sources';

// Your MSK cluster arn
const clusterArn = 'arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4';

// The Kafka topic you want to subscribe to
const topic = 'some-cool-topic';

declare const myFunction: lambda.Function;

// Create a Kafka DLQ destination
const kafkaDlq = new KafkaDlq('failure-topic');

myFunction.addEventSource(new ManagedKafkaEventSource({
  clusterArn,
  topic,
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
  onFailure: kafkaDlq,
  provisionedPollerConfig: {
    minimumPollers: 1,
    maximumPollers: 1,
  },
}));
python
from aws_cdk.aws_lambda_event_sources import ProvisionedPollerConfig
from aws_cdk.aws_lambda_event_sources import ManagedKafkaEventSource, KafkaDlq

# my_function: lambda.Function


# Your MSK cluster arn
cluster_arn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4"

# The Kafka topic you want to subscribe to
topic = "some-cool-topic"

# Create a Kafka DLQ destination
kafka_dlq = KafkaDlq("failure-topic")

my_function.add_event_source(ManagedKafkaEventSource(
    cluster_arn=cluster_arn,
    topic=topic,
    starting_position=lambda_.StartingPosition.TRIM_HORIZON,
    on_failure=kafka_dlq,
    provisioned_poller_config=ProvisionedPollerConfig(
        minimum_pollers=1,
        maximum_pollers=1
    )
))
java
import software.amazon.awscdk.services.lambda.eventsources.ManagedKafkaEventSource;
import software.amazon.awscdk.services.lambda.eventsources.KafkaDlq;

Function myFunction;


// Your MSK cluster arn
String clusterArn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4";

// The Kafka topic you want to subscribe to
String topic = "some-cool-topic";

// Create a Kafka DLQ destination
KafkaDlq kafkaDlq = new KafkaDlq("failure-topic");

myFunction.addEventSource(ManagedKafkaEventSource.Builder.create()
        .clusterArn(clusterArn)
        .topic(topic)
        .startingPosition(StartingPosition.TRIM_HORIZON)
        .onFailure(kafkaDlq)
        .provisionedPollerConfig(ProvisionedPollerConfig.builder()
                .minimumPollers(1)
                .maximumPollers(1)
                .build())
        .build());
csharp
using Amazon.CDK.AWS.Lambda.EventSources;

Function myFunction;


// Your MSK cluster arn
var clusterArn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4";

// The Kafka topic you want to subscribe to
var topic = "some-cool-topic";

// Create a Kafka DLQ destination
var kafkaDlq = new KafkaDlq("failure-topic");

myFunction.AddEventSource(new ManagedKafkaEventSource(new ManagedKafkaEventSourceProps {
    ClusterArn = clusterArn,
    Topic = topic,
    StartingPosition = StartingPosition.TRIM_HORIZON,
    OnFailure = kafkaDlq,
    ProvisionedPollerConfig = new ProvisionedPollerConfig {
        MinimumPollers = 1,
        MaximumPollers = 1
    }
}));
go
import "github.com/aws/aws-cdk-go/awscdk"

var myFunction Function


// Your MSK cluster arn
clusterArn := "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4"

// The Kafka topic you want to subscribe to
topic := "some-cool-topic"

// Create a Kafka DLQ destination
kafkaDlq := awscdk.NewKafkaDlq(jsii.String("failure-topic"))

myFunction.AddEventSource(awscdk.NewManagedKafkaEventSource(&ManagedKafkaEventSourceProps{
	ClusterArn: jsii.String(ClusterArn),
	Topic: jsii.String(Topic),
	StartingPosition: lambda.StartingPosition_TRIM_HORIZON,
	OnFailure: kafkaDlq,
	ProvisionedPollerConfig: &ProvisionedPollerConfig{
		MinimumPollers: jsii.Number(1),
		MaximumPollers: jsii.Number(1),
	},
}))

The same approach works with self-managed Kafka:

ts
import { SelfManagedKafkaEventSource, KafkaDlq } from 'aws-cdk-lib/aws-lambda-event-sources';

const bootstrapServers = ['kafka-broker:9092'];
const topic = 'some-cool-topic';

declare const myFunction: lambda.Function;

myFunction.addEventSource(new SelfManagedKafkaEventSource({
  bootstrapServers,
  topic,
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
  onFailure: new KafkaDlq('error-topic'),
  provisionedPollerConfig: {
    minimumPollers: 1,
    maximumPollers: 1,
  },
}));
python
from aws_cdk.aws_lambda_event_sources import ProvisionedPollerConfig
from aws_cdk.aws_lambda_event_sources import SelfManagedKafkaEventSource, KafkaDlq

# my_function: lambda.Function


bootstrap_servers = ["kafka-broker:9092"]
topic = "some-cool-topic"

my_function.add_event_source(SelfManagedKafkaEventSource(
    bootstrap_servers=bootstrap_servers,
    topic=topic,
    starting_position=lambda_.StartingPosition.TRIM_HORIZON,
    on_failure=KafkaDlq("error-topic"),
    provisioned_poller_config=ProvisionedPollerConfig(
        minimum_pollers=1,
        maximum_pollers=1
    )
))
java
import software.amazon.awscdk.services.lambda.eventsources.SelfManagedKafkaEventSource;
import software.amazon.awscdk.services.lambda.eventsources.KafkaDlq;

Function myFunction;


String[] bootstrapServers = List.of("kafka-broker:9092");
String topic = "some-cool-topic";

myFunction.addEventSource(SelfManagedKafkaEventSource.Builder.create()
        .bootstrapServers(bootstrapServers)
        .topic(topic)
        .startingPosition(StartingPosition.TRIM_HORIZON)
        .onFailure(new KafkaDlq("error-topic"))
        .provisionedPollerConfig(ProvisionedPollerConfig.builder()
                .minimumPollers(1)
                .maximumPollers(1)
                .build())
        .build());
csharp
using Amazon.CDK.AWS.Lambda.EventSources;

Function myFunction;


var bootstrapServers = new [] { "kafka-broker:9092" };
var topic = "some-cool-topic";

myFunction.AddEventSource(new SelfManagedKafkaEventSource(new SelfManagedKafkaEventSourceProps {
    BootstrapServers = bootstrapServers,
    Topic = topic,
    StartingPosition = StartingPosition.TRIM_HORIZON,
    OnFailure = new KafkaDlq("error-topic"),
    ProvisionedPollerConfig = new ProvisionedPollerConfig {
        MinimumPollers = 1,
        MaximumPollers = 1
    }
}));
go
import "github.com/aws/aws-cdk-go/awscdk"

var myFunction Function


bootstrapServers := []*string{
	"kafka-broker:9092",
}
topic := "some-cool-topic"

myFunction.AddEventSource(awscdk.NewSelfManagedKafkaEventSource(&SelfManagedKafkaEventSourceProps{
	BootstrapServers: BootstrapServers,
	Topic: jsii.String(Topic),
	StartingPosition: lambda.StartingPosition_TRIM_HORIZON,
	OnFailure: awscdk.NewKafkaDlq(jsii.String("error-topic")),
	ProvisionedPollerConfig: &ProvisionedPollerConfig{
		MinimumPollers: jsii.Number(1),
		MaximumPollers: jsii.Number(1),
	},
}))

S3 Bucket Destination ​

You can also specify an S3 bucket as an "on failure" destination:

ts
import { ManagedKafkaEventSource, S3OnFailureDestination } from 'aws-cdk-lib/aws-lambda-event-sources';
import { IBucket } from 'aws-cdk-lib/aws-s3';

// Your MSK cluster arn
const clusterArn = 'arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4';

// The Kafka topic you want to subscribe to
const topic = 'some-cool-topic';

declare const bucket: IBucket;
declare const myFunction: lambda.Function;

const s3OnFailureDestination = new S3OnFailureDestination(bucket);

myFunction.addEventSource(new ManagedKafkaEventSource({
  clusterArn,
  topic,
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
  onFailure: s3OnFailureDestination,
}));
python
from aws_cdk.aws_lambda_event_sources import ManagedKafkaEventSource, S3OnFailureDestination
from aws_cdk.aws_s3 import IBucket

# bucket: IBucket
# my_function: lambda.Function


# Your MSK cluster arn
cluster_arn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4"

# The Kafka topic you want to subscribe to
topic = "some-cool-topic"

s3_on_failure_destination = S3OnFailureDestination(bucket)

my_function.add_event_source(ManagedKafkaEventSource(
    cluster_arn=cluster_arn,
    topic=topic,
    starting_position=lambda_.StartingPosition.TRIM_HORIZON,
    on_failure=s3_on_failure_destination
))
java
import software.amazon.awscdk.services.lambda.eventsources.ManagedKafkaEventSource;
import software.amazon.awscdk.services.lambda.eventsources.S3OnFailureDestination;
import software.amazon.awscdk.services.s3.IBucket;

IBucket bucket;
Function myFunction;


// Your MSK cluster arn
String clusterArn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4";

// The Kafka topic you want to subscribe to
String topic = "some-cool-topic";

S3OnFailureDestination s3OnFailureDestination = new S3OnFailureDestination(bucket);

myFunction.addEventSource(ManagedKafkaEventSource.Builder.create()
        .clusterArn(clusterArn)
        .topic(topic)
        .startingPosition(StartingPosition.TRIM_HORIZON)
        .onFailure(s3OnFailureDestination)
        .build());
csharp
using Amazon.CDK.AWS.Lambda.EventSources;
using Amazon.CDK.AWS.S3;

IBucket bucket;
Function myFunction;


// Your MSK cluster arn
var clusterArn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4";

// The Kafka topic you want to subscribe to
var topic = "some-cool-topic";

var s3OnFailureDestination = new S3OnFailureDestination(bucket);

myFunction.AddEventSource(new ManagedKafkaEventSource(new ManagedKafkaEventSourceProps {
    ClusterArn = clusterArn,
    Topic = topic,
    StartingPosition = StartingPosition.TRIM_HORIZON,
    OnFailure = s3OnFailureDestination
}));
go
import "github.com/aws/aws-cdk-go/awscdk"
import "github.com/aws/aws-cdk-go/awscdk"

var bucket IBucket
var myFunction Function


// Your MSK cluster arn
clusterArn := "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4"

// The Kafka topic you want to subscribe to
topic := "some-cool-topic"

s3OnFailureDestination := awscdk.NewS3OnFailureDestination(bucket)

myFunction.AddEventSource(awscdk.NewManagedKafkaEventSource(&ManagedKafkaEventSourceProps{
	ClusterArn: jsii.String(ClusterArn),
	Topic: jsii.String(Topic),
	StartingPosition: lambda.StartingPosition_TRIM_HORIZON,
	OnFailure: s3OnFailureDestination,
}))

Kafka Observability Features ​

AWS Lambda provides enhanced observability for Kafka event sources through logging and metrics configuration.

Important: Observability features (LogLevel and MetricsConfig) are only available when using provisioned mode.

Logging ​

You can configure the verbosity of logs generated by the polling infrastructure. This is particularly useful for troubleshooting connection issues, monitoring polling behavior, and understanding the internal operations of your event source mapping.

ts
import { ManagedKafkaEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';

// Your MSK cluster arn
const clusterArn = 'arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4';

declare const myFunction: lambda.Function;

// Configure INFO level logging for production monitoring
myFunction.addEventSource(new ManagedKafkaEventSource({
  clusterArn,
  topic: 'production-events',
  startingPosition: lambda.StartingPosition.LATEST,
  // Provisioned mode is required for observability features
  provisionedPollerConfig: {
    minimumPollers: 1,
    maximumPollers: 5,
  },
  logLevel: lambda.EventSourceMappingLogLevel.INFO
}));
python
from aws_cdk.aws_lambda_event_sources import ProvisionedPollerConfig
from aws_cdk.aws_lambda_event_sources import ManagedKafkaEventSource

# my_function: lambda.Function


# Your MSK cluster arn
cluster_arn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4"

# Configure INFO level logging for production monitoring
my_function.add_event_source(ManagedKafkaEventSource(
    cluster_arn=cluster_arn,
    topic="production-events",
    starting_position=lambda_.StartingPosition.LATEST,
    # Provisioned mode is required for observability features
    provisioned_poller_config=ProvisionedPollerConfig(
        minimum_pollers=1,
        maximum_pollers=5
    ),
    log_level=lambda_.EventSourceMappingLogLevel.INFO
))
java
import software.amazon.awscdk.services.lambda.eventsources.ManagedKafkaEventSource;

Function myFunction;


// Your MSK cluster arn
String clusterArn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4";

// Configure INFO level logging for production monitoring
myFunction.addEventSource(ManagedKafkaEventSource.Builder.create()
        .clusterArn(clusterArn)
        .topic("production-events")
        .startingPosition(StartingPosition.LATEST)
        // Provisioned mode is required for observability features
        .provisionedPollerConfig(ProvisionedPollerConfig.builder()
                .minimumPollers(1)
                .maximumPollers(5)
                .build())
        .logLevel(EventSourceMappingLogLevel.INFO)
        .build());
csharp
using Amazon.CDK.AWS.Lambda.EventSources;

Function myFunction;


// Your MSK cluster arn
var clusterArn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4";

// Configure INFO level logging for production monitoring
myFunction.AddEventSource(new ManagedKafkaEventSource(new ManagedKafkaEventSourceProps {
    ClusterArn = clusterArn,
    Topic = "production-events",
    StartingPosition = StartingPosition.LATEST,
    // Provisioned mode is required for observability features
    ProvisionedPollerConfig = new ProvisionedPollerConfig {
        MinimumPollers = 1,
        MaximumPollers = 5
    },
    LogLevel = EventSourceMappingLogLevel.INFO
}));
go
import "github.com/aws/aws-cdk-go/awscdk"

var myFunction Function


// Your MSK cluster arn
clusterArn := "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4"

// Configure INFO level logging for production monitoring
myFunction.AddEventSource(awscdk.NewManagedKafkaEventSource(&ManagedKafkaEventSourceProps{
	ClusterArn: jsii.String(ClusterArn),
	Topic: jsii.String("production-events"),
	StartingPosition: lambda.StartingPosition_LATEST,
	// Provisioned mode is required for observability features
	ProvisionedPollerConfig: &ProvisionedPollerConfig{
		MinimumPollers: jsii.Number(1),
		MaximumPollers: jsii.Number(5),
	},
	LogLevel: lambda.EventSourceMappingLogLevel_INFO,
}))

Metrics Configuration ​

Enhanced metrics provide detailed insights into your Kafka event source performance. Metrics include event processing rates, error counts, and Kafka-specific metrics like consumer lag.

ts
import { ManagedKafkaEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';

// Your MSK cluster arn
const clusterArn = 'arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4';

declare const myFunction: lambda.Function;

// Enable basic event and error metrics
myFunction.addEventSource(new ManagedKafkaEventSource({
  clusterArn,
  topic: 'basic-monitoring',
  startingPosition: lambda.StartingPosition.LATEST,
  // Provisioned mode is required for observability features
  provisionedPollerConfig: {
    minimumPollers: 2,
    maximumPollers: 10,
  },
  metricsConfig: {
    metrics: [
      lambda.MetricType.EVENT_COUNT,
      lambda.MetricType.ERROR_COUNT
    ]
  }
}));
python
from aws_cdk.aws_lambda_event_sources import ProvisionedPollerConfig
from aws_cdk.aws_lambda_event_sources import ManagedKafkaEventSource

# my_function: lambda.Function


# Your MSK cluster arn
cluster_arn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4"

# Enable basic event and error metrics
my_function.add_event_source(ManagedKafkaEventSource(
    cluster_arn=cluster_arn,
    topic="basic-monitoring",
    starting_position=lambda_.StartingPosition.LATEST,
    # Provisioned mode is required for observability features
    provisioned_poller_config=ProvisionedPollerConfig(
        minimum_pollers=2,
        maximum_pollers=10
    ),
    metrics_config=lambda.MetricsConfig(
        metrics=[lambda_.MetricType.EVENT_COUNT, lambda_.MetricType.ERROR_COUNT
        ]
    )
))
java
import software.amazon.awscdk.services.lambda.eventsources.ManagedKafkaEventSource;

Function myFunction;


// Your MSK cluster arn
String clusterArn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4";

// Enable basic event and error metrics
myFunction.addEventSource(ManagedKafkaEventSource.Builder.create()
        .clusterArn(clusterArn)
        .topic("basic-monitoring")
        .startingPosition(StartingPosition.LATEST)
        // Provisioned mode is required for observability features
        .provisionedPollerConfig(ProvisionedPollerConfig.builder()
                .minimumPollers(2)
                .maximumPollers(10)
                .build())
        .metricsConfig(MetricsConfig.builder()
                .metrics(List.of(MetricType.EVENT_COUNT, MetricType.ERROR_COUNT))
                .build())
        .build());
csharp
using Amazon.CDK.AWS.Lambda.EventSources;

Function myFunction;


// Your MSK cluster arn
var clusterArn = "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4";

// Enable basic event and error metrics
myFunction.AddEventSource(new ManagedKafkaEventSource(new ManagedKafkaEventSourceProps {
    ClusterArn = clusterArn,
    Topic = "basic-monitoring",
    StartingPosition = StartingPosition.LATEST,
    // Provisioned mode is required for observability features
    ProvisionedPollerConfig = new ProvisionedPollerConfig {
        MinimumPollers = 2,
        MaximumPollers = 10
    },
    MetricsConfig = new MetricsConfig {
        Metrics = new [] { MetricType.EVENT_COUNT, MetricType.ERROR_COUNT }
    }
}));
go
import "github.com/aws/aws-cdk-go/awscdk"

var myFunction Function


// Your MSK cluster arn
clusterArn := "arn:aws:kafka:us-east-1:0123456789019:cluster/SalesCluster/abcd1234-abcd-cafe-abab-9876543210ab-4"

// Enable basic event and error metrics
myFunction.AddEventSource(awscdk.NewManagedKafkaEventSource(&ManagedKafkaEventSourceProps{
	ClusterArn: jsii.String(ClusterArn),
	Topic: jsii.String("basic-monitoring"),
	StartingPosition: lambda.StartingPosition_LATEST,
	// Provisioned mode is required for observability features
	ProvisionedPollerConfig: &ProvisionedPollerConfig{
		MinimumPollers: jsii.Number(2),
		MaximumPollers: jsii.Number(10),
	},
	MetricsConfig: &MetricsConfig{
		Metrics: []MetricType{
			lambda.MetricType_EVENT_COUNT,
			lambda.MetricType_ERROR_COUNT,
		},
	},
}))

Set configuration for provisioned pollers that read from the event source. When specified, allows control over the minimum and maximum number of pollers that can be provisioned to process events from the source.

ts
import { ManagedKafkaEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';

// Your MSK cluster arn
declare const clusterArn: string;

// The Kafka topic you want to subscribe to
const topic = 'some-cool-topic';

declare const myFunction: lambda.Function;
myFunction.addEventSource(new ManagedKafkaEventSource({
  clusterArn,
  topic,
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
  provisionedPollerConfig: {
    minimumPollers: 1,
    maximumPollers: 3,
  },
}));
python
from aws_cdk.aws_lambda_event_sources import ProvisionedPollerConfig
from aws_cdk.aws_lambda_event_sources import ManagedKafkaEventSource

# Your MSK cluster arn
# cluster_arn: str

# my_function: lambda.Function


# The Kafka topic you want to subscribe to
topic = "some-cool-topic"
my_function.add_event_source(ManagedKafkaEventSource(
    cluster_arn=cluster_arn,
    topic=topic,
    starting_position=lambda_.StartingPosition.TRIM_HORIZON,
    provisioned_poller_config=ProvisionedPollerConfig(
        minimum_pollers=1,
        maximum_pollers=3
    )
))
java
import software.amazon.awscdk.services.lambda.eventsources.ManagedKafkaEventSource;

// Your MSK cluster arn
String clusterArn;

Function myFunction;


// The Kafka topic you want to subscribe to
String topic = "some-cool-topic";
myFunction.addEventSource(ManagedKafkaEventSource.Builder.create()
        .clusterArn(clusterArn)
        .topic(topic)
        .startingPosition(StartingPosition.TRIM_HORIZON)
        .provisionedPollerConfig(ProvisionedPollerConfig.builder()
                .minimumPollers(1)
                .maximumPollers(3)
                .build())
        .build());
csharp
using Amazon.CDK.AWS.Lambda.EventSources;

// Your MSK cluster arn
string clusterArn;

Function myFunction;


// The Kafka topic you want to subscribe to
var topic = "some-cool-topic";
myFunction.AddEventSource(new ManagedKafkaEventSource(new ManagedKafkaEventSourceProps {
    ClusterArn = clusterArn,
    Topic = topic,
    StartingPosition = StartingPosition.TRIM_HORIZON,
    ProvisionedPollerConfig = new ProvisionedPollerConfig {
        MinimumPollers = 1,
        MaximumPollers = 3
    }
}));
go
import "github.com/aws/aws-cdk-go/awscdk"

// Your MSK cluster arn
var clusterArn string

var myFunction Function


// The Kafka topic you want to subscribe to
topic := "some-cool-topic"
myFunction.AddEventSource(awscdk.NewManagedKafkaEventSource(&ManagedKafkaEventSourceProps{
	ClusterArn: jsii.String(ClusterArn),
	Topic: jsii.String(Topic),
	StartingPosition: lambda.StartingPosition_TRIM_HORIZON,
	ProvisionedPollerConfig: &ProvisionedPollerConfig{
		MinimumPollers: jsii.Number(1),
		MaximumPollers: jsii.Number(3),
	},
}))

You can reduce costs by sharing provisioned pollers across multiple Kafka event sources using the pollerGroupName property. This is particularly useful when you have multiple Kafka topics that don't require dedicated polling capacity.

ts
import { ManagedKafkaEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';

declare const clusterArn: string;
declare const ordersFunction: lambda.Function;

// Orders processing function
ordersFunction.addEventSource(new ManagedKafkaEventSource({
  clusterArn,
  topic: 'orders-topic',
  startingPosition: lambda.StartingPosition.LATEST,
  provisionedPollerConfig: {
    minimumPollers: 2,
    maximumPollers: 10,
    pollerGroupName: 'shared-kafka-pollers',
  },
}));
python
from aws_cdk.aws_lambda_event_sources import ProvisionedPollerConfig
from aws_cdk.aws_lambda_event_sources import ManagedKafkaEventSource

# cluster_arn: str
# orders_function: lambda.Function


# Orders processing function
orders_function.add_event_source(ManagedKafkaEventSource(
    cluster_arn=cluster_arn,
    topic="orders-topic",
    starting_position=lambda_.StartingPosition.LATEST,
    provisioned_poller_config=ProvisionedPollerConfig(
        minimum_pollers=2,
        maximum_pollers=10,
        poller_group_name="shared-kafka-pollers"
    )
))
java
import software.amazon.awscdk.services.lambda.eventsources.ManagedKafkaEventSource;

String clusterArn;
Function ordersFunction;


// Orders processing function
ordersFunction.addEventSource(ManagedKafkaEventSource.Builder.create()
        .clusterArn(clusterArn)
        .topic("orders-topic")
        .startingPosition(StartingPosition.LATEST)
        .provisionedPollerConfig(ProvisionedPollerConfig.builder()
                .minimumPollers(2)
                .maximumPollers(10)
                .pollerGroupName("shared-kafka-pollers")
                .build())
        .build());
csharp
using Amazon.CDK.AWS.Lambda.EventSources;

string clusterArn;
Function ordersFunction;


// Orders processing function
ordersFunction.AddEventSource(new ManagedKafkaEventSource(new ManagedKafkaEventSourceProps {
    ClusterArn = clusterArn,
    Topic = "orders-topic",
    StartingPosition = StartingPosition.LATEST,
    ProvisionedPollerConfig = new ProvisionedPollerConfig {
        MinimumPollers = 2,
        MaximumPollers = 10,
        PollerGroupName = "shared-kafka-pollers"
    }
}));
go
import "github.com/aws/aws-cdk-go/awscdk"

var clusterArn string
var ordersFunction Function


// Orders processing function
ordersFunction.AddEventSource(awscdk.NewManagedKafkaEventSource(&ManagedKafkaEventSourceProps{
	ClusterArn: jsii.String(ClusterArn),
	Topic: jsii.String("orders-topic"),
	StartingPosition: lambda.StartingPosition_LATEST,
	ProvisionedPollerConfig: &ProvisionedPollerConfig{
		MinimumPollers: jsii.Number(2),
		MaximumPollers: jsii.Number(10),
		PollerGroupName: jsii.String("shared-kafka-pollers"),
	},
}))

Schema Registry ​

Lambda can validate the key, the value, or both of each Kafka record against a schema registry before it invokes your function. Set schemaRegistryConfig on ManagedKafkaEventSource or SelfManagedKafkaEventSource to an AWS Glue schema registry with GlueSchemaRegistry, or to a Confluent schema registry, cloud or self-managed, with ConfluentSchemaRegistry. The registry can validate Avro, Protobuf and JSON Schema records.

Schema registries need provisioned pollers, so also set provisionedPollerConfig. Then choose:

  • eventRecordFormat: JSON passes the validated parts to your function as JSON, which suits languages such as Python and Node.js; SOURCE passes them in their original format, which suits Avro or Protobuf generated classes.
  • schemaValidationConfigs: whether to validate the key, the value, or both.

For Confluent, give the registry URL, an authentication type and the Secrets Manager secret that goes with it; the function gets read access to the secret. For Glue, pass the registry; the function gets the Glue permissions it needs to read it.

For details, see Using schema registries with Kafka event sources in Lambda in the AWS Lambda Developer Guide.

This example uses a Confluent schema registry with Amazon MSK:

ts
import { ManagedKafkaEventSource, ConfluentSchemaRegistry } from 'aws-cdk-lib/aws-lambda-event-sources';
import { Secret } from 'aws-cdk-lib/aws-secretsmanager';

// Your MSK cluster arn
declare const clusterArn: string;

// The Kafka topic you want to subscribe to
const topic = 'some-cool-topic';

// The secret with the credentials for your schema registry
declare const secret: Secret;

declare const myFunction: lambda.Function;
myFunction.addEventSource(new ManagedKafkaEventSource({
  clusterArn,
  topic,
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
  provisionedPollerConfig: {
    minimumPollers: 1,
    maximumPollers: 3,
  },
  schemaRegistryConfig: new ConfluentSchemaRegistry({
    schemaRegistryUri: 'https://example.com',
    eventRecordFormat: lambda.EventRecordFormat.JSON,
    authenticationType: lambda.KafkaSchemaRegistryAccessConfigType.BASIC_AUTH,
    secret: secret,
    schemaValidationConfigs: [{ attribute: lambda.KafkaSchemaValidationAttribute.KEY }],
  }),
}));
python
from aws_cdk.aws_lambda_event_sources import ProvisionedPollerConfig
from aws_cdk.aws_lambda_event_sources import ManagedKafkaEventSource, ConfluentSchemaRegistry
from aws_cdk.aws_secretsmanager import Secret

# Your MSK cluster arn
# cluster_arn: str

# The secret with the credentials for your schema registry
# secret: Secret

# my_function: lambda.Function


# The Kafka topic you want to subscribe to
topic = "some-cool-topic"
my_function.add_event_source(ManagedKafkaEventSource(
    cluster_arn=cluster_arn,
    topic=topic,
    starting_position=lambda_.StartingPosition.TRIM_HORIZON,
    provisioned_poller_config=ProvisionedPollerConfig(
        minimum_pollers=1,
        maximum_pollers=3
    ),
    schema_registry_config=ConfluentSchemaRegistry(
        schema_registry_uri="https://example.com",
        event_record_format=lambda_.EventRecordFormat.JSON,
        authentication_type=lambda_.KafkaSchemaRegistryAccessConfigType.BASIC_AUTH,
        secret=secret,
        schema_validation_configs=[lambda.KafkaSchemaValidationConfig(attribute=lambda_.KafkaSchemaValidationAttribute.KEY)]
    )
))
java
import software.amazon.awscdk.services.lambda.eventsources.ManagedKafkaEventSource;
import software.amazon.awscdk.services.lambda.eventsources.ConfluentSchemaRegistry;
import software.amazon.awscdk.services.secretsmanager.Secret;

// Your MSK cluster arn
String clusterArn;

// The secret with the credentials for your schema registry
Secret secret;

Function myFunction;


// The Kafka topic you want to subscribe to
String topic = "some-cool-topic";
myFunction.addEventSource(ManagedKafkaEventSource.Builder.create()
        .clusterArn(clusterArn)
        .topic(topic)
        .startingPosition(StartingPosition.TRIM_HORIZON)
        .provisionedPollerConfig(ProvisionedPollerConfig.builder()
                .minimumPollers(1)
                .maximumPollers(3)
                .build())
        .schemaRegistryConfig(ConfluentSchemaRegistry.Builder.create()
                .schemaRegistryUri("https://example.com")
                .eventRecordFormat(EventRecordFormat.JSON)
                .authenticationType(KafkaSchemaRegistryAccessConfigType.BASIC_AUTH)
                .secret(secret)
                .schemaValidationConfigs(List.of(KafkaSchemaValidationConfig.builder().attribute(KafkaSchemaValidationAttribute.KEY).build()))
                .build())
        .build());
csharp
using Amazon.CDK.AWS.Lambda.EventSources;
using Amazon.CDK.AWS.SecretsManager;

// Your MSK cluster arn
string clusterArn;

// The secret with the credentials for your schema registry
Secret secret;

Function myFunction;


// The Kafka topic you want to subscribe to
var topic = "some-cool-topic";
myFunction.AddEventSource(new ManagedKafkaEventSource(new ManagedKafkaEventSourceProps {
    ClusterArn = clusterArn,
    Topic = topic,
    StartingPosition = StartingPosition.TRIM_HORIZON,
    ProvisionedPollerConfig = new ProvisionedPollerConfig {
        MinimumPollers = 1,
        MaximumPollers = 3
    },
    SchemaRegistryConfig = new ConfluentSchemaRegistry(new ConfluentSchemaRegistryProps {
        SchemaRegistryUri = "https://example.com",
        EventRecordFormat = EventRecordFormat.JSON,
        AuthenticationType = KafkaSchemaRegistryAccessConfigType.BASIC_AUTH,
        Secret = secret,
        SchemaValidationConfigs = new [] { new KafkaSchemaValidationConfig { Attribute = KafkaSchemaValidationAttribute.KEY } }
    })
}));
go
import "github.com/aws/aws-cdk-go/awscdk"
import "github.com/aws/aws-cdk-go/awscdk"

// Your MSK cluster arn
var clusterArn string

// The secret with the credentials for your schema registry
var secret Secret

var myFunction Function


// The Kafka topic you want to subscribe to
topic := "some-cool-topic"
myFunction.AddEventSource(awscdk.NewManagedKafkaEventSource(&ManagedKafkaEventSourceProps{
	ClusterArn: jsii.String(ClusterArn),
	Topic: jsii.String(Topic),
	StartingPosition: lambda.StartingPosition_TRIM_HORIZON,
	ProvisionedPollerConfig: &ProvisionedPollerConfig{
		MinimumPollers: jsii.Number(1),
		MaximumPollers: jsii.Number(3),
	},
	SchemaRegistryConfig: awscdk.NewConfluentSchemaRegistry(&ConfluentSchemaRegistryProps{
		SchemaRegistryUri: jsii.String("https://example.com"),
		EventRecordFormat: lambda.EventRecordFormat_JSON(),
		AuthenticationType: lambda.KafkaSchemaRegistryAccessConfigType_BASIC_AUTH(),
		Secret: secret,
		SchemaValidationConfigs: []KafkaSchemaValidationConfig{
			&KafkaSchemaValidationConfig{
				Attribute: lambda.KafkaSchemaValidationAttribute_KEY(),
			},
		},
	}),
}))

This example uses an AWS Glue schema registry with Amazon MSK:

ts
import { CfnRegistry } from 'aws-cdk-lib/aws-glue';
import { ManagedKafkaEventSource, GlueSchemaRegistry } from 'aws-cdk-lib/aws-lambda-event-sources';

// Your MSK cluster arn
declare const clusterArn: string;

// The Kafka topic you want to subscribe to
const topic = 'some-cool-topic';

// Your Glue Schema Registry
const glueRegistry = new CfnRegistry(this, 'Registry', {
  name: 'schema-registry',
  description: 'Schema registry for event source',
});

declare const myFunction: lambda.Function;
myFunction.addEventSource(new ManagedKafkaEventSource({
  clusterArn,
  topic,
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
  provisionedPollerConfig: {
    minimumPollers: 1,
    maximumPollers: 3,
  },
  schemaRegistryConfig: new GlueSchemaRegistry({
    schemaRegistry: glueRegistry,
    eventRecordFormat: lambda.EventRecordFormat.JSON,
    schemaValidationConfigs: [{ attribute: lambda.KafkaSchemaValidationAttribute.KEY }],
  }),
}));
python
from aws_cdk.aws_lambda_event_sources import ProvisionedPollerConfig
from aws_cdk.aws_glue import CfnRegistry
from aws_cdk.aws_lambda_event_sources import ManagedKafkaEventSource, GlueSchemaRegistry

# Your MSK cluster arn
# cluster_arn: str

# my_function: lambda.Function


# The Kafka topic you want to subscribe to
topic = "some-cool-topic"

# Your Glue Schema Registry
glue_registry = CfnRegistry(self, "Registry",
    name="schema-registry",
    description="Schema registry for event source"
)
my_function.add_event_source(ManagedKafkaEventSource(
    cluster_arn=cluster_arn,
    topic=topic,
    starting_position=lambda_.StartingPosition.TRIM_HORIZON,
    provisioned_poller_config=ProvisionedPollerConfig(
        minimum_pollers=1,
        maximum_pollers=3
    ),
    schema_registry_config=GlueSchemaRegistry(
        schema_registry=glue_registry,
        event_record_format=lambda_.EventRecordFormat.JSON,
        schema_validation_configs=[lambda.KafkaSchemaValidationConfig(attribute=lambda_.KafkaSchemaValidationAttribute.KEY)]
    )
))
java
import software.amazon.awscdk.services.glue.CfnRegistry;
import software.amazon.awscdk.services.lambda.eventsources.ManagedKafkaEventSource;
import software.amazon.awscdk.services.lambda.eventsources.GlueSchemaRegistry;

// Your MSK cluster arn
String clusterArn;

Function myFunction;


// The Kafka topic you want to subscribe to
String topic = "some-cool-topic";

// Your Glue Schema Registry
CfnRegistry glueRegistry = CfnRegistry.Builder.create(this, "Registry")
        .name("schema-registry")
        .description("Schema registry for event source")
        .build();
myFunction.addEventSource(ManagedKafkaEventSource.Builder.create()
        .clusterArn(clusterArn)
        .topic(topic)
        .startingPosition(StartingPosition.TRIM_HORIZON)
        .provisionedPollerConfig(ProvisionedPollerConfig.builder()
                .minimumPollers(1)
                .maximumPollers(3)
                .build())
        .schemaRegistryConfig(GlueSchemaRegistry.Builder.create()
                .schemaRegistry(glueRegistry)
                .eventRecordFormat(EventRecordFormat.JSON)
                .schemaValidationConfigs(List.of(KafkaSchemaValidationConfig.builder().attribute(KafkaSchemaValidationAttribute.KEY).build()))
                .build())
        .build());
csharp
using Amazon.CDK.AWS.Glue;
using Amazon.CDK.AWS.Lambda.EventSources;

// Your MSK cluster arn
string clusterArn;

Function myFunction;


// The Kafka topic you want to subscribe to
var topic = "some-cool-topic";

// Your Glue Schema Registry
var glueRegistry = new CfnRegistry(this, "Registry", new CfnRegistryProps {
    Name = "schema-registry",
    Description = "Schema registry for event source"
});
myFunction.AddEventSource(new ManagedKafkaEventSource(new ManagedKafkaEventSourceProps {
    ClusterArn = clusterArn,
    Topic = topic,
    StartingPosition = StartingPosition.TRIM_HORIZON,
    ProvisionedPollerConfig = new ProvisionedPollerConfig {
        MinimumPollers = 1,
        MaximumPollers = 3
    },
    SchemaRegistryConfig = new GlueSchemaRegistry(new GlueSchemaRegistryProps {
        SchemaRegistry = glueRegistry,
        EventRecordFormat = EventRecordFormat.JSON,
        SchemaValidationConfigs = new [] { new KafkaSchemaValidationConfig { Attribute = KafkaSchemaValidationAttribute.KEY } }
    })
}));
go
import "github.com/aws/aws-cdk-go/awscdk"
import "github.com/aws/aws-cdk-go/awscdk"

// Your MSK cluster arn
var clusterArn string

var myFunction Function


// The Kafka topic you want to subscribe to
topic := "some-cool-topic"

// Your Glue Schema Registry
glueRegistry := awscdk.NewCfnRegistry(this, jsii.String("Registry"), &CfnRegistryProps{
	Name: jsii.String("schema-registry"),
	Description: jsii.String("Schema registry for event source"),
})
myFunction.AddEventSource(awscdk.NewManagedKafkaEventSource(&ManagedKafkaEventSourceProps{
	ClusterArn: jsii.String(ClusterArn),
	Topic: jsii.String(Topic),
	StartingPosition: lambda.StartingPosition_TRIM_HORIZON,
	ProvisionedPollerConfig: &ProvisionedPollerConfig{
		MinimumPollers: jsii.Number(1),
		MaximumPollers: jsii.Number(3),
	},
	SchemaRegistryConfig: awscdk.NewGlueSchemaRegistry(&GlueSchemaRegistryProps{
		SchemaRegistry: glueRegistry,
		EventRecordFormat: lambda.EventRecordFormat_JSON(),
		SchemaValidationConfigs: []KafkaSchemaValidationConfig{
			&KafkaSchemaValidationConfig{
				Attribute: lambda.KafkaSchemaValidationAttribute_KEY(),
			},
		},
	}),
}))

This example uses a self-managed Confluent schema registry with a self-managed Kafka cluster:

ts
import { SelfManagedKafkaEventSource, ConfluentSchemaRegistry } from 'aws-cdk-lib/aws-lambda-event-sources';
import { ISecret } from 'aws-cdk-lib/aws-secretsmanager';

// The list of Kafka brokers
const bootstrapServers = ['kafka-broker:9092'];

// The Kafka topic you want to subscribe to
const topic = 'some-cool-topic';

// The secret that allows access to your self hosted Kafka cluster
declare const kafkaSecret: ISecret;

// The secret with the client certificate and private key for your schema registry
declare const registrySecret: ISecret;

declare const myFunction: lambda.Function;
myFunction.addEventSource(new SelfManagedKafkaEventSource({
  bootstrapServers,
  topic,
  secret: kafkaSecret,
  startingPosition: lambda.StartingPosition.TRIM_HORIZON,
  provisionedPollerConfig: {
    minimumPollers: 1,
    maximumPollers: 3,
  },
  schemaRegistryConfig: new ConfluentSchemaRegistry({
    schemaRegistryUri: 'https://schema-registry.example.com',
    eventRecordFormat: lambda.EventRecordFormat.SOURCE,
    authenticationType: lambda.KafkaSchemaRegistryAccessConfigType.CLIENT_CERTIFICATE_TLS_AUTH,
    secret: registrySecret,
    schemaValidationConfigs: [{ attribute: lambda.KafkaSchemaValidationAttribute.VALUE }],
  }),
}));
python
from aws_cdk.aws_lambda_event_sources import ProvisionedPollerConfig
from aws_cdk.aws_lambda_event_sources import SelfManagedKafkaEventSource, ConfluentSchemaRegistry
from aws_cdk.aws_secretsmanager import ISecret

# The secret that allows access to your self hosted Kafka cluster
# kafka_secret: ISecret

# The secret with the client certificate and private key for your schema registry
# registry_secret: ISecret

# my_function: lambda.Function


# The list of Kafka brokers
bootstrap_servers = ["kafka-broker:9092"]

# The Kafka topic you want to subscribe to
topic = "some-cool-topic"
my_function.add_event_source(SelfManagedKafkaEventSource(
    bootstrap_servers=bootstrap_servers,
    topic=topic,
    secret=kafka_secret,
    starting_position=lambda_.StartingPosition.TRIM_HORIZON,
    provisioned_poller_config=ProvisionedPollerConfig(
        minimum_pollers=1,
        maximum_pollers=3
    ),
    schema_registry_config=ConfluentSchemaRegistry(
        schema_registry_uri="https://schema-registry.example.com",
        event_record_format=lambda_.EventRecordFormat.SOURCE,
        authentication_type=lambda_.KafkaSchemaRegistryAccessConfigType.CLIENT_CERTIFICATE_TLS_AUTH,
        secret=registry_secret,
        schema_validation_configs=[lambda.KafkaSchemaValidationConfig(attribute=lambda_.KafkaSchemaValidationAttribute.VALUE)]
    )
))
java
import software.amazon.awscdk.services.lambda.eventsources.SelfManagedKafkaEventSource;
import software.amazon.awscdk.services.lambda.eventsources.ConfluentSchemaRegistry;
import software.amazon.awscdk.services.secretsmanager.ISecret;

// The secret that allows access to your self hosted Kafka cluster
ISecret kafkaSecret;

// The secret with the client certificate and private key for your schema registry
ISecret registrySecret;

Function myFunction;


// The list of Kafka brokers
String[] bootstrapServers = List.of("kafka-broker:9092");

// The Kafka topic you want to subscribe to
String topic = "some-cool-topic";
myFunction.addEventSource(SelfManagedKafkaEventSource.Builder.create()
        .bootstrapServers(bootstrapServers)
        .topic(topic)
        .secret(kafkaSecret)
        .startingPosition(StartingPosition.TRIM_HORIZON)
        .provisionedPollerConfig(ProvisionedPollerConfig.builder()
                .minimumPollers(1)
                .maximumPollers(3)
                .build())
        .schemaRegistryConfig(ConfluentSchemaRegistry.Builder.create()
                .schemaRegistryUri("https://schema-registry.example.com")
                .eventRecordFormat(EventRecordFormat.SOURCE)
                .authenticationType(KafkaSchemaRegistryAccessConfigType.CLIENT_CERTIFICATE_TLS_AUTH)
                .secret(registrySecret)
                .schemaValidationConfigs(List.of(KafkaSchemaValidationConfig.builder().attribute(KafkaSchemaValidationAttribute.VALUE).build()))
                .build())
        .build());
csharp
using Amazon.CDK.AWS.Lambda.EventSources;
using Amazon.CDK.AWS.SecretsManager;

// The secret that allows access to your self hosted Kafka cluster
ISecret kafkaSecret;

// The secret with the client certificate and private key for your schema registry
ISecret registrySecret;

Function myFunction;


// The list of Kafka brokers
var bootstrapServers = new [] { "kafka-broker:9092" };

// The Kafka topic you want to subscribe to
var topic = "some-cool-topic";
myFunction.AddEventSource(new SelfManagedKafkaEventSource(new SelfManagedKafkaEventSourceProps {
    BootstrapServers = bootstrapServers,
    Topic = topic,
    Secret = kafkaSecret,
    StartingPosition = StartingPosition.TRIM_HORIZON,
    ProvisionedPollerConfig = new ProvisionedPollerConfig {
        MinimumPollers = 1,
        MaximumPollers = 3
    },
    SchemaRegistryConfig = new ConfluentSchemaRegistry(new ConfluentSchemaRegistryProps {
        SchemaRegistryUri = "https://schema-registry.example.com",
        EventRecordFormat = EventRecordFormat.SOURCE,
        AuthenticationType = KafkaSchemaRegistryAccessConfigType.CLIENT_CERTIFICATE_TLS_AUTH,
        Secret = registrySecret,
        SchemaValidationConfigs = new [] { new KafkaSchemaValidationConfig { Attribute = KafkaSchemaValidationAttribute.VALUE } }
    })
}));
go
import "github.com/aws/aws-cdk-go/awscdk"
import "github.com/aws/aws-cdk-go/awscdk"

// The secret that allows access to your self hosted Kafka cluster
var kafkaSecret ISecret

// The secret with the client certificate and private key for your schema registry
var registrySecret ISecret

var myFunction Function


// The list of Kafka brokers
bootstrapServers := []*string{
	"kafka-broker:9092",
}

// The Kafka topic you want to subscribe to
topic := "some-cool-topic"
myFunction.AddEventSource(awscdk.NewSelfManagedKafkaEventSource(&SelfManagedKafkaEventSourceProps{
	BootstrapServers: BootstrapServers,
	Topic: jsii.String(Topic),
	Secret: kafkaSecret,
	StartingPosition: lambda.StartingPosition_TRIM_HORIZON,
	ProvisionedPollerConfig: &ProvisionedPollerConfig{
		MinimumPollers: jsii.Number(1),
		MaximumPollers: jsii.Number(3),
	},
	SchemaRegistryConfig: awscdk.NewConfluentSchemaRegistry(&ConfluentSchemaRegistryProps{
		SchemaRegistryUri: jsii.String("https://schema-registry.example.com"),
		EventRecordFormat: lambda.EventRecordFormat_SOURCE(),
		AuthenticationType: lambda.KafkaSchemaRegistryAccessConfigType_CLIENT_CERTIFICATE_TLS_AUTH(),
		Secret: registrySecret,
		SchemaValidationConfigs: []KafkaSchemaValidationConfig{
			&KafkaSchemaValidationConfig{
				Attribute: lambda.KafkaSchemaValidationAttribute_VALUE(),
			},
		},
	}),
}))

Roadmap ​

Eventually, this module will support all the event sources described under Supported Event Sources in the AWS Lambda Developer Guide.

Types ​

Classes ​

  • ApiEventSource Class
  • ConfluentSchemaRegistry Class
    Confluent schema registry configuration for a Lambda event source.
  • DynamoEventSource Class
    Use an Amazon DynamoDB stream as an event source for AWS Lambda.
  • GlueSchemaRegistry Class
    Glue schema registry configuration for a Lambda event source.
  • KafkaDlq Class
    A Kafka topic dead letter queue destination configuration for a Lambda event source.
  • KinesisConsumerEventSource Class
    Use an Amazon Kinesis stream consumer as an event source for AWS Lambda.
  • KinesisEventSource Class
    Use an Amazon Kinesis stream as an event source for AWS Lambda.
  • ManagedKafkaEventSource Class
    Use a MSK cluster as a streaming source for AWS Lambda.
  • S3EventSource Class
    Use S3 bucket notifications as an event source for AWS Lambda.
  • S3EventSourceV2 Class
    S3EventSourceV2 Use S3 bucket notifications as an event source for AWS Lambda.
  • S3OnFailureDestination Class
    An S3 dead letter bucket destination configuration for a Lambda event source.
  • SelfManagedKafkaEventSource Class
    Use a self hosted Kafka installation as a streaming source for AWS Lambda.
  • SnsDlq Class
    An SNS dead letter queue destination configuration for a Lambda event source.
  • SnsEventSource Class
    Use an Amazon SNS topic as an event source for AWS Lambda.
  • SqsDlq Class
    An SQS dead letter queue destination configuration for a Lambda event source.
  • SqsEventSource Class
    Use an Amazon SQS queue as an event source for AWS Lambda.
  • StreamEventSource Class
    Use an stream as an event source for AWS Lambda.

Enums ​

Structs ​