Table of Contents

Class DynamoDbCdcProcessor

Namespace
Excalibur.Cdc.DynamoDb
Assembly
Excalibur.Cdc.DynamoDb.dll

DynamoDB Streams CDC processor implementation.

[SuppressMessage("Maintainability", "CA1506:Avoid excessive class coupling", Justification = "CDC processors inherently couple with many SDK types.")]
public sealed class DynamoDbCdcProcessor : IDynamoDbCdcProcessor, ICdcStreamProcessor<DynamoDbDataChangeEvent, DynamoDbCdcPosition>, ICdcProcessor<DynamoDbDataChangeEvent>, IAsyncDisposable, IDisposable
Inheritance
DynamoDbCdcProcessor
Implements
Inherited Members

Remarks

This processor uses the DynamoDB Streams API to capture changes. It manages multiple shards concurrently and handles resharding events.

Shard iterators expire after 15 minutes, so processing should not pause for extended periods. The processor automatically refreshes expired iterators using saved sequence numbers.

Constructors

DynamoDbCdcProcessor(IAmazonDynamoDB, IAmazonDynamoDBStreams, IDynamoDbCdcStateStore, IOptions<DynamoDbCdcOptions>, ILogger<DynamoDbCdcProcessor>, IOptions<CdcFatalErrorOptions<DynamoDbDataChangeEvent>>?, IMessageFailureClassifier?, TimeProvider?, CdcHealthState?)

Initializes a new instance of the DynamoDbCdcProcessor class.

public DynamoDbCdcProcessor(IAmazonDynamoDB dynamoClient, IAmazonDynamoDBStreams streamsClient, IDynamoDbCdcStateStore stateStore, IOptions<DynamoDbCdcOptions> options, ILogger<DynamoDbCdcProcessor> logger, IOptions<CdcFatalErrorOptions<DynamoDbDataChangeEvent>>? fatalErrorOptions = null, IMessageFailureClassifier? failureClassifier = null, TimeProvider? timeProvider = null, CdcHealthState? healthState = null)

Parameters

dynamoClient IAmazonDynamoDB

The DynamoDB client for table operations.

streamsClient IAmazonDynamoDBStreams

The DynamoDB Streams client.

stateStore IDynamoDbCdcStateStore

The state store for position tracking.

options IOptions<DynamoDbCdcOptions>

The CDC options.

logger ILogger<DynamoDbCdcProcessor>

The logger.

fatalErrorOptions IOptions<CdcFatalErrorOptions<DynamoDbDataChangeEvent>>

Optional fatal-error handling. When omitted (or its handler is null), a fatal error rethrows and stops the processor (fail-loud — never an infinite silent reconnect loop).

failureClassifier IMessageFailureClassifier

Optional shared classifier deciding whether a processing error is fatal (non-retryable) or transient. When omitted, a conservative built-in fallback is used.

timeProvider TimeProvider

The clock the reconnect backoff waits on and measures stable connections with; defaults to the system clock.

healthState CdcHealthState

Where consecutive reconnect failures are reported for the CDC health check, when one is registered.

Methods

ConfirmPositionAsync(DynamoDbCdcPosition, CancellationToken)

Confirms that all events up to the specified position have been processed.

public Task ConfirmPositionAsync(DynamoDbCdcPosition position, CancellationToken cancellationToken)

Parameters

position DynamoDbCdcPosition

The position to confirm.

cancellationToken CancellationToken

A token to observe for cancellation requests.

Returns

Task

A task representing the asynchronous operation.

Remarks

This persists the position to the state store. On restart, processing will resume from this confirmed position. When using StartAsync(Func<TEvent, CancellationToken, Task>, CancellationToken), position confirmation is typically automatic. Use this method for manual control in batch scenarios.

Dispose()

Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources.

public void Dispose()

DisposeAsync()

Performs application-defined tasks associated with freeing, releasing, or resetting unmanaged resources asynchronously.

public ValueTask DisposeAsync()

Returns

ValueTask

A task that represents the asynchronous dispose operation.

GetCurrentPositionAsync(CancellationToken)

Gets the current processing position.

public Task<DynamoDbCdcPosition> GetCurrentPositionAsync(CancellationToken cancellationToken)

Parameters

cancellationToken CancellationToken

A token to observe for cancellation requests.

Returns

Task<DynamoDbCdcPosition>

The current position, or a beginning position if none exists.

ProcessBatchAsync(Func<DynamoDbDataChangeEvent, CancellationToken, Task>, CancellationToken)

Processes a single batch of CDC changes and returns.

public Task<int> ProcessBatchAsync(Func<DynamoDbDataChangeEvent, CancellationToken, Task> eventHandler, CancellationToken cancellationToken)

Parameters

eventHandler Func<DynamoDbDataChangeEvent, CancellationToken, Task>

A delegate that handles each change event.

cancellationToken CancellationToken

A token to observe for cancellation requests.

Returns

Task<int>

The number of events processed in this batch.

Remarks

Use this method for serverless scenarios or when you need explicit control over batch processing cadence.

Delivery semantics are provider-specific and this contract does not strengthen them. Providers differ in whether a batch that ends abnormally — a handler that throws, a cancelled token, a process that stops — redelivers the changes it had already read, and in whether a durable position is advanced on this path at all. Consult the documentation of the provider you register before relying on redelivery, and prefer ICdcStreamProcessor<TEvent, TPosition> where the provider offers it: its position management is explicit, whereas this method's is not part of the shared contract.

Consumer obligation: handlers MUST be idempotent. A provider that does redeliver does so without coordinating with the handler, so a handler that succeeded before a batch failed can be called again with the same change.

A return of zero does not necessarily mean "no changes were available". A provider that holds per-stream state MAY serve overlapping calls single-flight: while one call is processing, a second returns 0 immediately without touching the stream, and the work is picked up by a later invocation. This is specified behaviour rather than a failure — overlapping invocations are ordinary for the documented timer-trigger usage, and refusing them with an exception would surface as an unobserved task exception in most hosts. A provider that does this MUST log each skip so it is distinguishable from a stalled processor. Treat the return value as "changes processed by THIS call", never as "changes outstanding".

StartAsync(Func<DynamoDbDataChangeEvent, CancellationToken, Task>, CancellationToken)

Starts continuous CDC processing until cancellation.

public Task StartAsync(Func<DynamoDbDataChangeEvent, CancellationToken, Task> eventHandler, CancellationToken cancellationToken)

Parameters

eventHandler Func<DynamoDbDataChangeEvent, CancellationToken, Task>

A delegate that handles each change event.

cancellationToken CancellationToken

A token to observe for cancellation requests.

Returns

Task

A task that completes when processing stops due to cancellation.

Remarks

This method runs continuously until cancelled. It automatically handles connection management, reconnection, and position tracking. Position is confirmed after each successfully processed event. Use ProcessBatchAsync(Func<TEvent, CancellationToken, Task>, CancellationToken) for manual batch control.