Interface IStreamConsumerHandler<TDocument>
Defines a handler that consumes a stream of documents.
public interface IStreamConsumerHandler<in TDocument> where TDocument : IDispatchDocument
Type Parameters
TDocumentThe type of documents in the stream.
- Extension Methods
Examples
public class BatchImportHandler : IStreamConsumerHandler<DataRow>
{
private readonly IDatabase _database;
public BatchImportHandler(IDatabase database) => _database = database;
public async Task HandleAsync(
IAsyncEnumerable<DataRow> documents,
CancellationToken cancellationToken)
{
var batch = new List<DataRow>();
await foreach (var row in documents.WithCancellation(cancellationToken)
.ConfigureAwait(false))
{
batch.Add(row);
if (batch.Count >= 1000)
{
await _database.BulkInsertAsync(batch, cancellationToken)
.ConfigureAwait(false);
batch.Clear();
}
}
if (batch.Count > 0)
{
await _database.BulkInsertAsync(batch, cancellationToken)
.ConfigureAwait(false);
}
}
}
Remarks
Implement this interface to process an incoming stream of documents. This enables memory-efficient batch processing where documents arrive incrementally. The handler controls consumption rate, providing natural backpressure. Common use cases include:
- Batch imports: processing CSV/JSON rows as they are parsed
- ETL sinks: writing transformed records to storage
- Message queue consumers: processing message streams
- Real-time data ingestion: handling continuous data feeds
- Aggregation pipelines: computing statistics over streams
The contravariant in modifier on TDocument allows handlers
to accept streams of derived document types.
Handlers are resolved from dependency injection and should be registered with scoped lifetime to maintain state within a single stream processing operation.
Methods
HandleAsync(IAsyncEnumerable<TDocument>, CancellationToken)
Handles the incoming stream of documents asynchronously.
Task HandleAsync(IAsyncEnumerable<in TDocument> documents, CancellationToken cancellationToken)
Parameters
documentsIAsyncEnumerable<TDocument>The stream of documents to process.
cancellationTokenCancellationTokenThe cancellation token to observe throughout processing.
Returns
- Task
A task that completes when the stream has been fully processed.
Remarks
The handler controls consumption rate by the speed at which it iterates through
documents. This provides natural backpressure to upstream producers.
Use WithCancellation(cancellationToken) when iterating to ensure proper
cancellation propagation. Use ConfigureAwait(false) on all await expressions
when implementing in library code.
The task completes when all documents have been processed. If cancellation is requested, the task should throw OperationCanceledException.