Skip to main content

Elasticsearch Provider

The Elasticsearch provider offers full-text search and analytics with a resilient client wrapper, index lifecycle management, projection store integration, and health monitoring.

Before You Start​

  • .NET 10.0
  • An Elasticsearch cluster (local, Elastic Cloud, or AWS OpenSearch)
  • Familiarity with data access and projections

Installation​

dotnet add package Excalibur.Data.ElasticSearch

Dependencies: Excalibur.Data.Abstractions, Elastic.Clients.Elasticsearch

Quick Start​

using Microsoft.Extensions.DependencyInjection;

services.AddElasticsearchServices(configuration);

Registration Options​

Basic Registration​

// From configuration
services.AddElasticsearchServices(configuration, registry: null);

// With pre-configured client
services.AddElasticsearchServices(elasticsearchClient, registry: null);

// With client settings callback
services.AddElasticsearchServices(configuration, registry: null, configureSettings: settings =>
{
    settings.DisableDirectStreaming();
});

Resilient Client​

Adds Polly-based retry and circuit breaker policies:

services.AddResilientElasticsearchServices(configuration);

Monitoring​

services.AddElasticsearchMonitoring(configuration);

Combined Resilient + Monitoring​

services.AddMonitoredResilientElasticsearchServices(configuration);

Index Management​

services.AddElasticsearchIndexManagement(configuration);

Projection Store​

// Register all projections together (shared cluster)
services.AddElasticSearchProjections("https://es.example.com:9200", projections =>
{
    projections.Add<OrderSummary>();
    projections.Add<CustomerProfile>(o => o.Index.IndexName = "customers");
});

Projections are stored flat as the document root — no envelope wrapper. Custom repositories using ElasticRepositoryBase<T> can query the same index with natural field names. See Projections — Document Storage Format for details.

Builder Chain Integration​

When using the AddExcalibur composition root, register ElasticSearch projections inside the event sourcing builder:

services.AddExcalibur(excalibur => excalibur
    .AddEventSourcing(es => es
        .AddElasticSearchProjections("https://es.example.com:9200", projections =>
        {
            projections.Add<OrderSummary>();
            projections.Add<CustomerProfile>(o => o.Index.IndexName = "customers");
        })));

Or register a single projection store directly:

services.AddExcalibur(excalibur => excalibur
    .AddEventSourcing(es => es
        .AddElasticSearchProjectionStore<OrderSummary>(opts =>
        {
            opts.NodeUri = "https://es.example.com:9200";
            opts.Index.IndexPrefix = "orders";
        })));

Index Mapping Conventions​

By default, Excalibur infers Elasticsearch field mappings from CLR property types using DefaultIndexMappingConvention:

CLR TypeElasticsearch Type
stringkeyword
int, longlong
decimal, doubledouble
boolboolean
DateTime, DateTimeOffsetdate

To customize mappings (e.g., full-text search with analyzers), implement IIndexMappingConvention:

public class TextSearchConvention : IIndexMappingConvention
{
    public Properties ConfigureMappings(Type projectionType, Properties inferredProperties)
    {
        // Modify inferred mappings or replace entirely
        // Example: change string fields to text+keyword multi-field
        return inferredProperties;
    }
}

// Apply via options
services.AddElasticSearchProjections("https://es.example.com:9200", projections =>
{
    projections.Add<ProductSearch>(o =>
        o.Index.IndexMappingConvention = new TextSearchConvention());
});

Health Checks​

services.AddHealthChecks()
    .AddElasticHealthCheck("elasticsearch", timeout: TimeSpan.FromSeconds(5));

Security​

services.AddElasticsearchSecurity(configuration);

Performance Optimizations​

services.AddResilientElasticsearchServices(configuration);

Resilient Client​

The IResilientElasticsearchClient wraps the Elasticsearch client with retry and circuit breaker policies:

public interface IResilientElasticsearchClient
{
    bool IsCircuitBreakerOpen { get; }

    Task<SearchResponse<TDocument>> SearchAsync<TDocument>(SearchRequest request, CancellationToken cancellationToken);
    Task<IndexResponse> IndexAsync<TDocument>(IIndexRequest<TDocument> request, CancellationToken cancellationToken);
    Task<UpdateResponse<TDocument>> UpdateAsync<TDocument>(IUpdateRequest<TDocument, object> request, CancellationToken cancellationToken);
    Task<DeleteResponse> DeleteAsync(IDeleteRequest request, CancellationToken cancellationToken);
    Task<BulkResponse> BulkAsync(IBulkRequest request, CancellationToken cancellationToken);
    Task<GetResponse<TDocument>> GetAsync<TDocument>(IGetRequest request, CancellationToken cancellationToken);
    Task<bool> IsHealthyAsync(CancellationToken cancellationToken);
}

Index Lifecycle Management​

Manage indices, templates, aliases, and ILM policies:

  • IIndexInitializer — Bootstrap indices on startup
  • IIndexTemplateManager — Manage index templates
  • IIndexLifecycleManager — Configure ILM policies
  • IIndexOperationsManager — CRUD operations on indices
  • IIndexAliasManager — Manage index aliases

Index Field Mappings​

By default, Elasticsearch uses dynamic mapping to guess field types from the first document indexed. This can produce incorrect types — for example, mapping a numeric string as long instead of keyword, or missing full-text search capability on name fields.

Excalibur provides a three-tier mapping strategy that gives you control over how fields are mapped:

Implement IElasticIndexConfiguration<T> on your document class for full control:

using Elastic.Clients.Elasticsearch.Mapping;
using Excalibur.Data.ElasticSearch;

public sealed class CustomerSearchProjection : IElasticIndexConfiguration<CustomerSearchProjection>
{
    public Guid CustomerId { get; set; }
    public string Name { get; set; } = string.Empty;
    public string Email { get; set; } = string.Empty;
    public decimal TotalSpent { get; set; }
    public bool IsActive { get; set; }
    public DateTimeOffset CreatedAt { get; set; }
    public List<string> Tags { get; set; } = [];

    public static Properties ConfigureIndex() => new()
    {
        { "customerId", new KeywordProperty() },
        { "name", new TextProperty
            {
                Fields = new Properties
                {
                    { "keyword", new KeywordProperty { IgnoreAbove = 256 } }
                }
            }
        },
        { "email", new TextProperty
            {
                Fields = new Properties
                {
                    { "keyword", new KeywordProperty { IgnoreAbove = 256 } }
                }
            }
        },
        { "totalSpent", new DoubleNumberProperty() },
        { "isActive", new BooleanProperty() },
        { "createdAt", new DateProperty() },
        { "tags", new KeywordProperty() }
    };
}

Use explicit mapping when you need:

  • Full-text search fields (TextProperty) with keyword sub-fields
  • Nested object queries (NestedProperty)
  • Custom analyzers or field-specific settings

Tier 2: Reflection-Inferred Mapping (Good Default)​

When IElasticIndexConfiguration<T> is not implemented, the framework reflects over public properties and maps them to appropriate Elasticsearch types:

.NET TypeElasticsearch Type
string, Guid, enumskeyword
int, short, byte, long (and unsigned variants)long
float, double, decimaldouble
DateTime, DateTimeOffset, DateOnlydate
boolboolean
List<string>, string[], IReadOnlyList<string>keyword
Complex nested typesSkipped (ES dynamic mapping)

Nullable types are unwrapped — int? maps to long, DateTime? maps to date, etc.

This tier is automatic and requires no code changes. It is suitable when all string fields are exact-match (IDs, codes, statuses) and no full-text search is needed.

Tier 3: Dynamic Mapping (Fallback)​

If both explicit and inferred mapping are bypassed, Elasticsearch uses its own dynamic mapping rules. This is not recommended for production — it can produce incorrect types (e.g., mapping "12345" as long instead of keyword).

Using Mappings with ElasticRepositoryBase​

For non-projection documents using ElasticRepositoryBase<T>, call InitializeIndexWithMappingsAsync in your override:

public class CustomerRepository : ElasticRepositoryBase<CustomerDocument>
{
    public CustomerRepository(ElasticsearchClient client)
        : base(client, "customers") { }

    public override async Task InitializeIndexAsync(CancellationToken cancellationToken)
    {
        await InitializeIndexWithMappingsAsync(
            numberOfShards: 1,
            numberOfReplicas: 1,
            cancellationToken).ConfigureAwait(false);
    }
}

The projection store (ElasticSearchProjectionStore<T>) uses the three-tier strategy automatically during index creation — no additional configuration is needed.

Cursor-Based Pagination​

Excalibur provides cursor-based (keyset) pagination that maps directly to Elasticsearch's search_after API — delivering consistent, scalable paging without the performance cliff of deep from + size offsets.

Core Components​

ComponentPackagePurpose
CursorEncoderExcalibur.EventSourcing.AbstractionsBackend-agnostic Base64url cursor encoding/decoding
ElasticSearchCursorHelperExcalibur.Data.ElasticSearchConverts between CursorEncoder primitives and ES FieldValue sort values
CursorPagedResult<T>Excalibur.EventSourcing.AbstractionsResult type with items, total count, and opaque forward/backward cursors

How It Works​

  1. First request — no cursor, query returns the first page sorted by your chosen fields
  2. Subsequent requests — pass the opaque cursor from the previous response (NextCursor to go forward, PreviousCursor to go back); the framework decodes it into search_after sort values
  3. Boundaries — NextCursor is null on the last page (HasMore == false); PreviousCursor is null on the first page (HasPrevious == false)

Usage in a Controller​

[HttpGet("search")]
public async Task<CursorPagedResult<OrderSearchProjection>> Search(
    CancellationToken cancellationToken,
    [FromQuery] string? query,
    [FromQuery] int pageSize = 20,
    [FromQuery] string? cursor = null,
    [FromQuery] PageNavigation navigation = PageNavigation.Next)
{
    // Decode cursor into ES sort values (null for first page / First & Last navigation)
    var searchAfter = ElasticSearchCursorHelper.DecodeCursor(cursor);

    // For Previous/Last, reverse the sort so search_after walks backward;
    // ToCursorResult flips the items back to display order.
    var reverse = navigation is PageNavigation.Previous or PageNavigation.Last;
    var primary = reverse ? SortOrder.Asc : SortOrder.Desc;
    var tiebreak = reverse ? SortOrder.Desc : SortOrder.Asc;

    var searchRequest = new SearchRequestDescriptor<OrderSearchProjection>()
        .Index("orders")
        .Sort(s => s.Field(f => f.CreatedAt, new FieldSort { Order = primary }))
        .Sort(s => s.Field("_id", new FieldSort { Order = tiebreak }));

    if (!string.IsNullOrWhiteSpace(query))
    {
        searchRequest.Query(q => q.MultiMatch(m => m
            .Query(query)
            .Fields(new[] { "customerName", "status" })));
    }

    // Set page size (over-fetching one peek row) and apply the cursor. First/Last
    // navigate without a cursor; Next/Previous pass the decoded search_after.
    var pageCursor = navigation is PageNavigation.Next or PageNavigation.Previous
        ? searchAfter
        : null;
    ElasticSearchCursorHelper.ApplyCursorPaging(searchRequest, pageSize, pageCursor);

    var response = await elasticClient.SearchAsync(searchRequest, cancellationToken);

    // Build a bidirectional result: NextCursor (forward) + PreviousCursor (backward),
    // peek row trimmed.
    return ElasticSearchCursorHelper.ToCursorResult(response, pageSize, navigation);
}

Bidirectional Pagination​

ToCursorResult is given the PageNavigation the query was issued for and returns both NextCursor and PreviousCursor. For Previous and Last, reverse the sort order before searching (as shown above) — ToCursorResult detects the reversed navigation and flips the items back into display order automatically. No reverseItems flag is needed.

Supported Sort Value Types​

The CursorEncoder handles all common Elasticsearch sort value types:

TypeEncodingRound-trip behavior
stringJSON stringExact round-trip
long, intJSON numberDecoded as long
double, float, decimalJSON numberDecoded as long (if integer) or double
boolJSON true/falseExact round-trip
nullJSON nullExact round-trip
DateTimeOffset, DateTimeUnix epoch millisecondsDecoded as long
DateOnly, TimeOnlyISO 8601 stringDecoded as string

Cursors are Base64url-encoded (URL-safe, no padding) and opaque to consumers — the internal format may change between versions.

Design Notes​

  • Backend-agnostic: CursorEncoder and CursorPagedResult<T> live in the Excalibur.EventSourcing namespace (package Excalibur.EventSourcing.Abstractions) with no Elasticsearch dependency. They work with any store that supports keyset pagination (SQL Server, CosmosDB, etc.).
  • Corrupt cursors are safe: Invalid or tampered cursors return null from DecodeCursor, causing the query to start from the beginning rather than failing.
  • Always include a tiebreaker sort: Use _id or another unique field as the last sort criterion to ensure deterministic ordering when primary sort values are identical.

Audit Exporter​

A separate package indexes audit events into Elasticsearch for search, dashboards, and alerting:

dotnet add package Excalibur.AuditLogging.Elasticsearch
using Excalibur.AuditLogging.Elasticsearch;

// Single node
services.AddElasticsearchAuditExporter(es =>
{
    es.NodeUri(new Uri("https://es.example.com:9200"))
      .IndexName("dispatch-audit");
});

// Or a cluster -- every retry attempt round-robins to the next node
services.AddElasticsearchAuditExporter(es =>
{
    es.NodeUris([new Uri("https://es1:9200"), new Uri("https://es2:9200"), new Uri("https://es3:9200")])
      .IndexName("dispatch-audit");
});

// Or bind the whole option set from configuration
services.AddElasticsearchAuditExporter(es => es.BindConfiguration("AuditExporter:Elasticsearch"));

The builder covers connection (NodeUri, NodeUris, CloudId), index naming (IndexName), and configuration binding (BindConfiguration). The remaining settings on ElasticsearchExporterOptions -- ApiKey, BulkBatchSize, RefreshPolicy, ApplicationName, MaxRetryAttempts, RetryBaseDelay, Timeout -- come from the bound configuration section, or from a Configure call registered after the exporter:

using Excalibur.AuditLogging.Elasticsearch;

services.AddElasticsearchAuditExporter(es => es.NodeUri(new Uri("https://es.example.com:9200")));
services.Configure<ElasticsearchExporterOptions>(o =>
{
    o.ApiKey = configuration["Elasticsearch:ApiKey"];
    o.BulkBatchSize = 500;
    o.ApplicationName = "MyApp"; // fallback when AuditEvent.ApplicationName is null
});
info

Elasticsearch is a search/analytics projection of the audit trail, not a compliance-grade audit store: the package registers IAuditLogExporter and never IAuditStore. Use SQL Server for tamper-evident hash-chained storage. See provider compliance boundary and Audit Logging Providers.

See Also​