Audit Logging

Dispatch provides tamper-evident audit logging with cryptographic hash chaining for compliance requirements including SOX, HIPAA, and GDPR accountability.

Before You Start

• .NET 8.0+ (or .NET 9/10 for latest features)
• Install the required packages:

  dotnet add package Excalibur.Dispatch.Security

• Familiarity with security overview and audit logging providers

Overview

flowchart LR
    subgraph Application
        E1[Event 1] --> H1[Hash 1]
        E2[Event 2] --> H2[Hash 2]
        E3[Event 3] --> H3[Hash 3]
    end

    subgraph Chain["Hash Chain"]
        H1 --> H2
        H2 --> H3
    end

    subgraph Verification
        V[Verify Integrity]
        H3 --> V
    end

Each audit event includes a hash of the previous event, creating an immutable chain that detects tampering.

Quick Start

Configuration

// Development/testing — in-memory store (no persistence)
services.AddAuditLogging();

// Production — SQL Server with inline options configuration
// Package: Excalibur.Dispatch.AuditLogging.SqlServer
services.AddSqlServerAuditStore(options =>
{
    options.ConnectionString = builder.Configuration.GetConnectionString("Compliance");
    options.SchemaName = "compliance";
    options.RetentionPeriod = TimeSpan.FromDays(7 * 365); // 7 years for SOC2
    options.EnableHashChain = true;
});

// Custom store — implement IAuditStore and register the type
services.AddAuditLogging<MyCustomAuditStore>();

// Annotations — enrich stored events with tags, bookmarks, notes
services.AddSqlServerAuditAnnotationStore(options =>
{
    options.ConnectionString = configuration.GetConnectionString("Compliance");
});

// Assertions — scoped audit context for handlers
services.AddAuditContext();

Log an Event

public class OrderService
{
    private readonly IAuditStore _auditStore;

    public async Task<Order> CreateOrderAsync(
        CreateOrderCommand command,
        CancellationToken ct)
    {
        var order = new Order(command);
        await _repository.SaveAsync(order, ct);

        // Log the audit event
        await _auditStore.StoreAsync(new AuditEvent
        {
            EventId = Guid.NewGuid().ToString(),
            EventType = AuditEventType.DataModification,
            Action = "Order.Create",
            ActorId = _currentUser.Id,
            Outcome = AuditOutcome.Success,
            Timestamp = DateTimeOffset.UtcNow,
            ResourceId = order.Id.ToString(),
            ResourceType = "Order",
            TenantId = _currentTenant.Id
        }, ct);

        return order;
    }
}

Audit Events

Event Structure

public sealed record AuditEvent
{
    // Required fields
    public required string EventId { get; init; }
    public required AuditEventType EventType { get; init; }
    public required string Action { get; init; }
    public required AuditOutcome Outcome { get; init; }
    public required DateTimeOffset Timestamp { get; init; }
    public required string ActorId { get; init; }

    // Optional actor details
    public string? ActorType { get; init; }
    public string? IpAddress { get; init; }
    public string? UserAgent { get; init; }
    public string? SessionId { get; init; }

    // Optional resource details
    public string? ResourceId { get; init; }
    public string? ResourceType { get; init; }
    public DataClassification? ResourceClassification { get; init; }

    // Context
    public string? TenantId { get; init; }
    public string? ApplicationName { get; init; }
    public string? CorrelationId { get; init; }
    public string? Reason { get; init; }

    // Metadata (must not contain sensitive data values)
    public IReadOnlyDictionary<string, string>? Metadata { get; init; }

    // Hash chain (set by the audit store)
    public string? PreviousEventHash { get; init; }
    public string? EventHash { get; init; }
}

Event Types

public enum AuditEventType
{
    System = 0,
    Authentication = 1,
    Authorization = 2,
    DataAccess = 3,
    DataModification = 4,
    ConfigurationChange = 5,
    Security = 6,
    Compliance = 7,
    Administrative = 8,
    Integration = 9
}

Outcomes

public enum AuditOutcome
{
    Success = 0,
    Failure = 1,
    Denied = 2,
    Error = 3,
    Pending = 4
}

Querying Audit Logs

Basic Query

var query = new AuditQuery
{
    StartDate = DateTimeOffset.UtcNow.AddDays(-7),
    EndDate = DateTimeOffset.UtcNow,
    MaxResults = 100
};

var events = await _auditStore.QueryAsync(query, ct);

Filter by User

var query = new AuditQuery
{
    ActorId = "user-12345",
    StartDate = DateTimeOffset.UtcNow.AddDays(-30),
    EndDate = DateTimeOffset.UtcNow
};

var userActivity = await _auditStore.QueryAsync(query, ct);

Filter by Application

In shared audit backends, filter events by the producing application:

var query = new AuditQuery
{
    ApplicationName = "OrderService",
    StartDate = DateTimeOffset.UtcNow.AddDays(-7),
    EndDate = DateTimeOffset.UtcNow
};

var appEvents = await _auditStore.QueryAsync(query, ct);

tip

ApplicationName is set automatically from ApplicationContext.ApplicationName when not provided explicitly on the AuditEvent. Configure it once via hosting and all audit events will carry the application identity.

Filter by Resource

var query = new AuditQuery
{
    ResourceId = "order-abc123",
    ResourceType = "Order"
};

var resourceHistory = await _auditStore.QueryAsync(query, ct);

Filter by Event Type

var query = new AuditQuery
{
    EventTypes = [AuditEventType.DataModification, AuditEventType.Administrative],
    Outcomes = [AuditOutcome.Success]
};

var modifications = await _auditStore.QueryAsync(query, ct);

Pagination

var query = new AuditQuery
{
    StartDate = startDate,
    EndDate = endDate,
    MaxResults = 50,
    Skip = 100,  // Skip first 100 results
    OrderByDescending = true  // Newest first
};

Count Results

var count = await _auditStore.CountAsync(query, ct);

Query Performance

Audit queries are optimized for indexed fields:

Field	Indexed	Recommended Use
StartDate/EndDate	Yes	Always include time range
ActorId	Yes	User activity reports
TenantId	Yes	Multi-tenant isolation
ApplicationName	Yes	Multi-app shared backends
ResourceId	Yes	Resource history
CorrelationId	Yes	Request tracing
EventType	Yes	Filter by category
ResourceClassification	Yes	Sensitive data access

Performance Target

Queries should complete in under 5 seconds for 1M records when using indexed fields:

var query = new AuditQuery
{
    StartDate = DateTimeOffset.UtcNow.AddDays(-1),
    EndDate = DateTimeOffset.UtcNow,
    ActorId = "user-12345",
    MaxResults = 1000
};

// Uses index on (ActorId, Timestamp DESC)
var events = await _auditStore.QueryAsync(query, ct);

Integrity Verification

Verify Hash Chain

var result = await _auditStore.VerifyChainIntegrityAsync(
    startDate: DateTimeOffset.UtcNow.AddMonths(-1),
    endDate: DateTimeOffset.UtcNow,
    ct);

if (result.IsValid)
{
    _logger.LogInformation(
        "Audit chain verified: {TotalEventsValidated} events, no tampering detected",
        result.TotalEventsValidated);
}
else
{
    _logger.LogCritical(
        "AUDIT TAMPERING DETECTED: {CorruptedEvents} corrupted events found. {Message}",
        result.CorruptedEvents,
        result.Message);
}

Integrity Result

public sealed record AuditIntegrityResult
{
    public required string ValidationId { get; init; }
    public required bool IsValid { get; init; }
    public required long TotalEventsValidated { get; init; }
    public required long CorruptedEvents { get; init; }
    public required DateTimeOffset ValidatedAt { get; init; }
    public required long ExecutionTimeMs { get; init; }
    public string? Message { get; init; }
    public IReadOnlyList<string> CorruptedEventIds { get; init; }
    public IReadOnlyDictionary<string, string>? Details { get; init; }
    public IReadOnlyList<string>? Errors { get; init; }
}

Multi-Tenant Isolation

Each tenant has an isolated hash chain:

// Store with tenant isolation
await _auditStore.StoreAsync(new AuditEvent
{
    TenantId = "tenant-abc",
    // ...other properties
}, ct);

// Query within tenant
var query = new AuditQuery
{
    TenantId = "tenant-abc"
};

// Verify chain integrity (covers all tenants in the date range)
var result = await _auditStore.VerifyChainIntegrityAsync(
    startDate, endDate, ct);

Database Schema

SQL Server

CREATE SCHEMA [audit];

CREATE TABLE [audit].[AuditEvents] (
    -- Identity and ordering
    [SequenceNumber] BIGINT IDENTITY(1,1) NOT NULL,
    [EventId] NVARCHAR(64) NOT NULL,

    -- Event classification
    [EventType] INT NOT NULL,
    [Action] NVARCHAR(100) NOT NULL,
    [Outcome] INT NOT NULL,
    [Timestamp] DATETIMEOFFSET(7) NOT NULL,

    -- Actor information
    [ActorId] NVARCHAR(256) NOT NULL,
    [ActorType] NVARCHAR(50) NULL,

    -- Resource information
    [ResourceId] NVARCHAR(256) NULL,
    [ResourceType] NVARCHAR(100) NULL,
    [ResourceClassification] INT NULL,

    -- Context and correlation
    [TenantId] NVARCHAR(64) NULL,
    [ApplicationName] NVARCHAR(256) NULL,
    [CorrelationId] NVARCHAR(64) NULL,
    [SessionId] NVARCHAR(64) NULL,

    -- Source information
    [IpAddress] NVARCHAR(45) NULL,
    [UserAgent] NVARCHAR(500) NULL,

    -- Additional context
    [Reason] NVARCHAR(1000) NULL,
    [Metadata] NVARCHAR(MAX) NULL, -- JSON

    -- Hash chain integrity
    [PreviousEventHash] NVARCHAR(64) NULL, -- SHA-256 hex
    [EventHash] NVARCHAR(64) NOT NULL, -- SHA-256 hex

    CONSTRAINT [PK_AuditEvents] PRIMARY KEY CLUSTERED ([SequenceNumber] ASC),
    CONSTRAINT [UQ_AuditEvents_EventId] UNIQUE NONCLUSTERED ([EventId])
);

-- Performance indices
CREATE INDEX [IX_AuditEvents_Timestamp]
ON [audit].[AuditEvents] ([Timestamp] DESC)
INCLUDE ([EventId], [EventType], [ActorId], [Outcome]);

CREATE INDEX [IX_AuditEvents_ActorId_Timestamp]
ON [audit].[AuditEvents] ([ActorId], [Timestamp] DESC)
INCLUDE ([EventType], [Action], [ResourceId]);

CREATE INDEX [IX_AuditEvents_TenantId_Timestamp]
ON [audit].[AuditEvents] ([TenantId], [Timestamp] DESC)
WHERE [TenantId] IS NOT NULL;

CREATE INDEX [IX_AuditEvents_ApplicationName_Timestamp]
ON [audit].[AuditEvents] ([ApplicationName], [Timestamp] DESC)
WHERE [ApplicationName] IS NOT NULL;

CREATE INDEX [IX_AuditEvents_ResourceId_Timestamp]
ON [audit].[AuditEvents] ([ResourceId], [Timestamp] DESC)
WHERE [ResourceId] IS NOT NULL;

CREATE INDEX [IX_AuditEvents_CorrelationId]
ON [audit].[AuditEvents] ([CorrelationId])
WHERE [CorrelationId] IS NOT NULL;

Integration Patterns

Middleware Integration

public class AuditMiddleware : IDispatchMiddleware
{
    private readonly IAuditStore _auditStore;

    public async ValueTask<IMessageResult> InvokeAsync(
        IDispatchMessage message,
        IMessageContext context,
        DispatchRequestDelegate next,
        CancellationToken ct)
    {
        var result = await next(message, context, ct);

        await _auditStore.StoreAsync(new AuditEvent
        {
            EventId = Guid.NewGuid().ToString(),
            EventType = DetermineEventType(message),
            Action = message.GetType().Name,
            ActorId = context.UserId,
            Outcome = result.IsSuccess
                ? AuditOutcome.Success
                : AuditOutcome.Failure,
            Timestamp = DateTimeOffset.UtcNow,
            ResourceId = ExtractResourceId(message),
            ResourceType = ExtractResourceType(message),
            CorrelationId = context.CorrelationId
        }, ct);

        return result;
    }
}

Decorator Pattern

public class AuditingOrderService : IOrderService
{
    private readonly IOrderService _inner;
    private readonly IAuditStore _auditStore;

    public async Task<Order> CreateOrderAsync(
        CreateOrderCommand command,
        CancellationToken ct)
    {
        var order = await _inner.CreateOrderAsync(command, ct);

        await _auditStore.StoreAsync(new AuditEvent
        {
            EventId = Guid.NewGuid().ToString(),
            EventType = AuditEventType.DataModification,
            Action = "Order.Create",
            ActorId = _currentUser.Id,
            Outcome = AuditOutcome.Success,
            Timestamp = DateTimeOffset.UtcNow,
            ResourceId = order.Id.ToString(),
            ResourceType = "Order"
        }, ct);

        return order;
    }
}

Testing

In-Memory Store

[Fact]
public async Task Should_Query_Events_By_Date_Range()
{
    // Arrange
    var store = new InMemoryAuditStore();
    var now = DateTimeOffset.UtcNow;

    await store.StoreAsync(new AuditEvent
    {
        EventId = "evt-1",
        EventType = AuditEventType.DataAccess,
        Action = "Test.Old",
        ActorId = "user-1",
        Outcome = AuditOutcome.Success,
        Timestamp = now.AddDays(-2)
    }, CancellationToken.None);

    await store.StoreAsync(new AuditEvent
    {
        EventId = "evt-2",
        EventType = AuditEventType.DataAccess,
        Action = "Test.New",
        ActorId = "user-1",
        Outcome = AuditOutcome.Success,
        Timestamp = now
    }, CancellationToken.None);

    // Act
    var query = new AuditQuery
    {
        StartDate = now.AddDays(-1),
        EndDate = now.AddDays(1)
    };
    var results = await store.QueryAsync(query, CancellationToken.None);

    // Assert
    results.ShouldHaveSingleItem();
    results[0].Action.ShouldBe("Test.New");
}

[Fact]
public async Task Should_Detect_Tampering()
{
    // Arrange
    var store = new InMemoryAuditStore();
    // ... store events ...

    // Tamper with an event
    // ...

    // Act
    var result = await store.VerifyChainIntegrityAsync(
        startDate, endDate, CancellationToken.None);

    // Assert
    result.IsValid.ShouldBeFalse();
}

Best Practices

Practice	Recommendation
Time range	Always include StartDate/EndDate in queries
Retention	Match regulatory requirements (7 years for SOX)
Integrity checks	Run daily verification of hash chain
Sensitive data	Mask PII/PHI before logging
Performance	Use indexed fields for queries
Multi-tenant	Always include TenantId for isolation

Compliance Mapping

Standard	Requirement	Feature
SOX	Audit trail for financial systems	Full event logging with hash chain
HIPAA	Access logs for PHI	ActorId, ResourceId, Classification
GDPR	Processing records	Timestamp, Action, Outcome
PCI-DSS	Cardholder data access logs	ResourceType filtering

Provider Compliance Boundary

ADR-290: Not All Backends Are Compliance-Grade

Elasticsearch and OpenSearch are audit sinks -- write-only, search-optimized projections. They do not implement IAuditStore and cannot provide tamper-evident hash chain verification.

Only backends that can guarantee monotonic sequencing, document immutability, and transactional atomicity qualify as IAuditStore implementations:

Backend	Role	Hash Chain	Tamper-Evident	Compliance-Grade
SQL Server	IAuditStore	Yes	Yes (IDENTITY + DENY)	Yes
Elasticsearch	Audit Sink	No	No (mutable documents)	No
OpenSearch	Audit Sink	No	No (mutable documents)	No

Why Elasticsearch/OpenSearch Cannot Be Compliance Stores

1. No monotonic sequencing -- wall-clock timestamps, not database IDENTITY columns
2. Documents are mutable -- anyone with cluster access can PUT/DELETE
3. Eventually consistent reads -- NRT refresh delay means stale hash chain reads
4. No transactional atomicity -- HTTP calls, not database transactions
5. ILM/ISM can delete indexes -- silently destroying audit records

Recommended Architecture

flowchart LR
    AE[Audit Events] --> SQL["SqlServerAuditStore<br/>(compliance, hash-chained)"]
    AE --> ES["ElasticsearchAuditSink<br/>(search, dashboards)"]
    AE --> OS["OpenSearchAuditSink<br/>(search, dashboards)"]

    SQL --> V[Verify Chain Integrity]
    ES --> K[Kibana / Dashboards]
    OS --> OSD[OpenSearch Dashboards]

Consumers who need both compliance and search should register SQL as their IAuditStore and ES/OS as an audit sink. The sink receives copies for fast full-text search, dashboards, and alerting. SQL is the source of truth for chain verification and regulatory compliance.

For full details, see ADR-290 in management/architecture/adr-290-elasticsearch-audit-sink-not-store.md.

Audit Event Annotations

Annotations let auditors enrich stored events with tags, bookmarks, and notes without modifying the original event or its hash chain. Each annotation is a separate record linked by event ID.

Packages

Package	Purpose
Excalibur.Dispatch.AuditLogging	In-memory store + RBAC decorator
Excalibur.Dispatch.AuditLogging.SqlServer	SQL Server persistence

Configuration

// In-memory (development/testing)
services.AddAuditAnnotations();

// SQL Server (production)
// Package: Excalibur.Dispatch.AuditLogging.SqlServer
services.AddSqlServerAuditAnnotationStore(options =>
{
    options.ConnectionString = builder.Configuration.GetConnectionString("Compliance");
    options.SchemaName = "audit";        // default
    options.TableName = "AuditAnnotations"; // default
    options.CommandTimeoutSeconds = 30;  // default
});

// With RBAC enforcement (requires IAuditRoleProvider registration)
services.AddRbacAuditAnnotationStore();

Tagging Events

Tags are shared labels applied to audit events. Duplicate tags are idempotent.

public class ComplianceReviewService
{
    private readonly IAuditAnnotationStore _annotations;

    public async Task ReviewEventAsync(
        string eventId,
        CancellationToken ct)
    {
        // Tag an event for compliance review
        await _annotations.TagAsync(
            eventId,
            ["reviewed", "sox-relevant"],
            ct);
    }
}

Bookmarking Events

Bookmarks are personal markers with an optional label. Each actor has at most one bookmark per event (replace semantics).

// Bookmark an event
await _annotations.BookmarkAsync(eventId, "Follow up Monday", ct);

// Remove a bookmark
await _annotations.RemoveBookmarkAsync(eventId, ct);

Adding Notes

Notes are free-text annotations with actor identity and timestamp.

var annotationId = await _annotations.AnnotateAsync(
    eventId,
    "Verified with finance team -- legitimate transaction",
    ct);

Querying by Annotation

Find events matching annotation criteria using AuditAnnotationQuery:

// Find all tagged events
var taggedEvents = await _annotations.QueryByAnnotationAsync(
    new AuditAnnotationQuery
    {
        Tags = ["sox-relevant"],
        MaxResults = 50
    },
    ct);

// Find bookmarked events by a specific actor
var myBookmarks = await _annotations.QueryByAnnotationAsync(
    new AuditAnnotationQuery
    {
        IsBookmarked = true,
        ActorId = "auditor-jane",
        Since = DateTimeOffset.UtcNow.AddDays(-7)
    },
    ct);

// Find events with notes, paginated
var annotated = await _annotations.QueryByAnnotationAsync(
    new AuditAnnotationQuery
    {
        HasNotes = true,
        Skip = 100,
        MaxResults = 50
    },
    ct);

Retrieving Annotations

Get all annotations for a single event, grouped by type:

AuditAnnotations result = await _annotations.GetAnnotationsAsync(eventId, ct);

// result.Tags      — IReadOnlyList<string>
// result.Bookmarks — IReadOnlyList<AuditAnnotation>
// result.Notes     — IReadOnlyList<AuditAnnotation>

RBAC Enforcement

When AddRbacAuditAnnotationStore() is registered, annotation access is controlled by AuditLogRole:

Role	Tag	Bookmark	Annotate	View Others' Annotations
Developer	No	No	No	No
SecurityAnalyst	Yes	Yes	Yes	Shared only
ComplianceOfficer	Yes	Yes	Yes	All
Administrator	Yes	Yes	Yes	All

Annotation creation automatically emits a meta-audit event (AuditEventType.Administrative) for traceability.

Annotations Database Schema

CREATE TABLE [audit].[AuditAnnotations] (
    [Id] NVARCHAR(32) NOT NULL,
    [EventId] NVARCHAR(64) NOT NULL,
    [AnnotationType] INT NOT NULL,        -- 0=Tag, 1=Bookmark, 2=Note
    [Content] NVARCHAR(MAX) NOT NULL,
    [ActorId] NVARCHAR(256) NOT NULL,
    [CreatedAt] DATETIMEOFFSET(7) NOT NULL,
    [Visibility] INT NOT NULL,            -- 0=Personal, 1=Shared
    CONSTRAINT [PK_AuditAnnotations] PRIMARY KEY ([Id])
);

CREATE INDEX [IX_AuditAnnotations_EventId]
ON [audit].[AuditAnnotations] ([EventId]);

CREATE INDEX [IX_AuditAnnotations_ActorId]
ON [audit].[AuditAnnotations] ([ActorId]);

tip

Annotations never modify the original AuditEvent or its hash chain. They are stored in a separate table and linked by EventId.

Conditional Audit Assertions

IAuditContext provides a scoped, handler-injected service for emitting domain-aware audit events. It inherits pipeline context (correlation ID, actor, tenant) automatically -- handlers only supply the condition and message.

Configuration

// Register audit context with defaults
services.AddAuditContext();

// Or configure options
services.AddAuditContext(options =>
{
    options.DefaultEventType = AuditEventType.Compliance;
    options.IncludeMessageTypeName = true;   // default
    options.MaxAssertionsPerScope = 25;      // default
});

note

AddAuditContext() registers AuditContextMiddleware which populates scope context before handler execution. Requires an IAuditActorProvider implementation and an IAuditStore registration.

Assertions

AssertAsync records an audit event only when the condition is true. When false, it returns null with zero I/O overhead.

public class ProcessOrderHandler : IMessageHandler<ProcessOrder>
{
    private readonly IAuditContext _audit;

    public async Task HandleAsync(
        ProcessOrder message,
        IMessageContext context,
        CancellationToken ct)
    {
        // Only records if the threshold is actually exceeded
        await _audit.AssertAsync(
            message.Amount > 10_000m,
            $"High-value order: {message.Amount:C}",
            AuditEventType.Compliance,
            ct);

        // Process the order...
    }
}

Observations

ObserveAsync unconditionally records an audit event. Use for events that do not depend on a boolean condition.

await _audit.ObserveAsync(
    "Order exported to external system",
    AuditEventType.Integration,
    AuditOutcome.Success,
    ct);

Resource Association and Metadata

Use fluent methods to attach resource identity and metadata before assertions:

await _audit
    .ForResource(order.Id.ToString(), "Order")
    .WithMetadata("region", order.Region)
    .WithMetadata("currency", order.Currency)
    .AssertAsync(
        order.RequiresEscalation,
        "Order requires manager approval",
        AuditEventType.Authorization,
        ct);

How It Works

1. Pipeline begins processing a message
2. AuditContextMiddleware populates the scope: CorrelationId, ActorId (from IAuditActorProvider), TenantId, Timestamp (from TimeProvider)
3. Handler receives pre-configured IAuditContext via constructor injection
4. Assertions and observations inherit all scope data automatically
5. Events are stored via IAuditLogger with hash-chain integrity

Safety Guards

• False assertions are free: AssertAsync(false, ...) returns null immediately with no allocation and no I/O
• Max assertions per scope: Excess assertions beyond MaxAssertionsPerScope (default: 25) are logged as warnings and dropped -- never thrown
• Actor fallback: If IAuditActorProvider is not registered or throws, the actor defaults to "system"
• No aggregate dependency: Works in any handler -- command, query, or integration event

Next Steps

• Data Masking - PII/PHI protection
• GDPR Erasure - Right to be forgotten

See Also

• Security Overview - Security architecture and threat model
• Compliance Overview - Compliance framework capabilities
• GDPR Erasure - Right to be forgotten with cryptographic deletion
• Audit Logging Providers - Provider configuration for audit sinks