Leader Election

Leader election ensures only one instance in a distributed system performs a specific task at a time. Excalibur provides pluggable leader election for scenarios like background job processing, outbox publishing, and scheduled tasks.

Before You Start

.NET 8.0+ (or .NET 9/10 for latest features)

Install the required packages:

dotnet add package Excalibur.LeaderElection
dotnet add package Excalibur.LeaderElection.SqlServer  # or Redis/Postgres provider

A distributed storage backend (SQL Server or Redis) for lease management
Familiarity with Dispatch hosting

Packages

Package	Purpose
`Excalibur.Dispatch.LeaderElection.Abstractions`	Core interfaces: `ILeaderElection`, `ILeaderElectionFactory`, `IHealthBasedLeaderElection`
`Excalibur.LeaderElection`	Registration, telemetry decorator, and health check
`Excalibur.LeaderElection.SqlServer`	SQL Server-based leader election
`Excalibur.Data.Postgres`	PostgreSQL advisory lock-based leader election
`Excalibur.LeaderElection.Redis`	Redis-based leader election
`Excalibur.LeaderElection.Consul`	Consul-based leader election
`Excalibur.LeaderElection.Kubernetes`	Kubernetes lease-based leader election
`Excalibur.LeaderElection.InMemory`	In-memory leader election (testing/development)

When to Use Leader Election

Scenario	Why Leader Election
Outbox message publishing	Prevent duplicate message sends
Scheduled job processing	Run cron jobs exactly once
Cache warming	Single instance warms cache
Event projection updates	Prevent duplicate projections
Singleton background services	Only one active instance

Core Concepts

The Leader Election Pattern

Instance A --+-- Acquires Lock --> Becomes Leader --> Processes Work
             |
Instance B --+-- Waits ----------> Standby ---------> Ready to Take Over
             |
Instance C --+-- Waits ----------> Standby ---------> Ready to Take Over

If Instance A fails:
Instance B --> Acquires Lock --> Becomes Leader --> Processes Work

Leader Responsibilities

Acquire Leadership: Obtain exclusive lock
Renew Leadership: Keep lock alive with heartbeats
Perform Work: Execute the singleton workload
Release Leadership: Clean up on shutdown

The ILeaderElection Interface

public interface ILeaderElection
{
    /// <summary>
    /// Event raised when this instance becomes the leader.
    /// </summary>
    event EventHandler<LeaderElectionEventArgs>? BecameLeader;

    /// <summary>
    /// Event raised when this instance loses leadership.
    /// </summary>
    event EventHandler<LeaderElectionEventArgs>? LostLeadership;

    /// <summary>
    /// Event raised when the leader changes (any instance).
    /// </summary>
    event EventHandler<LeaderChangedEventArgs>? LeaderChanged;

    /// <summary>
    /// Gets the unique identifier for this election participant.
    /// </summary>
    string CandidateId { get; }

    /// <summary>
    /// Gets a value indicating whether this instance is currently the leader.
    /// </summary>
    bool IsLeader { get; }

    /// <summary>
    /// Gets the current leader's identifier.
    /// </summary>
    string? CurrentLeaderId { get; }

    /// <summary>
    /// Starts participating in leader election.
    /// </summary>
    Task StartAsync(CancellationToken cancellationToken);

    /// <summary>
    /// Stops participating and relinquishes leadership if held.
    /// </summary>
    Task StopAsync(CancellationToken cancellationToken);
}

Registration

Builder API (Recommended)

Use the fluent builder pattern to configure leader election with your chosen provider:

using Microsoft.Extensions.DependencyInjection;

// SQL Server with health checks
services.AddExcaliburLeaderElection(le => le
    .UseSqlServer(connectionString, "my-app-leader")
    .WithHealthChecks()
    .WithFencingTokens());

// Postgres
services.AddExcaliburLeaderElection(le => le
    .UsePostgres(opts =>
    {
        opts.ConnectionString = "Host=localhost;Database=myapp;";
        opts.LockKey = 12345;
    })
    .WithHealthChecks());

// Redis
services.AddExcaliburLeaderElection(le => le
    .UseRedis("myapp:leader")
    .WithHealthChecks());

// Consul
services.AddExcaliburLeaderElection(le => le
    .UseConsul(opts =>
    {
        opts.ConsulAddress = "http://localhost:8500";
        opts.SessionTTL = TimeSpan.FromSeconds(30);
    }));

// Kubernetes
services.AddExcaliburLeaderElection(le => le
    .UseKubernetes());

// In-memory (testing/development)
services.AddExcaliburLeaderElection(le => le
    .UseInMemory());

// Configure options via the builder
services.AddExcaliburLeaderElection(le => le
    .UseRedis("myapp:leader")
    .WithOptions(opts =>
    {
        opts.LeaseDuration = TimeSpan.FromSeconds(30);
        opts.RenewInterval = TimeSpan.FromSeconds(10);
    })
    .WithHealthChecks());

The builder automatically registers LeaderElectionOptions with ValidateDataAnnotations and ValidateOnStart.

Pre-Built Options

For scenarios where options are constructed externally (e.g., from configuration or testing):

var options = new LeaderElectionOptions
{
    LeaseDuration = TimeSpan.FromSeconds(30),
    RenewInterval = TimeSpan.FromSeconds(10)
};
services.AddExcaliburLeaderElection(options);

note

The pre-built options overload uses Options.Create() directly, which bypasses ValidateOnStart. Ensure your options are valid before passing them.

Available Builder Extensions

Method	Package	Purpose
`UseInMemory()`	`Excalibur.LeaderElection.InMemory`	Testing and development
`UseRedis(lockKey)`	`Excalibur.LeaderElection.Redis`	Redis-based leader election
`UseRedisFactory()`	`Excalibur.LeaderElection.Redis`	Redis factory for multiple elections
`UseSqlServer(conn, lock)`	`Excalibur.LeaderElection.SqlServer`	SQL Server-based leader election
`UseSqlServerFactory(conn)`	`Excalibur.LeaderElection.SqlServer`	SQL Server factory for multiple elections
`UsePostgres(opts)`	`Excalibur.Data.Postgres`	PostgreSQL advisory lock-based leader election
`UsePostgresFactory(opts)`	`Excalibur.Data.Postgres`	PostgreSQL factory for multiple elections
`UseConsul(opts?)`	`Excalibur.LeaderElection.Consul`	Consul session-based leader election
`UseKubernetes(opts?)`	`Excalibur.LeaderElection.Kubernetes`	Kubernetes Lease-based leader election
`WithHealthChecks()`	`Excalibur.LeaderElection`	Registers health check integration
`WithFencingTokens()`	`Excalibur.LeaderElection`	Registers fencing token middleware
`WithOptions(configure)`	`Excalibur.LeaderElection`	Configures `LeaderElectionOptions`

Direct Registration (Legacy)

Provider-specific Add*LeaderElection methods are still available for backward compatibility. The builder API above is preferred for new code.

When to Use Factory vs Single Registration

Excalibur offers two ways to register leader election: a single registration (ILeaderElection) and a factory (ILeaderElectionFactory). Choose based on how many independent leadership scopes your application needs.

Single Registration

Register a single ILeaderElection when your application has one leadership scope -- all background services share the same leader/follower state:

// One leader election for the entire application
services.AddExcaliburLeaderElection(le => le
    .UseSqlServer(connectionString, "my-app-leader")
    .WithHealthChecks());

Use cases:

Simple background worker: One app instance runs all background tasks
Application-level singleton: "This instance is the active one"
Small services: A microservice with 1-2 background jobs that should all run on the same instance

public class MyWorker : BackgroundService
{
    private readonly ILeaderElection _leaderElection;

    public MyWorker(ILeaderElection leaderElection)
    {
        _leaderElection = leaderElection;
    }

    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        await _leaderElection.StartAsync(ct);

        try
        {
            while (!ct.IsCancellationRequested)
            {
                if (_leaderElection.IsLeader)
                    await DoWorkAsync(ct);

                await Task.Delay(TimeSpan.FromSeconds(1), ct);
            }
        }
        finally
        {
            await _leaderElection.StopAsync(CancellationToken.None);
        }
    }
}

Factory Registration

Register ILeaderElectionFactory when your application has multiple independent leadership scopes -- different resources can have different leaders across your instances:

// Factory for creating per-resource elections
services.AddExcaliburLeaderElection(le => le
    .UseSqlServerFactory(connectionString));

Use cases:

Independent workloads: Outbox publishing, projection updates, and cleanup jobs should each have their own leader (Instance A leads outbox, Instance B leads projections)
Shared resource protection: Multiple resources each need exactly one writer
Fine-grained load distribution: Spread leadership across instances instead of concentrating all work on one node

public class DistributedWorker : BackgroundService
{
    private readonly ILeaderElectionFactory _factory;

    public DistributedWorker(ILeaderElectionFactory factory)
    {
        _factory = factory;
    }

    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        // Each resource gets its own independent election
        var outboxElection = _factory.CreateElection("outbox-processor", candidateId: null);
        var projectionElection = _factory.CreateElection("event-projector", candidateId: null);
        var cleanupElection = _factory.CreateElection("cleanup-job", candidateId: null);

        var elections = new[] { outboxElection, projectionElection, cleanupElection };
        await Task.WhenAll(elections.Select(e => e.StartAsync(ct)));

        try
        {
            while (!ct.IsCancellationRequested)
            {
                if (outboxElection.IsLeader)
                    await ProcessOutboxAsync(ct);

                if (projectionElection.IsLeader)
                    await ProcessProjectionsAsync(ct);

                if (cleanupElection.IsLeader)
                    await RunCleanupAsync(ct);

                await Task.Delay(TimeSpan.FromSeconds(1), ct);
            }
        }
        finally
        {
            await Task.WhenAll(elections.Select(e => e.StopAsync(CancellationToken.None)));
        }
    }
}

Decision Matrix

Question	Single	Factory
How many leadership scopes?	1	2+
Can different instances lead different workloads?	No -- one leader does all	Yes -- each resource has its own leader
Registration complexity	Lower	Slightly higher
Lock count in the backend	1	1 per resource
Load distribution	Concentrated on leader	Spread across instances

Shared Resource Scenarios

The factory pattern is especially valuable when multiple independent resources require exclusive access. Each resource gets its own election, so leadership is distributed naturally:

Scenario: E-commerce platform with multiple background processors

public class ECommerceWorkerService : BackgroundService
{
    private readonly ILeaderElectionFactory _factory;
    private readonly IServiceProvider _services;

    public ECommerceWorkerService(
        ILeaderElectionFactory factory,
        IServiceProvider services)
    {
        _factory = factory;
        _services = services;
    }

    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        // Each processor contends independently for its resource lock
        var orderOutbox = _factory.CreateElection("order-outbox", candidateId: null);
        var inventorySync = _factory.CreateElection("inventory-sync", candidateId: null);
        var priceUpdater = _factory.CreateElection("price-updater", candidateId: null);
        var reportGenerator = _factory.CreateElection("daily-reports", candidateId: null);

        var elections = new[]
        {
            orderOutbox, inventorySync, priceUpdater, reportGenerator
        };

        await Task.WhenAll(elections.Select(e => e.StartAsync(ct)));

        try
        {
            while (!ct.IsCancellationRequested)
            {
                // With 3 instances, leadership distributes across them:
                // Instance A: order-outbox, daily-reports
                // Instance B: inventory-sync
                // Instance C: price-updater
                if (orderOutbox.IsLeader)
                    await ProcessOrderOutboxAsync(ct);

                if (inventorySync.IsLeader)
                    await SyncInventoryAsync(ct);

                if (priceUpdater.IsLeader)
                    await UpdatePricesAsync(ct);

                if (reportGenerator.IsLeader)
                    await GenerateReportsAsync(ct);

                await Task.Delay(TimeSpan.FromSeconds(1), ct);
            }
        }
        finally
        {
            await Task.WhenAll(elections.Select(e => e.StopAsync(CancellationToken.None)));
        }
    }
}

Scenario: Multi-tenant background processing

When each tenant's data should be processed by exactly one instance:

public class TenantProcessorService : BackgroundService
{
    private readonly ILeaderElectionFactory _factory;
    private readonly ITenantRegistry _tenants;

    public TenantProcessorService(
        ILeaderElectionFactory factory,
        ITenantRegistry tenants)
    {
        _factory = factory;
        _tenants = tenants;
    }

    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        var tenantIds = await _tenants.GetActiveTenantIdsAsync(ct);
        var elections = tenantIds
            .Select(id => (
                TenantId: id,
                Election: _factory.CreateElection($"tenant-{id}", candidateId: null)))
            .ToList();

        await Task.WhenAll(elections.Select(e => e.Election.StartAsync(ct)));

        try
        {
            while (!ct.IsCancellationRequested)
            {
                foreach (var (tenantId, election) in elections)
                {
                    if (election.IsLeader)
                        await ProcessTenantAsync(tenantId, ct);
                }

                await Task.Delay(TimeSpan.FromSeconds(5), ct);
            }
        }
        finally
        {
            await Task.WhenAll(elections.Select(e => e.Election.StopAsync(CancellationToken.None)));
        }
    }
}

Basic Usage

Event-Based Leadership Pattern

public class OutboxProcessor : BackgroundService
{
    private readonly ILeaderElection _leaderElection;
    private readonly IOutboxStore _outbox;
    private readonly ILogger<OutboxProcessor> _logger;

    public OutboxProcessor(
        ILeaderElection leaderElection,
        IOutboxStore outbox,
        ILogger<OutboxProcessor> logger)
    {
        _leaderElection = leaderElection;
        _outbox = outbox;
        _logger = logger;

        // Subscribe to leadership events
        _leaderElection.BecameLeader += (_, args) =>
            _logger.LogInformation("Became leader: {CandidateId}", args.CandidateId);

        _leaderElection.LostLeadership += (_, args) =>
            _logger.LogInformation("Lost leadership: {CandidateId}", args.CandidateId);
    }

    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        // Start participating in leader election
        await _leaderElection.StartAsync(ct);

        try
        {
            while (!ct.IsCancellationRequested)
            {
                // Only process if we're the leader
                if (_leaderElection.IsLeader)
                {
                    await _outbox.ProcessPendingMessagesAsync(ct);
                }

                await Task.Delay(TimeSpan.FromSeconds(1), ct);
            }
        }
        finally
        {
            // Stop participating on shutdown
            await _leaderElection.StopAsync(CancellationToken.None);
        }
    }
}

Scheduled Job Runner

public class ScheduledJobRunner : BackgroundService
{
    private readonly ILeaderElection _leaderElection;
    private readonly ILogger<ScheduledJobRunner> _logger;

    public ScheduledJobRunner(
        ILeaderElection leaderElection,
        ILogger<ScheduledJobRunner> logger)
    {
        _leaderElection = leaderElection;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        await _leaderElection.StartAsync(ct);

        try
        {
            while (!ct.IsCancellationRequested)
            {
                if (_leaderElection.IsLeader)
                {
                    _logger.LogInformation("Running scheduled jobs as leader");
                    await RunScheduledJobsAsync(ct);
                }

                await Task.Delay(TimeSpan.FromMinutes(1), ct);
            }
        }
        finally
        {
            await _leaderElection.StopAsync(CancellationToken.None);
            _logger.LogInformation("Stopped leader election");
        }
    }

    private Task RunScheduledJobsAsync(CancellationToken ct)
    {
        // Execute scheduled jobs
        return Task.CompletedTask;
    }
}

Implementations

SQL Server Leader Election

Uses database locks for coordination:

// Installation
dotnet add package Excalibur.LeaderElection.SqlServer

// Configuration
builder.Services.AddSqlServerLeaderElection(
    connectionString,
    "my-app-leader",  // Lock resource name
    options =>
    {
        options.LeaseDuration = TimeSpan.FromSeconds(30);
        options.RenewInterval = TimeSpan.FromSeconds(10);
    });

SQL Server implementation features:

Uses sp_getapplock for distributed locking
Automatic heartbeat renewal
Graceful handoff on shutdown
Works with existing SQL Server infrastructure

PostgreSQL Leader Election

Uses Postgres advisory locks for coordination:

// Installation
dotnet add package Excalibur.Data.Postgres

// Direct registration
builder.Services.AddPostgresLeaderElection(options =>
{
    options.ConnectionString = "Host=localhost;Database=myapp;";
    options.LockKey = 12345;
});

// Or via builder API
builder.Services.AddExcaliburLeaderElection(le => le
    .UsePostgres(options =>
    {
        options.ConnectionString = "Host=localhost;Database=myapp;";
        options.LockKey = 12345;
    }));

PostgreSQL implementation features:

Uses pg_try_advisory_lock for distributed locking (session-level)
Lock automatically released when connection closes
Health-based leader election with candidate health tracking table
Factory pattern for multiple independent elections
AOT-safe with [GeneratedRegex] SQL identifier validation and JsonSerializerContext
ValidateDataAnnotations + ValidateOnStart on all options
IValidateOptions<LeaderElectionOptions> cross-property validator (RenewInterval < LeaseDuration)

Health-Based Election (Postgres)

PostgreSQL supports health-based leader election where unhealthy leaders automatically step down:

// Register health-based election
builder.Services.AddPostgresLeaderElection(
    pgOptions =>
    {
        pgOptions.ConnectionString = "Host=localhost;Database=myapp;";
        pgOptions.LockKey = 12345;
    },
    leOptions =>
    {
        leOptions.LeaseDuration = TimeSpan.FromSeconds(30);
        leOptions.RenewInterval = TimeSpan.FromSeconds(10);
    });

// Add health check
builder.Services.AddHealthChecks()
    .AddPostgresLeaderElectionHealthCheck();

Health-based options:

Option	Default	Description
`SchemaName`	`"public"`	Schema for the health tracking table
`TableName`	`"leader_election_health"`	Health tracking table name
`AutoCreateTable`	`true`	Auto-create schema and table on first start
`StepDownWhenUnhealthy`	`true`	Leader voluntarily releases lock when unhealthy
`HealthExpirationSeconds`	`60`	Stale health record expiration
`CommandTimeoutSeconds`	`5`	SQL command timeout

Redis Leader Election

Uses Redis for high-performance coordination:

// Installation
dotnet add package Excalibur.LeaderElection.Redis

// First register Redis connection
builder.Services.AddSingleton<IConnectionMultiplexer>(
    ConnectionMultiplexer.Connect("localhost:6379"));

// Configuration
builder.Services.AddRedisLeaderElection(
    "myapp:leader",  // Redis lock key
    options =>
    {
        options.LeaseDuration = TimeSpan.FromSeconds(30);
        options.RenewInterval = TimeSpan.FromSeconds(10);
    });

Redis implementation features:

Uses SET NX EX for atomic lock acquisition
Lua scripts for atomic operations
Low latency lock acquisition
Suitable for high-frequency leadership checks

Consul Leader Election

Uses Consul sessions for coordination:

// Installation
dotnet add package Excalibur.LeaderElection.Consul

// Configuration
builder.Services.AddConsulLeaderElection(options =>
{
    options.ConsulAddress = "http://localhost:8500";
    options.SessionTTL = TimeSpan.FromSeconds(30);
});

// Register a singleton election for a specific resource
builder.Services.AddConsulLeaderElectionForResource("my-processor");

Kubernetes Leader Election

Uses Kubernetes Lease objects for coordination:

// Installation
dotnet add package Excalibur.LeaderElection.Kubernetes

// Configuration with hosted service
builder.Services.AddExcaliburKubernetesLeaderElectionHostedService(
    "my-processor",
    options =>
    {
        options.LeaseDurationSeconds = 15;
        options.RenewIntervalMilliseconds = 10_000;
    });

Kubernetes implementation features:

Uses native Kubernetes Lease objects
Auto-detects in-cluster vs local kubeconfig
Integrates as a hosted service for automatic lifecycle management

In-Memory (Testing)

For unit tests and local development:

builder.Services.AddInMemoryLeaderElection();

note

AddInMemoryLeaderElection registers ILeaderElectionFactory, not ILeaderElection directly. Use the factory to create election instances for specific resources:

var factory = serviceProvider.GetRequiredService<ILeaderElectionFactory>();
var election = factory.CreateElection("my-resource", candidateId: null);

Configuration Options

public class LeaderElectionOptions
{
    /// <summary>
    /// How long a lease is valid before it expires.
    /// </summary>
    public TimeSpan LeaseDuration { get; set; } = TimeSpan.FromSeconds(15);

    /// <summary>
    /// How often to renew the lease (should be less than LeaseDuration).
    /// </summary>
    public TimeSpan RenewInterval { get; set; } = TimeSpan.FromSeconds(5);

    /// <summary>
    /// Retry interval when not leader.
    /// </summary>
    public TimeSpan RetryInterval { get; set; } = TimeSpan.FromSeconds(2);

    /// <summary>
    /// Grace period before declaring leadership lost.
    /// </summary>
    public TimeSpan GracePeriod { get; set; } = TimeSpan.FromSeconds(5);

    /// <summary>
    /// Unique identifier for this instance.
    /// </summary>
    public string InstanceId { get; set; } = Environment.MachineName;

    /// <summary>
    /// Enable health-based leader election.
    /// </summary>
    public bool EnableHealthChecks { get; set; } = true;

    /// <summary>
    /// Minimum health score (0.0 to 1.0) required to become or remain leader.
    /// </summary>
    public double MinimumHealthScore { get; set; } = 0.8;

    /// <summary>
    /// Automatically step down when health drops below MinimumHealthScore.
    /// </summary>
    public bool StepDownWhenUnhealthy { get; set; } = true;

    /// <summary>
    /// Custom metadata for this candidate (e.g., region, version).
    /// </summary>
    public IDictionary<string, string> CandidateMetadata { get; } = new Dictionary<string, string>();
}

Patterns and Best Practices

Graceful Shutdown

Release leadership cleanly on application shutdown:

public class LeaderAwareService : BackgroundService
{
    private readonly ILeaderElection _leaderElection;
    private readonly ILogger<LeaderAwareService> _logger;

    public LeaderAwareService(
        ILeaderElection leaderElection,
        ILogger<LeaderAwareService> logger)
    {
        _leaderElection = leaderElection;
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        await _leaderElection.StartAsync(ct);

        try
        {
            while (!ct.IsCancellationRequested)
            {
                if (_leaderElection.IsLeader)
                {
                    await DoWorkAsync(ct);
                }

                await Task.Delay(TimeSpan.FromSeconds(1), ct);
            }
        }
        finally
        {
            // Always stop on shutdown to release leadership
            await _leaderElection.StopAsync(CancellationToken.None);
            _logger.LogInformation("Released leadership on shutdown");
        }
    }
}

Handling Leadership Loss

React appropriately when leadership is lost using events:

public class LeadershipAwareProcessor : BackgroundService
{
    private readonly ILeaderElection _leaderElection;
    private readonly IWorkQueue _queue;
    private volatile bool _isLeader;

    public LeadershipAwareProcessor(ILeaderElection leaderElection, IWorkQueue queue)
    {
        _leaderElection = leaderElection;
        _queue = queue;

        // Track leadership state via events
        _leaderElection.BecameLeader += (_, _) => _isLeader = true;
        _leaderElection.LostLeadership += (_, _) => _isLeader = false;
    }

    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        await _leaderElection.StartAsync(ct);

        try
        {
            while (!ct.IsCancellationRequested)
            {
                if (_isLeader)
                {
                    var item = await _queue.DequeueAsync(ct);

                    // Check leadership before processing
                    if (!_isLeader)
                    {
                        await _queue.RequeueAsync(item);
                        continue;
                    }

                    await ProcessItemAsync(item, ct);
                }
                else
                {
                    await Task.Delay(TimeSpan.FromSeconds(1), ct);
                }
            }
        }
        finally
        {
            await _leaderElection.StopAsync(CancellationToken.None);
        }
    }
}

Multiple Leader Elections

Use the factory for multiple independent elections. See When to Use Factory vs Single above for guidance on choosing between single registration and factory.

// Register factory for multiple lock resources
builder.Services.AddSqlServerLeaderElectionFactory(connectionString);

public class MultiResourceLeaderService : BackgroundService
{
    private readonly ILeaderElectionFactory _factory;

    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        // Create separate leader elections for each resource
        // candidateId: null uses the default InstanceId from LeaderElectionOptions
        var outboxElection = _factory.CreateElection("outbox-processor", candidateId: null);
        var projectorElection = _factory.CreateElection("event-projector", candidateId: null);
        var cleanupElection = _factory.CreateElection("cleanup-job", candidateId: null);

        var elections = new[] { outboxElection, projectorElection, cleanupElection };

        // Start all elections
        await Task.WhenAll(elections.Select(e => e.StartAsync(ct)));

        try
        {
            while (!ct.IsCancellationRequested)
            {
                // Process each workload if we're leader for it
                if (outboxElection.IsLeader)
                    await ProcessOutboxAsync(ct);

                if (projectorElection.IsLeader)
                    await ProcessProjectionsAsync(ct);

                if (cleanupElection.IsLeader)
                    await RunCleanupAsync(ct);

                await Task.Delay(TimeSpan.FromSeconds(1), ct);
            }
        }
        finally
        {
            await Task.WhenAll(elections.Select(e => e.StopAsync(CancellationToken.None)));
        }
    }
}

Health Checks

A built-in health check is provided in the Excalibur.LeaderElection package. Register it with the standard ASP.NET Core health checks builder:

builder.Services.AddSqlServerLeaderElection(connectionString, "my-resource");

builder.Services.AddHealthChecks()
    .AddLeaderElectionHealthCheck();

The built-in LeaderElectionHealthCheck reports:

Healthy: This instance is the leader, or a valid leader is observed
Degraded: No leader is detected, but the service is running
Unhealthy: An exception occurs when querying leader election state

It is provider-agnostic and works with any ILeaderElection implementation (SQL Server, Redis, Consul, Kubernetes).

Common Use Cases

Outbox Pattern

Ensure only one instance publishes outbox messages:

public class OutboxPublisher : BackgroundService
{
    private readonly ILeaderElection _leaderElection;
    private readonly IOutboxStore _outbox;
    private readonly IMessageBus _messageBus;

    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        await _leaderElection.StartAsync(ct);

        try
        {
            while (!ct.IsCancellationRequested)
            {
                if (_leaderElection.IsLeader)
                {
                    var messages = await _outbox.GetPendingAsync(100, ct);

                    foreach (var message in messages)
                    {
                        await _messageBus.PublishAsync(message, ct);
                        await _outbox.MarkAsPublishedAsync(message.Id, ct);
                    }
                }

                await Task.Delay(TimeSpan.FromMilliseconds(100), ct);
            }
        }
        finally
        {
            await _leaderElection.StopAsync(CancellationToken.None);
        }
    }
}

Scheduled Tasks

Run scheduled jobs on exactly one instance:

public class DailyReportGenerator : BackgroundService
{
    private readonly ILeaderElection _leaderElection;

    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        await _leaderElection.StartAsync(ct);

        try
        {
            while (!ct.IsCancellationRequested)
            {
                // Wait until midnight
                var now = DateTime.UtcNow;
                var nextRun = now.Date.AddDays(1);
                await Task.Delay(nextRun - now, ct);

                // Only the leader generates the report
                if (_leaderElection.IsLeader)
                {
                    await GenerateDailyReportAsync(ct);
                }
            }
        }
        finally
        {
            await _leaderElection.StopAsync(CancellationToken.None);
        }
    }
}

Event Projection Processing

Single instance processes event projections:

public class ProjectionWorker : BackgroundService
{
    private readonly ILeaderElection _leaderElection;
    private readonly IEventStore _eventStore;
    private readonly ICheckpointStore _checkpointStore;
    private readonly IProjectionRegistry _projections;

    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        await _leaderElection.StartAsync(ct);

        try
        {
            while (!ct.IsCancellationRequested)
            {
                if (_leaderElection.IsLeader)
                {
                    var position = await _checkpointStore.GetPositionAsync("main", ct);

                    await foreach (var @event in _eventStore.ReadAllAsync(position, ct))
                    {
                        // Check leadership before each event
                        if (!_leaderElection.IsLeader)
                            break;

                        foreach (var projection in _projections)
                        {
                            await projection.HandleAsync(@event, ct);
                        }

                        await _checkpointStore.SavePositionAsync("main", @event.Position, ct);
                    }
                }

                await Task.Delay(TimeSpan.FromSeconds(1), ct);
            }
        }
        finally
        {
            await _leaderElection.StopAsync(CancellationToken.None);
        }
    }
}

Health-Based Elections

IHealthBasedLeaderElection extends standard elections with health awareness. Unhealthy leaders automatically step down, and only healthy candidates can become leader.

var election = _electionFactory.CreateHealthBasedElection(
    resourceName: "critical-processor",
    candidateId: null);

await election.StartAsync(ct);

// Report health status
await election.UpdateHealthAsync(
    isHealthy: true,
    metadata: new Dictionary<string, string>
    {
        ["cpu"] = "45%",
        ["memory"] = "2.1GB",
        ["queue_depth"] = "12"
    },
    ct);

// Query all candidates' health
var candidates = await election.GetCandidateHealthAsync(ct);

foreach (var candidate in candidates)
{
    // candidate.CandidateId
    // candidate.IsHealthy
    // candidate.HealthScore (0.0 to 1.0)
    // candidate.IsLeader
    // candidate.LastUpdated
    // candidate.Metadata
}

CandidateHealth

Each candidate exposes health information:

Property	Type	Description
`CandidateId`	`string`	Unique candidate identifier
`IsHealthy`	`bool`	Whether the candidate is healthy
`HealthScore`	`double`	Score from 0.0 to 1.0
`IsLeader`	`bool`	Whether this candidate is the current leader
`LastUpdated`	`DateTimeOffset`	When health was last reported
`Metadata`	`IDictionary<string, string>`	Custom health metadata

Troubleshooting

Lock Not Released

If a process crashes without releasing the lock, the lease will automatically expire after LeaseDuration. Another instance will then acquire leadership.

Split Brain

Configure appropriate timeouts to prevent split brain:

builder.Services.AddSqlServerLeaderElection(
    connectionString,
    "my-resource",
    options =>
    {
        // Lease duration must be longer than renewal interval
        options.LeaseDuration = TimeSpan.FromSeconds(30);
        options.RenewInterval = TimeSpan.FromSeconds(10);

        // Account for network latency and clock skew
        // Renewal should happen at least 2-3 times before expiry
    });

Next Steps

Event Sourcing - Use with event projections
CQRS - Coordinate read model updates
Dispatch Introduction - Background service patterns

Before You Start​

Packages​

When to Use Leader Election​

Core Concepts​

The Leader Election Pattern​

Leader Responsibilities​

The ILeaderElection Interface​

Registration​

Builder API (Recommended)​

Pre-Built Options​

Available Builder Extensions​

Direct Registration (Legacy)​

When to Use Factory vs Single Registration​

Single Registration​

Factory Registration​

Decision Matrix​

Shared Resource Scenarios​

Basic Usage​

Event-Based Leadership Pattern​

Scheduled Job Runner​

Implementations​

SQL Server Leader Election​

PostgreSQL Leader Election​

Health-Based Election (Postgres)​

Redis Leader Election​

Consul Leader Election​

Kubernetes Leader Election​

In-Memory (Testing)​

Configuration Options​

Patterns and Best Practices​

Graceful Shutdown​

Handling Leadership Loss​

Multiple Leader Elections​

Health Checks​

Common Use Cases​

Outbox Pattern​

Scheduled Tasks​

Event Projection Processing​

Health-Based Elections​

CandidateHealth​

Troubleshooting​

Lock Not Released​

Split Brain​

Next Steps​

See Also​