Skip to main content

Consumer Conformance Toolkit

When you implement one of Excalibur's provider contracts — a custom IEventStore, IOutboxStore, ISagaStore, ILeaderElection, ICdcStateStore, IDataRequestRetryPolicy, and so on — the conformance toolkit gives you a starting suite that exercises your implementation against the behaviors we test our own providers with.

The kits encode our current understanding of each contract. They are not a specification of it, and a kit can be incomplete or wrong. A passing kit means your implementation agrees with ours on the behaviors that kit exercises. It is not certification. Where a contract carries a security or isolation guarantee, verify that guarantee against the contract's own documentation and your own tests — a kit can pass an implementation that does not hold it.

If a kit fails your implementation, check the contract's documentation before changing your code. A kit can encode a rule the contract does not require — and at least one currently does, in a way a correct implementation cannot satisfy.

Known gap

The in-memory data-inventory store registered by AddInMemoryDataInventoryStore() is not tenant-isolated: reads are not scoped to the calling tenant. Use a database-backed provider where tenant isolation matters.

The toolkit is provided by these packages:

PackageWhat it provides
Excalibur.Testing.ConformanceAbstract {Contract}ConformanceTestKit base classes — framework-agnostic, no Docker dependency.
Excalibur.Dispatch.TestingReceiver smoke checks and optional settlement outcome/effect helpers for ITransportReceiver.
Excalibur.Testing.ContainersOpt-in TestContainers fixtures (SqlServerContainerFixture, PostgresContainerFixture) for running kits against a real backend.

The Containers package is kept separate so that if you only test in-memory, you never pull a Docker or TestContainers dependency.

Testing your implementation against a kit​

Each conformance kit is an abstract class with a small number of factory hooks you implement, plus a set of public virtual conformance methods. Inherit the kit, supply your implementation through the hooks, and expose the checks to your test framework:

using Excalibur.Data.Resilience;
using Excalibur.Testing.Conformance;
using Xunit;

public sealed class MyRetryPolicyConformanceTests : RetryPolicyConformanceTestKit
{
protected override IDataRequestRetryPolicy CreatePolicy(int maxRetryAttempts) =>
new MyRetryPolicy(maxRetryAttempts);

protected override Exception CreateRetryableException() => new TimeoutException();
protected override Exception CreateNonRetryableException() => new ArgumentException();

[Fact] public void MaxRetryAttempts_Match() => MaxRetryAttempts_ShouldMatchConfiguredValue();
[Fact] public void ShouldRetry_Transient() => ShouldRetry_WithRetryableException_ReturnsExpectedResult();
[Fact] public void ShouldRetry_Permanent() => ShouldRetry_WithNonRetryableException_ReturnsFalse();
}

The kits carry no [Fact]/[Theory] attributes themselves, so they work with xUnit, NUnit, MSTest, or any runner — you add the attributes on the thin overrides. A failing check throws TestFixtureAssertionException with a message describing the contract violation.

An arm you do not wrap does not run, and a green suite cannot tell you

Because nothing in a kit makes an arm execute, an arm you forget to wrap never runs — and an arm that never runs cannot fail, so it is indistinguishable in the results from one that passed. A kit with 70 arms of which you wrapped 60 produces a green suite over ten unverified behaviours.

ConformanceSuite_ShouldWireEveryArm() is the guard against exactly that: wrap it like any other arm and it fails, naming each arm you have not wired. Wire it first, before you trust a green.

Receiver settlement observations​

Excalibur.Dispatch.Testing provides TransportReceiverConformanceTests for implementations of ITransportReceiver. Supply CreateReceiverAsync and SeedMessagesAsync, then expose the protected checks through your test framework. Import Excalibur.Dispatch.Testing.Transport for the base class and Excalibur.Dispatch.Transport for the transport contracts and settlement outcome types. This receiver kit is separate from the data-provider kits above.

The original VerifyAcknowledgeSucceeds, VerifyRejectSucceeds, and VerifyRejectWithRequeueSucceeds checks are smoke checks: they establish that the operation returns without an exception. They do not establish that settlement changed anything. For an observable-effect check, use VerifyAcknowledgeHasObservableEffect or VerifyRejectHasObservableEffect and supply an observer that asserts against independently owned fixture state:

public Task Acknowledgement_IsRecorded() =>
VerifyAcknowledgeHasObservableEffect((receiver, message) =>
{
var observed = _fixture.AcknowledgedMessages;
Assert.Single(observed);
Assert.Same(message, observed[0]);
Assert.Empty(_fixture.RejectedMessages);
return Task.CompletedTask;
});

The observer runs after settling exactly one received message, while the receiver is still alive. Completing the observer without assertions does not prove an effect. Pair the healthy implementation with a control that suppresses settlement: the same observer must reject that no-op control. Observe the requested rejection disposition as well as the message identity when using the reject helper.

An in-memory acknowledgement record proves a local state change. A mocked RabbitMQ channel's exact acknowledgement or rejection call proves native operation selection. A real broker-state observation proves a different property; neither of the first two certifies durable broker acceptance or redelivery. Use real infrastructure when that property is required and report unavailable infrastructure as unverified.

Test unknown or retired receipts, duplicate settlement, cancellation ownership, native failure, and independent settlement of messages from the same batch against the provider's documented policy. Providers differ in whether they retain a failed receipt and whether settlement observes caller cancellation. The kit does not impose a common retry or cancellation policy beyond the transport contract. Existing smoke checks remain available; adopting the effect checks adds no abstract factory requirements.

The additional outcome helpers let the fixture declare its supported policy while the kit executes the settlement calls and checks their results:

HelperFixture responsibility
VerifySettlementRefusalArrange an unknown or expired/retired receipt, observe its lack of effect, and independently observe settlement of an unrelated valid receipt. Model expiry through a real fixture transition; malformed metadata alone does not prove expiry.
VerifyRepeatedSettlementObserve the first successful settlement, then the declared duplicate outcome and native/state effects. Stateless duplicate success and local-claim refusal are different supported policies.
VerifyNativeSettlementFailureInject a specific native exception and independently observe the failed operation. The helper requires that exact exception as the portable failure's inner cause.
VerifySettlementCancellationPause at the native boundary and provide an idempotent release hook. The helper cancels its own token and checks the declared outcome and independent effects.
VerifyCompetingSettlementsProve that the first call is pending and the second has been attempted before releasing the first. Observe both dispositions and exact message effects.
VerifySettlementBatchRetryInject failure for one of two distinct receipts, observe that failure before its healthy peer settles, and declare and observe the failed receipt's retry policy.

Use SettlementExpectation.Completed, SettlementExpectation.CallerCanceled, or SettlementExpectation.Refused(redelivery, retryability) where the helper accepts an expectation. Requested caller cancellation must carry the helper's exact token. Portable failures must preserve the declared redelivery and retry axes. A provider using an independent settlement token may complete even after caller cancellation; a provider that forwards the token may cancel instead.

Preparation hooks arrange faults or receipt transitions; they do not perform settlement. Observation hooks assert independent effects. All fixture hooks must return promptly with a task representing their work. Boundary hooks passively expose existing fixture signals and must not launch active fixture work. Synchronization hooks establish the intended overlap without sleeps. Canceling a boundary observation only cancels the kit's wait, not the borrowed signal or provider operation. A competing call may finish before its native-entry signal, including a local-claim refusal; its outcome must still be checked. Every asynchronous stage is bounded: the simpler helpers use 30 seconds per stage, while cancellation, competition, and batch helpers accept a positive finite timeout. Settlement is invoked on the thread pool so a synchronous native pause cannot prevent the harness from reaching its release boundary. These checks do not certify execution tied to the caller's synchronization context.

A timeout bounds the harness's wait; it does not prove the underlying operation was canceled. Release and observation of all started settlement tasks are part of cleanup. An operation that cannot be drained leaves explicit cleanup uncertainty and fails the check. Pending borrowed boundary signals need not complete, and canceling their observation must not mask a fault or count as provider cancellation.

These are opt-in protected helpers. Existing derived suites still compile without exposing them, so add explicit tests for the applicable scenarios. Pair them with controls for missing effects, incorrect classification or cause, wrong delivery, and missing overlap. Neither helper availability nor a green fixture for a different provider establishes your provider's behavior.

What a green run actually covered​

Some kit arms exercise an optional capability of the contract — the outbox and dead-letter kits both have arms that need the store's administrative facet, for example. An arm whose capability is unavailable has to do something, and the obvious thing — return — is the one thing it must not do silently: every test runner reports an arm that returned early exactly as it reports an arm that ran and passed.

The kits now record that distinction instead of erasing it. ConformanceArmLedger collects every arm that ran its body and every arm that did not, with the capability it needed and why it was unavailable:

using Excalibur.Testing.Conformance;

ConformanceArmLedger.Reset(); // process-wide and additive; reset before a run you intend to read

// ... run your derived kit ...

foreach (var arm in ConformanceArmLedger.Executed) // "Suite.Arm" keys
{
Console.WriteLine($"verified: {arm}");
}

foreach (var skip in ConformanceArmLedger.Skipped) // ConformanceArmSkip records
{
Console.WriteLine($"NOT verified: {skip.Suite}.{skip.Arm} — {skip.Capability?.Name} — {skip.Reason}");
}

Console.WriteLine(ConformanceArmLedger.Describe()); // both lists, formatted
A run that was green before may now report skips

This is not a regression, and it does not mean your provider got worse. An arm that reported a pass while never reaching its assertions now reports as unverified. The skip is the kit declining to overstate what it checked — the coverage is the same as it always was, and only the reporting has changed. Read the skips and decide, per capability, whether an unverified arm is acceptable for your certification.

The ledger is a reporting surface, not an assertion: it records, and you decide what an unverified arm means. To surface skips natively in your runner — or to make one a failure — override OnArmSkipped on the kit. The hook comes from ConformanceTestKit, which the capability-gated kits derive from — OutboxStoreConformanceTestKit and DeadLetterStoreConformanceTestKit today:

public sealed class MyOutboxConformanceTests : OutboxStoreConformanceTestKit
{
// Surface the skip in the runner instead of letting the arm complete quietly.
protected override void OnArmSkipped(ConformanceArmSkip skip) =>
Assert.Skip($"{skip.Arm}: {skip.Reason}"); // xUnit v3; Assert.Ignore under NUnit

// Or certify that every capability your store provides was actually reached:
// protected override void OnArmSkipped(ConformanceArmSkip skip) =>
// throw new InvalidOperationException($"unverified arm {skip.Arm}: {skip.Reason}");
}

If your store is decorated​

The outbox kit discovers capabilities through GetService(Type), which the contract requires and which a well-behaved decorator forwards — so a decorated outbox store is discovered correctly.

IDeadLetterStore has no capability-resolution method, so its kit can only discover the administrative facet from the store's own type. That is sound for a store handed to the kit directly and unsound for a decorated one: a wrapper's type does not carry the capabilities of what it wraps, so the facet becomes invisible and every arm needing it is skipped. If you certify a decorated dead-letter store, override ResolveAdminFacet to return the facet the wrapper holds:

public sealed class MyDeadLetterConformanceTests : DeadLetterStoreConformanceTestKit
{
protected override IDeadLetterStoreAdmin? ResolveAdminFacet(IDeadLetterStore store) =>
store is MyDeadLetterDecorator decorator
? decorator.Inner as IDeadLetterStoreAdmin
: base.ResolveAdminFacet(store);
}

Without the override the arms still report — as skips, in the ledger, rather than as silent passes — but a recorded absence is still an uncertified capability.

Which of our own providers these kits are run against​

The same honesty applies to the providers we ship, so here is the state of it rather than an implication that every provider is equally verified.

KeyManagementProviderConformanceTestKit is run against three providers — the in-memory one, AWS KMS against a real KMS API, and HashiCorp Vault against a real Vault. Azure Key Vault is not among them, and its conformance is UNVERIFIED rather than absent. Azure Key Vault has no emulator, so there is nothing to run a real-backend suite against, and a suite that skipped when its backend was unavailable would report the same green as one that ran — which is the defect this whole section exists to name. We would rather record the gap than close it with an arm that cannot fail.

One consequence is worth stating in behavioural terms, because it is a real difference a consumer can observe rather than a testing detail:

Azure Key Vault exposes no ordinal key version. Its versions are opaque identifiers, while the framework's key metadata carries an integer, so AzureKeyVaultProvider derives that integer from the opaque identifier. The derivation is deterministic and stable across processes, but it is not injective: two of a key's versions can derive the same number. When that happens, a version lookup refuses — it raises rather than resolving one of the candidates, because returning one would answer about a version the caller did not ask for, and the number is bound into the encryption's associated data, so the mismatch would surface later as a failed authentication tag on a value whose key is perfectly healthy.

The derivation cannot be changed compatibly: the number it produces is recorded on every envelope encrypted under an Azure-backed key and is part of what authenticates that envelope, so altering it would make existing values undecryptable. It is therefore guarded rather than replaced.

Testing against a real backend​

For providers backed by a database, run the same kit against a real engine with an opt-in fixture:

using Excalibur.Testing.Containers;
using Xunit;

public sealed class MySqlServerOutboxConformanceTests
: OutboxStoreConformanceTestKit, IClassFixture<SqlServerContainerFixture>
{
private readonly SqlServerContainerFixture _fixture;

public MySqlServerOutboxConformanceTests(SqlServerContainerFixture fixture) => _fixture = fixture;

protected override Task<IOutboxStore> CreateStoreAsync() =>
Task.FromResult<IOutboxStore>(new MySqlServerOutboxStore(_fixture.ConnectionString));
}

SqlServerContainerFixture (and PostgresContainerFixture) derive from ContainerFixtureBase, which manages the Docker container lifecycle — startup with a bounded timeout, transient-failure retry, and best-effort teardown. Container timeouts scale with the TEST_TIMEOUT_MULTIPLIER environment variable.

A runnable example​

A complete, runnable worked example lives at samples/09-advanced/ConsumerConformanceExample. It runs a conformance kit against a custom retry policy (no Docker needed) and, with --with-container, spins up a real SQL Server fixture:

dotnet run # conformance checks against a custom implementation
dotnet run -- --with-container # also start a real SQL Server (requires Docker)