The worker that ran every 15 seconds and did nothing

A background worker in one of our MicroFrame services woke up every 15 seconds, queried its database, found nothing to do and went back to sleep. It logged nothing, because there was nothing to report. Three records it was supposed to pick up sat in the table as pending the whole time. A second path in the same service had a different problem: since the day it was deployed, it had not written a single row. Both bugs were silent, and 127 unit tests passed against the code that contained them.

The service records consent: whether a person has agreed to receive commercial messages on a channel, and whether that decision has been reported to Türkiye’s commercial message registry (İYS). The fixes were small. What is worth writing down is why neither bug made any noise.

Bug one: the context that vanished on return

Every MicroFrame service scopes its queries by realm, the tenant boundary. A request that arrives through the gateway carries its realm, middleware puts it into an ambient context, and every query filter reads it from there. Code never passes the realm around by hand.

The context lives in an AsyncLocal. That works for requests, because middleware creates the context at the top of the flow. A background worker has no middleware. Ours fanned out over realms like this:

foreach (var realm in realms)
{
    await ConsentRealmScope.EnterAsync(realm.Id);   // sets the ambient realm
    var due = await repository.ListDueAsync(ct);    // realm-filtered query
    // ...
}

EnterAsync set the realm inside an async method. In .NET, changes an async method makes to its ExecutionContext do not flow back to the caller when it returns. Our context implementation keeps one mutable root per flow: if a root exists, setting a value mutates the root and the caller sees it. If no root exists, the first set creates one, and that root belonged to the helper’s own execution context. It was discarded the moment EnterAsync returned.

So the query ran with an empty realm, the filter matched nothing, and the worker concluded there was no work. An empty result is a valid answer, which is why nothing complained.

The fix is to create the root before anything sets a value. It is a single call at the top of the method that processes one realm, so one realm’s context cannot leak into the next:

private async Task ProcessRealmAsync(Guid realmId, CancellationToken ct)
{
    AmbientContext.EnsureRoot();                     // the flow now owns a root
    await ConsentRealmScope.EnterAsync(realmId);     // this set reaches the caller
    var due = await repository.ListDueAsync(ct);     // and the filter sees the realm
    // ...
}

Bug two: a write with no one behind it

The second path consumed events from Kafka. The handler set the realm correctly, then created a consent record. Every aggregate in the platform carries created_by, and the column is NOT NULL. A request has a user. A Kafka handler has none unless someone gives it one.

The insert failed with a not-null violation, the bus retried three times and moved the message to the dead-letter queue. From the outside, the service looked healthy: the pod was ready, the handler was running, and the consent table simply stayed empty.

The fix was already written. A system principal provider existed, was registered in dependency injection, and was never called. Background flows now take the system principal as their actor before they write.

Why the tests missed both

The tests used Entity Framework’s in-memory provider. It does not enforce NOT NULL, so the write without an actor succeeded in every test. It also does not reproduce how the worker is hosted, so the lost context never showed up either. The tests checked rows and return values; the defects were in what surrounded them.

The guards we added pin the context, not a row:

• Two tests run an awaited helper that sets the realm, once without a root and once with one. The first asserts that the realm is lost, the second that it reaches the caller. They document the behaviour the bug depended on.
• A test reads every worker’s source file and fails if AmbientContext.EnsureRoot() is missing. A worker that skips it produces silence, and silence does not show up in code review.
• A test asserts that entering a realm in a background flow sets an actor, not only the realm.

The in-memory provider is still useful for fast tests of logic. It is not evidence that a write will succeed in PostgreSQL.

How to tell this apart from a data problem

The clue that cut the investigation short: another worker on the same pod, which handles registry webhooks, was working. It read with IgnoreQueryFilters and carried its identity in a scoped object, so it never depended on the ambient context. When one path that bypasses ambient state works and its neighbour that relies on it returns nothing, suspect the context before the data.

What we took from it

• A background flow has no request, so nothing sets up its context for it. Give every flow a root and an actor before the first read or write.
• “Ran, found nothing” and “ran, failed to look” produce the same log. If a worker can legitimately find nothing, log what it searched, not only what it found.
• A test provider that ignores constraints tests a different database than the one you deploy.