Endatix.Modules.Jobs 0.8.0

Endatix.Modules.Jobs

Durable background job queue for long-running tenant operations.

Installation:

dotnet add package Endatix.Modules.Jobs

Recommended Usage:

For running and hosting the Endatix Platform, Endatix.Api.Host is the recommended main package as it simplifies the installation and setup process.

dotnet add package Endatix.Api.Host

The module is registered automatically by EndatixBuilder.UseDefaults() — no host code change is needed.

What it is

Two stores in one jobs schema, with one job each:

• The BackgroundJobs table is the record of every job: tenant, status, progress, attempts, error, trace and retention. It is what GET jobs/{jobId} reads, and the only place a job's state lives.
• Quartz.NET 4.2.2, clustered through the same database (tables jobs.qrtz_*), is the queue: it decides when and on which node a job runs, and re-runs the work of a node that stopped. A retry is a trigger the job wrapper stores itself.

Every job has exactly one row and one Quartz trigger, keyed by the job's id; the trigger carries nothing but that id.

Module layout

The abstractions live in Endatix.Core rather than here so that assemblies which cannot reference this module — Endatix.Infrastructure, most notably — can still enqueue and handle.

Schema

Database schema: jobs

The schema carries its own __EFMigrationsHistory, so job migrations advance independently of app-schema migrations. Quartz's tables are created by the jobs migrations only; see Quartz schema and upgrades.

Status

Pending → Processing → Completed | Failed | DeadLettered | Retrying | Canceled, and Retrying → Processing. A job cancelled before it is claimed never runs.

Failed and DeadLettered are separate because one status cannot express both "do not retry this" and "retried and gave up", and operators need to tell them apart.

Enqueueing

Feature code builds a request from a typed payload:

public sealed record SubmissionExportPayload(long FormId, long ExportFormatId) : IBackgroundJobPayload
{
    // Persisted on the row and in Quartz's job keys: never change it.
    public static string JobType => "SubmissionExport";
}

var jobId = await backgroundJobQueue.EnqueueAsync(
    BackgroundJobRequest.Create(new SubmissionExportPayload(formId, formatId), tenantId, userId),
    cancellationToken);

Idempotent enqueue. A request may carry a DedupKey naming its unit of work — for a fan-out, {outboxMessageId}:{subscriber}. It is unique per tenant and job type (a filtered unique index, IX_BackgroundJobs_DedupKey), and enqueueing a key that already exists returns the existing job's id without a second row or trigger, even when two enqueues race. Requests without a key never collide. It names the work, never the attempt: no timestamps or counters in it.

Use EnqueueManyAsync for fan-out — one job per webhook endpoint, say. The batch commits in a single transaction, so a partial fan-out cannot deliver to some destinations and silently drop the rest.

Writing a handler

Derive from BackgroundJobHandler<TPayload> and register it with services.AddBackgroundJobHandler<THandler, TPayload>(), which keys it by job type so a job run builds only its own handler. The job type comes from the payload; handlers may live in any assembly. Two handlers declaring the same job type fail startup.

A job type that needs other settings than the global defaults declares them at the same call, so every host runs it as tuned: AddBackgroundJobHandler<THandler, TPayload>(new BackgroundJobTypeDefaults { MaxAttempts = 8, MaxConcurrency = 4 }). A handler registered by a factory declares them with services.AddBackgroundJobTypeDefaults(jobType, defaults). A value the host sets in configuration wins over it (Configuration).

internal sealed class SubmissionExportJobHandler(..., ILogger<SubmissionExportJobHandler> logger)
    : BackgroundJobHandler<SubmissionExportPayload>(logger)
{
    protected override Task<Result> ExecuteAsync(
        BackgroundJobContext job, SubmissionExportPayload payload, CancellationToken cancellationToken) => ...;
}

Payload rules: the job type is a string literal declared once, on the payload; a payload written by one release must deserialize in the next (adding an optional property is fine; renaming, removing or retyping one is a new job type); keep payloads thin — ids plus the minimum non-personal data. Input that cannot be read ends the job Failed without a retry, and the base class logs why through the logger it is given.

Four obligations, each invisible until it hurts in production:

1. Return a failure Result for deterministic errors; throw only for transient ones. This is the only retry signal there is. Throwing on a permanent error re-runs expensive work until the attempt budget is gone.
2. Scope every query to the job's TenantId explicitly. Outside a request the ambient tenant filter is permissive, not restrictive — a handler that queries as if it were in a request reads every tenant's data.
3. Honour the CancellationToken, or the job cannot be cancelled or time-limited.
4. Do not hold one DbContext for the length of the job. Open a scope per chunk via IServiceScopeFactory; a change tracker held for minutes accumulates every row streamed through it.

Handlers never reference a Quartz type.

Runtime

Every host that registers the module builds a Quartz scheduler named endatix-jobs against the shared store. Endatix:BackgroundJobs:RunInProcess decides what it does with it:

That is what allows API and worker roles to be deployed separately from the same image.

Each job type is its own Quartz execution group, capped per node at JobTypes:{JobType}:MaxConcurrency (default 1). Every group this host has no handler for is capped at 0, so a node never takes a job it cannot run; the job waits for a node that can. The thread pool holds every registered job type's full cap at once, so it is sized as the sum of the caps, and there is no global concurrency setting. Trigger acquisition reads, for each group with a free slot on the node, at most as many of its oldest triggers as it has free slots, so a backlog of one job type never holds up another. One acquisition takes up to a trigger per free thread (the scheduler's batch size is the pool size) and never more of a job type than its free slots. The triggers Quartz creates to recover a dead node's jobs carry no group; they are given their job type's when read for acquisition, so recovered jobs count against the cap like any other. If a node ever fires a job type it has no handler for, the wrapper declines it without touching the row and offers the trigger again 30 seconds later.

Backlog warning. A job that waited past MisfireThresholdSeconds for a slot increments endatix.jobs.misfired (tag job_type), and again at every threshold while it waits. The warning Background job {JobId} of type {JobType} waited past the misfire threshold; {Misfires} triggers of this type misfired since the last warning. is logged at most once per job type per threshold. A job type no running node can handle shows up here: deploy the module that handles it.

Dashboard. With Dashboard:Enabled, the Quartz dashboard is served at /quartz and its HTTP API at /quartz-api, both behind the PlatformAdmin policy and read-only unless Dashboard:AllowWrites is set. It is an operator tool: a caller that may write can schedule jobs on the host, and it shows every tenant's triggers. Even with writes on, only Endatix's own job class may be named.

Each job type the host has a handler for gets one durable Quartz job, which requests recovery, so a job cut off by a stopped or crashed node runs again on another. A host enqueueing a job type checks once per process that its durable job is in the store, and stores it again if it has since been deleted. Every node sharing the store must run the same Quartz version, and nodes' clocks must agree within about a second.

Trigger lock. Quartz serializes trigger state across the cluster through one row lock (TRIGGER_ACCESS). Enqueueing does not take it (LockOnInsert is off): on PostgreSQL a trigger is invisible to every node's acquisition until its transaction commits. Each acquisition takes it once for its whole batch, and so does firing that batch; each firing takes it once more as it completes, and an idle node takes it once per idle wait. A failed attempt with attempts left takes it once more, to store its retry's trigger: that replaces the trigger that fired, and Quartz takes the lock for every replacement, LockOnInsert or not.

Retention

JobRetentionJob runs on the Retention:Cron schedule, on one node at a time, in an execution group of its own with one thread beyond the job types' caps. Each run deletes finished rows whose ExpiresAt has passed, BatchSize at a time, for at most MaxBatchesPerRun batches. It never deletes a row that is not finished. Every terminal write sets ExpiresAt to its time plus the job type's RetentionDays when the row has none. Quartz deletes its own finished triggers.

Execution

BackgroundJobExecution is the only Quartz job class. It only orchestrates each firing, through one class per step:

1. Admits the firing (JobFiringAdmission). A firing that carries no job id, as one an operator fires by hand from the dashboard does, is ignored. A job type this host has no handler for is declined without touching the row, and its trigger is offered again 30 seconds later.
2. Claims the row (JobAttemptClaimer) with a compare-and-swap from Pending/Retrying to Processing that increments AttemptCount. When Quartz reports a recovered firing, or the firing was scheduled to take over an attempt that left its row unsettled, it re-claims the row from Processing, fenced on the attempt it read, and dead-letters a job that has no attempt left instead. A claim that changes nothing ends the firing. The firing's trigger is the job's only one, so a claim that throws re-fires the job, to be claimed again, rather than ending it.
3. Runs the handler (JobHandlerRunner) in its own DI scope, under an Endatix.Jobs activity whose parent is the trace captured at enqueue, with one token linked from the runtime ceiling (MaxRuntimeMinutes), the cancellation watcher and the host's shutdown.
4. Records the outcome (JobOutcomeRecorder) with one write fenced on the claimed attempt, tried again a few times if it throws, and records the lifecycle metrics only when it lands. A firing whose outcome still cannot be written is re-fired shortly by UnrecordedJobRefire. The wrapper never throws to Quartz. Every retry gets a trigger of the job's own, stored for the next attempt's time before the row says Retrying:

A re-fire is a trigger too, and storing it can fail. While the scheduler is stopping it refuses every new trigger but still completes the firings it waits for, which would delete the job's last trigger; the firing is held instead until the scheduler lets go of it, and the next node to check in recovers it. Once the scheduler has let go, its job store is shut down and refuses every completion, so a firing that ends then leaves its trigger in place and stores nothing. A re-fire that fails on a node that keeps running is logged at Error, then at most once a minute, and tried again, after a wait that doubles from 0.5 s up to 5 s; the firing does not complete before then, so the row is never left Processing with no trigger. Each try sets the trigger 5 s after that try, so one stored late is not a misfire. A held firing occupies a worker thread of its job type's cap until its trigger is stored.

A failure the database caused is tried until the trigger is stored. Any other refusal is tried 3 times, with the job type's durable job stored again first if it has gone missing, and then the row is dead-lettered, if it is still as the firing left it, and the firing completes, so a rejection that will not clear does not hold a worker for good. Should that write fail too, the firing goes on trying as before.

The row's AttemptCount, against the job type's MaxAttempts on the node that runs the attempt, decides dead-lettering, and the retry's backoff is that node's too. Nothing about retries is fixed at enqueue, so a change to either applies to jobs already waiting. Exception text never reaches ErrorMessage. NextAttemptAt is the time the retry's trigger was stored for.

Triggers from earlier versions carry a Quartz retry policy, because the wrapper used to throw for Quartz to retry. It no longer throws, so such a trigger ends its firing like any other, and the retry runs on a trigger of the job's own, which carries none. Should a firing on such a trigger throw anyway until its policy is spent while the row is unfinished, JobTriggerListener re-fires the job 5 s later to take it over, logs The scheduler ran out of retries for background job … and counts endatix.jobs.retries_exhausted_mismatch. Every trigger of a job carries the job's id as its key, so the take-over replaces any trigger the job has and never adds a second.

Shutdown. A stopping host gives running jobs ShutdownWaitSeconds to finish and record their outcome. Jobs still running then are left as they are: their handlers are told to stop, nothing is recorded, and Quartz re-runs them on the next node to check in.

Cancellation. Setting a row to Canceled reaches a running handler through the wrapper's watcher, which re-reads the status every CancellationPollSeconds, on whichever node runs the job.

IJobMetrics is public so a host can replace it. The default records on the Endatix.Jobs meter (JobsModule.MeterName), and EndatixTelemetryBuilder subscribes the host's metrics pipeline to it.

Configuration

Under Endatix:BackgroundJobs, with per-job-type settings under JobTypes:{JobType}.

Anything set in configuration wins over code. Each per-type setting of a job type is looked up on its own, and the first of these that has a value is used:

1. the job type's own key, JobTypes:{JobType}:{Setting}, in appsettings or an environment variable;
2. the global key, {Setting}, if the host set it;
3. the job type's default, declared in code when its handler was registered (the table below);
4. the global default (the second table below).

MaxConcurrency has no global key, so it takes the job type's own key, then the job type's default, then 1.

For example, a host whose appsettings.json has

"Endatix": {
  "BackgroundJobs": {
    "RetentionDays": 30,
    "JobTypes": { "WebHookDelivery": { "MaxAttempts": 10 } }
  }
}

keeps every finished job's row for 30 days, webhook deliveries included, and tries a webhook delivery 10 times; the other webhook settings stay as declared in code. The same with environment variables:

Endatix__BackgroundJobs__RetentionDays=30
Endatix__BackgroundJobs__JobTypes__WebHookDelivery__MaxAttempts=10

The options validator checks the values each job type runs with, and names the key each one came from; a value from code is named by the JobTypes key that replaces it.

Registration

Registered via EndatixBuilder.UseDefaults() → UseModule(JobsModule.Instance). Capabilities: IEndatixModule, IHasFeatureFlag, IHasDbMigrations, IHasFastEndpoints. Do not also Api.ScanAssemblies this assembly (bypasses the flag).

Gated by Endatix:FeatureFlags:JobsModule, off by default. The module owns a DbContext and its own migrations, so registering it where nothing enqueues would create a schema no code writes to.

Migrations

PostgreSQL is currently the only supported provider. With the flag off — the default — nothing is registered and other providers are unaffected. With the flag on and a different provider configured, the host fails at startup naming the constraint. The SQL Server context and its migration already exist, so a SQL Server database can be migrated and checked, but the module does not run on SQL Server yet.

Persistence is provider-split: JobsPostgreSqlDbContext and JobsSqlServerDbContext derive from JobsDbContextBase, and each owns its migrations and model snapshot under Persistence/Migrations/<Provider>. EF Core keeps one model snapshot per context type, so adding a provider means adding a derived context, its own design-time factory, its own Config/<Provider>/ configuration and its own migrations folder — never reusing an existing one.

The migrations were reset once, in the change that moved scheduling onto Quartz, to a single InitialBackgroundJobs migration. v0.7.6 and the canaries before this change shipped the earlier ones (AddBackgroundJobs, RequireRealTenantOnBackgroundJobs), and EF cannot upgrade a database from them: it would try to create jobs."BackgroundJobs" again and fail. Any database where Endatix:FeatureFlags:JobsModule was turned on under those releases needs DROP SCHEMA jobs CASCADE once before upgrading. That deletes every job row and the schema's migration history. Those rows never ran, because no release executed jobs, and they cannot be carried over, because each job now needs a Quartz trigger as well as its row. Back up jobs."BackgroundJobs" first if you need a record of them. From here on jobs migrations are append-only.

SQL Server's chain starts with its own InitialBackgroundJobs, which creates everything the three PostgreSQL migrations create, including the trigger acquisition index. It needs SQL Server 2025 or later, or Azure SQL Database, for the native json column type.

Run the commands from the repository root, with Endatix.WebHost as the startup project. Each design-time factory pins its provider and reads ConnectionStrings:DefaultConnection, so set a connection string of that provider.

# PostgreSQL
dotnet ef migrations add <Name> \
  --startup-project src/Endatix.WebHost \
  --project src/Endatix.Modules.Jobs \
  --context JobsPostgreSqlDbContext \
  --output-dir Persistence/Migrations/PostgreSql

# SQL Server
dotnet ef migrations add <Name> \
  --startup-project src/Endatix.WebHost \
  --project src/Endatix.Modules.Jobs \
  --context JobsSqlServerDbContext \
  --output-dir Persistence/Migrations/SqlServer

Migrations apply automatically at startup when Endatix:Data:EnableAutoMigrations is enabled.

Quartz schema and upgrades

Quartz never migrates its own schema. The module keeps Quartz's SchemaProvisioning at Validate: a node checks the tables at startup and refuses to start when one is missing or outdated, but never creates or changes one. Only the jobs EF migrations do.

Both providers' InitialBackgroundJobs run Quartz's create script for their dialect, create_postgres.sql or create_sqlServer.sql, read at migration time from the Quartz.NET assembly the module references (QuartzSchemaScripts), with the table prefix jobs.qrtz_. The module keeps no copy of either script, so a fresh database always gets the schema that Quartz version validates.

Quartz 4.2.2 embeds only its create scripts, not its upgrade scripts. Quartz's upgrade scripts (database/migrations/<version>/ in the Quartz.NET repository, such as 4.2/add_continuations_<dialect>.sql) are additive and guarded (IF NOT EXISTS, IF COL_LENGTH(…) IS NULL and the like), so they are no-ops on a database whose tables the create script just made.

When a Quartz bump changes the schema, add one EF migration per provider. The upgrade scripts do not use the create script's {0}/{1} placeholders: they name Quartz's default tables, so each statement has to be pointed at the module's tables (QuartzRegistration.TablePrefix, jobs.qrtz_) by hand.

1. Copy Quartz's guarded upgrade SQL for that provider inline with migrationBuilder.Sql("""…"""), under a comment linking the file at the Quartz release tag, for example https://github.com/quartznet/quartznet/blob/v<version>/database/migrations/<version>/<file>.sql. AddTriggerAcquisitionIndex shows the style. Keep every guard, adapted as below.
   • SQL Server (…_sqlServer.sql): point every reference to a Quartz table at jobs.qrtz_<TABLE>, in whatever form it appears: [dbo].[QRTZ_<TABLE>] becomes [jobs].[qrtz_<TABLE>], OBJECT_ID('dbo.QRTZ_<TABLE>') becomes OBJECT_ID('jobs.qrtz_<TABLE>'), COL_LENGTH('QRTZ_<TABLE>', …) becomes COL_LENGTH('jobs.qrtz_<TABLE>', …), and the same for any REFERENCES. An unqualified or dbo reference resolves in the default schema and never sees the module's table, so its guard runs a statement that is not needed. Also rename constraint and index names from PK_/FK_/IDX_QRTZ_… to PK_/FK_/IDX_qrtz_…, in the sys.indexes guard and the CREATE alike: the create script names them with the qrtz_ prefix, so an unrenamed guard misses the existing index and the CREATE fails (or, under a case-sensitive collation, adds a duplicate). The script separates batches with GO, which is a client command, not T-SQL: put each batch in its own migrationBuilder.Sql call.
   • PostgreSQL (…_postgres.sql): names are lower-case and unquoted. Replace each table reference qrtz_<table> with jobs.qrtz_<table> (not the qrtz_ inside index names, which already match the create script's once lower-cased), and add table_schema = 'jobs' to each information_schema check while keeping its table_name = 'qrtz_<table>' literal unqualified, because table_name never includes the schema; only the ALTER TABLE / CREATE … ON targets become jobs.qrtz_<table>. Without the schema check, the guard matches a same-named table in any schema. Each DO $$ … $$; block can stay in one call.
2. If a later Quartz version embeds its upgrade scripts, load them from the package the way QuartzSchemaScripts loads the create script, once they carry its placeholders; until then, adapt them as above.
3. A fresh database runs InitialBackgroundJobs with the new create script first, so the guarded upgrade then changes nothing.

Keep the acquisition index. Trigger acquisition reads qrtz_TRIGGERS through an index of the module's own, idx_endatix_qrtz_t_acquire, keyed on the group each trigger belongs to (COALESCE(EXECUTION_GROUP, JOB_NAME)). On SQL Server that group is the persisted computed column ENDATIX_ACQUIRE_GROUP, and the SQL Server acquisition filters on the column by name; on PostgreSQL the index is on the expression itself. An upgrade migration whose Quartz script rebuilds qrtz_TRIGGERS (drops and re-creates it, or copies it into a new table) must re-create, in the same EF migration, ENDATIX_ACQUIRE_GROUP and idx_endatix_qrtz_t_acquire on SQL Server and idx_endatix_qrtz_t_acquire on PostgreSQL. Quartz validates its own columns only, so it accepts the extra column. On SQL Server they also block ALTER TABLE … ALTER COLUMN: the column on EXECUTION_GROUP and JOB_NAME, the index on its other key and included columns (JOB_GROUP, MISFIRE_INSTR, PREFERRED_NODE and the rest), except where an nvarchar key or included column only widens (a primary-key column never may). An upgrade migration that alters one of them drops idx_endatix_qrtz_t_acquire, then ENDATIX_ACQUIRE_GROUP, first, and re-creates both with the same definitions in the same migration.

See the Quartz.NET database documentation and its operations page.

Third-party licence

Quartz.NET is Apache-2.0 licensed. Its licence text ships in THIRD-PARTY-NOTICES, both in this package and in the API image (/app/THIRD-PARTY-NOTICES).