cid a8a17132a17d199ddb78dacbf6f650c2defe94411cff054a39657c31e4c84aec · original · open access · via openalex
Title: Rule Engine Core: a declarative complex event processing runtime with deterministic replay and at-least-once delivery Authors: Diogo Ribeiro Venue: Zenodo (CERN European Organization for Nuclear Research) Year: 2026 Canonical: doi:10.5281/zenodo.22001698 URL: https://doi.org/10.5281/zenodo.22001698 Collector: openalex OA: true ABSTRACT Rule Engine Core is a compact, embeddable complex event processing (CEP) runtime written in Python. Detection logic is expressed declaratively in YAML, validated against a formal schema at load time, compiled into executable runtime objects, and evaluated over timestamped, entity-keyed event streams. The package is domain-neutral: events carry an entity identifier, a source type, arbitrary numeric fields, and a timestamp, and all domain semantics live in the rules rather than in the runtime. Rule model. Five trigger families are supported: event — a matching event satisfies a condition; window — an aggregate over a time window satisfies a condition; absence — no matching event arrives before a timeout elapses; composite — per-source absence timers combine under the rule's AND/OR operator; scheduled — a cron expression elapses. Window rules support count, sum, mean, minimum, maximum, standard deviation, delta, rate, and percentile aggregations, optionally bucketed by a sub-window. Trigger fields, condition operators, duration literals, cron expressions, and sink declarations are all validated during compilation, so malformed rules fail fast with path-aware errors instead of surfacing partway through a run. Determinism. Timer-driven rules advance on an explicit watermark rather than on wall-clock time. Replaying the same events against the same initial watermark therefore yields identical output, which makes rule behaviour reproducible, diffable, and testable with golden-file fixtures. This event-time discipline follows the model established for unbounded, out-of-order stream processing. Delivery. Emitted alerts are dispatched through a sink layer providing at-least-once semantics: a versioned delivery envelope carrying a deterministic idempotency key, bounded retries with configurable backoff, dead-letter recording (in-memory or file-backed, with optional retention limits and fsync-oriented persistence), and a delivery metrics snapshot covering per-sink counts, retry activity, unsupported routes, and measured latency. Five adapters are implemented: standard output, file, webhook (with authentication headers and HMAC body signing), queue, and object storage, the latter two over pluggable transports. Event time and recovery. Event-time order is enforced rather than assumed. A rule may declare an allowed_lateness tolerance, and events arriving within it are folded into rule state in place without rewinding the watermark or re-firing timers; events later than any rule tolerates are rejected or dropped by explicit policy, and counted either way. Engine state — watermark, per-entity rule state, pending timers, in-flight window buffers, and late-event counters — can be captured as a versioned, JSON-serializable snapshot and restored, so a resumed run emits the same alerts as an uninterrupted one. Each snapshot records a structural fingerprint per rule, so restoring into a rule whose windows or timers changed fails loudly instead of reinterpreting state that no longer means the same thing. Alert lifecycle and rule lifecycle. A rule may declare an emission policy giving its alerts an episode model: a cooldown throttling repeats, a timer-driven reminder interval that fires without new events, and an optional resolution emitted when the condition clears. Every delivered payload carries a lifecycle label and a correlation identifier stable across one episode, so a consumer can join a resolution back to the alert that opened it. Rule sets can be swapped on a live engine under an explicit state-migration policy — discard state, preserve it where the rule's structure is unchanged, or keep the previous definition running for entities with an open episode until it resolves — with the swap optionally staged until the watermark reaches a chosen instant. Patterns, partitions, and inspection. Beyond conditions over single events and aggregates, rules can match ordered temporal patterns: a bounded sequence of event types, optionally cancelled by a negating event. Bounding each pattern is what keeps partial-match state bounded per key, and therefore serializable with the rest of the engine state. State keys are declarative rather than assumed: a rule may be partitioned by any combination of event fields, with ordering and isolation guaranteed inside each partition. Two inspection surfaces sit on top of the deterministic core. An explain facility reports, for any event, what each rule would do and why, naming the predicate that stopped a rule that would not fire together with the value observed, without altering engine state. A simulation facility backtests a stream against a rule set in a clean engine, reporting per-rule evaluation, alert, suppression and resolution counts, and diffing two rule versions over the same stream so the effect of a change can be measured before it is deployed. Reconciling late data. Lateness is measured against each entity's own progress rather than a single engine clock, so an entity reporting ahead of the others cannot make their in-order events appear late; explicit clock advancement still applies to every entity, since it asserts that time has moved rather than observing one source. Timer progress deliberately remains engine-wide: an entity that falls silent never advances its own clock, so per-entity timers would prevent the very absence alert such an entity should raise. Where configured, a tolerated late event also reopens windows that have already closed, retracting an alert whose window no longer holds under the originating episode's correlation identifier, firing one that now holds, and emitting nothing when the verdict is unchanged. Engineering. The runtime separates compilation from execution, so compiled rules can be embedded directly without going through the command-line interface. Public results — rule metadata, evaluation results, delivery reports, and metrics snapshots — are typed objects with structured dictionary and JSON exports for downstream tooling. The package targets Python 3.11 or later, depends only on PyYAML at runtime, and is linted and type-checked in continuous integration.