Events
Events are the spine
A change in one platform is captured, identified, validated, processed and passed on, without creating duplicates or losing state.
- id
- evt_7Q2K
- type
- order.paid
- source
- storefront
- at
- 2026-10-18T09:14:03Z
Automation demos are easy. Reliable automation is not. The engineering starts when a webhook arrives twice, two systems disagree, an API times out, a record disappears, a workflow runs out of order, or a model returns something unexpected.
Five rules
Most of this is unglamorous. It is also the difference between a demo and a system a business can run on.
Events
A change in one platform is captured, identified, validated, processed and passed on, without creating duplicates or losing state.
Idempotency
If the same event arrives twice, the business action happens once. Unique event IDs, state tracking and safe writes make sure of it.
Reconciliation
One platform reports 47 units, another reports 42. A useful integration detects the gap, records the evidence, applies a resolution policy and leaves an audit trail.
Failure
Timeouts, rate limits, malformed payloads, partial updates and outages are operating conditions. Good architecture makes them visible and recoverable.
Observability
Logs, metrics, event histories and clear ownership turn debugging from archaeology into operations.
In one line
Reliable automation is software engineering with consequences in the real world.
Trace
A webhook arrives, a model times out, and the sender delivers the same event twice. Replay it and watch the counts: delivered twice, applied once, nothing lost.
| T+ ms | Stage | What happened | Result |
|---|---|---|---|
| 0 | Receive | Webhook for order A-1042 arrives with event ID evt_7Q2K | Accepted |
| 4 | Validate | Payload matches schema v3 | Pass |
| 9 | Deduplicate | evt_7Q2K has not been seen before | New |
| 18 | Enrich | Customer and stock looked up in the source systems | OK |
| 26 | Rules | No credit hold, stock available | Pass |
| 31 | Model | Classify the delivery note, with a 2 s budget | Sent |
| 2,031 | Timeout | No answer within the budget | Timeout |
| 4,090 | Retry 1 | After a 2 s backoff the model answers, confidence 0.93 | OK |
| 4,104 | Write back | Order updated in the ERP using evt_7Q2K as the request key | Applied |
| 4,118 | Receive | The sender delivers the same webhook again | Duplicate |
| 4,120 | Deduplicate | evt_7Q2K was already applied, so nothing is written | Skipped |
| 4,141 | Close | Trace stored: 12 steps, 1 retry, 1 duplicate ignored | Done |
T+4,141 ms
Illustrative IDs and timings. Every OTLEN system keeps a trace with this shape for every event, so the four questions in U02 always have answers.
Failure catalogue
Each failure has a planned response and a place where it shows up. Nothing fails silently.
| Code | Condition | What the system does | Shows up as |
|---|---|---|---|
| 504 | Timeout | Retries with exponential backoff. After the last attempt the event moves to the dead-letter queue. | Retry count on the trace |
| 429 | Rate limited | Slows down, honours the Retry-After header and keeps work in the queue. | Queue depth |
| 422 | Malformed payload | Rejects the event to the dead-letter queue with the validation error attached. | Dead-letter entry with reason |
| 409 | Duplicate delivery | Skips by event ID. Nothing is written twice. | Skipped step |
| 503 | Upstream outage | Opens the circuit breaker, holds work in the queue and alerts the owner. | Alert and held count |
| — | Partial update | Reconciles against the system of record and repairs the gap. | Reconciliation report |
| — | Unexpected model output | Validates the answer against a schema. If it fails, a person gets the case with its context. | Escalation |