Execution Log
Every execution in Obelisk maintains a persistent, append-only execution log. The log is the single source of truth for the execution’s history: it records how the execution was created, every lock acquisition, every child execution request, every result, and the final outcome.
The execution log enables two critical capabilities:
- Crash recovery — after a restart, a workflow is replayed from its log to reconstruct its state.
- Observability — the full history of an execution is available for inspection via the REST API and the Web UI.
Events
Each entry in the execution log is an event with a monotonically increasing version number (starting at 0) and a timestamp. The following table lists all event types and their effect on the execution state.
| Event | Description | State Transition |
|---|---|---|
| Created | The execution was submitted. Records the function (FFQN), parameters, parent execution (if any), scheduled time, and component assignment. | → PendingAt |
| Locked | An executor acquired the execution. Records the executor ID, run ID, lock expiry, component ID, and retry configuration. | PendingAt → Locked |
| Unlocked | The executor voluntarily released the execution (e.g., shutting down or resource limits). Records a backoff_expires_at time and a reason. | Locked → PendingAt |
| ComponentUpgradeFinished | A manual or automatic workflow component upgrade finished. Records the target component digest, deployment, and success or failure outcome. | No change |
| TemporarilyFailed | An activity encountered a retryable error (e.g., a WASM trap). Records backoff expiry, reason, and optional HTTP client traces. | Locked → PendingAt |
| TemporarilyTimedOut | An activity timed out but has retries remaining. Records backoff expiry and optional HTTP client traces. | Locked → PendingAt |
| Finished | The execution completed. Records the return value or error. This is a terminal event — no further events can be appended. | Any → Finished |
| HistoryEvent | A workflow’s internal decision or interaction with the runtime. See History Events below. | Varies |
| Paused | The execution was paused via CLI or API. | Any non-Finished → Paused |
| Unpaused | The execution was resumed. | Paused → (previous underlying state) |
| CancellationRequested | Cancellation was requested for an activity or cancellable workflow. | Any non-Finished → Cancelling |
See Cancellation for the structured-concurrency teardown rules.
History Events
History events are created by the executor while the execution is in Locked state. They capture the deterministic decisions made by a workflow during execution. During replay, the runtime compares the workflow’s requests against the recorded history events to detect nondeterminism.
| History Event | Description | State Effect |
|---|---|---|
| Persist | Records a generated pseudorandom value (random u64 or random string). Used for determinism: on replay the same value is returned. | No change |
| JoinSetCreate | A new join set was created. | No change |
| JoinSetRequest | A request was added to a join set. This is either a ChildExecutionRequest (spawn a child activity or workflow) or a DelayRequest (persistent sleep). | No change |
| JoinNext | The workflow is waiting for the next response in a join set. If no response is available, the execution transitions to BlockedByJoinSet. If a response is already available, the execution stays Locked (or becomes PendingAt after lock expiry). | Locked → BlockedByJoinSet (if no response available) |
| JoinNextTry | Non-blocking poll of a join set. Records whether a response was found, is still pending, or all requests were processed. | No change |
| JoinNextTooMany | The workflow attempted to await more responses than were submitted to the join set. | No change |
| Schedule | The workflow scheduled a child execution outside of a join set (fire-and-forget). | No change |
| Stub | The workflow wrote a result to a stub execution. | No change |
Responses
In addition to the event log, each execution maintains a list of responses. Responses arrive asynchronously — they are written by child executions or the delay timer, not by the workflow itself:
| Response | Description |
|---|---|
| ChildExecutionFinished | A child activity or workflow in a join set completed. Contains the child execution ID, join set ID, and the result. |
| DelayFinished | A delay request in a join set expired. Contains the delay ID and join set ID. |
When a response arrives and the execution is in BlockedByJoinSet state waiting on that join set, the execution transitions to PendingAt, making it eligible for an executor to pick up and continue.
Versioning
Each event is assigned a version number equal to its index in the log (starting at 0 for the
Created event). The version serves as an optimistic concurrency control token: when an executor
appends events, it must specify the expected current version. If it does not match, the write is
rejected.
Once an execution reaches the Finished state, the version counter stops advancing.
Example: Activity Lifecycle
The following shows a typical activity execution log:
Version 0: Created(ns:pkg/ifc.my-activity, scheduled_at=2025-01-15T10:00:00Z)
→ State: PendingAt(2025-01-15T10:00:00Z)
Version 1: Locked(executor=E1, run=R1, expires_at=2025-01-15T10:00:30Z)
→ State: Locked(E1, R1)
Version 2: Finished(Ok([42]))
→ State: Finished(ok)Example: Activity with Retries
An activity that fails temporarily before succeeding:
Version 0: Created(ns:pkg/ifc.flaky-activity, scheduled_at=...)
→ State: PendingAt
Version 1: Locked(executor=E1, run=R1, expires_at=...)
→ State: Locked
Version 2: TemporarilyFailed(backoff_expires_at=+1s, reason="connection refused")
→ State: PendingAt(+1s)
Version 3: Locked(executor=E1, run=R2, expires_at=...)
→ State: Locked
Version 4: TemporarilyFailed(backoff_expires_at=+2s, reason="connection refused")
→ State: PendingAt(+2s)
Version 5: Locked(executor=E1, run=R3, expires_at=...)
→ State: Locked
Version 6: Finished(Ok("success"))
→ State: Finished(ok)Example: Workflow with Child Execution
A workflow that spawns a child activity and waits for its result:
Version 0: Created(ns:pkg/ifc.my-workflow, scheduled_at=...)
→ State: PendingAt
Version 1: Locked(executor=E1, run=R1, expires_at=T+30s)
→ State: Locked
Version 2: HistoryEvent(JoinSetCreate(o:1))
→ State: Locked (no change)
Version 3: HistoryEvent(JoinSetRequest(o:1, ChildExecutionRequest(child_id, ns:pkg/ifc.my-activity)))
→ State: Locked (no change)
Version 4: HistoryEvent(JoinNext(o:1, run_expires_at=T+30s))
→ State: BlockedByJoinSet(o:1) [no response available yet]
--- Response arrives: ChildExecutionFinished(child_id, Ok(42)) ---
→ State: PendingAt(max(T+30s, response_time))
Version 5: Locked(executor=E1, run=R1, expires_at=T+60s) [lock extension]
→ State: Locked
Version 6: HistoryEvent(JoinNextTry(o:1, Found))
→ State: Locked (no change)
Version 7: Finished(Ok(42))
→ State: Finished(ok)Note: if the original executor is still holding the WASM instance warm (see Join Next Blocking Strategy), it may continue directly without an additional Locked event.
Inspecting the Execution Log
REST API
# Get execution events
curl -H 'Accept: application/json' \
http://127.0.0.1:5005/v1/executions/{id}/events
# Include backtrace IDs (for source mapping in the Web UI)
curl -H 'Accept: application/json' \
'http://127.0.0.1:5005/v1/executions/{id}/events?include_backtrace_id=true'The response contains a list of events, each with its version, timestamp, and event type.
CLI
obelisk execution status {id}
obelisk execution result {id}Web UI
The execution detail view shows the full event log as an interactive timeline. Each event is displayed with its version number, timestamp, and details. The time-traveling debugger uses the execution log to reconstruct and visualize the execution at any point in its history.