Skip to documentation
Documentation navigation

Documentation navigation

Documentation / guides

Cancellation and Shutdown

Describe Hustle cancellation and session shutdown.

developer

Cancellation belongs to the run owner and is separate from the caller’s observation context. Shutdown is a session lifecycle transition that drains owned Hustle activity before closing the resources those runs depend on.

Cancellation stages

StageEffect
Preflight, before ownershipRequest is rejected; no RunID, audit pair, or finalizer.
Queued owned runLane removes or closes the node, emits a queue failure, and invokes the finalizer once.
Active inferenceRuntime cancels the worker context. A terminal HustleFailed records canceled or timeout reason as appropriate.
FinalizationThe finalizer receives the terminal outcome under its bounded finalization context; cleanup errors remain typed.

Caller cancellation is not a license to drop an owned run. The controller keeps ownership through terminal audit and finalization.

Proof: execution ownership and cancellation tests.

Shutdown order

Controller.Close(ctx) closes both admissions, cancels active execution, finishes queued owned runs through their finalizers, waits for worker drain, and reports any finalizer failures as CloseError. Session shutdown keeps the session context alive while Hustle activity, terminal audit, and finalization complete; only then does it publish idle/stopped lifecycle and release the session lease.

%%{init: {"theme":"dark"}}%%
flowchart TD
    S[session shutdown] --> C[close blocking and background admission]
    C --> X[cancel active runs]
    X --> Q[finish owned queued runs]
    Q --> F[terminal audit and finalizers]
    F --> D[worker drain]
    D --> R[close loops and resources]

Proof: controller Close and Hustle shutdown ordering tests.

Re-entry and finalizer safety

Finalizers run once for every owned run. A finalizer must not synchronously invoke session shutdown from the finalizer context; the runtime returns the typed HustleShutdownReentryError in that case. Finalizer, activity release, audit, and cleanup failures remain inspectable through errors.As and do not erase the original terminal reason.

Proof: finalizer boundary and shutdown re-entry test.

Source and proof

← back to documentation