Documentation / guides
Retries
Describe retry behavior for Hustle inference.
Retries are an immutable policy on a Hustle definition. They are distinct from Inference client retries: Hustle retries restart classified background work, while transport retries repeat eligible model-provider operations. The default
RetryPolicyNone performs one attempt. RetryPolicyClassifiedOnce allows one
restart only for the closed evidence-backed classifications accepted by the
runtime.
Policy API
// package hustle
type RetryPolicy uint8
const (
RetryPolicyNone RetryPolicy = iota
RetryPolicyClassifiedOnce
)
definition, err := hustle.Define(
// Classified retry requires an evidence-enabled, blocking definition.
hustle.WithRetryPolicy(hustle.RetryPolicyClassifiedOnce),
// Other required options omitted here for brevity.
)
_ = definition
_ = err
WithRetryPolicy is a singleton option. Descriptor validation rejects an
unknown value and rejects classified retry without an evidence policy.
Proof: retry policy type and option and definition validation.
What may retry
The runtime may retry exactly once after a clean restart for a closed set of
transient inference or recoverable terminal-output failures. It does not retry
context cancellation, timeout, controller poison, unsafe result decisions,
basis mismatches, or arbitrary consumer errors. A strict adapter may mark a
malformed terminal with hustle.NewRecoverableTerminalValidationError() and a
trusted caller can test it with IsRecoverableTerminalValidationError.
The retry keeps one owned RunID and one lifecycle audit record pair. It does not create a second public operation or call the finalizer twice.
Proof: retry execution policy and retry tests.
Failure stage matrix
hustle.ReasonAllowed(stage, reason) is a closed matrix used by audit
validation:
| Stage | Allowed reason families |
|---|---|
StageQueue | rejected, canceled, timeout, internal |
StageModelResolution | canceled, timeout, model resolution, internal |
StageInference | canceled, timeout, inference, internal |
StageOutput | canceled, timeout, invalid output, internal |
StageTerminal | timeout, terminal, internal |
StageFinalization | timeout, finalization, internal |
Proof: ReasonAllowed matrix and audit validation tests.