Overview

inference.Client is the provider-neutral seam above codecs, routes, transports, and authentication. It exposes one complete-response method and one pull-based streaming method.

API surface

type Client interface {
	Invoke(ctx context.Context, req Request) (*Response, error)
	Stream(ctx context.Context, req Request) (*stream.StreamReader[content.Chunk], error)
}

The request and response types are also provider-neutral:

type Request struct {
	Model             model.Model
	System            string
	Messages          content.AgenticMessages
	TransientMessages int
	Tools             []Tool
	Output            *OutputSchema
	ToolChoice        ToolChoice
	Override          *model.Sampling
}

type Tool struct {
	Name        string
	Description string
	Schema      json.RawMessage
}

Implementations own HTTP resources and should accept a caller’s context for cancellation and deadlines. A client is not required to expose provider-specific methods; those belong below this interface.

Choosing a method

Use Invoke when the caller needs one *Response and does not need incremental display. Use Stream for latency-sensitive UI or when the caller needs content.Chunk deltas. Both paths validate request feature combinations before codec encoding.

Proof

• Source: inference/client.go
• Tests: inference/client_test.go includes a compile-time client implementation and request field coverage.

Related: Complete inference, Streaming inference, Requests, and the Harness models and inference loop guide.