OpenAI-compatible models

The model-provider seam. create_language_model_client returns a client satisfying the LanguageModelClient protocol, routing to Azure OpenAI or to any OpenAI-compatible host (Ollama, vLLM) without the caller knowing which.

OpenAISDKClient is the shared base both concrete clients inherit from — it holds the chat/embedding calls, retry logic and token-metric logging, so a provider class only supplies its own SDK construction.

Provider-agnostic language-model client seam.

Workbench issues all chat/embedding traffic through a small, provider-neutral surface — call_chat(messages) -> str and call_embedding(batch) -> list[list[float]]. This module defines that surface as the LanguageModelClient Protocol and a factory that constructs the right concrete client for a chosen provider, so call sites can be routed to Azure OpenAI or any OpenAI-compatible host (OpenAI directly, Ollama, vLLM, hosted open models) without changing the callers.

LanguageModelClient

Bases: Protocol

Structural interface every language-model client conforms to.

Callers depend only on this surface — never on provider-specific SDK objects. Both AzureOpenAIClient and OpenAICompatibleClient satisfy it (as does the test double MockAzureOpenAIClient).

call_chat

call_chat(messages, max_retries=5, max_completion_tokens=None)

Return the chat completion text for messages.

call_embedding

call_embedding(batch, max_retries=5)

Return one embedding vector per string in batch.

close

close()

Release any underlying connections.

ModelProvider

Bases: StrEnum

Supported language-model hosting providers.

create_language_model_client

create_language_model_client(provider=None, **config)

Construct a language-model client for the given provider.

Parameters:

Name	Type	Description	Default
`provider`	`ModelProvider \| str \| None`	`ModelProvider` (or its string value). If omitted, defaults to the `MODEL_PROVIDER` environment variable, then Azure OpenAI. This preserves existing behaviour: with no provider set, calls go to Azure exactly as before.	`None`
`**config`	`Any`	Connection/config kwargs. Keys not relevant to the selected provider are ignored, so a single loosely-populated config dict can drive any provider. Azure keys: `api_key`, `azure_endpoint`, `api_version`, `model`, `response_format`, `temperature`, `timeout`. OpenAI-compatible keys: `api_key`, `base_url`, `model`, `response_format`, `temperature`, `timeout`.	`{}`

Returns:

Type	Description
`LanguageModelClient`	A concrete client satisfying `LanguageModelClient`.

Raises:

Type	Description
`ValueError`	If `provider` is unrecognised, or required config is missing.

OpenAISDKClient

Shared implementation for clients backed by an OpenAI-style SDK.

Both the Azure OpenAI SDK (openai.AzureOpenAI) and the plain OpenAI SDK (openai.OpenAI, which also fronts OpenAI-compatible hosts such as Ollama or vLLM via a custom base_url) expose identical chat.completions.create and embeddings.create interfaces and return identically-shaped response objects. This base therefore holds the retry/backoff, token-metric logging and response parsing once; concrete subclasses only differ in how they construct self.client and resolve self.model.

Subclasses must call super().__init__(...) with an already-constructed SDK client and the resolved per-instance model, response_format and temperature.

Attributes:

Name	Type	Description
`client`	`AzureOpenAI \| OpenAI`	The underlying `AzureOpenAI` or `OpenAI` SDK client.
`model`	`str \| None`	Model / deployment name used for chat and embedding calls.
`response_format`	`dict[Any, Any] \| None`	Optional response format, e.g. `{"type": "json_object"}`.
`temperature`	`float \| None`	Sampling temperature; `None` defers to the API default.

del

__del__()

Destructor to ensure cleanup if close() wasn't called.

enter

__enter__()

Context manager entry point.

exit

__exit__(exc_type, exc_val, exc_tb)

Context manager exit point - ensures connections are cleaned up.

call_chat

call_chat(messages, max_retries=5, max_completion_tokens=None)

Call the chat completions API.

Parameters:

Name	Type	Description	Default
`messages`	`list[dict[str, Any]]`	List of dictionary objects specifying the messages to send. Messages must adhere to the prompting standard.	required
`max_retries`	`int`	Number of times to call the API before raising an error	`5`
`max_completion_tokens`	`int \| None`	Maximum number of tokens to generate in the response. If None, uses API default.	`None`

Returns:

Type	Description
`str`	Response from the chat API.

Raises:

Type	Description
`RuntimeError`	If attempts exceeds `max_retries`

call_embedding

call_embedding(batch, max_retries=5)

Call the embeddings API.

Parameters:

Name	Type	Description	Default
`batch`	`list[str]`	List of strings to embed	required
`max_retries`	`int`	Number of times to call the API before raising an error	`5`

Returns:

Type	Description
`list[list[float]]`	List of embeddings

Raises:

Type	Description
`RuntimeError`	If attempts exceeds `max_retries`

close

close()

Explicitly close the underlying SDK client and release all connections.

Call this method when you're done using the client to ensure connections are properly cleaned up, especially in high-concurrency scenarios.

OpenAICompatibleClient

Bases: OpenAISDKClient

A client for any OpenAI-compatible chat/embeddings endpoint.

Uses the plain openai.OpenAI SDK pointed at a custom base_url, which covers OpenAI directly, local runtimes such as Ollama (http://localhost:11434/v1) and vLLM, and most hosted open-model providers. Chat/embedding calls, retry logic and token-metric logging are inherited from the shared OpenAISDKClient base, so token counts land under the same counts.chat.* / counts.embeddings.* metric paths as the Azure client.

Attributes:

Name	Type	Description
`api_key`	`str`	API key for the endpoint. Defaults to environment variable `OPENAI_COMPATIBLE_API_KEY`, then `OPENAI_API_KEY`. Many local runtimes ignore this; a placeholder is used if none is set.
`base_url`	`str`	The endpoint base URL, e.g. `http://localhost:11434/v1`. Defaults to environment variable `OPENAI_COMPATIBLE_BASE_URL`.
`model`	`str \| None`	Model name to request. Defaults to environment variable `OPENAI_COMPATIBLE_MODEL`.
`response_format`	`dict[Any, Any] \| None`	The type of response to request. For example for JSON: { "type": "json_object" }.
`client`	`AzureOpenAI \| OpenAI`	The `OpenAI` SDK client.

init

__init__(api_key=None, base_url=None, model=None, response_format=None, temperature=0.1, timeout=600.0)

Initialize the OpenAI-compatible client.

Parameters:

Name	Type	Description	Default
`api_key`	`str \| None`	API key for the endpoint. If not provided, defaults to `OPENAI_COMPATIBLE_API_KEY`, then `OPENAI_API_KEY`. If still unset a placeholder (`"not-needed"`) is used, since many local runtimes (e.g. Ollama) do not require a key.	`None`
`base_url`	`str \| None`	The endpoint base URL, e.g. `http://localhost:11434/v1`. If not provided, defaults to `OPENAI_COMPATIBLE_BASE_URL`.	`None`
`model`	`str \| None`	Model name to request. If not provided, defaults to `OPENAI_COMPATIBLE_MODEL`.	`None`
`response_format`	`dict[Any, Any] \| None`	The type of response to request, e.g. `{"type": "json_object"}`.	`None`
`temperature`	`float \| None`	Sampling temperature. Pass `None` to defer to the endpoint default.	`0.1`
`timeout`	`float`	Read/write timeout in seconds (default: 600.0 / 10 minutes).	`600.0`