Model backends

For developers who connect a client to OpenBot or extend its server. The examples need the access stated on this page. Start with the developer manual.

For project contributors who change or publish OpenBot.

Built-in models

When you run OpenBot from your own program, you can give models in code with the builtinModels option. See Run from code.

Every organization can use them.
The API cannot change them.
They show in the model list next to the stored providers.

builtinModels has three fields:

Field	What it is
`providers`	The providers to show, such as `{ id, name }`.
`models`	The models to show, as `Config.Model` values.
`resolve`	A function that gets a selection and returns an Effect with a `LanguageModel`.

The builtinModels.defaultModel option names one of these models. A new Chat gets it when its Bot names no model. Only a simulated platform, such as development or a test, sets it. On a real platform, each Bot names its model.

This example gives one Anthropic model with a key from the environment of your program:

import { NodeRuntime, NodeServices } from "@effect/platform-node"
import { Config, Model, Provider } from "@openbot-build/core"
import {
  AnthropicProvider,
  EnvironmentConfiguration,
  Server,
} from "@openbot-build/openbot"
import { Context, Effect, Layer, Redacted } from "effect"
import { LanguageModel } from "effect/ai"
import { FetchHttpClient } from "effect/http"

const selection = Config.Selection.make({
  provider: Provider.Id.make("house"),
  model: Model.Id.make("claude-sonnet-4-5"),
})

NodeRuntime.runMain(
  Effect.scoped(
    Effect.gen(function* () {
      const settings = yield* EnvironmentConfiguration.load
      // Build the model once. It lives as long as the program.
      const layer = Layer.unwrap(
        AnthropicProvider.make({ max_tokens: 4096 }).model(selection),
      ).pipe(
        Layer.provide(
          AnthropicProvider.layerClient({
            apiKey: Redacted.make(process.env.ANTHROPIC_API_KEY ?? ""),
          }),
        ),
        Layer.provide(FetchHttpClient.layer),
      )
      const claude = Context.get(
        yield* Layer.build(layer),
        LanguageModel.LanguageModel,
      )
      const builtinModels = {
        providers: [{ id: selection.provider, name: "House models" }],
        models: [
          Config.Model.make({
            provider: selection.provider,
            id: selection.model,
            name: "Claude Sonnet 4.5",
            reasoningEfforts: [],
            serviceTiers: [],
          }),
        ],
        resolve: (requested: Config.Selection) =>
          requested.model === selection.model
            ? Effect.succeed(claude)
            : Effect.fail(
                new Config.ModelConfigurationError({
                  reason: "model-not-found",
                  message: "This model is not available",
                }),
              ),
      }
      return yield* Layer.launch(
        Server.layerLive({ ...settings, builtinModels }).pipe(
          Layer.provide(Layer.merge(NodeServices.layer, FetchHttpClient.layer)),
        ),
      )
    }),
  ),
)

When resolve fails, starting a Turn returns 503. The development entry point uses builtinModels for its test models. See packages/openbot/src/app/DevelopmentModel.ts for a full example.

Provider modules

A provider module turns a model name into a LanguageModel for one kind of API. @openbot-build/openbot exports three of them. Each one has the same parts:

Part	What it does
`make(defaults?)`	Makes a provider. `defaults` are request options for every call.
`provider.model(selection)`	Returns the model Layer for one selection. It sends no requests.
`layerClient(options)`	Makes the HTTP client. Pass `apiKey`, and `apiUrl` for another address.
`layer(tag, defaults?)`	Puts the provider in a service that you name.

Module	Provider ID	API
`OpenAiProvider`	`openai`	OpenAI Responses API. `ResponsesCompaction` adds native context compaction for your own programs.
`OpenAiCompatProvider`	`openai-compat`	OpenAI Chat Completions API, and services that copy it.
`AnthropicProvider`	`anthropic`	Anthropic Messages API.

OpenBot uses these modules for the stored providers too. Context compaction makes a long history shorter so that it fits in the model’s context window. OpenBot does this for every Chat with a summary that the model of the Chat writes. It does not use ResponsesCompaction by default. See Compaction. See Packages for a short example of each provider.

For Cloudflare Workers AI, use an openai-compat provider with the account endpoint https://api.cloudflare.com/client/v4/accounts/<account-id>/ai/v1. Workers AI rejects null text in an assistant message that calls a tool. OpenBot sends an empty string for that message, so the next model call can read the tool result. This change applies only to this Cloudflare endpoint. Other providers keep the same messages.

Development models

The development server (devenv shell -- pnpm dev) does not call a real model. It uses test models instead. They give the same answer every time, so you can test clients without cost.

The provider ID is development. The model IDs are:

Model	What it does
`mock-text`	The default. Returns fixed text at once.
`mock-slow`	Returns text slowly over ten seconds. Use it to test streams and interrupts.
`mock-tool`	Calls a harmless test tool that needs a decision. See Requests.
`mock-app`	Calls a tool of the test App and returns its real content.
`mock-failure`	Sends some text, then fails on purpose with a model error.

The token counts in their answers are not real. To pick one, save it on a Chat:

curl -X PATCH -H "Authorization: Bearer $OPENBOT_TOKEN" -H 'Content-Type: application/json' \
  -d '{"expected":1,"model":{"_tag":"Config.Selection","provider":"development","model":"mock-slow"}}' \
  http://localhost/v1/chats/$CHAT_ID

The development entry point gives these models with builtinModels, so every organization of the development platform can use them. It gives mock-text as builtinModels.defaultModel, so a new Chat gets it when its Bot names no model. The production program never uses them.