AkurAI-Framework / site/content/docs/assistant-api.md

Latest change a81af83654d182e3becb8e2beaed6d61a3bd8e6b - feat(llm): streaming OpenAI-compatible client + /api/assistant SSE endpoint by Ólafur Búi Ólafsson

# Assistant API

The `POST /api/assistant` endpoint is the streaming backend for an AI chat UI. It
takes a list of chat messages, calls any OpenAI-compatible
`/v1/chat/completions` endpoint with streaming enabled, and relays the model's
output to the browser as **Server-Sent Events** — token by token, plus assembled
tool calls. It is built on the framework's own pure-`std` `akurai-llm` client and
its existing SSE support; there are no external dependencies and no bundler.

## Request

```
POST /api/assistant
Content-Type: application/json
```

```json
{
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "Hello!" }
  ],
  "tools": [
    {
      "type": "function",
      "function": { "name": "get_weather", "description": "…", "parameters": {} }
    }
  ]
}
```

- `messages` (required) — a non-empty array of `{ "role", "content" }` objects.
  Both fields must be strings. The roles are passed straight through to the model
  (`system`, `user`, `assistant`, …).
- `tools` (optional) — an OpenAI `tools` array, forwarded verbatim in the
  upstream request. Omit it (or pass `[]`) to disable tool calling.

A non-`POST` method returns `405`. A malformed body does **not** fail the
request — it streams a single `error` event (see below) so the client only ever
has to handle one response shape.

## Response — Server-Sent Events

The response is `text/event-stream`. Each event has a named `event:` type and a
`data:` payload:

| Event       | `data:` payload                                  | Meaning                                            |
| ----------- | ------------------------------------------------ | -------------------------------------------------- |
| `delta`     | the token text (raw string)                      | A streamed chunk of assistant text. Concatenate.   |
| `tool_call` | JSON `{ "id", "name", "arguments" }`             | A fully-assembled tool call (one per call).        |
| `done`      | `[DONE]`                                          | The stream finished; no more events will arrive.   |
| `error`     | JSON `{ "error": "<message>" }`                  | A graceful failure; the stream then ends.          |

Ordering: zero or more `delta` events, then zero or more `tool_call` events,
then exactly one terminal `done` — unless an `error` is emitted instead. Tool
calls are reassembled server-side from the upstream's streamed fragments, so each
`tool_call` event carries the complete `id`, `name`, and `arguments` JSON string.

Example stream:

```
event: delta
data: Hello

event: delta
data: , world

event: tool_call
data: {"id":"call_1","name":"get_weather","arguments":"{\"location\":\"NYC\"}"}

event: done
data: [DONE]
```

Consume it from the browser with `EventSource` (for `GET`) or a `fetch` +
`ReadableStream` reader (for this `POST`), dispatching on the event name.

## Configuration

The endpoint and model are read from the environment at request time — nothing is
hardcoded:

| Variable           | Example                       | Purpose                                  |
| ------------------ | ----------------------------- | ---------------------------------------- |
| `AKURAI_LLM_URL`   | `http://localhost:8080`       | Base URL of the OpenAI-compatible server |
| `AKURAI_LLM_MODEL` | `gemma-3-12b`                 | Model name sent in the request body      |

If either variable is unset (or empty), the endpoint streams a single `error`
event — `LLM endpoint not configured (set AKURAI_LLM_URL and AKURAI_LLM_MODEL)` —
and ends. This keeps the chat UI functional and debuggable even before a backend
is wired up.

```sh
AKURAI_LLM_URL=http://localhost:8080 \
AKURAI_LLM_MODEL=gemma-3-12b \
akurai serve
```

## No TLS — point at a plain-HTTP origin

The `akurai-llm` client is built on `std::net::TcpStream` and speaks **plain
HTTP/1.1 only**. The Rust standard library has no TLS, and the framework will not
pull in a dependency for it. An `https://` value for `AKURAI_LLM_URL` is rejected
with a clear error event rather than silently failing.

This is by design and matches the rest of the framework: **terminate TLS at the
edge** (Caddy, nginx, a cloud load balancer) and point `AKURAI_LLM_URL` at the
plain-HTTP origin behind it — typically a local model server
(`http://localhost:8080`) or an internal address on a trusted network. The
endpoint understands both unchunked and `Transfer-Encoding: chunked` streaming
responses, which is what local OpenAI-compatible servers normally send.