Revision history for Langertha-Skeid

0.004     2026-10-24 04:04:43Z

  [Status]
    - Live backends: t/950-live-docling.t runs the document-conversion relay against a real
      docling-serve 1.37.0 (document.json_content.pages keyed "1", "2" is counted as pages) and
      t/951-live-speaches.t runs a realtime session against a real speaches node, with the
      recorded frames in t/data/realtime/speaches.jsonl (flat session.update, usage as a
      duration, no delta events). Both skip without SKEID_LIVE_*_URL; xt/live/skeid-live
      starts and watches the Python services (Perl for the management, Python for the AI
      work) and xt/live/record-realtime records a session. Still open: vLLM's realtime
      dialect and usage block, GPU nodes, the cost of decoding realtime frames and parsing
      pages (not measured).

    - Tests: a scenario suite for the core features against Langertha::Virtual::Server
      nodes (t/200-core-scenarios.t, t/scenarios/*.yml, t/lib/Test/Skeid/Virtual.pm):
      chat in the three faces, routing, weighted round-robin, aliases and tiers, key
      policy, a 429 node backed off, error pass-through, client_auth, the admin API,
      draining a node, the registry snapshot, usage and cost metered from the node's
      usage block and from streams in all three faces (streamed reads in the helper). It skips as a whole
      when the installed Langertha has no Engine::Virtual with its HTTP endpoint.

    - Document conversion through docling-serve: POST /v1/convert/file and
      POST /v1/convert/source under docling's own paths, relayed (multipart
      part for part from the spooled upload, a source body decoded once and
      written again; the JSON or zip answer byte for byte), the synchronous
      calls only. A request has no model: nodes get an option kind
      (documents; default chat, unknown value fails the load) and the route
      is planned over nodes of that kind under the routing key documents,
      which a key policy's models grants (403 without it) and an alias of
      that name tiers; chat routes, /v1/models and the manifest never see a
      documents node and the reverse. New node option api_key_header
      (RFC 7230 token, checked before the old node is removed): the node's key
      goes raw in that header, X-Api-Key for docling, the client's credentials
      still dropped. http_sources, callbacks and a target that is not
      inbody/zip are 400 before admission (the field's name, never its value;
      all spellings, a duplicate key) unless documents.allow_remote_sources.
      uploads.documents_max_bytes (default 100 MiB) is the limit of both
      routes, 413/401 before the body. New nullable usage field pages, read
      off document.json_content.pages of a JSON answer up to 8 MiB (absent
      otherwise, never 0): column in both schemas and by the additive
      migration, jsonlog, the spool, totals/by_key/by_model, skeid usage.
      documents counts the request's input documents. Relay hook
      max_parse_bytes leaves a larger answer undecoded (k94, ADR 0028,
      ADR 0021 Update).
    - GET /v1/realtime: live transcription over the OpenAI realtime protocol, a
      WebSocket relayed to the node's /realtime in the node's own events, both
      client dialects (session.update and transcription_session.update) as
      sent. A relayed route that is a session: routed, admitted and
      authenticated like a request at the handshake, the node opened before the
      client is upgraded (a refusing node is a 502, with no slot left), one
      slot held until the session closes (max_conns of a realtime node means
      concurrent sessions), one usage event when it closes (length, summed
      usage, how it ended). model in the query is required. A browser's key may
      come in the openai-insecure-api-key.<key> subprotocol, behind the same
      client gate, never echoed, logged or forwarded. Every client frame is
      decoded: a model other than the session's is refused with an error event
      and not forwarded; a non-object or binary frame closes 1003. New realtime:
      section (idle_timeout, max_session_seconds, max_message_bytes,
      max_queued_bytes; hot-reloaded); a graceful stop closes sessions 1001
      first. Not checked against a deployed vLLM or speaches node: the test
      fixtures are hand-written (ADR 0029).

    - MCP gateway, review follow-up: the server's response headers are an allowlist
      (Content-Type, Cache-Control, Retry-After, Mcp-*), no longer everything
      but hop-by-hop ones; an event stream is no longer cut at the user agent's
      16 MiB message limit; a stream cut by the request timeout is recorded
      ok 0 with error_type upstream_timeout instead of a clean 200; the
      request body has its own limit, mcp.max_request_bytes (default 10 MiB,
      413 over it). The client query going upstream as it came and the absent
      OPTIONS/HEAD/CORS are documented (k103, ADR 0022).

    - GET /openapi.json: an OpenAPI 3.1 description of the client routes the
      configuration enables (manifest and MCP routes only with their section),
      with the client_auth bearer scheme and example model names from the same
      list as /v1/models; behind client_auth like the other client routes. The
      admin routes are a separate document, GET /skeid/openapi.json, behind
      the admin key. Routes are now registered from one route table
      (Langertha::Skeid::Proxy->route_table) that also builds the documents,
      and a test fails for a route without an entry (k105, ADR 0025).
    - Optional Swagger UI at /docs, off by default (openapi.swagger_ui,
      openapi.cache_dir). Not shipped: the first request fetches the pinned
      swagger-ui-dist 5.17.14 non-blocking, verifies it against compiled-in
      SHA-256 digests, caches it and serves it locally; a failed fetch is a
      502, then 503 for a minute, never a retry storm (k105, ADR 0025).

    - /v1/messages: the tool-result images and the client's own text of one
      Anthropic message go upstream as one user message instead of two in a
      row, which a strict-alternation chat template may refuse. Nothing is
      merged across Anthropic messages; earlier turns are unchanged (k102).

    - /v1/messages: an empty top-level `system: ""` is dropped like an empty
      system block array, instead of going upstream as an empty system
      message that a strict chat template may refuse (k101).

    - request.finish has a fourth outcome, `rejected`, for a request the
      client got wrong (upstream 400, 404, 413, 422, or a request refused
      after admission as unsendable). It no longer counts as a node error:
      not in the error counter, errors_in_window or last_failure_at. 401,
      403, 429 and 5xx stay node errors. The registry snapshot carries the
      count in a new node field, rejected_in_window; version stays 1 and the
      existing fields keep their meaning (ADR 0024).
    - The request goes upstream as the client spelled it. /v1/chat/completions
      and /v1/embeddings forward the client's bytes with only model replaced,
      instead of decoding the body and encoding it again -- which wrote every
      object's keys sorted. A node renders tool definitions into the prompt in
      the order they arrive, so the model got a prompt the client did not
      write: a JSON schema as properties, required, type instead of type,
      properties, required. One node answered such a prompt in gibberish. The
      Anthropic and Ollama faces translate the body and so cannot forward it;
      they carry each tool's JSON schema over as the client wrote it, in a
      tool definition spelled in wire order (ADR 0023).
    - A body that names model twice has both replaced by the served model, so
      no node reads a model the request was not routed for.
    - The usage report lists by_key and by_model rows of equal cost in a
      stable order (key id resp. model name, ascending) instead of Perl hash
      order; the sqlite/postgresql ORDER BY gained the same final key.
    - A node's hop-by-hop response headers (Connection, Keep-Alive, ...) are
      no longer relayed to the client on the JSON and stream routes; a node's
      `Connection: close` made Skeid drop the customer's keep-alive
      connection. One filter now serves the JSON, stream and MCP relays.
    - add_node validates a node's engine before removing the node it
      replaces, so a re-add with an unknown engine (POST /skeid/nodes,
      nodes.add) no longer drops the working node from the inventory.
    - POST /v1/audio/transcriptions and /v1/audio/translations on the OpenAI
      face, for Whisper nodes (vLLM, speaches). The multipart form is routed by
      its model field like a chat request (aliases, tiers, key policy,
      max_conns, health, client_auth) and relayed to the node's own
      /audio/... endpoint part for part; only model is rewritten when an alias
      tier serves another one. The upload is sent from the spooled file, never
      copied in memory. The answer comes back as the node gave it: json,
      verbose_json, text, srt, vtt, or with stream=true the node's event
      stream byte for byte, in whichever dialect it speaks. Nothing is
      translated (ADR 0021).
    - uploads: { max_bytes: N } limits the body of the upload routes (default
      26214400, 25 MiB; hot reloaded). Over it: 413, no node touched, no usage
      event. The limit holds while the body arrives, and with client_auth a
      caller that is not on the list gets its 401 before the upload is
      received. Every other route keeps the server's own request limit.
    - Usage events of the audio routes carry audio_seconds when the node
      reports a duration (usage.seconds of a usage.type duration block, else
      the duration of a verbose_json answer) and no such key when it does not.
      Tokens are recorded when the node reports them. jsonlog writes the field,
      the DBI stores get a nullable audio_seconds column (added to an existing
      table on start), reports sum it in totals, by_key and by_model, and
      skeid usage prints it where there is some. Seconds are recorded, not
      priced.
    - POST /v1/rerank and its alias POST /rerank on the OpenAI face, for
      reranker nodes. The JSON body (model, query, documents, top_n,
      return_documents, ...) is routed by model like a chat request (aliases,
      tiers, key policy, max_conns, health, client_auth) and relayed to
      {node url}/rerank with only model replaced; the answer comes back as the
      node gave it. That covers vLLM, infinity (started with --url-prefix
      /v1), Jina and Cohere, whose differences are not evened out. A body that
      is no JSON object, or has no model, no string query or no documents, is
      400 before a node is touched.
    - Node option rerank_format: tei for Hugging Face text-embeddings-inference:
      the request is translated to POST /rerank at the server root (query,
      texts, return_text, truncate) and TEI's bare array back into results
      sorted by score, cut to top_n, with document { text } when asked for and
      usage.total_tokens from the x-compute-tokens header. A document that is
      not text is 400. An unknown rerank_format fails the config load.
    - Usage events of the rerank route carry documents, the number of documents
      the request held, when the node answered. The node's tokens are recorded
      as input tokens, priced by input_per_million, from usage.prompt_tokens,
      usage.total_tokens, meta.tokens.input_tokens (Cohere) or x-compute-tokens
      (TEI). jsonlog writes documents, the DBI stores get a nullable documents
      column (added to an existing table on start), reports sum it in totals,
      by_key and by_model, and skeid usage prints it where there is some.
      Documents are recorded, not priced.
    - An optional client_auth: { keys: [...] } section turns the client routes
      into an allowlist. Each entry is a names: entry or a key id (k_...),
      never a key. With it, a missing key or one that is not listed gets 401
      with WWW-Authenticate: Bearer realm="skeid" in the dialect of the route
      (OpenAI invalid_api_key, Anthropic authentication_error, Ollama error
      string) on every /v1/*, /api/* route and /.well-known/langertha.json,
      before the body is read or any node is touched. /health stays open;
      the /skeid/* admin routes keep their own bearer authentication and are
      not gated by it. Without the section nothing is checked. The section is hot
      reloaded; keys: [] lets nobody in. GET /v1/models, /api/tags, /api/ps and
      the manifest now pick up a changed config themselves.
    - The release builds, tags and pushes the Docker image through
      Dist::Zilla::Plugin::Docker::API (docker_image in dist.ini) over the
      container engine's HTTP API instead of shelling out to docker build /
      docker push, so a release works against rootless Podman.
      SKEID_DOCKER_BUILD_ARGS is gone.
    - The GitHub release is published by the bundle's GitHub::CreateRelease
      after the tag is pushed, with the Changes section as release notes. The
      gh calls in run_after_release are gone: they ran before the tag existed,
      so the release tag landed on the commit before the release commit.
    - A system message inside messages on /v1/messages (Claude Code sends one
      after the conversation started) no longer reaches the node in the middle
      of the conversation, where chat templates such as vLLM's Qwen3.x answer
      400 "System message must be at the beginning.". The node gets at most one
      system message, at index 0: top-level system plus the system messages
      before the first turn. A later one is wrapped in <system-reminder> and
      becomes text of the adjacent user turn, never between an assistant's
      tool_calls and their tool messages, so the prompt prefix stays stable
      for prefix caching. Requests without such a message are translated as
      before.
    - MCP gateway: an optional mcp: section assigns MCP servers to customer key
      ids (servers: { name: { url, description, api_key_ref, api_key_env,
      headers, keys: [...] } }). POST, GET and DELETE /mcp/<name>, and paths
      below it, are relayed to the server's url for a key the server is
      assigned to: the body as it came, Content-Type, Accept, Last-Event-ID
      and the Mcp-* headers up; status, headers and body down, an event stream
      unbuffered and opened as soon as the server's headers arrive. Nothing of
      MCP is read and no tool lists are merged (ADR 0022). The client's
      Authorization never goes upstream; the server gets its own key as
      Authorization: Bearer from the key broker or a variable, like a node,
      and is not called when a named key source yields none (503). A name
      that is unknown, assigned to another key, a stdio entry or followed by a
      path with a .. segment is one and the same 404. A server's 401 and a
      redirect from it are answered 502 and logged. No key is 401; with
      client_auth its list applies first.
    - GET /v1/mcp/servers lists the MCP servers assigned to the presented key:
      name, transport, description and the public url (mcp.public_url, else
      the request's) of a relayed server, command, args and env of a
      transport: stdio entry, which is a hint for a server on the customer's
      machine and is never proxied. Upstream URLs, key references and static
      headers are not shown.
    - Each relayed MCP request writes a usage event with api_format mcp and
      endpoint /mcp/<name>: customer key id, status, duration, the server's
      url as route_url; no model, tokens or cost. No store schema change.
      Without the mcp: section both routes are unknown paths, as before.
    - A stream relay now meters and translates upstreams that write
      `data:{...}` without a space; before, such a stream was billed as zero
      tokens and answered empty on the Anthropic and Ollama faces.
    - Fix: a client's Expect: 100-continue (curl sends it above 1 MiB) is no
      longer forwarded to the node on any route. A node answering it with
      100 Continue could leave a streamed chat answer empty while the usage
      event said ok.
    - customer_source: reads customer key ids (key_id, name, policy, models)
      from a PostgreSQL table or view through DBD::Pg. The connect and the
      query run in a child process, so a database that does not answer never
      blocks the event loop; a second child LISTENs. The rows are fetched in
      the background every refresh_interval_s and sooner
      on LISTEN/NOTIFY, and merged into the file's config at reload. The ids
      join client_auth; the file wins per key id. Bad rows are rejected and
      reported by name or position only, never by content. When the database
      is unreachable the last good set is kept (max_stale_s bounds how long);
      a failed fetch at startup is degraded and the start fails closed.
      Status is on /skeid/config and /health (ADR 0026). GET /skeid/config
      carries customer_names and customer_report. Under --workers N each
      worker fetches (and opens its LISTEN connection) at start. t/90 checks
      the source against a real PostgreSQL when SKEID_TEST_PG_* is set.
    - usage_store: { spool_dir: DIR } (sqlite and postgresql; off by default)
      keeps the usage events the database cannot take while it is
      unreachable: they are appended to a JSON Lines file in DIR (fsync'd,
      one file per process, the event's columns only -- key ids, never a key
      or the database password) instead of being reported lost, and replayed
      after the next successful write and at start, 500 per batch, without a
      second row for an event already in the table (request_id and
      created_at). A store with a spool also starts while the database is
      down and migrates on its first connect. An event the database refuses
      is still reported lost (ADR 0027).
    - A failed connect to the usage database is no longer repeated for every
      event or flush: the next attempt waits 1s, doubling up to 60s, and
      writes in between fail (or spool) at once without touching the network.
    - The usage schema gains a non-unique index on usage_events(request_id),
      created on start with auto_migrate.

0.003     2026-10-01 21:30:07Z

    - Docker image stops on SIGQUIT so write-behind usage events are flushed;
      serve handles SIGQUIT in single-process mode; the image build fails
      without Cpanel::JSON::XS
    - A SQLite or PostgreSQL usage store can write behind the request:
      usage_store.flush_interval_ms queues each event and writes the queue in
      one transaction per interval, so the answer no longer waits for the
      database. Off by default (0 = write each event at once). Queued events
      are written on reload, shutdown, before a report and by flush_usage; a
      process killed without a flush loses them. An event a flush cannot write
      is logged as "usage event lost", as before
    - GET /v1/models and GET /api/tags list the alias names as well as the
      node models, one entry per name (an alias named like a model is listed
      once), and follow the policy of the key presented -- no key means the
      default policy. A name the key's models do not grant, an alias whose
      tiers are all denied and a model only denied nodes serve are left out.
      Both faces build the list from one Langertha::Skeid method, list_models
    - A client that hangs up is counted apart from node errors: the slot is
      given back as a third outcome, aborted, which shows in the node metrics
      as "aborted" and does not raise "error", the registry snapshot's
      errors_in_window or last_failure_at. The usage event is ok 0,
      status_code 499, error_type client_abort
    - An Ollama request (/api/chat, /api/generate) that the translator cannot read
      is answered with a 400 in Ollama's error shape, before anything is routed
      or metered, instead of an HTML 500. The exception's text is neither sent
      nor logged
    - An Anthropic request (/v1/messages) whose translation fails for a reason
      of the translator's own is answered with the fixed text "Invalid request"
      instead of the exception's text, which can quote the request. A deliberate
      refusal (a provider built-in tool, an unsupported image source) keeps its
      worded message
    - Every answer carries an x-request-id header: on all three faces, on
      refusals, upstream errors and translator failures, and on streams before
      the first byte. The id is fixed when the request arrives -- the client's
      own x-request-id if it sent one, else a generated one -- and it is the
      request_id of the usage event and of the "usage event lost" log line. An
      x-request-id from the upstream never replaces it on the relayed answer
    - A translator that dies on the Anthropic or Ollama face is answered in
      that face's own error shape: an HTTP 500 while nothing was sent, an
      error event or line once the stream is open. The upstream is cancelled,
      the slot is given back, and one usage event is written as failed with
      status_code 500 and error_type translation_error. The exception's text
      is neither logged nor sent to the client
    - A client that hangs up before its answer is complete now ends its
      request. It used to go unnoticed: a request waiting for capacity kept
      polling, then took a slot it never gave back; a request already at a
      node kept the node generating and its slot taken until the node was
      done, and then lost its usage event; on the Anthropic and Ollama
      streaming faces the slot was never given back at all, and an abandoned
      stream kept its controller and everything still queued for the client
      in memory. Now the wait stops without taking a slot, the upstream
      connection is closed so the node stops generating, the slot is given
      back exactly once, and one usage event is written as failed with
      status_code 499 and error_type client_abort, priced from the usage
      the stream had reported until then. A client that leaves while the
      node key is still being resolved never reaches the node and records
      no usage event, also when that node's key turns out to be
      unavailable
    - Complete POD for every module and the skeid command, including a full
      configuration and environment reference in Langertha::Skeid and a
      route/auth/error reference in Langertha::Skeid::Proxy.
    - examples/README.md documents every example script with its options
      and environment, and carries the Avatar, one-box and Vast lab recipes
    - Docker images no longer take a KNARR_SRC build argument; Skeid does not
      depend on Langertha::Knarr. LANGERTHA_SRC still installs a specific
      Langertha before the cpanfile
    - Ollama streaming (/api/chat) now carries tool calls: fragmented
      arguments, parallel calls and Unicode are accumulated through
      Langertha's stream parser and rendered as message.tool_calls on the
      closing line, matching what non-streamed /api/chat already returned.
      A stream that ends without completing a pending tool call, or whose
      translator fails while finalizing, closes with a safe Ollama error
      line instead of a false done:true or invented empty arguments. A
      translated stream is now finalized before its Usage event and
      admission are closed, so a translator error discovered only at the
      end of the stream is billed and recorded as failed, not as ok
    - Routing at saturation checks each node at most once per selection
      instead of once per unit of weight: two fully busy nodes weighted
      1000:1000 used to cost 3000 admission checks to fail over, now 2.
      Normal weighted round-robin sequencing, cursor progress across a
      wrap, and partial saturation are unchanged
    - The routing cache and its round-robin cursors are bounded at 256
      entries per inventory generation, FIFO-evicted together, so a stream
      of client-chosen model names that never resolve to a node can no
      longer grow either one without limit. Any real inventory change
      (a node added, removed, moved or its health flipped) still clears
      every cursor and restarts its fairness, but calling set_node_health
      with a node's current value is now a no-op and no longer does
    - A config file that fails to parse is retried with the same bounded
      back-off as a failing config_loader (1s doubling to 60s) instead of
      being re-parsed on every dispatch. The last good config stays in
      force; a different mtime, including one that changed again while a
      failed version was being retried, is always read immediately
    - An upstream HTTP error (a 429 with Retry-After, or any other 4xx/5xx)
      is now observed for capacity on every face -- OpenAI JSON, and the
      Anthropic/Ollama streaming paths -- before the error is returned, not
      only on success. A rate-limited node backs off instead of receiving
      the very next request immediately
    - Fix jsonlog event ids colliding under load: a dir-mode write is created
      exclusively (O_EXCL), retried up to 16 times, and the id is built from
      timestamp, pid, a per-process random nonce and a sequence, all
      refreshed after a fork, so two events sharing a second and a pid never
      overwrite each other. A write, fsync, close or lock failure is now
      reported as failed instead of a silently truncated or lost event
    - Fix a request-lifecycle leak: the recursive routing callback, and on a
      streamed response the drain callback and the completed upstream read
      listener, kept the finished controller (and its upstream connection)
      alive after the response had ended. Each callback now clears its own
      self-reference once its work is done, so a completed request releases
      its controller instead of accumulating until the process is restarted
    - The admin API key is checked in constant time, like the registry
      read key and the snapshot signature; all three share one comparison
      (Langertha::Skeid::Secret)
    - Optional Skeid-to-Skeid registry. A downstream with registry.enabled
      publishes a signed snapshot of its nodes (inflight, max_conns,
      health, recent errors, current capacity reading; never URLs, key
      references, customer key ids or usage) on the route
      GET /skeid/registry/snapshot, HMAC-SHA256 with the secret named by
      registry.secret_env. The route takes the registry read key named
      by registry.read_key_env, which opens no other route, or the admin
      API key. A fronting Skeid reads it with capacity probe registry
      (read_key_env or admin_key_env, secret_env, optional url, tags,
      interval_ms, max_skew_s), preferring the read key and never
      falling back to the admin key, and forgets the reading on a bad
      signature, a stale, future or replayed snapshot, or a missing
      secret. Off by default; the route answers 404 until enabled.
      Enabling it needs a read key or an admin API key and a secret of
      at least 32 bytes. Serve the route over TLS only
    - When two sources report capacity for one node, the tighter reading
      wins while it is current: while it carries a pending backoff, or
      is younger than the longer of the two sources' poll intervals. A stale
      rate-limit reading from the last response no longer keeps a probe
      that reports the node empty out. A probe that fails or stops
      forgets only its own reading, so a 429 backoff survives it. A
      config reload that drops a node drops its reading too
    - A capacity probe whose interval_ms (times the worker count) is not
      below capacity_max_age_ms warns at start
    - Errors on the Ollama routes (/api/chat, /api/generate, /api/tags,
      /api/ps) are answered in Ollama's shape, {"error": "<message>"} with
      the failure's HTTP status, instead of the OpenAI error object an
      Ollama client cannot decode: invalid body, key policy, no capacity,
      no node, and upstream errors with the upstream's own message. A
      stream that fails after it opened ends with an Ollama error line and
      no done:true line
    - Ollama format reaches the model on /api/chat and /api/generate:
      "json" is sent upstream as response_format json_object, a JSON schema
      as response_format json_schema (named ollama_format, no strict). The
      Ollama face of the provider manifest publishes
      response_format_json_object and response_format_json_schema
    - A non-streamed Ollama /api/chat answer sends done as the JSON
      boolean true, as Ollama does, instead of the number 1 that typed
      clients (Go, Rust serde, pydantic) reject
    - Ollama POST /api/generate is served, with the same key policy,
      routing, usage event and pricing as /api/chat. system and prompt,
      with the request's images, become one chat conversation upstream;
      the answer comes back in generate's shape (response, done,
      done_reason, prompt_eval_count, eval_count), streamed as NDJSON
      unless stream is false. think, suffix, template, raw, context and
      keep_alive are not forwarded, and no context is returned
    - Images inside an Anthropic tool_result reach the model. An OpenAI
      tool message cannot carry them, so the tool message keeps the
      result's other blocks and one user message after the tool messages
      carries every result's images, each labelled "Images from tool
      result <tool_use_id>:". A Files API image there is answered with 400
    - Images reach the model through the Anthropic and Ollama faces. An
      Anthropic image block (base64 or url source) and an Ollama message's
      images become OpenAI image_url parts, text and images in the order
      the client sent them; Ollama's raw base64 gets a data URL typed from
      its magic bytes (PNG, JPEG, GIF, WebP, else PNG). An Anthropic image
      from the Files API is answered with 400. The provider manifest may
      claim image_input for a model on every face, when the installed
      Langertha's manifest knows the flag
    - Streamed requests are priced: their usage event stores the same
      tokens and costs as the same usage answered in one piece, on the
      OpenAI, Anthropic and Ollama faces. A stream cut after its usage frame
      is billed from it and still recorded as failed
    - A model's pricing rule may set cached_input_per_million and
      cache_write_per_million. A request's usage event then prices
      prompt-cache reads and writes at those rates into the new
      cost_cache_read_usd and cost_cache_write_usd, both part of
      cost_total_usd, with cost_input_usd covering only the uncached input;
      OpenAI-, Anthropic- and /anthropic-shim-shaped usage each price every
      token once. A rule without them bills as before. Every usage event
      also records the prompt-cache write count as cache_write_tokens. The
      DBI stores add the three fields as nullable columns on the existing
      table. A rate that is not a
      number >= 0 fails the config load. Needs a Langertha newer than
      0.503; an older one ignores the two keys with a one-time warning
    - Taking a node in or out of rotation through the admin API no longer
      restarts every capacity probe and drops their readings. Probes restart
      only when a probed node is added, removed, moved to another URL or
      given a different capacity block, or the worker count changes; a probe
      that is stopped ignores the answer to a poll still in flight
    - Customer key ids (skeid keyid) are the key's full SHA-1 digest, k_ plus
      40 hex digits, instead of its first 12 hex digits, which are a prefix of
      the new id. A config that still names a customer by the short id in
      keys: or names: keeps routing, billing and serving the manifest to that
      key, with a one-time deprecation warning; listing both the short id and
      the full id it prefixes is a load error. Usage events keep the id they
      were recorded under, so a report by api_key_id splits at the upgrade:
      query both ids for one customer's full history
    - A config_loader runs at most once per config_reload_interval (default
      1s, env SKEID_CONFIG_RELOAD_INTERVAL) instead of on every dispatch, and
      may return ($config, $version). A load whose version, or else canonical
      digest, matches the last applied config changes nothing; an unchanged
      nodes section keeps the node list, so capacity probes are not restarted
      and health set through the admin API survives. A config file touched
      without a change is likewise a no-op
    - GET /.well-known/langertha.json serves a provider manifest per customer
      key: only the models that key's keys: entry lists under manifest.models,
      on the openai, anthropic (anthropic-compat) and ollama faces under
      manifest.public_url, each claiming only the capabilities that face
      carries upstream, never a node URL or upstream key. Off until
      manifest.enabled; 401 without a key, 403 for a key without a grant, 404
      when disabled or on a Langertha without Langertha::Manifest
    - A config reload that fails leaves the previous config fully in force
      instead of half-applied, and no longer fails the request that
      triggered it: the request is served under the kept config, the failure
      is logged, and a failing config_loader is retried with a back-off (up
      to a minute) without reapplying the same broken result. GET /health
      shows config_reload ok and failed_at; the admin route GET /skeid/config
      (and the config.status function) also gives the error
    - A streamed request's usage event carries content_bytes, the UTF-8
      byte count of the content relayed or translated (non-ASCII included),
      on every face; it is optional (absent on non-streamed events), never
      turned into a token estimate, and stored in a new nullable
      content_bytes column added to existing sqlite/postgresql tables
    - An Ollama client's replayed tool round-trip reaches the OpenAI upstream
      in OpenAI's shape on /api/chat: tool_calls arguments objects become JSON
      strings (non-ASCII intact), calls without an id get call_skeid_N, and
      each tool message is tied to its call by tool_call_id (matched by
      tool_name, else in order)
    - Non-ASCII text in tool_use input and structured tool_result content
      reaches the upstream intact on /v1/messages instead of double-encoded into
      mojibake
    - A streamed text block now opens with data type content_block_start,
      as Anthropic specifies, so Anthropic SDKs that dispatch on the data
      type see the text block begin
    - The Anthropic face reports stop_reason tool_use when the upstream
      reply carries tool calls but finishes with stop (gpt-oss on
      vLLM-style servers, e.g. AKI.IO), on /v1/messages and in the
      streamed message_delta alike, so Anthropic clients run the tools
    - Relay a streaming upstream that answers with exactly
      Content-Type: text/event-stream (no charset) and an unchunked body
      (Content-Length or close-delimited). Mojolicious parsed such a body
      itself, so the client got an empty stream and the request was metered
      as served with no tokens
    - Answer every error on /v1/messages in Anthropic's shape,
      {type: "error", error: { type, message } }, so Anthropic SDKs parse it and
      raise the matching exception: invalid JSON, translation failures, 403,
      429, 503 and upstream errors alike. error.type follows the HTTP status
      per Anthropic's documented error reference (400 invalid_request_error,
      401 authentication_error, 402 billing_error, 403 permission_error,
      404 not_found_error, 409 conflict_error, 413 request_too_large,
      429 rate_limit_error, 500 api_error, 504 timeout_error,
      529 overloaded_error; checked against the docs, not live). Statuses it
      does not list, such as Skeid's own 502 and 503, fall back to api_error
      (5xx) or invalid_request_error (4xx). A streamed request the upstream
      refuses gets that HTTP error before any event is sent; a stream that
      fails after it opened (dropped upstream connection, body shorter than
      its framing, upstream error chunk) ends with an `event: error` frame and
      no message_stop. An error chunk keeps the upstream's error.type when
      Anthropic uses the same one, else api_error. The OpenAI and Ollama
      faces keep their error shape
    - Upstream error responses carry the upstream's own error message
      ("Upstream error: context too long") instead of the HTTP reason phrase,
      on every face
    - Meter a stream whose upstream hangs up before the end of its chunked or
      Content-Length body as failed (ok = 0, and a failed request for the
      node) on every face
    - Answer an Anthropic request that carries a provider built-in tool
      (web_search_20250305, bash_*, text_editor_*, computer_*, mcp_toolset, ...)
      with a JSON 400 invalid_request_error naming the tool type and its
      category, instead of an HTML 500. Skeid forwards function tools only.
      Any other failure to translate a /v1/messages body is a JSON 400 too.
      Needs a Langertha with Langertha::Tool->classify
    - Record cached_tokens (the prompt-cache read count) on every usage event.
      It is read off the upstream usage payload (OpenAI's
      prompt_tokens_details.cached_tokens, plus a flat cached_tokens fallback) on
      both the streaming and non-streaming paths, stored in a new nullable
      cached_tokens column added additively to the sqlite and postgresql schemas
      (old tables migrate on prepare, like requested_model), and surfaced in the
      usage report, GET /skeid/usage and bin/skeid usage. Recording only: cost is
      still priced with no cache discount, so cached tokens currently bill at the
      normal input rate -- the pricing correction waits on Langertha::Pricing
      modelling a cache-discount rate (ADR 0013)
    - Partition max_conns across multiple Skeid frontends: routing.frontend_count
      (or SKEID_FRONTEND_COUNT) divides a node's max_conns among the separate
      Skeid hosts in front of it, the way worker_count divides it among prefork
      workers. The two compose -- a process admits max_conns/(frontend_count *
      worker_count) -- and a max_conns that cannot be split cleanly warns at
      startup. Default 1 leaves single-frontend deployments unchanged. It does
      not scale probe or vault timers: separate frontends each hold their own
      (ADR 0012)
    - Stream Anthropic tool_use in SSE: when the upstream OpenAI stream emits
      tool_calls, each one becomes its own content_block_start(type=tool_use),
      every arguments chunk becomes an input_json_delta with partial_json, and
      a content_block_stop closes it before message_delta. Parallel tool calls
      get distinct Anthropic indices, allocated in order of first appearance.
      content_block_start is now lazy: it fires on the first delta that fills
      the block, not on message_start, so a stream whose first content is a
      tool call no longer emits an empty text block before it
    - Add t/33-agent-flow.t, a Claude-Code-style scenario test: three
      sequential streamed turns at /v1/messages on a shared messages array,
      with a synthetic prior tool_use and tool_result, plus a standalone
      tool_use turn. Exercises translation, streaming, the usage accumulator
      and admission control together -- the path a real agent client walks
    - Fix SSE streaming, which was broken end to end: the relay finished the
      response on its first chunk, so clients received correct headers, a 200,
      and an empty body. Chunks are now queued and drained through write_chunk
      with a drain callback, and empty chunks are never relayed
    - Fix usage accounting for streamed responses: SSE frames split across read
      boundaries were dropped, which most often lost the final frame carrying
      the token counts
    - Support admin_api_key_env / admin.api_key_env, consistent with
      usage_store's password_env. The shipped service stack used api_key_env
      and silently ran with the admin API disabled
    - Split protocol translation into Langertha::Skeid::Protocol::Anthropic
      and ::Ollama, and the usage store into Langertha::Skeid::UsageStore
      with ::JsonLog and ::DBI backends
    - Fix jsonlog usage store ignoring a reconfigured path
    - Declare File::ShareDir, namespace::clean and HTTP::Tiny in cpanfile;
      all three were used at runtime but undeclared
    - Stop forwarding hop-by-hop headers upstream (RFC 7230). A client sending
      "Connection: close" made Skeid close its own upstream connection, so the
      connection pool never held anything
    - Raise the upstream connection pool from Mojo::UserAgent's default of 5 to
      100, overridable with SKEID_UPSTREAM_POOL
    - Remove the unused synchronous twins of the async request path
    - Nodes carry tags, and routing can select by them (select_nodes,
      nodes.select, and a tags argument on pick_node / route_state). Tags are
      the grouping the per-key routing policy of ADR 0008 is built on
    - Cache the node lists routing derives from the inventory, invalidated by
      any change to it
    - Model aliases: a client-facing model name resolves to an ordered list of
      tiers, each selecting nodes by tag, naming the model to ask them for, and
      carrying its own wait window. A saturated tier falls through to the next;
      a tier with no eligible node is skipped without waiting. An exhausted
      plan is 503 when nothing was ever eligible and 429 when everything was
      busy. A model without an alias routes exactly as before
    - Usage events record requested_model alongside model, so an alias cannot
      silently lose which product a request was billed for.
      Langertha::Skeid::UsageStore::DBI adds the column to a pre-existing table
    - Per-key routing policy: a customer key decides which models it may ask
      for and which node tags it may not be served from. Named profiles with a
      default_policy and sparse per-key overrides, resolved once at config load
      into shared immutable objects, so ten thousand identically-configured
      customers cost no key entries and a request costs one hash lookup. A
      refusal is 403, never a capacity code; running out of permitted capacity
      stays 429 rather than falling through to a denied node
    - deny_tags filters node selection, not just the routing plan. Denying the
      cloud tier of an alias was otherwise worthless: the same node still
      answered to its own model name
    - SECURITY: stop taking the customer key id from the client's
      x-skeid-key-id / x-api-key-id header. It selects the routing policy and
      the invoice, and any client could set it to another customer's. The id is
      now derived from the presented API key; the header is honoured only under
      routing.trust_key_id_header, for deployments that authenticate callers in
      front of Skeid. Deployments relying on the header must either set that
      option or move to derived ids
    - Add "skeid keyid", which prints the key id a customer key resolves to --
      the name a keys: entry uses, so the config never holds a customer key
    - Declare Digest::SHA in cpanfile; it was used at runtime but undeclared
    - Key resolution no longer blocks the event loop. Langertha::Skeid::KeyBroker
      gains key_async (the request path's only entry point), an in-memory TTL
      cache with a short negative cache, and coalescing of concurrent misses for
      one reference into a single resolution. A broker that only implements the
      blocking resolve_key keeps working; ::OpenBao resolves non-blocking via
      Mojo::UserAgent and renews its token on a timer rather than when a request
      discovers it expired. Verified with a test that the process still answers
      other requests while a resolution is outstanding
    - SECURITY: KeyBroker::OpenBao verifies the OpenBao TLS certificate. It
      hardcoded verify_ssl => 0, so anything able to intercept the connection
      could hand out the AppRole token and every secret resolved with it. Set
      OPENBAO_VERIFY_SSL=0 for a dev vault with a self-signed certificate
    - KeyBroker::OpenBao no longer puts a vault response body in a warning, and
      no longer leaks its token by keeping the renewal timer alive after the
      broker is gone
    - Node capacity is probed, not only counted (ADR 0009). inflight counts
      what this process sent, which undercounts the moment a second frontend, a
      prefork worker or a batch job shares the node -- each counter sees its own
      share and together they over-admit. Nodes can now carry a capacity block
      selecting a probe: ratelimit (reads x-ratelimit-* / anthropic-ratelimit-* /
      Retry-After off responses Skeid already has, so it costs nothing),
      prometheus (polls vLLM/SGLang/TGI metrics on a timer), or custom (a
      callback or a class, because Skeid fronts whatever an operator runs).
      Absent means inflight, so an existing config routes exactly as before
    - A capacity reading may only narrow what max_conns allows, never widen it,
      so a stale or broken probe cannot become an overload -- and for a rented
      node max_conns keeps its meaning as a spend limit
    - skeid serve --workers N runs Mojo::Server::Prefork. Measured at 170.7
      req/s with 4 workers against 128.1 with one, and TTFT p50 falls from
      120ms to 88ms and stops growing with concurrency (334 req/s at c=32).
      Only ~1.2ms per request is Skeid's own logic, so more loops was the only
      lever left
    - max_conns is divided among the workers (ADR 0010). inflight is
      per-process, so without this N workers each admit up to max_conns and a
      node configured for 8 sees 32. A max_conns below the worker count cannot
      be honoured and now warns at startup instead of quietly over-admitting
    - Capacity probe intervals are multiplied by the worker count, so the rate
      a node's metrics endpoint sees from the process group stays what was
      configured rather than N times it
    - Note for prefork deployments: the usage store becomes a multi-writer
      store. jsonlog in directory mode and postgresql are fine; SQLite is not
      recommended above one worker
    - The ratelimit probe reads request and token quotas separately and lets
      the tightest one decide. For an LLM API the token budget is usually what
      runs out first, and a node with requests to spare and no tokens left
      answers 429 all the same
    - Readings expire after capacity_max_age_ms (5s) and every failure reports
      nothing rather than something old, degrading to inflight. A provider's 429
      becomes a backoff that outlives the age limit and never touches the
      healthy flag: rate-limited is busy, not broken
    - Add bench/: a C fake-LLM server with configurable TTFT and token rate,
      plus a measuring client for TTFT and throughput distributions
    - The examples/service compose stack works as written: init-skeid.sh runs
      in the OpenBao image with the bao CLI (no curl or psql needed), grants
      the AppRole policy on the KV v2 paths so key reads are no longer denied,
      stores SKEID_GROQ_KEY for the sample node and no longer writes customer
      keys nothing reads; usage_schema.sql is gone, Skeid creates the
      usage_events table from share/sql on start. The skeid image tag is
      SKEID_IMAGE (default latest), SKEID_REMOTE_KEY_REF and the SQLite-only
      SKEID_USAGE_DB are dropped from the stack, and vast-start.sh prints the
      right base URL
    - GET /api/tags lists the same models as GET /v1/models: each distinct node
      model once, and no entry for a node configured without a model (it
      matches any requested name, so none reaches it in particular)
    - A usage event the store cannot write is now logged at error level
      ("usage event lost: request_id=... store=... api_key_id=... model=...
      status=...: reason") for a thrown error and for a store's { ok => 0 }
      answer alike, so a lost billing event is visible at the default log
      level; the response still completes. The DBI store reports a failing
      insert or report query as { ok => 0, error } instead of dying, with a
      password= from its DSN masked in the error text
    - skeid serve and skeid usage stop with "ERROR: config file not found:
      PATH" and exit code 2 when --config names a file that does not exist,
      instead of starting with no config and no nodes; the library croaks the
      same way for a missing config_file at construction and on config.reload.
      Without --config, ./skeid.yaml is still used when present. A config
      file that disappears while Skeid runs keeps the config in force and is
      warned about once. The Docker image's default command passes --config
      /etc/skeid/skeid.yaml, so the image now exits when no config is mounted
      there instead of starting empty
    - skeid usage reports a jsonlog store correctly: recent event ids print as
      strings (no numeric warning, no truncated id) and the Store line names
      the log path. New --log-path (alias --jsonlog) builds a jsonlog store
      from the command line, --backend takes jsonlog, and the jsonlog report
      carries log_path
    - The admin API key now has one order that every config reload re-applies:
      an explicit key (skeid serve --admin-api-key, build_app admin_api_key,
      new admin_api_key, the new set_admin_api_key) wins over the config's key
      (any of its four spellings; empty turns the admin API off), which wins
      over SKEID_ADMIN_API_KEY; with none the admin API is off. A CLI key is
      no longer replaced by the next changed config, and SKEID_ADMIN_API_KEY
      now also works alongside a config file that names no key
    - A config reload makes the running config equal to the file, as a
      restart with it would: a section the previous config declared and the
      new one drops (nodes, pricing, aliases, the policy sections, a routing
      key) is cleared, and pricing replaces the price list instead of merging
      per model. Sections no config ever declared stay with whoever set them
      (nodes pushed through the admin API, pricing.set). A usage_store taken
      out of the file is deliberately kept until restart and warned about
      once, so a reload cannot silently stop recording billing data; a
      changed usage_store is still swapped live
    - A SQLite or PostgreSQL usage store whose connection dropped reconnects
      once and retries the event, instead of losing every later usage event
      until restart. A failed reconnect is reported as a lost event, never
      thrown, and the next event tries again
    - Security: a node that names a key of its own (api_key_ref, api_key_env)
      and gets no key from it is no longer called with the client's key. When
      the key broker failed or was not running (a failed OpenBao login at
      start), or the variable was unset or empty, the request used to go
      upstream with the customer's own Authorization / x-api-key as its
      credential. It is now refused with 503 upstream_key_unavailable in the
      shape of the face that was called, streamed or not, recorded as a
      failed usage event and logged with the key reference. The same goes
      for a node that left the inventory after it was selected. When a node
      key is injected, the client's Authorization and x-api-key are now
      dropped in any spelling: a client's X-Api-Key used to travel upstream
      beside the node's key, only x-api-key was removed. A node that names
      no key source still forwards the client's header unchanged
    - A completion that takes longer than 30 seconds, or a stream whose
      first token or whose gap between two tokens does, is no longer cut by
      Skeid itself. Mojolicious closes a connection that was silent for 30s
      (server) or 40s (user agent), while the upstream was allowed 300s. Both
      sides now follow SKEID_UPSTREAM_TIMEOUT (seconds, default 300): an
      upstream request may take that long and be silent for all of it, and
      the client's connection of a request that calls an upstream is kept
      open as long on top of the server's inactivity timeout, so the client
      is still there for the answer or for the upstream's timeout error.
      Other routes keep the server's timeout
    - The Docker image runs Skeid as the unprivileged user skeid (uid and
      gid 10001) instead of root, and no longer carries build-essential
      and libpq-dev: the Dockerfile builds in one stage and ships another.
      DBI, DBD::Pg and DBD::SQLite are installed from the cpanfile's
      recommends instead of by a second, unversioned install, so the
      sqlite usage store now works in the image too. The user owns
      /var/log/skeid/events and /var/lib/skeid in the image; a host
      directory mounted over them has to be writable for uid 10001
    - The cpanfile recommends Cpanel::JSON::XS 4.20; measured: without it
      streaming throughput drops 43%

0.003    2026-05-16 17:10:00Z

    - Add OpenBao KeyBroker integration (Langertha::Skeid::KeyBroker::OpenBao)
      for secure API key management with AppRole auth
    - Fix OpenBao AppRole token lifecycle: initial token is renew-only,
      refresh() uses renew-self to get client_token for API calls
    - Fix needs_refresh() to trigger on first call when no client_token exists
    - Add streaming token tracking via SSE chunk parsing
    - Add content byte tracking for usage estimation when provider doesn't
      return usage in streaming chunks (GROQ compatibility)
    - Add service stack Docker Compose example with OpenBao + PostgreSQL
    - Add init-skeid.sh for automatic AppRole setup and customer key creation
    - Add usage_schema.sql for PostgreSQL usage events schema
    - Add skeid.yaml example for service stack configuration
    - Dockerfile: add libpq-dev, jq, explicit DBI/DBD::Pg install
    - cpanfile: fix DBI from recommends to requires for PostgreSQL users

0.002     2026-04-10 00:37:36Z

    - Drop Langertha::Knarr dependency. Skeid no longer goes through
      the Knarr namespace facades for format conversion or metrics —
      it talks to Langertha::Usage / Cost / Pricing / UsageRecord /
      Tool / ToolCall / ToolChoice directly from Langertha core. Bump
      Langertha floor to 0.400.
    - Skeid::Proxy hot path unchanged: still raw HTTP forwarding via
      Mojo::UserAgent + class-method calls on the new value objects,
      no per-request object construction overhead.
    - Skeid::Proxy: fix duplicate $choice variable in
      _openai_response_to_anthropic uncovered during the port.
    - dist.ini sets irc = #langertha (on irc.perl.org).

0.001     2026-03-15 01:22:05Z

    - Make usage storage layer pluggable: `record_usage` delegates to
      `_store_usage_event` and `usage_report` delegates to `_query_usage_report`
    - Add `store_usage_event` and `query_usage_report` constructor parameters
      for callback-based usage backend override (no subclassing required)
    - Add `jsonlog` usage backend: one JSON file per event in a directory
      (recommended, no DBI needed) or JSON-lines append to a single file
    - Make DBI and DBD::SQLite optional (moved to `recommends` in cpanfile);
      usage tracking is gracefully disabled when no backend is configured
    - Initial release extracted from Knarr as `Langertha::Skeid`
    - Add Skeid control-plane + proxy with OpenAI, Anthropic, and Ollama routes
    - Add weighted node routing with health checks, inflight/max_conns admission,
      and configurable wait timeout/poll behavior (`429` after timeout)
    - Add metrics/cost helpers via Knarr Input/Output/Metrics APIs
    - Add usage store support for SQLite/PostgreSQL, automatic schema setup from
      `share/sql`, and usage APIs (`usage.record`, `usage.report`)
    - Add `skeid usage` CLI subcommand for usage/cost reporting
    - Add admin routes with bearer-token protection (`/skeid/*`), hidden when
      no admin key is configured
    - Add engine ID mapping based on Langertha engine registry
      (`Langertha->available_engine_ids`) and reject legacy aliases
    - Make proxy request handling non-blocking with Mojolicious async upstream calls
    - Add Avatar smoke benchmark examples:
      `examples/avatar-skeid-single.yaml`, `examples/avatar-skeid-2nodes.yaml`,
      and `examples/skeid-parallel-smoke.pl`
    - Add one-box flush helper `examples/skeid-onebox-flush.sh` to prepare temp
      config, start Skeid, run smoke, and cleanup in one command