Data sources¶
A source is what a caller names with source. It is one of these:
- an id the operator configured under
sources - a
file:path insidefiles.allowDirs inline:, a table the caller sends with the request (MCP, REST). No exposure policy applies to it.
A configured source is one of these kinds:
| Kind | Configured with | Where it is read |
|---|---|---|
| File | file: a CSV, TSV, TXT, JSON or XLSX file |
In a worker thread, with the file limits (options) |
| HTTP | http: an operator's backend, its address with typed parameters |
Requested in the server's thread, a JSON answer parsed there, a CSV answer in a worker; computed in a worker; see HTTP sources |
| Code | load: a function of a library host |
Loaded in the host's thread, computed in a worker; see Code sources |
Every kind has the same exposure policy, scopes and field descriptions, and its results carry
dataAsOf (pivotkit-mcp, REST).
HTTP sources¶
An HTTP source asks an operator's backend for its rows. You write the address once in the
configuration. A parameter in braces marks where a caller's value goes. A caller, such as a model or a
program, passes only the values. pivotkit checks them against their declared type, pattern or list
before it sends anything. A caller never gives an address. When the API has an OpenAPI document,
source-from-openapi proposes the block for one of its operations.
{
"httpSources": {
"backends": {
"platform": {
"url": "https://api.example.com/v1",
"headers": { "accept": "application/json" },
"timeMs": 15000
}
}
},
"sources": {
"vms": {
"title": "Virtual machines",
"http": {
"backend": "platform",
"path": "/customers/{customer}/vms",
"query": { "state": "{state}", "limit": "500" },
"rows": "/items",
"asOf": "/generatedAt",
"paging": { "next": "/next" },
"maxPages": 10
},
"params": {
"customer": { "type": "string", "required": true, "pattern": "^[a-z0-9-]{1,40}$", "description": "The customer's short name", "example": "acme" },
"state": { "type": "string", "enum": ["running", "stopped"] }
},
"ttl": 300,
"policy": { "default": "hide", "fields": { "name": "show", "cpu": "show", "state": "show" } }
}
}
}
A call with "params": { "customer": "acme", "state": "running" } does this:
- It requests
https://api.example.com/v1/customers/acme/vms?state=running&limit=500. - It follows the
nextpages. - It reads the rows at
/itemsof each answer.
list_sources names the parameters with their type, description, example, list of values, pattern,
format, lengths and range.
A call without a required parameter answers INVALID_INPUT. The message names the parameter and
how to call again: "the source vms needs params.customer (The customer's short name,
for example acme); re-run with params.customer=acme". On the command line it says --param
customer=acme.
A value of the wrong form is refused. The message gives the parameter's description, its example and, when short, its pattern in one parenthesis: "params.customer is not in the expected form (The customer's short name, for example acme; pattern ^[a-z0-9-]{1,40}$)".
A value must have the parameter's type. pivotkit refuses a number for a string parameter, and a
text with lone surrogates. The fields of a source with required parameters are known only after a
load with values, so describe_source needs the values too.
Options¶
| Key | Default | Meaning |
|---|---|---|
httpSources.enabled |
true |
false leaves every HTTP source out (not listed, SOURCE_NOT_FOUND); check-config says how many are ignored |
httpSources.allowIpLiterals |
false |
An IP address as the host of a backend's or a source's address |
httpSources.backends.<name> |
none | A backend many sources share: url (an https:// address and path prefix, no query), intranet, headers, auth, timeMs, maxBytes, ttl, maxStale; a source's own value wins |
httpSources.backends.<name>.auth |
none | The backend's sign-in for all its sources (backend sign-in) |
httpSources.backends.<name>.intranet |
false |
The backend may be on a loopback or private address (an internal API); check-config lists each such backend |
sources.<id>.http.backend + .path |
- | A backend and the path after its url, with {name} for a parameter |
sources.<id>.http.url |
- | Instead of backend and path: a whole https:// address template |
sources.<id>.http.intranet |
false |
With url: as a backend's intranet |
.http.query |
none | Query parameters: each value a literal or exactly one {name}; a parameter without a value is left out |
.http.auth |
the backend's | The source's own sign-in: a header from a secret, client credentials or a token exchange (backend sign-in) |
.http.headers |
none | Static headers, over the backend's (literal values: no secret). authorization and cookie go to http.auth; host, content-length, transfer-encoding, connection, te, trailer, upgrade, keep-alive, accept-encoding, forwarded, proxy-* and x-forwarded-* are refused |
.http.rows |
"" (the body is the list) |
A JSON Pointer (RFC 6901) to the rows: "/items", "/data/rows" |
.http.format |
"json" |
"csv": the body is a CSV file, one page, read as a file is |
.http.asOf |
none | A JSON Pointer to the body's own time (ISO 8601 text or epoch seconds): the results' dataAsOf; else the time the last page arrived |
.http.paging |
none | How further pages are found (see Paging) |
.http.maxPages |
10 | 1 to 100; a source with more pages answers LIMIT_EXCEEDED |
.http.maxBytes |
20 MiB | All pages together, after decompression; over it LIMIT_EXCEEDED. A load reads at most limits.fileBytes (50 MiB unless set): a larger value is refused when limits.fileBytes is set, and a check-config warning otherwise |
sources.<id>.params |
none | The values a caller passes: type (string, integer, number, boolean), required, description, example, enum, pattern, minLength, maxLength (strings: default 64, at most 256), minimum, maximum, format: "date", default, claim (filled from the sign-in, claims) |
sources.<id>.timeMs |
15000 ms | All pages of one load, at most 600000 |
sources.<id>.ttl |
60 s | Seconds a load is reused for the same caller and values; 0 only shares loads that run at the same time |
sources.<id>.maxStale |
3600 s | How old the last good data may be to stand in after a transient failure; 0 turns it off |
An HTTP source's policy needs a default that is not show, under the server's policy and every
profile. Otherwise the server does not start. The reason: a field the backend adds later would be
shown without anyone deciding. Set policy.default to hide and list the fields to show.
A backend's sign-in goes to http.auth, never to http.headers. Its secret is a reference to a
variable or a file (backend sign-in). check-config warns about a header with a
literal value whose name looks like a secret's: key, token, secret, auth, pass. Everyone
who can read the configuration file reads that value too.
The address¶
- The address is
https://only. It has no user name or password, no fragment and no?. Query values go toquery. - An expression is allowed in the path only, never in the host or the port. Otherwise the message is "expressions in the URL are allowed in the path only; for a query value use http.query".
- A path segment holds at most one
{name}. Text around it is allowed ({id}.json,v{version}). Each{name}names a declared parameter. A parameter in the path must berequiredor have adefault. A segment.or.., also percent-encoded, is refused. - pivotkit encodes a value as RFC 6570 simple string expansion. It percent-encodes everything but
letters, digits,
-,.,_and~. So/,?,#,&,=,+,;and a space (%20) stay inside their segment or value. - pivotkit writes numbers and booleans itself, never as the caller typed them.
- The address must keep the template's host and its number of path segments. No value may make a
segment empty,
.or... - A
stringparameter withoutpatternorenumtakes letters, digits and. _ ~ -only, because some backends decode%2Fbefore they route. No value may hold..or a control character. - A
patternmatches the whole value: it is anchored. pivotkit refuses at start a pattern that can take long on some values. - Write no secret into the address or
query. The configuration file holds them in plain text. A backend's sign-in goes tohttp.auth. The log andcheck-sourcewrite each literal query value as..., but the path as it is. - Callers see a parameter's
description,example,enum,pattern,format, lengths and range, inlist_sourcesand in the message of a refused value. Write no internal names or secrets there. - Each query name appears once, without regard to case. A name the paging fills cannot also be a
parameter or a query value.
check-configwarns about a parameter that no part of the address uses.
Reading the answer¶
- The server parses the body as JSON in its own thread. Each page is at most
maxBytes. A body nested deeper than 64 levels, or one that is no JSON, answersSOURCE_UNAVAILABLE. - The rows' pointer reads the body's own keys only.
- Rows are objects or lists. In objects, the keys are the fields, in the order first seen. In lists, the first list is the header.
- A value that is an object or a list becomes its JSON text. Texts are cut at
limits.textLength. - Field names lose control and direction characters. A name that repeats another without regard to case gets " (2)".
- The rows have the limits a file has:
limits.rows(counted while paging),limits.columnsandlimits.importCells. Over them, the answer isLIMIT_EXCEEDED. - A worker drops hidden fields before it encodes the data. The operation runs in that worker too.
Paging¶
paging |
The next page |
|---|---|
{ "next": "/next" } |
The address at this pointer of the body (absolute or relative); none, null or "" ends |
{ "link": true } |
The Link header's rel="next" |
{ "page": "page", "start": 1 } |
The query parameter page, counted up from start until a page has no rows |
{ "offset": "from", "limit": "size", "size": 100 } |
from counted up by size (sent as size) until a page has fewer rows |
A next or link page must be on the same host and path as the page before, or the load fails.
Only its query may differ. A page named again ends the paging. pivotkit fills the names page,
offset and limit itself. They are never a parameter and never offered to a caller. A source
whose pages reach maxPages and still name a next page answers LIMIT_EXCEEDED.
What a caller gets¶
| Backend | Caller gets | retryable |
|---|---|---|
| 2xx | the data | |
| 400, 422 | INVALID_INPUT "the backend did not accept these parameters" |
false |
| 401, 403, 404 on a source with parameters | DATA_NOT_AVAILABLE (404) "nothing you may see for customer=xyz: check the value", one answer for all three so other customers' names cannot be probed |
false |
| 401, 403 on a source without parameters | SOURCE_UNAVAILABLE "the source's backend refused pivotkit's access; ask the operator and give the request id" |
false |
| 404 on a source without parameters, a redirect, an address it may not reach, an answer the settings do not fit | SOURCE_UNAVAILABLE "the source is not set up correctly; ask the operator and give the request id" |
false |
| 429, 502, 503, 504, a timeout, a network error | SOURCE_UNAVAILABLE "the source is not available right now; ...", Retry-After passed on |
true |
| anything else | SOURCE_UNAVAILABLE |
false |
The texts are pivotkit's, the same on REST and MCP. MCP adds the source's id and whether trying again may work. Every error names the request id. A backend's body never reaches a caller or the log.
The log has:
- the status, the address template and the source's id
- a warning with the failure's
cause(timeout,connection,tls,dns,blocked_address,http_status,bad_body,setup), the backend's name, its status and the time it took (operations) - with
log: "debug", also each request's path with every value written as{name}, and why a request failed
The network¶
Every request goes through the server's guarded fetch:
https://only.- It follows no redirect.
- It checks the address on the connection's own name lookup. A name that answers differently a second time cannot pass.
- It counts the body while it arrives.
- It refuses loopback and private addresses unless the backend is marked
intranet. - It always refuses link-local and cloud metadata addresses (169.254.0.0/16,
fd00:ec2::254,100.100.100.200and others).
A request has its own connection and uses no proxy of the environment.
A backend whose certificate comes from an internal certificate authority needs that authority's
certificate in NODE_EXTRA_CA_CERTS. That is a file of PEM certificates, which the server reads when
it starts. The container image takes it as an environment variable too. On a certificate error,
check-source says so. When the variable is set, it says that its file holds no authority that
signed the backend's certificate.
The server holds each page's body in memory while it reads and parses it, up to maxBytes per load.
The rows of a kept load stay in memory for ttl, bounded as for code sources. Keep maxBytes near
what a source needs.
Backend sign-in¶
You configure a backend's sign-in in a source's http.auth, or in a backend's auth for all its
sources. A source's own sign-in wins. It is exactly one of three kinds.
A secret is always a reference: { "env": "NAME" } or { "file": "path" }, relative to the
configuration file. pivotkit reads it when it uses it, so a rotated file or variable works without a
restart.
A static header, such as an API key:
{ "header": { "name": "authorization", "prefix": "Bearer ", "value": { "env": "PLATFORM_TOKEN" } } }
An OAuth access token for pivotkit itself (RFC 6749, client credentials), with a client secret
sent as HTTP Basic credentials (client_secret_basic):
{ "clientCredentials": {
"tokenUrl": "https://auth.example.com/application/o/token/",
"clientId": "pivotkit-backend",
"clientSecret": { "file": "secrets/backend-secret" },
"scope": "platform:read" } }
or with a JWT as the client's proof (RFC 7523 client_assertion), such as a projected Kubernetes
service account token, read again for every token request:
{ "clientCredentials": {
"tokenUrl": "https://auth.example.com/application/o/token/",
"clientId": "pivotkit-backend",
"assertion": { "file": "/var/run/secrets/tokens/pivotkit" } } }
An access token for the signed-in person (RFC 8693 token exchange), so the backend applies that person's rights:
{ "tokenExchange": {
"audience": "platform-api",
"scope": "platform:read",
"clientId": "pivotkit-backend",
"clientSecret": { "file": "secrets/backend-secret" },
"maxAgeSeconds": 300 } }
| Key | Default | Meaning |
|---|---|---|
header.name |
authorization |
The header's name; not one the connection sets (host, content-length, ...), and not also a static header |
header.prefix |
none | Text before the secret, such as Bearer |
header.value |
- | The secret reference |
clientCredentials.tokenUrl |
- | The token endpoint, https:// |
clientCredentials.clientId |
- | pivotkit's client at the token endpoint |
clientCredentials.clientSecret or .assertion |
- | One of the two, as a secret reference |
clientCredentials.scope, .audience, .resource |
none | Sent with the token request |
tokenExchange.tokenUrl |
the token_endpoint of auth.issuer's metadata |
An https:// address; the metadata's endpoint must be on the issuer's own origin. The metadata is read under the server's issuer rules: an issuer on a private address needs auth.intranet: true (OAuth 2.1) |
tokenExchange.clientId, .clientSecret |
- | pivotkit's confidential client (client_secret_basic) |
tokenExchange.audience or .resource |
- | The backend the exchanged token is for; at least one |
tokenExchange.scope |
none | The scopes asked for; ask for the narrowest |
tokenExchange.maxAgeSeconds |
300 | The longest an exchanged token is reused, 1 to 86400 |
.intranet (both OAuth kinds) |
false |
The token endpoint may be on a loopback or private address |
A token exchange needs auth.mode: "oauth" (see sign-in). With auth.mode
token or none, the server does not start with one.
A token exchange works only for a signed-in person: a caller whose token has a subject
(auth.subjectClaim). Two kinds of caller do not see the source: a caller whose token has no
subject, and a stdio session, which has no sign-in. For them the source is not listed and answers
SOURCE_NOT_FOUND. The log has an info line naming the source for the operator.
A header and client credentials work for every caller, also over stdio.
The person's token goes to the token endpoint only. pivotkit sends the caller's own access token
as subject_token to the token endpoint, and never to a backend or to an address taken from an
answer, as the MCP specification requires ("MCP servers MUST NOT accept or transit any other
tokens", Authorization, Token Handling; Security Best Practices, Token Passthrough).
Tokens kept¶
- Client credentials: one token per sign-in setting, shared by every caller. pivotkit keeps it
until 30 s before it expires (
expires_in). - Token exchange: one token per person, client and scopes. A token got through a broad client never
serves a restricted one. pivotkit keeps it until the first of these:
- 30 s before it expires
- the person's own token expires
maxAgeSeconds
- pivotkit sends one token request per key at a time. Everyone who waits for it shares it.
- pivotkit keeps a failed request for 5 s.
- A process keeps at most 1000 tokens, and the oldest goes out first. Tokens stay in memory only.
- When a backend answers 401, pivotkit drops the token it sent and asks once more with a new one.
- The token endpoint's answer must hold an
access_tokenoftoken_typeBearer. For a token exchange,N_Awith an access token asissued_token_typeworks too. - An
expires_inthat is missing or outside 1 to 86400 counts as 60 s. - When the token is a JWT, pivotkit compares its
audwith the requestedaudienceand logs a difference as a warning.
When the sign-in fails¶
| What happened | Caller gets | retryable |
|---|---|---|
The token endpoint refused (4xx), e.g. invalid_client, invalid_grant |
SOURCE_UNAVAILABLE "the source's backend refused pivotkit's access; ..." |
false |
| A secret is missing, the token endpoint cannot be reached by its address rules, its answer holds no bearer token | SOURCE_UNAVAILABLE "the source is not set up correctly; ..." |
false |
| The token endpoint failed (5xx, 429), its own time limit (10 s), a network error | SOURCE_UNAVAILABLE "the source is not available right now; ..." |
false |
The source's timeMs ended while pivotkit still waited for a token |
SOURCE_UNAVAILABLE "the source is not available right now; ...", or the same caller's last good data with stale |
true |
Last good data never stands in after a refused or failed token request. Such a request drops what
was kept. When the source's timeMs ends first, the source's time limit answers as for any slow
load.
pivotkit keeps a failed token request for 5 s. Calls in that time get the same answer without a new request. The log has an info line "token request failed (cached)" with each call's request id.
A secret that cannot be read is a warning in the log when the server starts. pivotkit logs a warning about the sign-in's settings at most once a minute, not on every call.
A backend that answers 401 to a source with parameters gives the caller DATA_NOT_AVAILABLE
("nothing you may see for ..."). That is the same answer as for a value the backend does not know.
So a wrong secret looks like an empty answer to the caller: look at the log's warning first. With a
header sign-in, the warning names the secret behind it.
The log names only the token endpoint's status and its RFC 6749 error code, and drops
error_description. It names a missing secret by its reference. It never names a secret or a
token.
Parameters bound to a claim¶
A backend without rights per person can still serve each person their own rows. pivotkit fills a
parameter with claim from that claim of the caller's verified token, never from the caller. This
works for HTTP and code sources and needs auth.mode: "oauth".
"params": {
"customer": { "type": "string", "required": true, "claim": "customer", "pattern": "^[a-z0-9-]{1,40}$" }
}
- A text parameter bound to a claim needs a
patternor anenum. pivotkit checks the claim's value as it checks a caller's value: texts, numbers and booleans only. It refuses a value with/,\,..,%,?, a line break or one outside the pattern (NOT_ALLOWED) before it sends anything. claim: "email"takes the email only when the token'semail_verifiedistrue.list_sourcesdoes not offer the parameter.- A caller that sends it gets
INVALID_INPUT("params.customer is set by the server from your sign-in; leave it out"). - A token without the claim gets
NOT_ALLOWED("your sign-in has no customer claim, which the source vms needs; ask the operator"). - pivotkit never says the claim's value back. A backend's 401, 403 or 404 names the caller's own values only.
- pivotkit reads only the claims a parameter names from a token. A code source sees them in
caller.claimstoo. Claims are never logged.
A source with a token exchange or a claim is listed only for a signed-in person, one with a subject.
draft_schedule answers NOT_SCHEDULABLE for it (schedulable: false in list_sources), because a
schedule runs later without a person signed in.
What the provider must allow¶
A token exchange makes pivotkit a deputy that asks for tokens on a person's behalf. Configure the provider so it can do no more than this source needs:
- Only pivotkit's client may exchange tokens, and it is a confidential client.
- Only tokens whose audience is pivotkit's own resource can be exchanged.
- Tokens are exchanged only for the named audience (the backend), with the narrowest scopes.
The settings below were checked against the providers' documentation in November 2026. These settings change, so check the current documentation of your provider's version.
Authentik's OAuth2 provider takes a JWT as client_assertion in a client credentials request.
The JWT must be signed by a source listed under Federated OIDC Sources of the provider. That
is an OAuth source configured with the Kubernetes API server's OpenID configuration URL or its
JWKS. Authentik then issues a token for a service account named after the provider and the JWT's
sub (Authentik documentation, "Client credentials").
"httpSources": {
"backends": {
"platform": {
"url": "https://platform.example.com/api",
"auth": { "clientCredentials": {
"tokenUrl": "https://auth.example.com/application/o/token/",
"clientId": "pivotkit-backend",
"assertion": { "file": "/var/run/secrets/tokens/pivotkit" } } }
}
}
}
The token file is the pod's service account token, for example a projected
serviceAccountToken volume. Group and others can usually read such files: Kubernetes' default
mode for a projected volume is 0644, so expect check-config's warning about a secret file
other users can read. The pod has no other users.
With the provider's client secret instead, Authentik issues the token for a service account it
makes itself, ak-<provider name>-client_credentials. The scopes the backend needs come from
scope mappings on the provider. Ask for them in scope (Authentik documentation, "Client
credentials"). Authentik's page shows the secret sent in the form body. pivotkit sends it as
HTTP Basic credentials (client_secret_basic), so check that your Authentik version takes
those.
"auth": { "clientCredentials": {
"tokenUrl": "https://auth.example.com/application/o/token/",
"clientId": "pivotkit-backend",
"clientSecret": { "file": "secrets/backend-secret" },
"scope": "platform:read" } }
Keycloak's standard token exchange works like this (Keycloak documentation, "Configuring and using token exchange"):
- Switch on Standard token exchange on pivotkit's client. The client must be confidential and in the same realm as the backend's client.
- The person's token must hold pivotkit's resource, which pivotkit checks (
auth.resource). The audience mapper of the Keycloak sign-in puts it there. - The person's token must also hold pivotkit's client id, because Keycloak exchanges a token
only for a client in its
aud. A second audience mapper adds it. - The
audienceis the backend's client id. Keycloak does not take theresourceparameter. - Without
tokenUrl, pivotkit takes the token endpoint from the metadata ofauth.issuer(the realm).
"auth": { "tokenExchange": {
"audience": "platform-api",
"scope": "platform:read",
"clientId": "pivotkit",
"clientSecret": { "file": "secrets/pivotkit-client-secret" } } }
Kept data¶
HTTP sources keep data as code sources do:
- pivotkit keeps data for
ttlseconds per caller and set of values. The caller is the person, a static token's name, orlocal. - Calls that need the same data while it loads share one load.
- A caller that goes away stops waiting. pivotkit cancels the requests when nobody waits any more.
- At most 4 loads of one source and 2 of one person run at once.
- Last good data stands in only after a transient failure, with
stale: { since }. A refusal drops it. - A load aborted because every caller went away keeps nothing. The next call loads again.
- A caller without an identity, an OAuth token without a usable subject, gets nothing kept and shares no load.
- For a source with a token exchange or a claim, pivotkit keeps the data per person, client and scopes, as it keeps the token.
pivotkit keeps an answer without the columns the caller's policy hides. A caller whose profile shows
more gets its own copy. It keeps a JSON answer's rows. It reads a CSV answer once in a worker into a
compact table and keeps that, as it keeps a code source's data. That read runs within
compute.timeMs and compute.memoryMb, after the source's timeMs. A kept answer goes to a worker
once. Later calls send its key only. The key holds the values and the caller only as a keyed
digest.
What a CSV answer takes in memory:
- The compact table holds each distinct text of a column once, as UTF-8. Each row takes a code of one, two or four bytes. Whole numbers take one, two or four bytes. Other numbers and dates take 8 bytes.
- The table is about the size of the answer or smaller. Two examples:
- A million rows of four columns with a distinct text in one (27.5 MB of CSV) take about 28 MB.
- A million rows of eight small whole numbers (19 MB) take 8 MB.
- Packing costs the first call some time. For that million rows with a distinct text it takes about 1.2 s instead of 1.1 s, measured on a laptop. Later calls are as fast.
- The table counts against
sourceCache.maxMbas a code source's table does (see code sources). pivotkit does not keep a table over a quarter of it. Then every call asks the backend again, and the log says so once an hour per source, namingsourceCache.maxMb. - When reading the answer fails or waits too long (
BUSY,TIMEOUT,LIMIT_EXCEEDED), pivotkit drops the answer. The next call asks the backend again. pivotkit never keeps the raw answer, which includes the hidden columns. - For a caller without an identity, pivotkit keeps nothing. The answer goes straight to the worker that computes the call, and that worker drops the hidden columns.
check-config and check-source¶
check-config prints each HTTP source without calling its backend:
HTTP sources (1):
vms: backend platform (https://api.example.com/v1)
GET /customers/{customer}/vms?state={state}&limit=...
rows at /items
next page at /next
up to 10 pages / 20 MiB / 15 s
kept 300 s per caller
last good data: up to 3600 s
create_link: none (no sources.<id>.link)
draft_schedule: drafts with the caller's params
params: customer (string, pattern, required), state (running|stopped)
backend authentication: client credentials as pivotkit-backend at
https://auth.example.com/application/o/token/, scope platform:read
(httpSources.backends.platform.auth)
client secret from the file secrets/backend-secret
(/etc/pivotkit/secrets/backend-secret)
The backend authentication lines say how the sign-in works and where its secrets come from: a
variable's name, or a file as written and with its full path. They never show a value or its
length.
A variable that is not set, or a file that cannot be read, is a PROBLEM. The server still starts.
The sources that use it answer SOURCE_UNAVAILABLE until it is there.
check-config warns about:
- a secret file other users of the machine can read
- a token endpoint on a loopback or private address without
intranet: true - a token exchange's
tokenUrlon another host thanauth.issuer, since the person's token goes there - a backend's
auththat no source uses, because each source has its ownhttp.auth
In the Sources and profile lines, an HTTP source's fields are known only after a load
(reading check-config).
pivotkit-server check-source <id> [--param name=value ...] [--config file] loads a source once as
the server would, without an AI client. The id and the options go in any order. It prints:
- each page's status, content type and size
- the rows and the fields with their types
- when the data is from
- what a caller would get when it fails
Where there is one, it adds the operator's hint:
- the
--paramto re-run with when a required value is missing or a value is refused. It fills in the declared example, else<value>. - when the backend answers 401 or 403: the secret behind
http.auth(its variable or file, never the value), or where to set one intranetfor a private addressNODE_EXTRA_CA_CERTSfor a certificate. When the variable is set, it says that its file holds no authority that signed the backend's certificate.
On success, the last line names the values it loaded with (OK with customer=acme). It exits 1 when
the load fails. It exits 2 when the id names no HTTP source, or when the command line is wrong.
It prints no cell value and no secret. Each value in the address is {name}, and each literal query
value is .... It prints field names, and the keys where the rows' pointer finds no list. These can
be the backend's data, so treat the output as you treat the backend's answers. A load without rows
prints a warning.
With a sign-in, it prints the token endpoint's status, its error code, the scope it granted and
expires_in. It never prints a token. A token exchange needs a person's token, which a command line
does not have. For a token exchange, check-source asks the token endpoint with the client's
credentials only. It says whether the endpoint answers and takes the client, and does not load the
source. A --param for a parameter bound to a claim stands in for the sign-in's claim.
check-source uses the same configuration, address rules and limits as the server:
$ pivotkit-server check-source vms --param customer=acme --config pivotkit.json
source vms: GET https://api.example.com/v1/customers/{customer}/vms?state={state}&limit=...
page 1: 200 application/json, 18342 bytes
1 page, 18342 bytes, 42 rows, 180 ms
fields: name (text), cpu (number), state (text), owner (text), tags (text)
data as of 2026-11-15T08:00:00Z (from the body)
OK with customer=acme
With a pointer that matches nothing:
source vms: GET https://api.example.com/v1/customers/{customer}/vms?state={state}&limit=...
page 1: 200 application/json, 18342 bytes
pointer /items matched nothing; the body has the keys: results, next
a caller gets: SOURCE_UNAVAILABLE: the source is not set up correctly; ask the operator and give the request id
FAILED
Code sources¶
A host that embeds pivotkit-server or pivotkit-mcp as a library can give a source a function instead
of a file. A configuration file cannot hold a function, so code sources exist only in options that
code passes to createContext, createMcpContext or listen.
import { createContext, SourceError } from "pivotkit-server";
const context = await createContext({
sources: {
tickets: {
title: "Open tickets",
load: async (params, { caller, signal }) => {
const res = await fetch(`https://tickets.example.com/api/teams/${encodeURIComponent(params.team)}`, {
signal,
headers: { "x-on-behalf-of": caller.subject ?? caller.client }
});
if (res.status === 403) throw new SourceError("NOT_ALLOWED", { publicMessage: "You may not see this team's tickets." });
if (res.status === 404) throw new SourceError("DATA_NOT_AVAILABLE", { publicMessage: "There is no such team." });
if (!res.ok) throw new SourceError("SOURCE_UNAVAILABLE", { status: res.status });
const body = await res.json();
return { rows: body.items, asOf: body.generatedAt };
},
params: { team: { type: "string", required: true, enum: ["blue", "red"], description: "The team" } },
timeMs: 20000,
ttl: 120,
maxStale: 3600,
policy: { default: "summary", fields: { Team: "show", Status: "values" } }
},
rates: {
// the same data for everyone: one copy, and load gets no caller
load: async (_params, { caller, signal }) => {
const res = await fetch("https://rates.example.com/today.json", { signal });
return { rows: caller === null ? await res.json() : [], asOf: new Date() };
},
cacheShared: true,
ttl: 600
}
}
});
The example compiles with strict TypeScript. pivotkit-server's types give load a caller of type
SourceCaller, or null with cacheShared: true. They export SourceCaller, SourceLoad,
SharedSourceLoad, SourceVersion, LoadResult and SourceParams for your own functions.
Options¶
| Key | Default | Meaning |
|---|---|---|
sources.<id>.load |
- | load(params, { caller, signal }): the data (see below) |
sources.<id>.version |
none | version(params, { caller }): a value (not a promise) that changes when the data changes; kept data is reused while it is the same. It is called at once on every call, never awaited, so keep it cheap |
sources.<id>.params |
none | The values a caller passes with the source (parameters); a parameter with claim is filled from the signed-in person's token instead (claims) |
sources.<id>.timeMs |
30000 ms | The longest load may take, at most 600000 (ten minutes); the signal is aborted then |
sources.<id>.ttl |
none (only version decides) |
Seconds a load is reused; 0 only shares loads that run at the same time |
sources.<id>.maxStale |
3600 s | How old the last good data may be to stand in after a transient failure; 0 turns it off |
sources.<id>.cacheShared |
false |
The data is the same for every caller: one copy for all, and load gets caller: null |
sourceCache.maxMb |
256 | Top level: the loaded data of code and HTTP sources the process keeps, in MiB (1 to 65536); see bounds |
sources.<id>.perPerson |
false |
The data differs per caller (each person sees their own): its reports count as closed for download links, so only the person who asked can open them (a sign-in link) or no link is given. Not with cacheShared |
checkConfig lists each code source with its time limit, how its data is kept, and whether last
good data may stand in. It first checks the options as createContext does, so it throws the same
ConfigError for options the server would refuse:
import { checkConfig } from "pivotkit-server";
const { text, problems } = await checkConfig(options, { name: "my-app" }); // name: the first line's
console.log(text); // (default "pivotkit-mcp")
What load gets and returns¶
params: the caller's values, which pivotkit checks and writes again. It is{}when the source declares none. A source withoutparamsrefuses any.-
caller: a frozen copy of who calls, without any token. It has these keys:client: a token's name or an OAuth client idprofilescopes: a list, ornullfor every scopegroupsclaims: only those the configuration namessubjectandissuerwith OAuth, elsenull
With
cacheShared: true,callerisnull. A source that reads it there fails at once instead of serving one person's rows to everyone. -
signal: anAbortSignal. pivotkit aborts it aftertimeMs(with aTimeoutError). It also aborts it when every caller waiting for this load went away: a REST request or an MCP call over HTTP whose connection closed. Pass it tofetchand your database client.- A load that ignores it runs on in the background and keeps its slot (see the load limits)
until it ends. After twice
timeMs, the log warns once. - A load that never ends holds its slot until the process restarts.
- A load that ignores it runs on in the background and keeps its slot (see the load limits)
until it ends. After twice
load returns one of these:
- rows (
[{ ... }]) { rows, asOf }- columnar data
{ columns, types?, data, asOf? }, with one array per column
It does not return encode()'s result, so pivotkit can drop hidden fields. asOf becomes the
results' dataAsOf. It is a Date, an ISO 8601 text or epoch milliseconds. Without it, dataAsOf
is the time load gave the data. A load() that takes no arguments works as it is.
The data has the limits a file has: limits.rows, limits.columns and limits.importCells. Over
them, the answer is LIMIT_EXCEEDED. pivotkit cuts texts longer than limits.textLength, on a
copy.
Errors¶
Throw a SourceError to say what went wrong in words a caller may read:
SourceError code |
The caller gets | Last good data |
|---|---|---|
NOT_ALLOWED |
403 with your publicMessage |
dropped |
INVALID_INPUT |
400 with your publicMessage |
dropped |
DATA_NOT_AVAILABLE |
404 with your publicMessage |
dropped |
SOURCE_UNAVAILABLE |
502 with pivotkit's fixed text, retryable as transient |
used when transient |
new SourceError(code, { publicMessage, transient, status, retryAfter, reason }):
publicMessageis the only text of yours that reaches a caller. pivotkit removes control characters and keeps at most 300 characters. A 502 always says one of pivotkit's fixed texts. WithoutpublicMessage, the caller gets the code's own words.
These options apply to SOURCE_UNAVAILABLE only:
transient: truesays the failure may pass.statusis a backend's HTTP status, also for the log. A status of 429, 502, 503 or 504 makes the failure transient unlesstransientsays otherwise.retryAfter, in whole seconds, becomes the answer'sRetry-After.reasonpicks pivotkit's text:"refused": "the source's backend refused pivotkit's access; ask the operator ...""setup": "the source is not set up correctly; ask the operator ..."- none: "the source is not available right now; ..."
A refusal (NOT_ALLOWED, INVALID_INPUT, DATA_NOT_AVAILABLE) is never transient.
Anything else load throws answers SOURCE_UNAVAILABLE with retryable: false and the fixed text.
That includes an OperationError of pivotkit's own codes. pivotkit logs the error's class, and its
message only with log: "debug".
A network error that may pass answers SOURCE_UNAVAILABLE with retryable: true. These are
ECONNRESET, ETIMEDOUT, ECONNREFUSED, ENOTFOUND, EAI_AGAIN, and undici's socket and timeout
errors, also as the cause of fetch's error.
Two more cases answer SOURCE_UNAVAILABLE: an error your code throws while pivotkit reads the
returned rows (a getter), and a version that returns a promise.
pivotkit reads your rows once into plain values. Getters and toJSON run then. Objects in cells
become JSON text, and every text is cut at limits.textLength.
Kept data, coalescing and last good data¶
-
Per caller. pivotkit keeps loaded data per caller. The key is made of:
- the person: with OAuth
subjectandissuer, a static token's name, orlocal - the client, its scopes, the groups and the claims
- the values of
params - the caller's policy levels
A caller without an identity, an OAuth token without a usable subject, gets nothing kept, shares no load and gets no last good data. With
cacheShared: true, all callers share one copy per set of values and policy levels. The key holds the values, groups and claims only as a keyed digest (random per process), not in clear text. - the person: with OAuth
-
Reuse. pivotkit reuses kept data while
versiongives the same value, and withinttlwhen both are set. Withoutversion, it reuses data forttlseconds. Without either, every call loads.ttlcounts from when the load ended. - Coalescing. Calls that need the same data while it loads wait for that one load. A caller that goes away stops waiting. pivotkit aborts the load only when every caller waiting for it went away. That is no failure of the source, and nothing kept is dropped.
- Load limits. At most 4 loads of one source run at once, and at most 2 of them for one person.
Callers without an identity count per client, so one person trying many values cannot take every
slot. A call over the limit answers
SOURCE_UNAVAILABLEwithretryable: trueandRetry-After: 1. A load that ignores the abort keeps its slot until it ends. - Time limit. After
timeMs, pivotkit aborts the signal. The call answersSOURCE_UNAVAILABLE(retryable: true), or last good data. pivotkit keeps a result that arrives later for the next call, unless a refusal for the same data came in between. -
Last good data. Only after a transient failure does the data last loaded for the same caller and values stand in, and only when it is at most
maxStaleseconds old. Transient failures are the time limit, a network error and aSourceErrorwithtransient.- The result keeps that data's
dataAsOfand addsstale: { since }, the time its load ended. render_view's message says "The data is from ...: the source could not be refreshed."- The log has a warning with the source's id.
Any other failure answers at once and drops what was kept for that caller and values. So the cache never covers a withdrawn access.
- The result keeps that data's
-
Bounds. The process keeps at most 64 loads and
sourceCache.maxMbof loaded tables (256 MiB by default). pivotkit counts each table as it holds it: each distinct text of a column once as UTF-8, whole numbers in one, two or four bytes, other numbers, dates and true/false values as typed arrays. One person keeps at most 8 loads and a quarter of the bytes. A shared source counts as one person. A person's new load pushes out that person's oldest first, so one caller cannot push out everyone else's data. Otherwise the least recently used go first. Kept data stays in the process's memory until it is pushed out, also aftermaxStalehas passed.check-configprints both limits. -
Tables up to a quarter of
sourceCache.maxMb(64 MiB by default) are kept. pivotkit never keeps a larger table. Every call loads it again, reads it and sends it to a worker, as withoutttl. The log warns once an hour per source with the table's size.For example, a million rows of eight columns with a distinct text of about 36 characters in each row take about 65 MiB, just over the default share. With
"sourceCache": { "maxMb": 512 }(tables up to 128 MiB), pivotkit keeps it, and a later call takes a few milliseconds. Give the process the memory for it.The process keeps one load cache. With several contexts in one process, the largest
sourceCache.maxMbof those open applies. When a context closes, the cache shrinks to the largest left.
Where it runs¶
load runs in the host's thread. pivotkit reads its rows there once into a compact table:
- The run limits above apply, and hidden fields are left out.
- The table holds each distinct text of a column once.
- Its texts and codes, numbers, dates and true/false values are in shared memory, so sending it to a worker copies no value.
The operation over that table runs in a worker, as for a file or an HTTP source:
timeMsboundsload.compute.timeMsbounds the computation (TIMEOUT).compute.memoryMbbounds its memory (LIMIT_EXCEEDED).- It waits up to
compute.queueMsfor a free worker (BUSY).
A load that is kept (ttl or version) goes to a worker once. Later calls go to that worker without
the table while it keeps it. Each worker keeps the data of its last four runs, at most a quarter of
compute.memoryMb.
Reading a million rows of eight columns takes about 0.5 s of the host's thread on a laptop. A call over kept data takes a few milliseconds, so keep the data a code source returns to what a report needs.
A code source is your code. pivotkit hands it a frozen copy of the caller and never a token. That guards against mistakes, not against code that means harm.
HTTP sources from an OpenAPI document¶
pivotkit-server source-from-openapi reads an API's OpenAPI document. It proposes the
configuration of an HTTP source for one of its GET operations:
- the address with its parameters
- the parameters' types and descriptions
- where the rows are
- the fields' descriptions
You review the result, fill in what it leaves to you and add it to your configuration. The command is for you at the command line. The server never reads the document.
pivotkit-server source-from-openapi <file or https URL> --operation <operationId | "GET /path"> [--id <source id>] [--backend <name>]
--operation names the operation by its operationId. For a document without them, it names the
method and path as the document writes it (--operation "GET /pet/{petId}"). An unknown name lists
the document's GET operations in both forms.
The command prints the block as JSON on stdout. On stderr it prints what you must still check and decide. It exits 0 when it wrote a block. It exits 2 on a usage error, a document it does not read, or an operation that cannot be a source.
--idnames the source. The default is theoperationId, else the path:pet-petId.--backendnames the backend. The default is the document'sinfo.titlein lower case with-, elseapi.
pivotkit-server source-from-openapi --help lists the options.
With the public Swagger Petstore (OpenAPI 3) document:
$ pivotkit-server source-from-openapi https://petstore3.swagger.io/api/v3/openapi.json --operation findPetsByStatus --backend petstore > petstore.json
{
"httpSources": {
"backends": {
"petstore": {
"url": "TODO: https://... (the document names the relative address /api/v3, which is https://petstore3.swagger.io/api/v3 from the document's address; check it)"
}
}
},
"sources": {
"findPetsByStatus": {
"title": "Finds Pets by status.",
"description": "Multiple status values can be provided with comma separated strings.",
"http": {
"backend": "petstore",
"path": "/pet/findByStatus",
"query": {
"status": "{status}"
},
"rows": ""
},
"params": {
"status": {
"type": "string",
"required": true,
"description": "Status values that need to be considered for filter",
"enum": [
"available",
"pending",
"sold"
]
}
},
"fields": {
"status": {
"description": "pet status in the store"
}
},
"policy": {
"default": "hide",
"fields": {
"id": "hide",
"name": "hide",
"category": "hide",
"photoUrls": "hide",
"tags": "hide",
"status": "hide"
}
}
}
}
}
The API host comes from the document; check it before adding credentials.
Still to decide: 1 value says TODO, and the server refuses the configuration until none does:
httpSources.backends.petstore.url: TODO: https://... (the document names the relative address /api/v3, which is https://petstore3.swagger.io/api/v3 from the document's address; check it)
Notes:
- The document's sign-in is not one pivotkit writes for you (petstore_auth: OAuth for people, not client credentials): configure the backend's auth by hand (https://pivotkit.eize.dev/reference/sources/#http-auth); for each person's own access, use tokenExchange.
- The query parameter "status"'s explode is left out: it makes no difference for a single value.
- Every field is hidden (sources.findPetsByStatus.policy): set the ones a caller may see to "show", "values" or "summary".
Review these as text the model will read:
sources.findPetsByStatus.title: "Finds Pets by status."
sources.findPetsByStatus.description: "Multiple status values can be provided with comma separated strings."
sources.findPetsByStatus.params.status.description: "Status values that need to be considered for filter"
sources.findPetsByStatus.params.status.enum: "available, pending, sold"
sources.findPetsByStatus.fields.status.description: "pet status in the store"
Add the block to your configuration, fill in every TODO, then run pivotkit-server check-config and check-source findPetsByStatus. Docs: https://pivotkit.eize.dev/reference/sources/#openapi
The document names its server as /api/v3, an address relative to the document's own, so the TODO
names both. Its sign-in is OAuth for people (an implicit flow). The command does not write that. You
configure a token exchange or another backend auth yourself.
What it reads¶
- OpenAPI 3.0 or 3.1 in JSON (
"openapi": "3.1.0", also"3.1"), at most 5 MiB and 128 levels deep. The command refuses a YAML document and says how to convert it, for examplenpx js-yaml openapi.yaml > openapi.json. js-yaml's command writes a YAML file as JSON. The command refuses Swagger 2.0. Convert it to OpenAPI 3 first. - A file, or an
https://URL. The command fetches a URL once, within 10 seconds and without credentials. It uses the guarded fetch of HTTP sources: no redirects, no cloud metadata or link-local addresses, and the size limit while it streams. You name the address yourself, so a private address is allowed here withoutintranet. For a document behind a sign-in or on plainhttp://, download it and pass the file. -
Local references only. The command follows
$refvalues that start with#/, with these limits:- own properties only
- at most 32 in a chain
allOfat most 8 deep, merging at most 1,000 properties. Each schema is merged once per depth, however often it is referenced.- one budget of 100,000 nodes for the whole run. Every property read counts against it.
- finding the operation has a budget of its own, about 200,000 paths.
A document over a limit is refused, or shortened with a note, quickly. The command never fetches a reference to another file or address (
file:,https:,http://169.254.169.254/...). It names it in a note and goes on without it. It names a circular reference the same way.
What it writes¶
| From the document | In the block |
|---|---|
The server address (servers, the operation's or path's first) |
The backend's url, as a TODO that names the address without a user name, password, query or fragment; an address with variables as written with their defaults (https://{region}.example.com (region=eu)); a relative address with what it is relative to the document's URL |
A server on a private address (a loopback or private IP address, localhost, a name of one label or ending in .local, .lan, .internal, .intranet, .corp, .home, .home.arpa, .localdomain) |
A note that the backend needs intranet: true; the block does not set it (no name is looked up). A link-local, unspecified or metadata IP address (169.254.0.0/16, fe80::/10, 0.0.0.0, a cloud's metadata address) gets a note that the server refuses it even with intranet |
| The operation's path | http.path, with each path parameter as {name}; a path over 2,000 characters or with an empty segment refuses the operation |
| Path parameters | Required params |
| Query parameters | params and http.query; required is kept |
A parameter's type, enum, minLength, maxLength, minimum, maximum, format: date, description, example |
The same keys of the parameter, as far as the server accepts them (an example outside the safe characters, a maxLength over 256 or an enum of more than 100 values is left out or capped, with a note) |
A parameter's pattern |
Never copied: a document's regular expression could take very long on a caller's value; a text parameter without a pattern takes letters, digits and . _ ~ - only. Write your own pattern where values need more |
A name that is no parameter's name (created-after, filter[name]) |
The parameter created_after, filter_name; the query keeps the document's name |
Header and cookie parameters, array and object values, a style or allowReserved |
Left out, with a note (a path parameter like that refuses the operation) |
| A JSON answer (the first 2xx) that is a list | http.rows: "" |
| An object with exactly one list | http.rows pointing at it (/items) |
The object's one property in date-time format, else its one property named like a time (generatedAt, updated_at, timestamp, ...; a text or a number) |
http.asOf pointing at it, with a note; two of a kind give none and a note |
| Any other answer | http.rows as a TODO |
A text/csv answer |
http.format: "csv" |
| The row's properties | fields.<name>.description where the document describes them, and every property in policy.fields as hide, under policy.default: "hide" |
The operation's summary and description |
The source's title and description |
| The security scheme | A skeleton of the backend's sign-in, see below |
Descriptions and other texts lose control characters, terminal escape sequences, bidi controls and other invisible characters. The joiners an emoji needs stay. Line breaks become spaces. Texts are cut at 500 characters, a title at 200.
The model reads these texts through list_sources and describe_source. An API's documentation can
hold instructions meant for a model. So the command prints each text under "Review these as text the
model will read".
Some parameters keep no example and no list of values, and a note names each one:
- a parameter whose name looks like a credential's:
key,token,secret,auth,pass - a parameter whose example or values look like a credential:
- a known token prefix such as
sk-,ghp_,hf_,xoxb-,AKIA,AIza,ya29.orSG.followed by at least 16 token characters - a JWT
- 32 or more token characters with letters and digits that are no UUID
- a known token prefix such as
The command leaves out a query parameter in which the document's own security scheme sends an API key. A credential never comes from a caller.
The command takes the sign-in from the operation's security, else from the document's. It uses the
first alternative of one scheme that maps:
| Security scheme | Backend auth |
|---|---|
apiKey in a header |
header with the header's name (listed for review; Cookie, Authorization, Origin, Referer or a header of the connection becomes a TODO) and value: { "env": "TODO: ..." } |
http with bearer |
header with prefix: "Bearer " |
http with basic |
header with prefix: "Basic "; the secret holds the base64 of user:password |
oauth2 with a clientCredentials flow |
clientCredentials with tokenUrl, clientId and clientSecret as TODOs and the operation's scopes (a scope starting with TODO is left out) |
| An API key in the query or a cookie, OAuth for people, OpenID Connect, mutual TLS, two schemes together | No auth, with a note: configure it by hand; for each person's own access use tokenExchange |
TODO values¶
Every value you must decide starts with TODO:, the same word everywhere, so grep TODO finds
them. The server refuses a configuration with a value under httpSources or a source's http that
still starts with TODO:. It checks this right after the keys and their types, before the other
rules. pivotkit-server check-config and serve exit 2 while one is left:
pivotkit-server: 1 value still says TODO: httpSources.backends.petstore.url; replace each with the real value
No address from the document becomes a value by itself. A document could name an attacker's host as
the API or the token endpoint, and that host would then receive your credentials. Check the host,
write it into url and tokenUrl, then add the credentials. Other texts, such as a title or a
description, are ordinary values. So is a value that is just TODO, such as a query value a ticket
API takes.
Then decide which fields a caller may see (policy.fields). Test the source with
pivotkit-server check-config and check-source. check-source also prints the
fields the backend really sends.