Operations and logs¶
pivotkit-server and pivotkit-mcp write one line per event to stderr and keep no log file. On stdio,
stdout is the MCP protocol. What happens to the lines is up to your container runtime or service
manager: docker logs, journald, a log driver or a cluster's log agent. This page says what a line
holds and how to get the lines where you search them. Whether a server is up and can take work is
for /healthz and /readyz, not the log (health checks).
Settings¶
Each setting is a key in the configuration file, a flag and an environment variable. A flag wins, then the file, then the variable.
| Key | Flag | Variable | Default | Values |
|---|---|---|---|---|
log.level |
--log |
PIVOTKIT_LOG_LEVEL |
info |
debug, info, warn, error, off |
log.format |
--log-format |
PIVOTKIT_LOG_FORMAT |
auto |
auto, text, json |
log.address |
--log-address |
PIVOTKIT_LOG_ADDRESS |
on |
on, off |
{ "log": { "level": "info", "format": "json", "address": "on" } }
"log": "debug" (a level alone) is short for { "level": "debug" }.
log.level: a line is written when its level reaches the setting.warnkeeps every failure and drops successful requests and calls.errorkeeps only server errors (5xx) and unexpected errors.- The
startedandstoppedlines are written at every level butoff. debugis for setting a server up. Its lines hold more, an error's own message among them, which may quote a value or a request's body, so do not leave it on.- Where you only look at what went wrong (a busy server, a log you pay for by volume),
warnis the level to run with.
log.format:jsonwrites one JSON object per line.textwriteskey=valuepairs (logfmt, in the style of Go'sslogTextHandler) for a person at a terminal.autowrites text when stderr is a terminal, and JSON otherwise.- Both formats carry the same fields, except that text leaves out
severity,service,version(thestartedline names it) andtraceId. - Text is coloured only on a terminal, and only when
NO_COLORis not set. - The container image sets
PIVOTKIT_LOG_FORMAT=json, becausedocker run -tgives the process a terminal.
-
log.address:on, the default, puts the client's address on every line of a request:address, the client as best known: withhttp.trustProxythe last entry ofX-Forwarded-For, else the socket's peerpeerAddress, the socket's peer, where it differs (behind a proxy, the proxy)
An address is personal data (privacy).
offleaves both out:"log": { "address": "off" },--log-address offorPIVOTKIT_LOG_ADDRESS=off.
started names the format it chose (logFormat), so a log that suddenly reads as text or JSON
tells you why.
An error at start, before the log exists, follows the same rule: for example a configuration that
cannot be read, or an unknown flag. It is a JSON line (event error, code CONFIG) where
stderr is no terminal or JSON is asked for. It is the plain line pivotkit-server: ... on a terminal
or with --log-format text. check-config prints the log's settings and does not log.
Lines¶
Every line has these fields:
| Field | What it holds |
|---|---|
time |
When, in UTC (ISO 8601 with milliseconds) |
level |
debug, info, warn or error |
severity |
The level as an RFC 5424 number: error 3, warn 4, info 6, debug 7 |
event |
What happened (events) |
msg |
A short sentence: describe_source failed, request refused: no_token, the text of a warning |
service |
pivotkit-server, or pivotkit-mcp when pivotkit-mcp runs on its own |
version |
The package's version |
A line written while a request or tool call runs adds its scope:
| Field | What it holds |
|---|---|
requestId |
11 random characters, the same on every line of the request, in its X-Request-Id answer header and in its error (correlation) |
traceId |
32 hex characters: the trace id of a valid W3C traceparent header, else a new one |
upstreamRequestId |
The X-Request-Id the request came with (a proxy's id), 1 to 128 letters, digits and _ . : + / = -. Any other value is not logged. The caller or a proxy chooses it, so it can hold a user id or a session id. No setting leaves it out (privacy) |
address |
The client as best known, unless log.address is off. With http.trustProxy, the last entry of X-Forwarded-For, or invalid when that is no IP address (the request is never put on the proxy's address). Without it, the socket's peer |
peerAddress |
The socket's peer, where it differs from address (behind a trusted proxy, the proxy) |
traceId and upstreamRequestId come from the caller's headers. They are checked for their form,
not for who sent them, so they are hints to join lines with other systems, not proof of where a
request came from.
Lines of calls, requests, downloads and refusals add:
| Field | On | What it holds |
|---|---|---|
operation, route |
request, download |
The REST operation and its route (POST /sources/describe) |
tool |
call |
The MCP tool |
client |
call, request, download |
The static token's name or the OAuth client id |
subject |
call, request, download |
The person, only with audit.subject |
source |
call, request, download |
A configured id, inline:, file: (any file by path) or unknown (anything else a caller typed). At most 64 characters |
status |
all four | The HTTP status. On a call line, the status REST answers the same error with |
outcome |
all four | ok or error. ok stays beside it on call lines |
code |
failures | The error code |
ms |
call, request |
The time the work took, in milliseconds |
saved |
call, request |
false for a render_view with save: false, which writes no file (inline only) |
reason |
refused |
Why the request was refused. The values are listed below the table |
limit, size |
refused |
For body_too_large: the bytes allowed and the declared size. For rate_limited after sign-in, limit is rate (the caller's own bucket), clientRate (the bucket of one OAuth client's people together, eight times as large), concurrency (the caller's own requests at once) or clientShare (the requests of one OAuth client's people together, eight times concurrency). A download line with code RATE_LIMITED is a sign-in callback over 20 a minute from one address. Behind a proxy, that needs http.trustProxy, or everyone shares the proxy's address. Limits count per person. The line names the person as subject only with audit.subject |
path |
refused |
The route family that refused it (/api/v1, /openapi.json, /mcp, /downloads), never the path the request asked for |
error, refusal, detail |
refused |
For a request the MCP protocol layer refused: its error's class, the SDK's fixed name for the refusal (such as name-header-mismatch) and, only at debug, its message, which can quote the request's body |
host, origin |
refused |
For reason host or origin: the Host or Origin the request sent (at most 100 characters) |
hint |
refused |
For reason host: how to allow the name (http.allowHosts, --allow-host or http.publicUrl) |
parserError |
refused |
For a request the HTTP parser refused (answered 400, 408 or 431 with an X-Request-Id): the parser's code, such as HPE_INVALID_HEADER_TOKEN. A CONNECT (a tunnel, which the server never opens) is answered 400 with a bad_request line without it |
no_token,bad_token,rate_limited,body_too_large,host,origin,insufficient_scopenot_allowed: the person's groupsnot_ready: the issuer is not loaded yetbusybad_request: a body or a request the HTTP parser cannot read, a method the server never takes such asTRACE, a body that is not JSON, or anmcp-protocol-versionthe MCP endpoint does not support. A method such asTRACEgets 400 and a line atwarnby design.unsupported_media_typeandnot_acceptable: the MCP endpoint'sContent-TypeandAcceptcheckstoken_in_addressinternal: a 5xx without a line of its own- At
debugonly:method_not_allowed: a method a route does not take, such as a GET of the MCP endpoint, which clients send to open a streamnot_found: a path nobody serves, also under/api/v1. Under/downloads, a link nobody has is atinfo, since a person asks you about it.
- Any other error answer without a line of its own gives its problem's code in lower case, or one of these by its status.
The level of a call, request, download or refused line follows its status: info below 400,
warn for 4xx, error for 5xx. A refused request is warn or error, except the two routine ones
above (method_not_allowed, not_found), which are debug.
Every error answer (4xx and 5xx) has one line under the requestId of its X-Request-Id: its own
call, request or download line, or a refused line. A 404 or 405 has its line only at
log.level debug. A request that fails unexpectedly after it logged a line below 500 gets an
internal refused line at error besides it.
A log line names the route family. The path, query and body a request sent are never logged. A host or origin refusal names the refused Host or Origin value, cut to 100 characters.
/healthz and /readyz write no line per probe. /readyz writes one when the server stops being
ready (warn, with what is missing) and one when it is ready again (info).
A body larger than the cap is answered 413 at once. One that keeps coming after four times the cap
(at least 64 KiB) has its connection cut. A client that sends with no Content-Length, such as a
proxy forwarding an HTTP/2 body, still gets the 413 with its X-Request-Id first. A proxy may report
a stream error to its own client instead. The refused line with body_too_large and size (the
declared size, when there is one) is written either way. Look it up by time and peerAddress.
A warning about a backend adds these fields. A backend is an HTTP source, its sign-in, or the upload store.
| Field | What it holds |
|---|---|
cause |
timeout, connection, tls, dns, blocked_address (an address it may not reach), http_status, bad_body (an answer that cannot be read as configured), setup (the request cannot be sent as configured), sign_in_failed (a backend's sign-in: the token endpoint refused or answered badly), upload_failed (an upload that failed otherwise) |
backend |
The backend's name in httpSources.backends, where the source uses one |
upstreamStatus |
The backend's (or the store's, the token endpoint's) status, where it answered |
upstreamMs |
The time since the request to it started, in milliseconds |
A warning's text (msg) is free text and may change between versions. Query by event, cause,
backend and source, which stay. The warning of an HTTP source's error status also holds
template: the source's configured address template with the backend's host, never a value of a
parameter.
No line holds a view, a value, a token, or a person's groups or other claims. A request's or a call's line holds no file path. Lines at start can name the operator's own file paths. Text a caller or a backend controls is cut, and its control characters (C0, C1, the bidi controls, the line separators) are removed in both formats, so it cannot start a line of its own, a field in the text format, or a terminal escape in a log viewer.
An example request line in each format:
{"time":"2026-11-15T11:12:45.066Z","level":"warn","severity":4,"event":"request","msg":"describe_source failed","service":"pivotkit-server","version":"<version>","requestId":"F0iFBi9CZjQ","traceId":"4bf92f3577b34da6a3ce929d0e0e4736","operation":"describe_source","route":"POST /sources/describe","client":"reports","address":"192.0.2.10","source":"web","status":404,"outcome":"error","code":"DATA_NOT_AVAILABLE","ms":15}
time=2026-11-15T11:12:45.066Z level=warn event=request msg="describe_source failed" requestId=F0iFBi9CZjQ operation=describe_source route="POST /sources/describe" client=reports address=192.0.2.10 source=web status=404 outcome=error code=DATA_NOT_AVAILABLE ms=15
Events¶
event |
When |
|---|---|
info with msg started |
The server listens: version, transport, url, what it serves, the auth mode, sources (how many) and sourceIds, logLevel, logFormat, over HTTP logAddress, and with static tokens their names (tokens), never a value. Ids and names at most 64 characters |
info with msg stopped |
The HTTP server has stopped after the running requests finished or the grace time ended (ms). Written after SIGTERM, before the process ends |
call |
An MCP tool call |
request |
A REST request that reached an operation |
download |
A file handed out: by the REST file route or a download link (via: "link") |
refused |
A request refused before it reached an operation, with its reason |
warning |
Something the operator should look at: a backend that failed, refused connections, a setting that does not fit, a static token's file that cannot be used, with the cause (at most once a minute per token) |
info |
Something that changed: the issuer's metadata loaded, old reports deleted (to make room, or by output.maxAgeDays: one line per sweep, "reports and schedule drafts no longer waiting older than N days deleted", with the counts reports and drafts and maxAgeDays) |
error |
An unexpected error: its class, its message only at debug, never its stack |
debug |
Only at debug: why a token was refused, what a backend answered |
message repeats msg, ok repeats outcome on call lines, and warnings have the event name
warning. Query by msg, outcome and level where you can.
Error codes in the log¶
code on a line is the code the caller got: REST's problem codes and MCP's
tool error codes, with the same names on both. A few to watch:
| Code | What it says about the server |
|---|---|
SOURCE_UNAVAILABLE |
A backend or a source's code failed. The warning line of the same requestId has the cause |
DATA_NOT_AVAILABLE |
The backend refused or did not know the data (401, 403, 404 of a parameterised source) |
TIMEOUT, BUSY |
The compute workers or the queue are at their limit (compute.*) |
INTERNAL |
An unexpected error. The error line of the same requestId has its class, and --log debug its message |
INVALID_TOKEN, RATE_LIMITED |
On refused lines with reason. Many from one address are someone guessing tokens. Behind a proxy, that holds only with http.trustProxy, else address is the proxy for every client. An address of invalid is a forwarded header the proxy should not have passed |
Correlation with a reverse proxy¶
Every answer carries X-Request-Id with the request's own id. A person who quotes the id of an
error finds every line of that request. The id is the server's own: an X-Request-Id the request
came with is logged as upstreamRequestId and never replaces it.
To find a proxy's access log line next to pivotkit's:
- Let the proxy send its own id as
X-Request-Id(nginx:proxy_set_header X-Request-Id $request_id;and$request_idin its log format) and search pivotkit's log forupstreamRequestId. - Or let the proxy log the answer's header (nginx:
$upstream_http_x_request_id) and search forrequestId. - A proxy or service mesh with W3C trace context propagation turned on (its OpenTelemetry tracing)
sends
traceparent. Its trace id becomestraceId, so lines join a trace in your tracing tool. Atraceparentthat does not parse strictly (version 00 with exactly four lower-case hex fields, or a later version) is ignored, and a new trace id is made. - With
http.trustProxy,addressis the client the proxy saw andpeerAddressthe proxy. Without it,addressis the proxy's own address. - When the proxy runs in a container whose port Docker publishes (
ports:in Compose,-p), the proxy itself may see the Docker network's gateway (such as172.17.0.1) instead of the client, depending on how Docker forwards the connection on that host.addressis then the gateway for every client, andhttp.trustProxycannot fix that: the proxy never learned the client's address. Check the proxy's own access log. If it shows the gateway, change how the proxy receives connections (for example host networking for the proxy's container).
Docker log drivers¶
The JSON lines go to Docker's logging driver as they are. Docker's own json-file and local
drivers keep them for docker logs. Give them a size, so they do not fill the disk. To send them to
a log server, use the gelf or syslog driver
(Docker's driver options).
GELF (Graylog, or anything that takes GELF over UDP or TCP), in compose.yml:
services:
pivotkit:
image: ghcr.io/eizedev/pivotkit:<version>
logging:
driver: gelf
options:
gelf-address: "udp://graylog.example.com:12201"
tag: "pivotkit"
docker run --log-driver gelf --log-opt gelf-address=udp://graylog.example.com:12201 --log-opt tag=pivotkit ...
syslog (rsyslog, syslog-ng, a SIEM, a NAS's log center), in RFC 5424 format. Over TLS:
services:
pivotkit:
image: ghcr.io/eizedev/pivotkit:<version>
logging:
driver: syslog
options:
syslog-address: "tcp+tls://logs.example.com:6514"
syslog-format: "rfc5424"
tag: "pivotkit"
To a plain receiver on port 514, over TCP or UDP, whichever the receiver listens on. rfc3164 is
the safe choice for a NAS's log center. Use rfc5424 for a receiver that expects it:
logging:
driver: syslog
options:
syslog-address: "tcp://nas.example:514" # or "udp://nas.example:514"
syslog-format: "rfc3164" # or "rfc5424"
tag: "pivotkit"
On a Synology NAS, Log Center receives syslog on port 514 and files it per sending host. Whether it takes UDP or TCP depends on the receiving rule in Log Center > Log Receiving.
- The Docker syslog driver sends each line as one message with facility
daemonand severityerrfor everything on stderr, where pivotkit logs. This holds whatever pivotkit's ownlevelsays, so map the level from the JSON. - The message text is pivotkit's JSON line.
- A BSD (RFC 3164) receiving rule accepts the driver's
rfc3164andrfc5424output over UDP and TCP. Check one test line in your Log Center before relying on how it shows the tag and severity.
Both drivers put pivotkit's whole JSON line into the message (GELF short_message, the syslog
message text). The driver's own level is the stream's, the same for every line of stderr, so it says
nothing. Let the log server parse the message as JSON and use pivotkit's level (or severity).
In Graylog, add a JSON extractor on the message (or a pipeline rule that parses it as JSON). Use its
level and severity fields instead of the GELF level the driver sets. time stays a text field
until you map it to a date, or use the message's own timestamp, which the driver sets when it
receives the line.
Without a container: journald and rsyslog¶
When the server runs as a systemd service, its stderr goes to the journal. Name the syslog identifier
(SyslogIdentifier). The log stays JSON, since stderr is no terminal there:
# /etc/systemd/system/pivotkit.service (the parts about the log)
[Service]
ExecStart=/usr/bin/npx pivotkit-server serve --config /etc/pivotkit/pivotkit.json
StandardError=journal
SyslogIdentifier=pivotkit
journalctl -u pivotkit -o cat | jq 'select(.level != "info")' # every warning and error
rsyslog reads the journal through its imjournal module and can pass the lines on over TCP. Many
distributions load the module already in /etc/rsyslog.conf. Keep the module line only where
yours does not, since loading it twice is an error.
The template RSYSLOG_SyslogProtocol23Format is rsyslog's IETF draft form (syslog-protocol-23),
close to RFC 5424. RSYSLOG_SyslogRFC5424Format is the strict one, but it ends a message without a
line feed, which a receiver reading lines over TCP may not split:
# /etc/rsyslog.d/pivotkit.conf
module(load="imjournal") # only where /etc/rsyslog.conf does not load it
if $programname == 'pivotkit' then @@logs.example.com:514;RSYSLOG_SyslogProtocol23Format
The state volume¶
When the server has a state directory (/var/lib/pivotkit in the container image,
--state-dir elsewhere), it keeps there:
- in
secrets/, the tokenadmin, the key of the download links and the audit key it made at its first start (made at the first start) - in
out/, the reports, whereoutput.dirnames it
Back the volume up with the rest of the installation. Keep the backup as secret as the tokens: with the files, anyone can call the server and open its download links. A restore brings the same token and keys back. Without one, the server makes new ones, so clients need the new token and older links stop working.
Rotate a value by replacing its file, without a restart:
docker exec pivotkit pivotkit-server token new --file /var/lib/pivotkit/secrets/admin.token --force
docker exec pivotkit pivotkit-server key new --for downloads --file /var/lib/pivotkit/secrets/downloads.key --force
The token's old value gets 401 at once, and links made with the old key stop working within a second. The audit key is read at start, so restart after replacing it. The same person then gets another hash. To make a value new at the next start instead, delete its file.
Kubernetes¶
Kubernetes has no log driver per pod. The container runtime writes each container's stderr to a file
on the node. kubectl logs reads that file, and a log agent of the cluster (Fluent Bit, Vector, the
OpenTelemetry Collector, Grafana Alloy) ships it.
- Keep
log.formatatjson, the image's setting. - Let the agent parse each line as JSON.
- Keep pivotkit's
levelandrequestIdas fields.
See Kubernetes for the deployment itself.
Privacy¶
What the log holds about people, and how long to keep it, is on the privacy page.