Kubernetes¶
This page runs the container image on Kubernetes. One Deployment
serves the REST service at /api/v1 and the MCP server
at /mcp on port 3333, from one configuration file. The example manifest
docker/kubernetes.example.yaml
holds every part shown below. The sections explain each one.
Set it up¶
-
Download the manifest:
curl -fsSLO https://raw.githubusercontent.com/eizedev/pivotkit/pivotkit-server@0.2.0/docker/kubernetes.example.yaml -
Edit your copy. Always set these:
<version>: a release of pivotkit-server, or the image's digest. See which tag.reports.example.com: your host name.- The CronJob's
SOURCE_URL, and the addresses of its host in the NetworkPolicypivotkit-fetch-data. - The namespace of your ingress controller in the NetworkPolicy
pivotkit.
Set these where needed:
- The Ingress's
ingressClassName, when the cluster has no default class. - The certificate's Secret
pivotkit-tls. You make it by hand or with a tool such as cert-manager. - The
storageClassNameofpivotkit-data, when the default class offers no ReadWriteMany.
-
Apply the manifest, fetch the data once, and read the token the server made at its first start:
kubectl apply -f kubernetes.example.yaml kubectl -n pivotkit create job --from=cronjob/pivotkit-fetch-data pivotkit-fetch-data-first kubectl -n pivotkit rollout status deployment/pivotkit TOKEN=$(kubectl -n pivotkit exec deploy/pivotkit -- pivotkit-server token show admin)The server makes the token
adminand the key of the download links on the volumepivotkit-state, so both survive a restart. See made at the first start. The pod's log names the samekubectl execcommand, but never the token. For tokens of your own as Secrets, see managed secrets.
Check it¶
Check it from your machine through a port forward. The port forward sends Host: localhost:3333,
and the server answers that host:
kubectl -n pivotkit port-forward svc/pivotkit 3333:3333 &
curl -s http://localhost:3333/readyz
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:3333/api/v1/sources
kill %1 # ends the port forward
/readyz answers {"ok":true,...}, and /api/v1/sources lists sales. A call to the source
answers 404 FILE_NOT_AVAILABLE until the job has written the file. The job's log shows its
progress: kubectl -n pivotkit logs job/pivotkit-fetch-data-first. Then check the same through the
Ingress:
curl -s -H "Authorization: Bearer $TOKEN" https://reports.example.com/api/v1/sources
Give the token to its users the way you hand out secrets.
Configuration and token¶
The configuration is a ConfigMap mounted at /config, where the image reads
/config/pivotkit.json. Its keys are those of the REST service and
the MCP server:
apiVersion: v1
kind: ConfigMap
metadata:
name: pivotkit-config
namespace: pivotkit
data:
pivotkit.json: |
{
"sources": {
"sales": { "file": "/data/sales.csv", "title": "Sales", "policy": { "fields": { "Region": "show", "Product": "show", "Customer": "hide" } } }
},
"output": { "dir": "/var/lib/pivotkit/out", "maxAgeDays": 7 },
"http": {
"publicUrl": "https://reports.example.com/",
"trustProxy": true
}
}
http.publicUrlis the address clients use. The server answers that host name, the localhost names and the names inhttp.allowHosts. Any otherHost, such as the pod's IP address, gets- Clients inside the cluster use the public address too, through the ingress controller. For the other way, see clients inside the cluster.
http.trustProxymakes the server read the client's address from the last entry ofX-Forwarded-For. The ingress controller writes that header. The address appears asaddressin the log. The log is JSON lines for the cluster's log agent. See operations and logs. Withhttp.trustProxy, only the controller may reach the pod, through the NetworkPolicy below. If other clients can reach the pod, a client could name any address.- The configuration names no token. At its first start the server makes
adminin/var/lib/pivotkit/secrets/, and the key of the download links beside it. To set a new value:kubectl -n pivotkit exec deploy/pivotkit -- pivotkit-server token new --file /var/lib/pivotkit/secrets/admin.token --force. The server reads the file on every request, so the old token gets 401 at once, without a restart. output.diris/var/lib/pivotkit/out, on the same volume.- With OAuth instead of tokens, add
authas shown there. The server then makes no token. - The server does not read a changed ConfigMap again. Restart the Deployment:
kubectl -n pivotkit rollout restart deployment/pivotkit.
Managed secrets¶
You can hand out tokens of your own: several, with scopes or profiles. Keep them in a Secret and name
them in auth.tokens. The server then makes no admin. The image makes the values, so you need no
tool on your machine:
kubectl -n pivotkit exec deploy/pivotkit -- pivotkit-server token new \
| kubectl -n pivotkit create secret generic pivotkit-tokens --from-file=reports=/dev/stdin
Mount the Secret as a directory: /run/secrets/pivotkit, with defaultMode: 0440, readable through
fsGroup. Add "auth": { "mode": "token", "tokens": [ { "name": "reports", "file": "/run/secrets/pivotkit/reports" } ] }
to the ConfigMap. After a restart, kubectl -n pivotkit exec deploy/pivotkit -- pivotkit-server token show reports
prints the token for its users.
The server reads a token file on every request. To rotate a token, run
kubectl create secret ... --dry-run=client -o yaml | kubectl apply -f - with a new token. The
kubelet updates the file after a short delay, its sync period and cache. The new token then works
without a restart. That holds only without subPath: a file mounted with subPath never changes.
A token has at least 32 characters. Keep it out of the manifest and out of environment variables,
since kubectl describe shows those.
The Deployment¶
apiVersion: apps/v1
kind: Deployment
metadata:
name: pivotkit
namespace: pivotkit
labels:
app.kubernetes.io/name: pivotkit
spec:
# one process per state volume: no second pod on pivotkit-state, not even during an update. More
# replicas need one ReadWriteMany volume they all share (per-pod volumes would each make their own token)
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: pivotkit
template:
metadata:
labels:
app.kubernetes.io/name: pivotkit
spec:
# pivotkit calls no Kubernetes API
automountServiceAccountToken: false
# else Kubernetes sets PIVOTKIT_PORT=tcp://<address>:3333 for the Service named pivotkit
enableServiceLinks: false
securityContext:
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
# the server lets running reports finish for up to 25 seconds (http.graceMs) after SIGTERM
terminationGracePeriodSeconds: 30
containers:
- name: pivotkit
image: ghcr.io/eizedev/pivotkit:<version>
ports:
- name: http
containerPort: 3333
env:
- name: PIVOTKIT_PORT
value: "3333"
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
# compute.workers (2) x (compute.memoryMb (1024) + 64 + the file read, up to 50 MB) + the
# main process; see the sizing section of the docs page
resources:
requests:
cpu: 500m
memory: 2560Mi
limits:
memory: 2560Mi
# a probe sends the pod's address as Host, which the server refuses: send localhost
startupProbe:
httpGet:
path: /healthz
port: http
httpHeaders:
- name: Host
value: localhost
periodSeconds: 2
failureThreshold: 30
readinessProbe:
httpGet:
path: /readyz
port: http
httpHeaders:
- name: Host
value: localhost
periodSeconds: 10
timeoutSeconds: 6
livenessProbe:
httpGet:
path: /healthz
port: http
httpHeaders:
- name: Host
value: localhost
periodSeconds: 30
timeoutSeconds: 6
volumeMounts:
- name: config
mountPath: /config
readOnly: true
- name: data
mountPath: /data
readOnly: true
- name: state
mountPath: /var/lib/pivotkit
- name: tmp
mountPath: /tmp
volumes:
- name: config
configMap:
name: pivotkit-config
- name: data
persistentVolumeClaim:
claimName: pivotkit-data
readOnly: true
- name: state
persistentVolumeClaim:
claimName: pivotkit-state
- name: tmp
emptyDir:
sizeLimit: 64Mi
What each setting is for:
| Setting | Why |
|---|---|
replicas: 1, strategy: Recreate |
One server process per state volume (pivotkit-state, ReadWriteOnce). One process keeps the count of the reports' files and cleans them up (output directory). A rolling update would run two pods on the volume for a while. |
enableServiceLinks: false |
With service links on, Kubernetes sets PIVOTKIT_PORT=tcp://<address>:3333 in every pod of the namespace for the Service named pivotkit. The server then stops with PIVOTKIT_PORT is "tcp://...", which Kubernetes sets for a Service named pivotkit. The example also sets PIVOTKIT_PORT to 3333. |
automountServiceAccountToken: false |
pivotkit calls no Kubernetes API. |
User and group 10001, fsGroup: 10001 |
10001 is the image's user. fsGroup makes the volumes readable and /var/lib/pivotkit writable for it. It also makes a Secret's files with mode 0440 readable, see managed secrets. fsGroupChangePolicy: OnRootMismatch changes the owner of a volume's files only when its top directory does not match yet. So a start does not walk a large volume every time. |
The namespace's label pod-security.kubernetes.io/enforce: restricted |
Kubernetes refuses any pod in the namespace that does not meet the restricted Pod Security Standard: non-root, no privilege escalation, no capabilities, a seccomp profile. Every pod of the example meets it. |
readOnlyRootFilesystem, no capabilities, RuntimeDefault seccomp |
The same as --read-only --cap-drop ALL in Run it. /tmp is an emptyDir. |
Probes with Host: localhost |
A probe sends the pod's IP address as Host, which the server refuses. /healthz answers while the process runs, for liveness and startup. /readyz answers 200 only when the server can take work: the OAuth issuer is loaded, the workers are up and the output directory is writable (Health). |
terminationGracePeriodSeconds: 30 |
After SIGTERM the server takes no new connection. It lets running reports finish for up to 25 seconds (http.graceMs). |
A source whose file is missing does not stop the server or /readyz. A call to that source answers
404 FILE_NOT_AVAILABLE until the file is there. pivotkit-server check-config reports such a file
as a PROBLEM the server starts with, and exits with 1. Run it with the flags of the container's
serve: --host 0.0.0.0, and --enable where you set it. Otherwise it checks another surface.
More than one replica¶
- Each process counts the limits on callers in its own memory:
rest.rateLimit,mcp.rateLimitand the refused tokens per address. With more than one replica, each counts on its own. - The MCP endpoint keeps no sessions, so any replica can answer any request.
- Replicas need one state volume they all share (ReadWriteMany). Replicas that start together end with one token and one link key. A download link then works on every replica.
- pivotkit does not support a volume per pod, such as a StatefulSet's volume claim template, with
the token made at the first start. Each pod would make its own. Give such replicas tokens and a
link key of your own instead: managed secrets and
"output": { "downloads": { "key": { "file": "/run/secrets/download-key" } } }. The server reads that key file again within a second of a change.
Memory and workers¶
A pool of worker threads computes the reports over a file. compute.* in
the configuration sets it up:
| Key | Default | What it does |
|---|---|---|
compute.workers |
2 (at most 16) | Reports computed at once |
compute.memoryMb |
1024 | The JavaScript heap a worker may use. A report that needs more fails with LIMIT_EXCEEDED, and the server replaces the worker |
compute.timeMs |
30000 | The longest a report may compute (TIMEOUT) |
compute.queueMs |
10000 | How long a call waits for a free worker before BUSY |
limits.fileBytes |
50 MB | The largest file the server reads |
compute.memoryMb bounds a worker's JavaScript heap, but not all of its memory:
- The bound covers the old generation. The young generation adds up to 64 MB.
- The bytes of the file the worker reads lie outside the heap.
- The encoded columns lie outside it too. They are typed arrays, up to 12 bytes per row and column.
- Each worker keeps the last four data sets it read, so its memory stays up after a report instead of going back down. The encoded columns of all four count.
As a starting point for the container's memory limit:
memory limit = main process + compute.workers x (compute.memoryMb + 64 MB + limits.fileBytes)
With the defaults that is about 150 MB + 2 x (1024 + 64 + 50) MB, about 2.4 GB. The example sets 2560Mi. Large files with many columns need more than this for the encoded columns the workers keep. Watch the pod's memory with your own data.
With a limit below the sum, a report that alone would fail with LIMIT_EXCEEDED becomes an OOM
kill of the whole pod, with every report running in it. For a smaller pod, lower
compute.memoryMb or the workers, not the limit alone. With compute.memoryMb 512 and 2 workers,
a limit of about 1.4 GB is a starting point.
With code or HTTP sources, the main process also keeps up to sourceCache.maxMb of loaded data,
256 MiB by default (bounds). Add it to the limit, or lower
sourceCache.maxMb.
For an idea of real use, here is one measurement. It ran once with Node 26 on macOS, not with the
image. Take it as a rough order and check your own data with kubectl top pod. The source was a CSV
file of 30 MB with 700,000 rows and 6 columns. The process used about 100 MB without work. Each
worker that had computed a report over the file added about 470 MB.
A report runs on one worker thread. Give the pod about one CPU per worker that should compute at the same time. The example requests half a CPU and sets no CPU limit.
Data in /data¶
The server reads its sources from files. On Kubernetes, something has to put them there.
The server notices a new file by its identity, size and time. So replace a file with a new one in one of these ways:
- Write it under another name in the same directory and rename it (
mv). - Flip a symbolic link to a new directory.
Never write into the file the server reads: a read in between gets half of it. The user 10001 must be able to read the files.
A CronJob on a ReadWriteMany volume¶
A job fetches the data on a schedule. It writes the data to a volume that the pivotkit pods mount
read-only. The job's pod and the server's pods may run on different nodes. So the volume must allow
ReadWriteMany, as NFS, CephFS or a cloud provider's file storage do. If the cluster's default class
does not allow it, set storageClassName to a class that does.
# The data: written by the CronJob, read by every pivotkit pod, so ReadWriteMany (NFS, CephFS, a cloud
# provider's file storage). Set storageClassName to such a class where the default class is not one.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pivotkit-data
namespace: pivotkit
spec:
accessModes: ["ReadWriteMany"]
resources:
requests:
storage: 1Gi
---
# The state: the token and the download links' key the server makes at its first start (secrets/),
# and the reports render_view writes (out/). One server process per volume.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pivotkit-state
namespace: pivotkit
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 2Gi
apiVersion: batch/v1
kind: CronJob
metadata:
name: pivotkit-fetch-data
namespace: pivotkit
spec:
schedule: "15 * * * *"
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 1
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 2
activeDeadlineSeconds: 600
template:
metadata:
labels:
app.kubernetes.io/name: pivotkit-fetch-data
spec:
restartPolicy: Never
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
fsGroupChangePolicy: OnRootMismatch
seccompProfile:
type: RuntimeDefault
containers:
- name: fetch
image: curlimages/curl:8.23.0
env:
- name: SOURCE_URL
value: https://data.example.com/exports/sales.csv
# HTTPS only (no redirects, which would leave an empty file), at most limits.fileBytes,
# only a 200 with content replaces the data: a new name, then a rename, so the server
# never reads half a file; a credential comes from a file, never from the URL or env
command:
- sh
- -c
- |
set -eu
tmp=/data/.sales.csv.tmp
trap 'rm -f "$tmp"' EXIT
set --
if [ -f /run/secrets/source/header ]; then set -- -H @/run/secrets/source/header; fi
code=$(curl --fail --silent --show-error --proto '=https' --max-time 300 \
--max-filesize 52428800 --write-out '%{http_code}' -o "$tmp" "$@" "$SOURCE_URL")
test "$code" = 200
test -s "$tmp"
chmod 0644 "$tmp"
mv -f "$tmp" /data/sales.csv
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
memory: 128Mi
volumeMounts:
- name: data
mountPath: /data
- name: source-secret
mountPath: /run/secrets/source
readOnly: true
volumes:
- name: data
persistentVolumeClaim:
claimName: pivotkit-data
# optional: a header such as "Authorization: Bearer ..." for the data's host
- name: source-secret
secret:
secretName: pivotkit-source
optional: true
defaultMode: 0440
What the job does:
- It takes only HTTPS (
--proto '=https'). - It follows no redirects. A redirect would give an empty file.
- It stops at 50 MB. That is
limits.fileBytes, the largest file the server reads. - It replaces the data only after an answer 200 with content. On any failure the old file stays, and the job removes the temporary one.
When the data's host needs a credential, put the header line into the Secret pivotkit-source. The
job mounts it as a file and sends it with -H @file. Never put the credential in SOURCE_URL or in
an environment variable, since kubectl describe shows those:
printf 'Authorization: Bearer %s\n' "$(cat source.token)" > header
kubectl -n pivotkit create secret generic pivotkit-source --from-file=header=./header
rm header
The job runs as the user 10001 too, so the server can read the files it writes. Some NFS servers
ignore fsGroup. There the user 10001 must already be able to write to the volume's directory. The
storage's administrator sets that, or a provisioner option does.
Fetch the first data right after you apply the manifest, with kubectl create job --from=cronjob/...
as above. Otherwise the source is missing until the schedule runs the job. The job's image and
command are an example. Any program that writes the file and renames it does the job.
A sidecar with an emptyDir¶
A second container in the pod fetches the data into an emptyDir. The server's container mounts it
read-only. You need no ReadWriteMany storage. Each pod has its own copy and fetches it again after a
restart.
The example uses git-sync (v4). It clones a Git repository and flips a symbolic link to each new commit. Start from the example manifest:
- Remove the CronJob, its NetworkPolicy
pivotkit-fetch-dataand the volume claimpivotkit-data. - In the ConfigMap, point the source at the link.
- Replace the Deployment's
datavolume, and add the container and its/tmp.
"sales": { "file": "/data/git/current/sales.csv", "title": "Sales", "policy": { "fields": { "Region": "show", "Product": "show", "Customer": "hide" } } }
containers:
- name: pivotkit
# as in the example; its mount of data at /data stays readOnly: true
- name: git-sync
image: registry.k8s.io/git-sync/git-sync:v4.8.0
args:
- --repo=https://git.example.com/reports/data.git
- --root=/data/git
- --link=current
- --period=5m
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
memory: 256Mi
volumeMounts:
- name: data
mountPath: /data
# the image's HOME
- name: git-tmp
mountPath: /tmp
volumes:
- name: data
emptyDir:
sizeLimit: 1Gi
- name: git-tmp
emptyDir:
sizeLimit: 64Mi
Mount the volume at the same path in both containers, so the link resolves the same way in each.
The server follows the link on every call. The next report reads the new commit after a flip. A call that reads the file in the moment of the flip may fail once.
git-sync's image sets HOME to /tmp. With the read-only root file system, /tmp therefore needs
the emptyDir.
If you use files.allowDirs, the server resolves it once at start. It does not start while a
directory named there is missing, so name the volume's mount (/data), which exists from the start.
Do not name a directory git-sync makes later (/data/git) or the link (/data/git/current).
A private repository needs credentials: git-sync's --password-file or --ssh-key-file, from a
Secret. Over SSH, git may need a passwd entry for the user 10001. git-sync's --add-user adds one.
That was not tried here.
The policy pivotkit needs egress to the Git server on port 443, as for
the issuer. The policy holds for both containers of the pod.
This sidecar setup follows git-sync's documentation and passes the schema check. It has not been run in a cluster here, so check the flags against the documentation of the version you run. Any other sidecar works the same way when it renames or flips a link.
The images of the CronJob and the sidecar are third-party images pinned by tag. Pin them by digest
(image@sha256:...) where you need the exact image every time, as for pivotkit's own.
Service and Ingress¶
apiVersion: v1
kind: Service
metadata:
name: pivotkit
namespace: pivotkit
spec:
selector:
app.kubernetes.io/name: pivotkit
ports:
- name: http
port: 3333
targetPort: http
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: pivotkit
namespace: pivotkit
spec:
# ingressClassName: the class of your controller, where the cluster has no default class
tls:
- hosts: ["reports.example.com"]
secretName: pivotkit-tls
rules:
- host: reports.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: pivotkit
port:
name: http
The Ingress sends everything on the host to the Service. The one process serves all of it: /mcp,
/api/v1, /openapi.json, the /.well-known/ addresses of OAuth and
the download links under /downloads/.
These answer without a token:
- the quick-start page at
/.http.page: falseturns it off. /healthzand/readyz- the download links under
/downloads/. The link is the credential.
Everything else needs a token.
The Ingress needs an ingress controller in the cluster. Where the cluster has no default class, it
also needs ingressClassName. pivotkit-tls is the certificate's Secret. You make it by hand or
with a tool such as cert-manager.
The controller must pass the request's Host on unchanged and write X-Forwarded-For. Its time
limit for an answer must be longer than compute.queueMs + compute.timeMs, which is 40 seconds
with the defaults. Check your controller's default.
To publish less, give the Ingress the paths /mcp, /api/v1 and /downloads instead of /. With
OAuth, add the prefix /.well-known/oauth-protected-resource.
The ingress controller may sit behind a LoadBalancer Service. Then the client's address reaches it
only with externalTrafficPolicy: Local, or with the PROXY protocol, depending on the controller.
Otherwise X-Forwarded-For holds a node's address, and the server's per-address limits count the
node.
NetworkPolicy¶
# Nothing in the namespace talks to anything unless a policy below allows it.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: pivotkit
spec:
podSelector: {}
policyTypes: ["Ingress", "Egress"]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: pivotkit
namespace: pivotkit
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: pivotkit
policyTypes: ["Ingress", "Egress"]
ingress:
# the ingress controller only: http.trustProxy believes its X-Forwarded-For
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: ingress-controller
ports:
- port: 3333
protocol: TCP
egress:
# name lookups
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP
---
# The CronJob's pods: name lookups and the data's host on 443, nothing coming in.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: pivotkit-fetch-data
namespace: pivotkit
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: pivotkit-fetch-data
policyTypes: ["Ingress", "Egress"]
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- port: 53
protocol: UDP
- port: 53
protocol: TCP
# the addresses data.example.com resolves to
- to:
- ipBlock:
cidr: 192.0.2.20/32
ports:
- port: 443
protocol: TCP
The three policies:
default-denystops all traffic of every pod in the namespace that the other policies do not allow.pivotkitlets the ingress controller in and name lookups out.pivotkit-fetch-datalets the job's pods look up names and reach the data's host on port 443. Put the addresses its host name resolves to there, since a policy cannot name a host.
Policies take effect only with a network plugin that enforces NetworkPolicy.
The namespace selector uses the label kubernetes.io/metadata.name, which Kubernetes sets on every
namespace. Put the name of your ingress controller's namespace there. ingress-controller is a
placeholder.
A controller that runs with hostNetwork arrives from the nodes' addresses instead of its pods'
addresses. Then allow the nodes' address range there as an ipBlock. Check how your cluster's
controller runs.
The kubelet's probes come from the node. Most network plugins let them through, whatever the policy says. If the pod never gets ready with the policy applied, check your plugin.
Clients inside the cluster¶
With the policy as it is, programs in the cluster call the public address through the ingress
controller. To let them call the Service directly (http://pivotkit.pivotkit.svc:3333):
- Allow their namespace in the policy's
ingress. - Add the names they call to
http.allowHosts:pivotkit,pivotkit.pivotkit.svcandpivotkit.pivotkit.svc.cluster.local. - Turn
http.trustProxyoff.
A direct caller can write any X-Forwarded-For. With trustProxy, the server would believe it,
and its limits per address count by that header. With trustProxy off, every request through the
controller counts with the controller's address.
Egress¶
With static tokens the server opens no connection of its own. The policy allows name lookups and
nothing else. Name lookups go to the cluster DNS, which is k8s-app: kube-dns in kube-system on
most clusters.
Clusters whose DNS is elsewhere need another rule for it. For example, NodeLocal DNSCache answers on
an address of the node, and OpenShift runs its DNS in the namespace openshift-dns.
Other features need more egress:
- OAuth needs the issuer too. See below.
- An HTTP source needs the addresses of its backend on port 443, as the issuer does.
- It also needs the addresses of its token endpoint when it signs in with client credentials or a token exchange. See backend sign-in.
Cluster addresses are private ones, so something inside the cluster needs a setting:
- A backend needs
intranet: trueon it. - A token endpoint needs
intranet: truein its sign-in's settings. - An issuer needs
auth.intranet: true. See below.
Sign-in with Authentik¶
With OAuth, the MCP endpoint takes access tokens of your identity provider instead of static tokens. It works with any provider that issues JWT access tokens. OAuth 2.1 has every key. With Authentik:
- Under Customization > Property Mappings, create a Scope Mapping. Give it the Scope name
pivotkitand an expression that returns{}. The scope adds no claims. Authentik drops a scope a client asks for when the provider has no scope mapping for it. A token withoutpivotkitthen gets 403. - Under Applications > Applications, choose New Application. Give it the slug
pivotkitand the provider type OAuth2/OIDC, with these provider settings:- Client Type: Public. An MCP client keeps no secret. It signs in with PKCE.
- Redirect URIs/Origins (RegEx): the callback addresses of the MCP clients you allow. Take
them from the clients' documentation. Give each the matching mode Strict, which matches
the address exactly. With Regex, the whole address must match the expression. Then
escape the dots (
https://app\.example\.com/callback) and use no.*. Otherwise a sign-in can be sent to an address you did not mean. - Signing Key: a certificate with one of the keys the server accepts by default
(
auth.algorithms): an RSA key (RS256), an EC P-256 key (ES256) or an Ed25519 key (EdDSA). For the other EC curves, Authentik signs with ES384 or ES512. Without a signing key, Authentik signs with HS256 and the client secret, which the server refuses. - Encryption Key: none. With one, Authentik encrypts the access token (JWE), and the server cannot read it.
- Access Token Validity: minutes, for example
minutes=10. The default is one hour. A token stays valid until it expires. - Scopes: add the
pivotkitmapping to the selected scopes.
- On the application's Policy / Group / User Bindings, bind the group that may use pivotkit (Bind existing group/user). You can bind users or a policy instead. An application without any binding lets every user of Authentik sign in to it.
- Note the provider's Client ID.
The server accepts every access token this provider issues for its audience with the required scopes. So Authentik decides who gets a token: by the bindings of step 3 and by the scopes.
The issuer is https://auth.example.com/application/o/pivotkit/. It ends with the application's
slug. That holds with the default issuer mode, where each provider has its own issuer. Authentik
puts the client ID into the access token's aud, and the granted scopes into its scope claim. The
auth part of the configuration:
{
"auth": { "mode": "oauth",
"issuer": "https://auth.example.com/application/o/pivotkit/",
"resource": "https://reports.example.com/mcp",
"audience": ["<the provider's client id>"],
"requiredScopes": ["pivotkit"] },
"http": { "publicUrl": "https://reports.example.com/", "trustProxy": true, "enable": ["mcp"] }
}
requiredScopesalso keeps Authentik's ID tokens out: they carry the sameaudbut noscope. Do not setauth.requireTyp: Authentik's access tokens carry notypofat+jwt.http.enable: ["mcp"]serves the MCP endpoint only. The REST service is a resource of its own, and its tokens must carry its audience (REST sign-in). With Authentik, every token of the provider carries the same client ID. To serve REST from the same provider, setrest.audienceto that client ID. Give REST a scope of its own withrest.requiredScopesand a second scope mapping. A token with both scopes then works on both endpoints.
Egress to the issuer¶
The server loads the issuer's metadata and keys over HTTPS. Until then, /readyz answers 503. A
NetworkPolicy cannot name a host, so allow the addresses that auth.example.com resolves to from
inside the cluster, on port 443. Add the rule to the policy's egress:
- to:
- ipBlock:
cidr: 192.0.2.10/32
ports:
- port: 443
protocol: TCP
Authentik may run in the same cluster, with its host name resolving to the ingress controller. Then
allow the controller's namespace instead of an ipBlock.
An issuer whose name resolves to a cluster address or another private address also needs
"intranet": true in auth. Without it, the server refuses loopback and private addresses for the
issuer, and check-config names the setting. The same setting covers the token endpoint of the
download sign-in.
These settings are based on the following sources, checked in November 2026:
- Authentik's OAuth2 provider documentation: issuer, signing key, PKCE.
- Authentik's source code at the release
version/2026.8.5:authentik/providers/oauth2/: the access token'saud,scopeand header, the signing algorithm per key, the default validity, scopes without a mapping, and redirect addresses matched in full.authentik/policies/engine.py: an application without bindings passes everyone.web/src/admin/: the labels of the forms.
Labels and defaults can change between Authentik releases. It was not checked whether Authentik
reads the resource parameter of RFC 8707. The server
does not need it, since it accepts the client ID as audience. Decode an access token of your own to
see what your setup issues.
Update¶
To update, write the new tag or digest into the Deployment and apply it again. See
Kubernetes in Update an installation. With strategy: Recreate, the old
pod stops before the new one starts.
The whole manifest¶
docker/kubernetes.example.yaml holds, in this order:
- the Namespace
- the ConfigMap
- the two volume claims
- the Deployment
- the Service
- the Ingress
- the three NetworkPolicies
- the CronJob
To check your changed copy against the Kubernetes API schemas without a cluster, use
kubeconform: kubeconform -strict -summary kubernetes.example.yaml.
With a cluster, use kubectl apply --dry-run=server.