# Kubernetes

This page runs the [container image](<https://pivotkit.eize.dev/reference/container/index.md>) on Kubernetes. One Deployment serves the [REST service](<https://pivotkit.eize.dev/reference/rest/index.md>) at `/api/v1` and the [MCP server](<https://pivotkit.eize.dev/reference/mcp/index.md>) at `/mcp` on port 3333, from one configuration file. The example manifest [docker/kubernetes.example.yaml](<https://github.com/eizedev/pivotkit/blob/pivotkit-server@0.2.0/docker/kubernetes.example.yaml>) holds every part shown below. The sections explain each one.

## Set it up

1. Download the manifest:

   ```
   curl -fsSLO https://raw.githubusercontent.com/eizedev/pivotkit/pivotkit-server@0.2.0/docker/kubernetes.example.yaml
   ```
2. Edit your copy. Always set these:

   - `<version>`: a release of pivotkit-server, or the image's digest. See [which tag](<https://pivotkit.eize.dev/getting-started/upgrading/#update-image>).
   - `reports.example.com`: your host name.
   - The CronJob's `SOURCE_URL`, and the addresses of its host in the NetworkPolicy `pivotkit-fetch-data`.
   - The namespace of your ingress controller in the NetworkPolicy `pivotkit`.

   Set these where needed:

   - The Ingress's `ingressClassName`, when the cluster has no default class.
   - The certificate's Secret `pivotkit-tls`. You make it by hand or with a tool such as cert-manager.
   - The `storageClassName` of `pivotkit-data`, when the default class offers no ReadWriteMany.
3. Apply the manifest, fetch the data once, and read the token the server made at its first start:

   ```
   kubectl apply -f kubernetes.example.yaml
   kubectl -n pivotkit create job --from=cronjob/pivotkit-fetch-data pivotkit-fetch-data-first
   kubectl -n pivotkit rollout status deployment/pivotkit
   TOKEN=$(kubectl -n pivotkit exec deploy/pivotkit -- pivotkit-server token show admin)
   ```

   The server makes the token `admin` and the key of the [download links](<https://pivotkit.eize.dev/reference/delivery/index.md>) on the volume `pivotkit-state`, so both survive a restart. See [made at the first start](<https://pivotkit.eize.dev/reference/rest/#first-start>). The pod's log names the same `kubectl exec` command, but never the token. For tokens of your own as Secrets, see [managed secrets](<#managed-secrets>).

## Check it

Check it from your machine through a port forward. The port forward sends `Host: localhost:3333`, and the server answers that host:

```
kubectl -n pivotkit port-forward svc/pivotkit 3333:3333 &
curl -s http://localhost:3333/readyz
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:3333/api/v1/sources
kill %1   # ends the port forward
```

`/readyz` answers `{"ok":true,...}`, and `/api/v1/sources` lists `sales`. A call to the source answers 404 `FILE_NOT_AVAILABLE` until the job has written the file. The job's log shows its progress: `kubectl -n pivotkit logs job/pivotkit-fetch-data-first`. Then check the same through the Ingress:

```
curl -s -H "Authorization: Bearer $TOKEN" https://reports.example.com/api/v1/sources
```

Give the token to its users the way you hand out secrets.

## Configuration and token

The configuration is a ConfigMap mounted at `/config`, where the image reads `/config/pivotkit.json`. Its keys are those of [the REST service](<https://pivotkit.eize.dev/reference/rest/#running>) and [the MCP server](<https://pivotkit.eize.dev/reference/mcp/#configuration>):

```
apiVersion: v1
kind: ConfigMap
metadata:
  name: pivotkit-config
  namespace: pivotkit
data:
  pivotkit.json: |
    {
      "sources": {
        "sales": { "file": "/data/sales.csv", "title": "Sales", "policy": { "fields": { "Region": "show", "Product": "show", "Customer": "hide" } } }
      },
      "output": { "dir": "/var/lib/pivotkit/out", "maxAgeDays": 7 },
      "http": {
        "publicUrl": "https://reports.example.com/",
        "trustProxy": true
      }
    }
```

- `http.publicUrl` is the address clients use. The server answers that host name, the localhost names and the names in `http.allowHosts`. Any other `Host`, such as the pod's IP address, gets
- Clients inside the cluster use the public address too, through the ingress controller. For the other way, see [clients inside the cluster](<#in-cluster>).
- `http.trustProxy` makes the server read the client's address from the last entry of `X-Forwarded-For`. The ingress controller writes that header. The address appears as `address` in the log. The log is JSON lines for the cluster's log agent. See [operations and logs](<https://pivotkit.eize.dev/reference/operations/#kubernetes>). With `http.trustProxy`, only the controller may reach the pod, through the [NetworkPolicy](<#networkpolicy>) below. If other clients can reach the pod, a client could name any address.
- The configuration names no token. At its first start the server makes `admin` in `/var/lib/pivotkit/secrets/`, and the key of the download links beside it. To set a new value: `kubectl -n pivotkit exec deploy/pivotkit -- pivotkit-server token new --file /var/lib/pivotkit/secrets/admin.token --force`. The server reads the file on every request, so the old token gets 401 at once, without a restart.
- `output.dir` is `/var/lib/pivotkit/out`, on the same volume.
- With [OAuth](<#oauth>) instead of tokens, add `auth` as shown there. The server then makes no token.
- The server does not read a changed ConfigMap again. Restart the Deployment: `kubectl -n pivotkit rollout restart deployment/pivotkit`.

### Managed secrets

You can hand out tokens of your own: several, with scopes or profiles. Keep them in a Secret and name them in `auth.tokens`. The server then makes no `admin`. The image makes the values, so you need no tool on your machine:

```
kubectl -n pivotkit exec deploy/pivotkit -- pivotkit-server token new \
  | kubectl -n pivotkit create secret generic pivotkit-tokens --from-file=reports=/dev/stdin
```

Mount the Secret as a directory: `/run/secrets/pivotkit`, with `defaultMode: 0440`, readable through `fsGroup`. Add `"auth": { "mode": "token", "tokens": [ { "name": "reports", "file": "/run/secrets/pivotkit/reports" } ] }` to the ConfigMap. After a restart, `kubectl -n pivotkit exec deploy/pivotkit -- pivotkit-server token show reports` prints the token for its users.

The server reads a token file on every request. To rotate a token, run `kubectl create secret ... --dry-run=client -o yaml | kubectl apply -f -` with a new token. The kubelet updates the file after a short delay, its sync period and cache. The new token then works without a restart. That holds only without `subPath`: a file mounted with `subPath` never changes.

A token has at least 32 characters. Keep it out of the manifest and out of environment variables, since `kubectl describe` shows those.

## The Deployment

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pivotkit
  namespace: pivotkit
  labels:
    app.kubernetes.io/name: pivotkit
spec:
  # one process per state volume: no second pod on pivotkit-state, not even during an update. More
  # replicas need one ReadWriteMany volume they all share (per-pod volumes would each make their own token)
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: pivotkit
  template:
    metadata:
      labels:
        app.kubernetes.io/name: pivotkit
    spec:
      # pivotkit calls no Kubernetes API
      automountServiceAccountToken: false
      # else Kubernetes sets PIVOTKIT_PORT=tcp://<address>:3333 for the Service named pivotkit
      enableServiceLinks: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        runAsGroup: 10001
        fsGroup: 10001
        fsGroupChangePolicy: OnRootMismatch
        seccompProfile:
          type: RuntimeDefault
      # the server lets running reports finish for up to 25 seconds (http.graceMs) after SIGTERM
      terminationGracePeriodSeconds: 30
      containers:
        - name: pivotkit
          image: ghcr.io/eizedev/pivotkit:<version>
          ports:
            - name: http
              containerPort: 3333
          env:
            - name: PIVOTKIT_PORT
              value: "3333"
          securityContext:
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          # compute.workers (2) x (compute.memoryMb (1024) + 64 + the file read, up to 50 MB) + the
          # main process; see the sizing section of the docs page
          resources:
            requests:
              cpu: 500m
              memory: 2560Mi
            limits:
              memory: 2560Mi
          # a probe sends the pod's address as Host, which the server refuses: send localhost
          startupProbe:
            httpGet:
              path: /healthz
              port: http
              httpHeaders:
                - name: Host
                  value: localhost
            periodSeconds: 2
            failureThreshold: 30
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
              httpHeaders:
                - name: Host
                  value: localhost
            periodSeconds: 10
            timeoutSeconds: 6
          livenessProbe:
            httpGet:
              path: /healthz
              port: http
              httpHeaders:
                - name: Host
                  value: localhost
            periodSeconds: 30
            timeoutSeconds: 6
          volumeMounts:
            - name: config
              mountPath: /config
              readOnly: true
            - name: data
              mountPath: /data
              readOnly: true
            - name: state
              mountPath: /var/lib/pivotkit
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: config
          configMap:
            name: pivotkit-config
        - name: data
          persistentVolumeClaim:
            claimName: pivotkit-data
            readOnly: true
        - name: state
          persistentVolumeClaim:
            claimName: pivotkit-state
        - name: tmp
          emptyDir:
            sizeLimit: 64Mi
```

What each setting is for:

| Setting | Why |
| --- | --- |
| `replicas: 1`, `strategy: Recreate` | One server process per state volume (`pivotkit-state`, ReadWriteOnce). One process keeps the count of the reports' files and cleans them up ([output directory](<https://pivotkit.eize.dev/reference/rest/#auth>)). A rolling update would run two pods on the volume for a while. |
| `enableServiceLinks: false` | With service links on, Kubernetes sets `PIVOTKIT_PORT=tcp://<address>:3333` in every pod of the namespace for the Service named `pivotkit`. The server then stops with `PIVOTKIT_PORT is "tcp://...", which Kubernetes sets for a Service named pivotkit`. The example also sets `PIVOTKIT_PORT` to 3333. |
| `automountServiceAccountToken: false` | pivotkit calls no Kubernetes API. |
| User and group 10001, `fsGroup: 10001` | 10001 is the image's user. `fsGroup` makes the volumes readable and `/var/lib/pivotkit` writable for it. It also makes a Secret's files with mode 0440 readable, see [managed secrets](<#managed-secrets>). `fsGroupChangePolicy: OnRootMismatch` changes the owner of a volume's files only when its top directory does not match yet. So a start does not walk a large volume every time. |
| The namespace's label `pod-security.kubernetes.io/enforce: restricted` | Kubernetes refuses any pod in the namespace that does not meet the restricted Pod Security Standard: non-root, no privilege escalation, no capabilities, a seccomp profile. Every pod of the example meets it. |
| `readOnlyRootFilesystem`, no capabilities, `RuntimeDefault` seccomp | The same as `--read-only --cap-drop ALL` in [Run it](<https://pivotkit.eize.dev/reference/container/#run>). `/tmp` is an `emptyDir`. |
| Probes with `Host: localhost` | A probe sends the pod's IP address as `Host`, which the server refuses. `/healthz` answers while the process runs, for liveness and startup. `/readyz` answers 200 only when the server can take work: the OAuth issuer is loaded, the workers are up and the output directory is writable ([Health](<https://pivotkit.eize.dev/reference/container/#health>)). |
| `terminationGracePeriodSeconds: 30` | After SIGTERM the server takes no new connection. It lets running reports finish for up to 25 seconds (`http.graceMs`). |

A source whose file is missing does not stop the server or `/readyz`. A call to that source answers 404 `FILE_NOT_AVAILABLE` until the file is there. `pivotkit-server check-config` reports such a file as a `PROBLEM` the server starts with, and exits with 1. Run it with the flags of the container's `serve`: `--host 0.0.0.0`, and `--enable` where you set it. Otherwise it checks another surface.

### More than one replica

- Each process counts the limits on callers in its own memory: `rest.rateLimit`, `mcp.rateLimit` and the refused tokens per address. With more than one replica, each counts on its own.
- The MCP endpoint keeps no sessions, so any replica can answer any request.
- Replicas need one state volume they all share (ReadWriteMany). Replicas that start together end with one token and one link key. A [download link](<https://pivotkit.eize.dev/reference/delivery/index.md>) then works on every replica.
- pivotkit does not support a volume per pod, such as a StatefulSet's volume claim template, with the token made at the first start. Each pod would make its own. Give such replicas tokens and a link key of your own instead: [managed secrets](<#managed-secrets>) and `"output": { "downloads": { "key": { "file": "/run/secrets/download-key" } } }`. The server reads that key file again within a second of a change.

## Memory and workers

A pool of worker threads computes the reports over a file. `compute.*` in [the configuration](<https://pivotkit.eize.dev/reference/mcp/#configuration>) sets it up:

| Key | Default | What it does |
| --- | --- | --- |
| `compute.workers` | 2 (at most 16) | Reports computed at once |
| `compute.memoryMb` | 1024 | The JavaScript heap a worker may use. A report that needs more fails with `LIMIT_EXCEEDED`, and the server replaces the worker |
| `compute.timeMs` | 30000 | The longest a report may compute (`TIMEOUT`) |
| `compute.queueMs` | 10000 | How long a call waits for a free worker before `BUSY` |
| `limits.fileBytes` | 50 MB | The largest file the server reads |

`compute.memoryMb` bounds a worker's JavaScript heap, but not all of its memory:

- The bound covers the old generation. The young generation adds up to 64 MB.
- The bytes of the file the worker reads lie outside the heap.
- The encoded columns lie outside it too. They are typed arrays, up to 12 bytes per row and column.
- Each worker keeps the last four data sets it read, so its memory stays up after a report instead of going back down. The encoded columns of all four count.

As a starting point for the container's memory limit:

```
memory limit  =  main process  +  compute.workers x (compute.memoryMb + 64 MB + limits.fileBytes)
```

With the defaults that is about 150 MB + 2 x (1024 + 64 + 50) MB, about 2.4 GB. The example sets 2560Mi. Large files with many columns need more than this for the encoded columns the workers keep. Watch the pod's memory with your own data.

With a limit below the sum, a report that alone would fail with `LIMIT_EXCEEDED` becomes an OOM kill of the whole pod, with every report running in it. For a smaller pod, lower `compute.memoryMb` or the workers, not the limit alone. With `compute.memoryMb` 512 and 2 workers, a limit of about 1.4 GB is a starting point.

With code or HTTP sources, the main process also keeps up to `sourceCache.maxMb` of loaded data, 256 MiB by default ([bounds](<https://pivotkit.eize.dev/reference/sources/#code-cache>)). Add it to the limit, or lower `sourceCache.maxMb`.

For an idea of real use, here is one measurement. It ran once with Node 26 on macOS, not with the image. Take it as a rough order and check your own data with `kubectl top pod`. The source was a CSV file of 30 MB with 700,000 rows and 6 columns. The process used about 100 MB without work. Each worker that had computed a report over the file added about 470 MB.

A report runs on one worker thread. Give the pod about one CPU per worker that should compute at the same time. The example requests half a CPU and sets no CPU limit.

## Data in /data

The server reads its sources from files. On Kubernetes, something has to put them there.

The server notices a new file by its identity, size and time. So replace a file with a new one in one of these ways:

- Write it under another name in the same directory and rename it (`mv`).
- Flip a symbolic link to a new directory.

Never write into the file the server reads: a read in between gets half of it. The user 10001 must be able to read the files.

### A CronJob on a ReadWriteMany volume

A job fetches the data on a schedule. It writes the data to a volume that the pivotkit pods mount read-only. The job's pod and the server's pods may run on different nodes. So the volume must allow ReadWriteMany, as NFS, CephFS or a cloud provider's file storage do. If the cluster's default class does not allow it, set `storageClassName` to a class that does.

```
# The data: written by the CronJob, read by every pivotkit pod, so ReadWriteMany (NFS, CephFS, a cloud
# provider's file storage). Set storageClassName to such a class where the default class is not one.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pivotkit-data
  namespace: pivotkit
spec:
  accessModes: ["ReadWriteMany"]
  resources:
    requests:
      storage: 1Gi
---
# The state: the token and the download links' key the server makes at its first start (secrets/),
# and the reports render_view writes (out/). One server process per volume.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pivotkit-state
  namespace: pivotkit
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 2Gi
```

```
apiVersion: batch/v1
kind: CronJob
metadata:
  name: pivotkit-fetch-data
  namespace: pivotkit
spec:
  schedule: "15 * * * *"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 1
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 600
      template:
        metadata:
          labels:
            app.kubernetes.io/name: pivotkit-fetch-data
        spec:
          restartPolicy: Never
          automountServiceAccountToken: false
          securityContext:
            runAsNonRoot: true
            runAsUser: 10001
            runAsGroup: 10001
            fsGroup: 10001
            fsGroupChangePolicy: OnRootMismatch
            seccompProfile:
              type: RuntimeDefault
          containers:
            - name: fetch
              image: curlimages/curl:8.23.0
              env:
                - name: SOURCE_URL
                  value: https://data.example.com/exports/sales.csv
              # HTTPS only (no redirects, which would leave an empty file), at most limits.fileBytes,
              # only a 200 with content replaces the data: a new name, then a rename, so the server
              # never reads half a file; a credential comes from a file, never from the URL or env
              command:
                - sh
                - -c
                - |
                  set -eu
                  tmp=/data/.sales.csv.tmp
                  trap 'rm -f "$tmp"' EXIT
                  set --
                  if [ -f /run/secrets/source/header ]; then set -- -H @/run/secrets/source/header; fi
                  code=$(curl --fail --silent --show-error --proto '=https' --max-time 300 \
                    --max-filesize 52428800 --write-out '%{http_code}' -o "$tmp" "$@" "$SOURCE_URL")
                  test "$code" = 200
                  test -s "$tmp"
                  chmod 0644 "$tmp"
                  mv -f "$tmp" /data/sales.csv
              securityContext:
                readOnlyRootFilesystem: true
                allowPrivilegeEscalation: false
                capabilities:
                  drop: ["ALL"]
              resources:
                requests:
                  cpu: 50m
                  memory: 64Mi
                limits:
                  memory: 128Mi
              volumeMounts:
                - name: data
                  mountPath: /data
                - name: source-secret
                  mountPath: /run/secrets/source
                  readOnly: true
          volumes:
            - name: data
              persistentVolumeClaim:
                claimName: pivotkit-data
            # optional: a header such as "Authorization: Bearer ..." for the data's host
            - name: source-secret
              secret:
                secretName: pivotkit-source
                optional: true
                defaultMode: 0440
```

What the job does:

- It takes only HTTPS (`--proto '=https'`).
- It follows no redirects. A redirect would give an empty file.
- It stops at 50 MB. That is `limits.fileBytes`, the largest file the server reads.
- It replaces the data only after an answer 200 with content. On any failure the old file stays, and the job removes the temporary one.

When the data's host needs a credential, put the header line into the Secret `pivotkit-source`. The job mounts it as a file and sends it with `-H @file`. Never put the credential in `SOURCE_URL` or in an environment variable, since `kubectl describe` shows those:

```
printf 'Authorization: Bearer %s\n' "$(cat source.token)" > header
kubectl -n pivotkit create secret generic pivotkit-source --from-file=header=./header
rm header
```

The job runs as the user 10001 too, so the server can read the files it writes. Some NFS servers ignore `fsGroup`. There the user 10001 must already be able to write to the volume's directory. The storage's administrator sets that, or a provisioner option does.

Fetch the first data right after you apply the manifest, with `kubectl create job --from=cronjob/...` as above. Otherwise the source is missing until the schedule runs the job. The job's image and command are an example. Any program that writes the file and renames it does the job.

### A sidecar with an emptyDir

A second container in the pod fetches the data into an `emptyDir`. The server's container mounts it read-only. You need no ReadWriteMany storage. Each pod has its own copy and fetches it again after a restart.

The example uses [git-sync](<https://github.com/kubernetes/git-sync>) (v4). It clones a Git repository and flips a symbolic link to each new commit. Start from the example manifest:

1. Remove the CronJob, its NetworkPolicy `pivotkit-fetch-data` and the volume claim `pivotkit-data`.
2. In the ConfigMap, point the source at the link.
3. Replace the Deployment's `data` volume, and add the container and its `/tmp`.

```
"sales": { "file": "/data/git/current/sales.csv", "title": "Sales", "policy": { "fields": { "Region": "show", "Product": "show", "Customer": "hide" } } }
```

```
      containers:
        - name: pivotkit
          # as in the example; its mount of data at /data stays readOnly: true
        - name: git-sync
          image: registry.k8s.io/git-sync/git-sync:v4.8.0
          args:
            - --repo=https://git.example.com/reports/data.git
            - --root=/data/git
            - --link=current
            - --period=5m
          securityContext:
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          resources:
            requests:
              cpu: 50m
              memory: 64Mi
            limits:
              memory: 256Mi
          volumeMounts:
            - name: data
              mountPath: /data
            # the image's HOME
            - name: git-tmp
              mountPath: /tmp
      volumes:
        - name: data
          emptyDir:
            sizeLimit: 1Gi
        - name: git-tmp
          emptyDir:
            sizeLimit: 64Mi
```

Mount the volume at the same path in both containers, so the link resolves the same way in each.

The server follows the link on every call. The next report reads the new commit after a flip. A call that reads the file in the moment of the flip may fail once.

git-sync's image sets `HOME` to `/tmp`. With the read-only root file system, `/tmp` therefore needs the `emptyDir`.

If you use `files.allowDirs`, the server resolves it once at start. It does not start while a directory named there is missing, so name the volume's mount (`/data`), which exists from the start. Do not name a directory git-sync makes later (`/data/git`) or the link (`/data/git/current`).

A private repository needs credentials: git-sync's `--password-file` or `--ssh-key-file`, from a Secret. Over SSH, git may need a passwd entry for the user 10001. git-sync's `--add-user` adds one. That was not tried here.

The policy `pivotkit` needs egress to the Git server on port 443, as for [the issuer](<#oauth-egress>). The policy holds for both containers of the pod.

This sidecar setup follows git-sync's documentation and passes the schema check. It has not been run in a cluster here, so check the flags against the documentation of the version you run. Any other sidecar works the same way when it renames or flips a link.

The images of the CronJob and the sidecar are third-party images pinned by tag. Pin them by digest (`image@sha256:...`) where you need the exact image every time, as for pivotkit's own.

## Service and Ingress

```
apiVersion: v1
kind: Service
metadata:
  name: pivotkit
  namespace: pivotkit
spec:
  selector:
    app.kubernetes.io/name: pivotkit
  ports:
    - name: http
      port: 3333
      targetPort: http
```

```
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: pivotkit
  namespace: pivotkit
spec:
  # ingressClassName: the class of your controller, where the cluster has no default class
  tls:
    - hosts: ["reports.example.com"]
      secretName: pivotkit-tls
  rules:
    - host: reports.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: pivotkit
                port:
                  name: http
```

The Ingress sends everything on the host to the Service. The one process serves all of it: `/mcp`, `/api/v1`, `/openapi.json`, the `/.well-known/` addresses of [OAuth](<https://pivotkit.eize.dev/reference/mcp/#oauth>) and the [download links](<https://pivotkit.eize.dev/reference/delivery/index.md>) under `/downloads/`.

These answer without a token:

- the quick-start page at `/`. `http.page: false` turns it off.
- `/healthz` and `/readyz`
- the [download links](<https://pivotkit.eize.dev/reference/delivery/index.md>) under `/downloads/`. The link is the credential.

Everything else needs a token.

The Ingress needs an ingress controller in the cluster. Where the cluster has no default class, it also needs `ingressClassName`. `pivotkit-tls` is the certificate's Secret. You make it by hand or with a tool such as cert-manager.

The controller must pass the request's `Host` on unchanged and write `X-Forwarded-For`. Its time limit for an answer must be longer than `compute.queueMs` + `compute.timeMs`, which is 40 seconds with the defaults. Check your controller's default.

To publish less, give the Ingress the paths `/mcp`, `/api/v1` and `/downloads` instead of `/`. With OAuth, add the prefix `/.well-known/oauth-protected-resource`.

The ingress controller may sit behind a LoadBalancer Service. Then the client's address reaches it only with `externalTrafficPolicy: Local`, or with the PROXY protocol, depending on the controller. Otherwise `X-Forwarded-For` holds a node's address, and the server's per-address limits count the node.

## NetworkPolicy

```
# Nothing in the namespace talks to anything unless a policy below allows it.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: pivotkit
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: pivotkit
  namespace: pivotkit
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: pivotkit
  policyTypes: ["Ingress", "Egress"]
  ingress:
    # the ingress controller only: http.trustProxy believes its X-Forwarded-For
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-controller
      ports:
        - port: 3333
          protocol: TCP
  egress:
    # name lookups
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP
---
# The CronJob's pods: name lookups and the data's host on 443, nothing coming in.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: pivotkit-fetch-data
  namespace: pivotkit
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: pivotkit-fetch-data
  policyTypes: ["Ingress", "Egress"]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - port: 53
          protocol: UDP
        - port: 53
          protocol: TCP
    # the addresses data.example.com resolves to
    - to:
        - ipBlock:
            cidr: 192.0.2.20/32
      ports:
        - port: 443
          protocol: TCP
```

The three policies:

- `default-deny` stops all traffic of every pod in the namespace that the other policies do not allow.
- `pivotkit` lets the ingress controller in and name lookups out.
- `pivotkit-fetch-data` lets the job's pods look up names and reach the data's host on port 443. Put the addresses its host name resolves to there, since a policy cannot name a host.

Policies take effect only with a network plugin that enforces NetworkPolicy.

The namespace selector uses the label `kubernetes.io/metadata.name`, which Kubernetes sets on every namespace. Put the name of your ingress controller's namespace there. `ingress-controller` is a placeholder.

A controller that runs with `hostNetwork` arrives from the nodes' addresses instead of its pods' addresses. Then allow the nodes' address range there as an `ipBlock`. Check how your cluster's controller runs.

The kubelet's probes come from the node. Most network plugins let them through, whatever the policy says. If the pod never gets ready with the policy applied, check your plugin.

### Clients inside the cluster

With the policy as it is, programs in the cluster call the public address through the ingress controller. To let them call the Service directly (`http://pivotkit.pivotkit.svc:3333`):

1. Allow their namespace in the policy's `ingress`.
2. Add the names they call to `http.allowHosts`: `pivotkit`, `pivotkit.pivotkit.svc` and `pivotkit.pivotkit.svc.cluster.local`.
3. Turn `http.trustProxy` off.

A direct caller can write any `X-Forwarded-For`. With `trustProxy`, the server would believe it, and its limits per address count by that header. With `trustProxy` off, every request through the controller counts with the controller's address.

### Egress

With static tokens the server opens no connection of its own. The policy allows name lookups and nothing else. Name lookups go to the cluster DNS, which is `k8s-app: kube-dns` in `kube-system` on most clusters.

Clusters whose DNS is elsewhere need another rule for it. For example, NodeLocal DNSCache answers on an address of the node, and OpenShift runs its DNS in the namespace `openshift-dns`.

Other features need more egress:

- OAuth needs the issuer too. See [below](<#oauth-egress>).
- An [HTTP source](<https://pivotkit.eize.dev/reference/sources/#http>) needs the addresses of its backend on port 443, as the issuer does.
- It also needs the addresses of its token endpoint when it signs in with client credentials or a token exchange. See [backend sign-in](<https://pivotkit.eize.dev/reference/sources/#http-auth>).

Cluster addresses are private ones, so something inside the cluster needs a setting:

- A backend needs `intranet: true` on it.
- A token endpoint needs `intranet: true` in its sign-in's settings.
- An issuer needs `auth.intranet: true`. See [below](<#oauth-egress>).

## Sign-in with Authentik

With OAuth, the MCP endpoint takes access tokens of your identity provider instead of static tokens. It works with any provider that issues JWT access tokens. [OAuth 2.1](<https://pivotkit.eize.dev/reference/mcp/#oauth>) has every key. With [Authentik](<https://goauthentik.io/>):

1. Under **Customization \> Property Mappings**, create a **Scope Mapping**. Give it the **Scope name** `pivotkit` and an expression that returns `{}`. The scope adds no claims. Authentik drops a scope a client asks for when the provider has no scope mapping for it. A token without `pivotkit` then gets 403.
2. Under **Applications \> Applications**, choose **New Application**. Give it the slug `pivotkit` and the provider type **OAuth2/OIDC**, with these provider settings:

   - **Client Type**: **Public**. An MCP client keeps no secret. It signs in with PKCE.
   - **Redirect URIs/Origins (RegEx)**: the callback addresses of the MCP clients you allow. Take them from the clients' documentation. Give each the matching mode **Strict**, which matches the address exactly. With **Regex**, the whole address must match the expression. Then escape the dots (`https://app\.example\.com/callback`) and use no `.*`. Otherwise a sign-in can be sent to an address you did not mean.
   - **Signing Key**: a certificate with one of the keys the server accepts by default (`auth.algorithms`): an RSA key (RS256), an EC P-256 key (ES256) or an Ed25519 key (EdDSA). For the other EC curves, Authentik signs with ES384 or ES512. Without a signing key, Authentik signs with HS256 and the client secret, which the server refuses.
   - **Encryption Key**: none. With one, Authentik encrypts the access token (JWE), and the server cannot read it.
   - **Access Token Validity**: minutes, for example `minutes=10`. The default is one hour. A token stays valid until it expires.
   - **Scopes**: add the `pivotkit` mapping to the selected scopes.
3. On the application's **Policy / Group / User Bindings**, bind the group that may use pivotkit (**Bind existing group/user**). You can bind users or a policy instead. An application without any binding lets every user of Authentik sign in to it.
4. Note the provider's **Client ID**.

The server accepts every access token this provider issues for its audience with the required scopes. So Authentik decides who gets a token: by the bindings of step 3 and by the scopes.

The issuer is `https://auth.example.com/application/o/pivotkit/`. It ends with the application's slug. That holds with the default issuer mode, where each provider has its own issuer. Authentik puts the client ID into the access token's `aud`, and the granted scopes into its `scope` claim. The `auth` part of the configuration:

```
{
  "auth": { "mode": "oauth",
    "issuer": "https://auth.example.com/application/o/pivotkit/",
    "resource": "https://reports.example.com/mcp",
    "audience": ["<the provider's client id>"],
    "requiredScopes": ["pivotkit"] },
  "http": { "publicUrl": "https://reports.example.com/", "trustProxy": true, "enable": ["mcp"] }
}
```

- `requiredScopes` also keeps Authentik's ID tokens out: they carry the same `aud` but no `scope`. Do not set `auth.requireTyp`: Authentik's access tokens carry no `typ` of `at+jwt`.
- `http.enable: ["mcp"]` serves the MCP endpoint only. The REST service is a resource of its own, and its tokens must carry its audience ([REST sign-in](<https://pivotkit.eize.dev/reference/rest/#auth>)). With Authentik, every token of the provider carries the same client ID. To serve REST from the same provider, set `rest.audience` to that client ID. Give REST a scope of its own with `rest.requiredScopes` and a second scope mapping. A token with both scopes then works on both endpoints.

### Egress to the issuer

The server loads the issuer's metadata and keys over HTTPS. Until then, `/readyz` answers 503. A NetworkPolicy cannot name a host, so allow the addresses that `auth.example.com` resolves to from inside the cluster, on port 443. Add the rule to the policy's `egress`:

```
    - to:
        - ipBlock:
            cidr: 192.0.2.10/32
      ports:
        - port: 443
          protocol: TCP
```

Authentik may run in the same cluster, with its host name resolving to the ingress controller. Then allow the controller's namespace instead of an `ipBlock`.

An issuer whose name resolves to a cluster address or another private address also needs `"intranet": true` in `auth`. Without it, the server refuses loopback and private addresses for the issuer, and `check-config` names the setting. The same setting covers the token endpoint of the [download sign-in](<https://pivotkit.eize.dev/reference/delivery/#sign-in>).

These settings are based on the following sources, checked in November 2026:

- Authentik's [OAuth2 provider documentation](<https://docs.goauthentik.io/add-secure-apps/providers/oauth2/>): issuer, signing key, PKCE.
- Authentik's source code at the release `version/2026.8.5`:

  - `authentik/providers/oauth2/`: the access token's `aud`, `scope` and header, the signing algorithm per key, the default validity, scopes without a mapping, and redirect addresses matched in full.
  - `authentik/policies/engine.py`: an application without bindings passes everyone.
  - `web/src/admin/`: the labels of the forms.

Labels and defaults can change between Authentik releases. It was not checked whether Authentik reads the `resource` parameter of [RFC 8707](<https://www.rfc-editor.org/rfc/rfc8707>). The server does not need it, since it accepts the client ID as audience. Decode an access token of your own to see what your setup issues.

## Update

To update, write the new tag or digest into the Deployment and apply it again. See [Kubernetes in Update an installation](<https://pivotkit.eize.dev/getting-started/upgrading/#update-image>). With `strategy: Recreate`, the old pod stops before the new one starts.

## The whole manifest

[docker/kubernetes.example.yaml](<https://github.com/eizedev/pivotkit/blob/pivotkit-server@0.2.0/docker/kubernetes.example.yaml>) holds, in this order:

- the Namespace
- the ConfigMap
- the two volume claims
- the Deployment
- the Service
- the Ingress
- the three NetworkPolicies
- the CronJob

To check your changed copy against the Kubernetes API schemas without a cluster, use [kubeconform](<https://github.com/yannh/kubeconform>): `kubeconform -strict -summary kubernetes.example.yaml`. With a cluster, use `kubectl apply --dry-run=server`.
