gitea-mcp always writes a lumberjack log (100MB rotation, 10 backups) and falls back to /tmp on a read-only root. A 16Mi in-memory emptyDir would fill and get the pod evicted; the log also counted against the memory limit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs gitea-mcp in -oauth mode with no server token: tokenless /mcp requests get 401 instead of reaching Gitea anonymously, and a plain PAT in the Authorization header is passed through to Gitea. Published on the NetBird proxy via netbird.renner.uno/public; LAN DNS stays on the Traefik VIP.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Master build of #1238 (pipeline 13250). Activates the proxy-only label, so
the gitea Ingress is published on the NetBird proxy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dns.renner.uno/public publishes a host on the NetBird proxy AND drops its
UDM record (external-dns-unifi labelFilter). For gitea that sends LAN SSH
to the proxy host's own sshd and node image pulls over the WAN, since the
hairpin covers only 80/443 and not the node VLAN.
netbird-service-sync now also selects netbird.renner.uno/public, which
external-dns-unifi does not filter on, so the host is published on the
proxy while the LAN record on the Traefik VIP stays. The gitea Ingress
switches to the new label; it takes effect once the master build of this
image is pinned in the deployment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Label the gitea Ingress dns.renner.uno/public so netbird-service-sync
publishes git.renner.uno and gitea.renner.uno on the NetBird proxy. The
gitea-ssh IngressRouteTCP stays unlabeled and LAN-only.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Embeds a labelled Paperless corpus and scores nearest-neighbour correspondent and document-type accuracy against ceiling and floor. Standard library only.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Point PAPERLESS_AI_LLM_* at api.mistral.ai (mistral-small-latest), key from paperless-secrets minstral-token. Embeddings stay on llama-embed; an explicit embedding key stops the Mistral token falling back to it.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- vm-verify: `kopia snapshot verify` read the loop's stdin, so anything
reading stdin would eat the remaining plan entries; it now reads
/dev/null. KOPIA_CHECK_FOR_UPDATES=false.
- cloud-init failing on the VM no longer waits out SSH_WAIT_S: ssh exit
codes other than 255 mean cloud-init itself failed, so the orchestrator
logs `cloud-init status --long` and gives up at once.
- A failure staging files on the VM is no longer fatal: the run still
prints the per-entry summary (no result from VM) and exits 1.
- A server create that times out after the server exists no longer
blocks the fallback type with a duplicate name: labelled servers are
deleted between attempts.
- cloud-config is rendered as JSON (a YAML subset) with jq, so the
multi-line host key is escaped by jq and the test can parse it.
- Tests: secrets parsing (with/without spaces around =), user-data
structure, every server type unavailable, retry after a leftover
server, push failure, cloud-init failure.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- AUTH_INSECURE_* is no longer excluded. #527 is done: every CephX key
is aes256k since 2026-09-23 (#931, #934, #935) and all six
AUTH_INSECURE_* checks report 0, so a new one is a real regression.
- The description now only says what to do: BLUESTORE_SLOW_OP_ALERT
outlives the stall by 24h; find the OSD in `ceph health detail` and
check that drive's SMART data. No history in the annotation.
- Comment: drop the AUTH_INSECURE paragraph and the restore-test note;
the "80-87% endurance used" claim was wrong for the OSDs (nvme2 at
2-3% per smartctl_device_percentage_used, the 83-91% drives are the
nvme1 EPHEMERAL disks), so it goes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The alert told the reader that BLUESTORE_SLOW_OP_ALERT stalls between
08:00-14:00Z on the 1st are expected because of the monthly Velero
restore-test sweep. restore-test was removed in #1233 (replaced by
backup-verify, which writes nothing to the OSDs), so a stall in that
window is no longer explained by a scheduled job and must not be
waved through. Expression, for and labels are unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
backup-verify passed its live runs on 2026-10-10:
- apps only: PASS=2, 61 s, VM torn down
- negative (forged snapshot IDs): FAIL=2 "snapshot not in repository",
Job Failed, VM torn down
- abort (Job deleted after ssh ready): TERM handled in 3 s, server,
firewall and key gone within 16 s; next run swept nothing
- full: PASS=39 FAIL=0 SKIP=2, 3688 s, cx23, ~410 GB verified, nothing
left behind
The CronJob is unsuspended. restore-test (CronJob, RBAC, script,
ConfigMap) and its Prometheus :9090 ingress rule in monitoring-ns go;
Flux prune removes them and the failed restore-test-29847360 Job with
its CronJob, which clears the open KubeJobFailed (velero) alert.
RECOVERY.md describes backup-verify instead of the OSD capacity gate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The k8s backend defaults to IfNotPresent, so a node kept serving a stale
:latest (Claude Code 2.1.218) after the refresh pipeline pushed a new one.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- TERM while waiting for cloud-init over SSH was deferred until the
probe returned; the probe now runs in the background and is waited
on, so a Job deleted during VM bring-up still tears down in time.
- /run/verify is its own 1500m tmpfs (cloud-init runcmd); the Kopia
cache no longer competes with the ~10% /run of a 4 GB VM.
- Summary rows carry backup name and snapshot ID; a final line reports
elapsed seconds and the server type used (spec section 5).
- kopia verify runs with --file-parallelism=8 (default NumCPU=2 is
latency-bound against S3).
- /work emptyDir 32Mi -> 128Mi (kubectl cache + Velero object dumps),
memory limit 256Mi -> 384Mi to cover it; overflow would evict the pod
without the teardown trap.
Each fix has a test that failed first (term during ssh wait, tmpfs
mount in user-data, summary row/elapsed line, verify parallelism).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs kubernetes/infrastructure/velero/backup-verify/tests/run.sh in the
ci-backup-verify image (bash, jq, ssh-keygen), offline.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- CronJob velero/backup-verify, monthly 0 8 1 * * UTC, suspended until
the live runs pass; 4 h deadline, 120 s grace for teardown; runs as
65532 with a read-only root fs, tmpfs /work.
- Role limited to get/list schedules, backups, datauploads in velero.
- ExternalSecret hcloud-backup-verify from OpenBao
infrastructure/hcloud-backup-verify.
- CiliumNetworkPolicy: apiserver, DNS, api.hetzner.cloud:443, tcp/22.
- Scripts packaged as ConfigMap backup-verify-scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
backup-verify.sh: sweep leftovers by label, build the plan from Velero,
create ssh-key/firewall/server (cx23, fallback cax11) with a pre-made
host key pinned in known_hosts, push check.jq + vm-verify.sh, stream
secrets on stdin, summarize, and delete everything labelled
managed-by=backup-verify on EXIT. The VM call runs in the background
and is waited on so a TERM (Job deadline/deletion) tears down at once.
FORGE_SNAPSHOT_ID replaces planned snapshot IDs for the live negative
test.
Tests stub hcloud/kubectl/ssh: happy path, type fallback, VM FAIL,
forged IDs, empty plan, and TERM mid-run (fails if the VM call runs in
the foreground).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
vm-verify.sh installs kopia 0.16.0 (sha256-pinned, x64/arm64), connects
read-only per namespace (kopia/<ns>/), runs check.jq and then
`snapshot verify --verify-files-percent=100` per snapshot root. Secrets
come on stdin only. Exactly one RESULT line per entry, then DONE;
connect or list failures turn every entry of that namespace into FAIL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
check.jq matches planned DataUploads against kopia snapshot list output:
missing snapshot, incomplete snapshot, entries failed at backup time and
size mismatch against DataUpload totalBytes are FAIL before any verify.
summary.jq turns a VERIFY entry without a VM result into FAIL and never
reports success when nothing was verified.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
plan.jq turns Schedules, Backups and DataUploads into VERIFY/FAIL/SKIP
entries: newest finished backup per schedule (an in-flight backup does
not count), PartiallyFailed/Failed -> FAIL, no DataUploads -> SKIP.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OpenProject Community hides every OIDC provider without an Enterprise
token (auth_plugin.rb, EnterpriseToken.allows_to?(:sso_auth_providers)),
so it cannot sign in through Authentik.
Removes the app, its Flux Kustomization, namespace, network policy,
op-pg PodMonitor, blackbox probe, Velero schedule and ClusterSecretStore
entry. Authentik drops the OIDC secret env and marks the OpenProject
application, provider and group absent. Gitea no longer allows
openproject.renner.uno as a webhook host.
Refs: REN-1363
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Alpine with bash, jq, kubectl, openssh and the hcloud CLI (v1.69.0,
copied from the official image). A verify user (uid 65532) exists
because the OpenSSH client refuses to run as a UID without a passwd
entry. Built by build-ci-images.yml with a dated tag; Flux image
automation tracks the newest date.
First step of docs/superpowers/plans/2026-10-10-backup-verify-cloud-vm.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Implementation plan for docs/superpowers/specs/2026-10-10-backup-verify-cloud-vm-design.md:
PR A image, PR B jq filters + scripts + manifests (CronJob suspended)
with offline fixture/stub tests, live runs, PR C cut-over.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
k.renner
issue-fix-bot
claude-botandClaude Opus 5.5
security-bot