status.renner.uno / status.k.renner.uno is a public, unauthenticated gatus
page. Its endpoints named the internal service estate (Authentik, Gitea,
Immich, Grafana, HA, Paperless, Proxmox, PBS, UDM DNS, Mosquitto) and gatus
exposed each check's target hostname/URL/port — and on failure the error
text (e.g. mosquitto.home-assistant.svc.cluster.local, 10.2.40.1) — via the
public results and /api/v1/endpoints/statuses API (all hide-* default false).
Per owner decision keep the page public (not Authentik-gated) but scrub the
disclosure:
- rename every endpoint to an opaque label (Web Service N / Infrastructure N),
removing product/vendor/hostname identifiers from the public `name`
- add a per-endpoint ui block with hide-hostname/hide-url/hide-port/
hide-conditions/hide-errors: true so gatus scrubs targets server-side
All checks keep running unchanged (real targets stay in `url`, visible only
to git/cluster operators). Public page now shows group + opaque label +
up/down + response time only. Maintainer opaque->real map kept as a comment.
Fixes#206
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The CI automation keys were deployed to root's (and ladmin's) authorized_keys
on every host with no key_options, giving any holder of a CI key unrestricted
root fleet-wide (issue #184).
- Restrict woodpecker-ci@homelab with from="10.244.0.0/16,fd10:244::/48" — a
network-of-origin restriction to the cluster pod CIDRs, so the key only
authenticates from within the CI's network segment. Determined by inspecting
Cilium: routing-mode=tunnel with enable-ipv4-masquerade=true but
ipv4-native-routing-cidr=10.0.0.0/8 (and v6 fd00::/8), which covers every
10.2.x / fd.. LAN SSH target, so pod egress to the fleet is NOT SNAT'd to the
node IP — pods present their real POD IP, BGP-advertised to the UDM
(cilium-bgp.yaml). The stable boundary is therefore the pod subnet
(talos cluster.yaml podSubnets: 10.244.0.0/16 + fd10:244::/48), which
survives node DHCP-lease churn and pod rescheduling across k8s-1/2/3 — unlike
pinning any node's current (DHCP) IP. Value lives in group_vars/all/vars.yml
(ci_ssh_from_cidrs) as the single source of truth.
- Remove ansible-runner@ci-new entirely from the play, the common role, and
common_ssh_public_keys — dead key, no such host exists anywhere in the repo.
- Keep krenner@fortress (human) unrestricted.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The velero-bootstrap-restore Job (restore-job.yaml + its RBAC) was never
wired into any Flux Kustomization — no `path` field in
kubernetes/clusters/homelab/ referenced it, and the parent
infrastructure/kustomization.yaml only aggregates `velero/`, not
`velero-bootstrap-restore/`. It never ran, creating false confidence that
app-data auto-restore-on-bootstrap existed.
Per owner decision on #200: delete it and rely on the documented manual
restore procedure in kubernetes/RUNBOOK-cold-rebuild.md. The real seed
path is the seed-* Kustomizations (rsync from legacy host) plus the
per-app restore steps in the runbook.
Also updates two now-dangling doc comments in
post-crds/velero-schedules.yaml that referenced the deleted Job's
SCHEDULES mechanism, pointing them at the runbook instead.
kubectl kustomize on infrastructure/ and infrastructure/post-crds/ both
build clean after removal.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Flux pulls manifests from the in-cluster Gitea, which is itself Flux-managed
(Ceph data, in-cluster DNS+Traefik), so a cold rebuild has no entry point.
Give it one:
- bootstrap.sh: FLUX_SOURCE=github selects 'flux bootstrap github' against the
mirror (github.com/konairius/Homelab, branch master) instead of
'flux bootstrap gitea'; defaults to gitea so steady-state is unchanged.
- kubernetes/RUNBOOK-cold-rebuild.md: single tofu->talos->flux->openbao->apps
runbook, with the break-glass github bootstrap and the manual OpenBao
snapshot-restore+unseal step wired in.
The GitHub mirror is already kept current by Gitea's own native push-mirror
(server-side, sync_on_commit=true) — no setup and no new credential required;
the runbook just references it. OpenBao restore stays manual (unseal key out
of CI, in pass).
Refs #187
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PR #231 scoped the openbao ClusterSecretStore to the 12 namespaces that
had ExternalSecrets at the time. PR #236 then added a new consumer in the
crowdsec namespace (the bouncer key), which the selector refused
(SecretSyncedError). Add crowdsec to the allowed namespaces.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The apps namespace (Gitea, Woodpecker, gatus) had no Velero schedule at
all, so the in-cluster Gitea (data-gitea-0, the Flux GitOps source of
truth) and Woodpecker (data-woodpecker-server-0) PVCs migrated from the
legacy LXCs were unprotected. The only PBS coverage (opentofu/backup_jobs.tf
VMID 114/231) targets the original LXCs, not the live cluster PVCs.
Add an apps-daily Schedule (06:45 local, next free stagger slot) mirroring
the existing PVC-only pattern; no CNPG cluster runs in this namespace so no
labelSelector exclusion is needed. Also register it in the bootstrap-restore
Job's SCHEDULES list so a fresh-cluster bootstrap restores it like the
others.
Fixes#186
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
The Talos bootstrap inline manifest (patches/cilium-cni.yaml) and its
standalone copy (cilium.yaml) embedded Secret/cilium-ca (RSA ca.key) and
Secret/hubble-server-certs (RSA tls.key) as plaintext base64 — rendered
Helm output from hubble.tls.auto.method=helm. Anyone with repo/history
read access could mint certs under the Cilium CA and impersonate/decrypt
the Hubble mTLS channel.
Switch to hubble.tls.auto.method=cronJob: replace the two committed-key
Secrets with the certgen ServiceAccount/Role/RoleBinding/Job/CronJob.
The Job generates (and the CronJob renews) Secret cilium-ca and issues
Secret/hubble-server-certs in-cluster at install. No private key material
is committed. Verified with `helm template cilium/cilium --version 1.19.2`:
the only delta between the two methods is exactly these 7 resources; the
ConfigMap and cilium-agent DaemonSet are unchanged, and the agent's
hubble-tls volume is already optional:true so it starts before certgen
runs. Both files and the inline block scalar validated as YAML.
This is the code half of the fix. Rotating the already-leaked live CA +
Hubble certs and scrubbing them from git history is an operator task
tracked on the issue.
Refs #188
Owner confirmed on #225 that all five roles flagged in #198 are legacy and
were decommissioned some time ago; the earlier inventory-presence liveness
check was wrong (inventory has stale entries for services migrated into the
k8s cluster). So instead of sanitizing committed secrets, remove the dead
code that held them.
Deleted role directories (all owner-confirmed legacy):
- ansible/roles/immich -> now runs in-cluster (kubernetes/apps/immich, secrets via OpenBao ESO)
- ansible/roles/grafana-observability -> dashboards now live in kubernetes/apps/monitoring/dashboards
- ansible/roles/postgres-simple-ha
- ansible/roles/keepalived-ntp
- ansible/roles/keepalived-vip
Deleted plays that exclusively orchestrate the deleted legacy roles:
- ansible/plays/install-immich.yml
- ansible/plays/configure-grafana-observability.yml
- ansible/plays/backup-immich-db.yml (one-time migration dump from legacy LXC 225)
- ansible/plays/configure-immich-db-password.yml (loads the deleted immich role)
Edited (kept, live):
- ansible/plays/update.yml -> dropped the `import_playbook: install-immich.yml`
line so the CI update path (.woodpecker/update-all.yml -> plays/update.yml)
keeps working.
NOT touched (ambiguous — reference companion roles the owner did NOT list as
legacy; flagged on #225 for owner decision, no dangling roles deleted here):
- install-ntp-ha.yml (also uses chrony-ntp-server)
- install-proxmox-dns-ha.yml (also uses proxmox-dns, which is still live)
- install-authentik-ha.yml (also uses authentik-stateless)
These three plays now reference deleted keepalived-*/postgres-simple-ha roles;
their companion roles + inventory groups need an owner call before removal.
Refs #198, #225. Coordinates with #208 (decommission sweep) for immich.
Closes gitea issue #197. The update-audit agent fetches untrusted
third-party release pages via WebFetch/WebSearch and its Write output
is posted verbatim to a public Gitea issue, so injected content in a
fetched page could try to redirect the agent's behavior. Harden the
tool surface (verified empirically against the pinned Claude Code CLI):
- Drop WebSearch: it cannot be domain-scoped like WebFetch, and the
prompt already gives explicit source URLs or a well-known registry
for everything it needs to look up.
- Add ci/agents/update-audit/settings.json (loaded via --settings) that
denies Read on /proc, /root, /home, /etc, and the repo's own .git/ —
closing the credential-file and env-var-dump read paths a
prompt-injected instruction could otherwise try. Verified this deny
mechanism actually blocks reads, unlike Write path-scoping, which
was tested and found NOT enforced under --permission-mode acceptEdits
(documented in the pipeline comment; a full WebFetch domain allowlist
was also considered and rejected, since Helm chart lookups legitimately
hit whichever domain each HelmRelease's own trusted source URL points
at).
- prompt.md now tells the agent fetched content is untrusted data, not
instructions, and to never echo it verbatim into the report.
Not fixed here: GITEA_TOKEN is still the broad registry/release-writing
token (no write:issue-only token exists yet) — that requires an
operator to mint one; tracked via a comment on #197 and a code comment
at the secret binding.
PR #235 (fix/issue-193-woodpecker-ci-hardening) already adds
kubernetes/infrastructure/network-policies/woodpecker-ci-ns.yaml, with a
deliberately different (and better-informed) egress design: default-allow,
because a past attempt at egress enforcement in this namespace broke
crowdsec (66dbe9f) — history this branch didn't have context on.
PR #231 (fix/issue-202-openbao-hardening) already adds an egress block to
openbao-ns.yaml as part of a broader OpenBao hardening pass, including a
"world" allowance (OIDC) this branch's version omitted.
Dropping both here to avoid shipping a conflicting/duplicate policy;
external-secrets-ns.yaml (this issue's third gap, not covered by any other
open PR) is the only change left on this branch.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
bouncer-key.yaml committed the CrowdSec<->ingress-nginx bouncer key in
cleartext across two namespaces, outside the ESO/OpenBao pattern used
elsewhere in the cluster. Replace both Secrets with ExternalSecrets
pulling the same value from a single OpenBao KV path
(infrastructure/crowdsec-bouncer-key), matching the existing
apps/infrastructure ExternalSecret convention.
Requires the owner to populate OpenBao before merge:
bao kv put -mount=homelab infrastructure/crowdsec-bouncer-key key=<value>
Fixes#212
Closes the gaps from #194: external-secrets and woodpecker-ci had no
CiliumNetworkPolicy at all (Cilium defaults to allow-all absent one), and
openbao-ns.yaml had an ingress-only CNP with no egress block, which leaves
egress from openbao pods completely unenforced rather than merely permissive.
- external-secrets-ns.yaml: default-deny-ingress; egress scoped to the
namespace, DNS, and the openbao KV API on :8200 (the only
ClusterSecretStore in this cluster) plus host/remote-node for
kube-apiserver (webhook admission, Secret watch/create, leader election).
- woodpecker-ci-ns.yaml: this namespace hosts only ephemeral, sometimes
privileged, pipeline-defined step pods (no static workloads) — the goal
is blocking lateral movement into unrelated namespaces while still
allowing the git clone (east-west via k8s_gateway -> traefik, since
gitea.renner.uno now has an in-cluster Ingress) and the broad internet
egress every CI system legitimately needs (image pulls, package
managers).
- openbao-ns.yaml: added an egress block (DNS, intra-namespace, host/
remote-node for the k8s auth method + kubelet) so egress enforcement is
actually turned on, not just missing.
music-assistant is deliberately NOT covered here: its only workload runs
hostNetwork: true, so a namespace-scoped CiliumNetworkPolicy would select
no enforceable pod-network endpoint and be a no-op (same documented
limitation as the rook-ceph Ceph daemons). Left for a comment on #194
rather than shipping a policy that looks protective but isn't.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Addresses #209. All four ci/images/*/Containerfile FROM lines were
tag-pinned only (alpine:3.21, node:22-alpine), letting the base silently
change between builds. Pin each to its current manifest-list digest.
ci/images/k8s/Containerfile downloaded talosctl, kubectl, and velero via
curl with no integrity check, and installed flux via `curl | bash`
(itself an unpinned, unverified fetch-and-execute). Replace the flux
install with a pinned-version tarball download verified against its
published sha256 checksum, same as talosctl/kubectl/velero now are.
kubectl's version was also resolved dynamically from dl.k8s.io's
stable.txt at build time instead of being pinned; add KUBECTL_VERSION.
ci/images/tofu/Containerfile's tofu.zip download gets the same
sha256 verification.
ci/images/claude/Containerfile's CLAUDE_CODE_VERSION defaulted to
`latest` and was never overridden by build-ci-images.yml, so the image
holding repo/issue-write tokens floated to whatever npm resolved at
build time. Pin it to the current release (2.1.209).
Checksums and digests verified against the respective registries/release
APIs at fix time (2026-07-14).
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Adds minAvailable:1 PDBs for the data-bearing single-replica StatefulSets
that had no eviction protection: gitea, woodpecker-server, and paperless's
protonmail-bridge + syncthing sidecars. On a 1-replica workload this makes
the eviction API refuse *voluntary* evictions, so a node drain blocks and
surfaces the pod instead of yanking a live SQLite DB / RWO-PVC pod mid-write.
Involuntary disruptions (node crash, OOM) are unaffected.
Deliberately excluded, with rationale:
- CNPG clusters im-pg and gf-pg (instances:1): CloudNativePG intentionally
omits a primary PDB for single-instance clusters (no failover target), and
handles graceful primary shutdown itself. A manual maxUnavailable:0 would
only wedge node drains with no HA benefit.
- Ceph mons: the Rook operator self-manages quorum-preserving mon/OSD PDBs
(disruptionManagement.managePodBudgets default). A hand-written PDB would
be redundant or fight the operator on each reconcile.
- Stateless single-replica Deployments (redis/tika/gotenberg caches, etc.):
a blocking PDB gives no recoverability benefit, only blocks drains.
Refs #201
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The certificates role fetched https://authentik.renner.uno with
validate_certs: false and fed whatever it got back into
update-ca-certificates on every Ansible-managed host — an on-path/DNS
attacker during that fetch could plant a rogue CA cluster-wide.
authentik.renner.uno also turned out to be a stale hostname: it isn't
routed by the current Traefik/cert-manager ingress. The live service
(auth.renner.uno) already gets a publicly-trusted Let's Encrypt cert via
cert-manager, which the OS default CA bundle already trusts. Nothing else
in the repo depends on the installed authentik.crt, so removing the task
(and its now-unused handler) closes the hole with no functional loss.
Fixes#185
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Issue #210: neither the SOPS age key nor ANSIBLE_VAULT_PASSWORD has a
documented offline backup location. The age-key archive was previously
bundled into the Phase 4 point-of-no-return checklist, blocking it on the
reconcile/unseal soak for no reason — backing it up to pass is safe to do
immediately since it only adds a copy. Also documents a pass location and
verification procedure for the Ansible Vault password, which gates PBS and
other Ansible-Vault-encrypted credentials and had no backup plan at all.
Co-Authored-By: Claude Bot <claude-bot@renner.uno>
Closes#205. The websecure entrypoint had zero response hardening and no
rate limiting — CrowdSec is detection-only since the 2026-06-27 IPv6
mass-ban outage, and no headers/rate-limit Middleware existed anywhere in
the cluster, so logins sat behind app-level throttling only.
Adds two Middlewares wired as entrypoint defaults via the same
additionalArguments mechanism the (still-detached) crowdsec-bouncer
Middleware documents for its own re-enable:
- security-headers: HSTS, nosniff, xss-filter, referrer-policy,
permissions-policy, and a conservative frame-ancestors 'self' CSP
(deliberately not a full default-src policy — apps here span Paperless,
Immich, Grafana, Gitea, Home Assistant, etc. with very different
script/style/font sources).
- rate-limit: generous defaults (100 req/s, burst 200) as a baseline
abuse/brute-force backstop rather than traffic shaping, keyed off the
real client IP via sourceCriterion.ipStrategy.excludedIPs (same trusted
pod CIDRs as forwardedHeaders/crowdsec-bouncer) so it doesn't collapse
every external client onto one shared budget — the same client-IP-
visibility gap that caused the CrowdSec outage this fix is otherwise
independent of.
CrowdSec enforcement itself stays disabled; that re-enable is separate,
documented, unaddressed work.
k.renner
security-bot
claude-botandClaude Opus 4.8