The three Matrix Ingresses had no dns.renner.uno/public label, so
netbird-service-sync never created a NetBird proxy service for them and the
public proxy rejects their SNI (tlsv1 alert internal error). Label them like
audiobooks/immich and add Gatus checks for the three hosts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
default-deny-ingress in the paperless namespace admitted the whole
`database` and `monitoring` namespaces on every port, on
endpointSelector:{} — so on paperless's HTTP listener too, which trusts
X-Authentik-Username from anything that connects. Both admits are
removed and the three load-bearing flows are kept as per-workload
policies:
pprl-pg-allow-cnpg-operator database/cloudnative-pg -> pprl-pg :8000
pprl-pg-allow-prometheus monitoring/prometheus -> pprl-pg :9187
paperless-allow-blackbox monitoring/blackbox -> paperless :8000
No ICMP rule on them (REN-144 option (a)); the baseline ICMP and the
host/remote-node L3 admit are untouched.
Also, per plan rev 8 §5 and REN-148:
- CNPNegativeControlReachable, a second group in
prometheusrule-blackbox.yaml, scoped to job=blackbox-negative-control.
- Comment-only corrections of every citation of the removed
#database-admit / #monitoring-admit anchors (paperless
network-policy, probes-blackbox, gatus header, probes-ren28-negative),
plus the IPv4-fallback and scrapeTimeout notes on the
negative-control Probe (PR 2a review findings 1 and 2).
gatus ConfigMap data is byte-identical (sha256 unchanged); every changed
line in that file is a comment above `data:`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comments that pointed at other YAML as `file.yaml:N` went stale on any
edit above the cited line; three drift fixes had been merged by hand
(REN-181). The cited block now carries `# anchor: <name>` and the
comment cites `path.yaml#<name>`.
- Sweep: all 32 line-number citations in YAML comments converted.
Several were already stale at head and now point at the right block
(see PR). Vendored files (Flux gotk-components, IMC CRD) get no
anchor; those two citations name the object/schema path instead.
- Guard: scripts/ci/yaml-citations.sh in ci-manifests-test.yml; fails
on a line-number citation or a path#name whose file/anchor is gone.
Self-test in ci-scripts-test.yml.
- CONTRIBUTING.md: the convention in three lines.
Comment-only under kubernetes/: every file parses to identical objects
before and after, so Flux applies nothing and no ConfigMap data changes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR #997 inserted 9 comment lines into `paperless-ns.yaml` and 21 into
`navidrome/deployment.yaml`. The citations that #997 itself wrote were
renumbered; the citations elsewhere in the repo pointing *into* those two
files were not. REN-116 (merged as a8c48b54) then added one more line to
`paperless-ns.yaml`, shifting everything below it by one again.
Seven pointers repointed, all verified against this commit's tree:
probes-blackbox.yaml:156 navidrome/deployment.yaml 61-74 -> 80-93
probes-blackbox.yaml:218 paperless-ns.yaml 24-26 -> 34-36
probes-blackbox.yaml:221 paperless/network-policy 8-12 -> 22-26
gatus/configmap.yaml:47 paperless-ns.yaml 24-26 -> 34-36
paperless/network-policy.yaml:15 paperless-ns.yaml 30-32 -> 31-33
paperless/network-policy.yaml:16 paperless-ns.yaml 33-35 -> 34-36
paperless/network-policy.yaml:17 paperless-ns.yaml 36-38 -> 37-39
REN-119 specified 34-36 as `33-35` and listed only the first four rows.
Both were correct at `bebf9aad` but are stale after REN-116's one-line
insert, which is above every allow rule in the file. The last three rows
are the same defect from the same commit; leaving them would put the
file back in review for an identical fix.
`paperless-ns.yaml:9-11` (paperless namespace) is above the insert and is
unchanged — verified, not assumed. `navidrome/lidarr.yaml:114-127`,
`lidarr.yaml:29` and `probes-blackbox.yaml:250` were re-checked and are
still correct.
Comments only, and nothing is applied to the cluster. Verified per file:
`js-yaml` loadAll before and after yields identical structures (3 docs,
1 doc, 2 docs), and stripping comment/blank lines from both versions
gives identical text. `gatus/configmap.yaml:47` is in the header above
`data:` (line 62), outside the `config.yaml: |` block scalar, so
ConfigMap `data["config.yaml"]` stays byte-identical at 6572 bytes and
`configmap.reloader.stakater.com/reload: gatus-config` does not roll the
gatus Deployment. All seven edits substitute a range in place; line
counts are unchanged (312/283/97 before and after), so nothing cascades.
Revert: `git revert <this commit>`. Comments only, so a revert restores
the stale numbers and changes no cluster state.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment-only. Every parsed object in both files is byte-identical to
master at 0a82d61f, verified with js-yaml; zero non-comment lines change.
Review follow-ups from REN-22 on the merged #976/#977 probes:
1. The CNP argument was right in its conclusion and wrong in its framing.
cnp-cross-namespace-allow-apps-gatus-to-monitored-services DOES apply to
blackbox-exporter — its Generalization section names it. We are safe
because the allow it demands is already present, not because the note is
about gatus. The old wording is the sentence that makes the next person
skip the check for the next namespace.
2. Records the load-bearing egress fact, which was written down nowhere.
Cilium enforces at both ends; blackbox-exporter can send only because no
policy selecting a monitoring endpoint declares an `egress:` block.
Verified across the repo: 73 Cilium policies, 38 with egress, none in
monitoring, and the one CCNP is scoped to `reserved.health`. 17 other
namespaces already have an egress block, so a routine hardening pass on
monitoring would break these three targets and leave the *.monitoring.svc
ones working.
3. Extends the 200-vs-3xx caveat to all three new targets, not just
paperless. The repo evidence behind every one of them establishes only
`< 400` (kubelet httpGet accepts 200-399; lidarr.yaml:29's `curl -f`
without -L exits 0 on a 302). It proves the paths are ungated, not that
they answer 200. A 3xx gets reverted, not widened.
4. Records the tested revert recipe for #976 in the file. `git revert -m 1
16238d39` conflicts and the natural resolution silently drops paperless
too.
5. Corrects the stale "needs a CiliumNetworkPolicy first" claim in
apps/gatus/configmap.yaml. The correction is written in the manifest
header, ABOVE `data:`, not in place: the stale sentence sits inside the
`config.yaml: |` block scalar, and gatus carries
configmap.reloader.stakater.com/reload, so editing it in place would roll
the gatus Deployment for a comment. ConfigMap data sha256 is unchanged.
Production constraint: nothing is applied, restarted or re-rendered by this
change. No workload, namespace, policy or probe target is touched.
Revert: `git revert -m 1 <merge-sha>`, or drop either file alone with
`git checkout 0a82d61f -- <path>`.
Refs REN-21, REN-22.
Co-Authored-By: Paperclip <noreply@paperclip.ing>
Three services that nothing watched. Each needed a different answer, and the
difference is what the comments record.
ical-proxy serves an unauthenticated /health returning 200 with
{"status":"healthy"}. That is real app health, not reachability. Its root path
answers 404 by design — it only serves calendar paths — which is why a naive
probe of / would have looked broken and a naive fix would have been a module
that accepts 404.
Loki has no usable external vantage at all: every path through its ingress
answers 401, so an outside probe could only ever assert that the ingress is up.
In-cluster /ready answers 200 unauthenticated and lives in the monitoring
namespace, so it rides the same intra-namespace allowance the existing
Prometheus and Alertmanager health probes depend on. Verified through the
apiserver proxy before committing.
Paperless goes on the status page rather than into blackbox, and only at gate
level. Every path including /api/ answers 302 to unauthenticated callers, so
there is nothing to probe that proves the app instead of the Authentik outpost.
The entry says so. Closing that blind spot means probing the ClusterIP from
in-cluster, which crosses a namespace boundary and needs a CiliumNetworkPolicy
— the monitoring probes avoid one only because their targets share a namespace
with the prober. That is a separate change.
Paperless takes a new slot rather than the retired Web Service 7, which the
legend used to misattribute to it. Reusing a public label would silently
reassign what it means.
The upstream image is FROM scratch with no USER directive, so the pod ran as
root and tripped restricted on all four counts: allowPrivilegeEscalation,
capabilities, runAsNonRoot and seccompProfile.
fsGroup is what makes this survivable rather than a crash. The only write path
is the sqlite store on the RWO volume — storage.type sqlite, path
/data/gatus.db in the ConfigMap — and fsGroup both chowns the existing volume
on mount and leaves the directory group-writable, which sqlite needs to create
its -wal and -journal files beside the database.
The uid is numeric because a scratch image has no /etc/passwd for the kubelet
to resolve a name against.
readOnlyRootFilesystem is deliberately not set: restricted does not require it,
and on a scratch image there is nothing to gain that would justify the risk of
finding out at runtime that the binary wants a scratch path.
#696 deleted the Ingress and ExternalName Service for the NAS file UI because
they pointed at nas-1, which is off. It did not delete the gatus check for the
same hostname, so the public status page has been showing Web Service 7 as
failing ever since — verified against the live API, success: false.
That was my omission in #696: I removed the route and left the thing watching
it.
The label is retired rather than renumbered, matching Infrastructure 1 and 2.
The maintainer legend claimed Web Service 7 was Paperless. It never was — the
slot held the NAS file UI, and nothing in this config checks Paperless at all.
I added the missing Web Service 10 entry to this same map in #699 without
verifying the rest of it, which is how a wrong line survived a pass that was
supposed to be about exactly this map. Every remaining entry is now checked
against its URL.
Paperless is genuinely unmonitored here and is left that way deliberately:
adding coverage is a separate decision from correcting a map that lies.
This is also the ConfigMap change that exercises the Reloader annotation from
#700. That annotation landed after the ConfigMap it watches had already
changed, so no event ever fired and the pod kept serving the pre-#699 endpoint
list, still probing pbs2. This edit should reload it and clear both.
Dropping the PBS 2 check in the previous commit reconciled cleanly and changed
nothing. The ConfigMap in the cluster no longer lists it; the running instance
still reports Infrastructure 1, because Gatus reads its config once at startup
and the pod is 26 hours old.
Nothing about that state looks wrong from the outside. Flux reports the
Kustomization applied, the ConfigMap matches git, and no resource is failing —
the only symptom is that the endpoint list served by the app disagrees with the
one in the repo. Adding a check would have been just as invisible as removing
one.
Reloader has been installed cluster-wide for 40 days and eleven other workloads
already carry its annotation, across the apps, home-assistant, woodpecker,
wordpress and authentik namespaces. Gatus was simply never given one. The
annotation form for a ConfigMap is confirmed against upstream rather than
recalled: configmap.reloader.stakater.com/reload takes the ConfigMap name.
The pod restarts on reload and the deployment uses Recreate for its RWO volume,
so the status page has a short gap whenever the config changes. That is the
correct trade against serving checks that silently no longer match the repo.
Not addressed: the deployment trips PodSecurity restricted on four counts
(allowPrivilegeEscalation, capabilities, runAsNonRoot, seccompProfile). That
predates this and is its own change.
pbs2.renner.uno was probed every 60 seconds as "Infrastructure 1". Both of its
declared clients are gone: the PVE cluster, retired to bare-metal Talos, and
nas-1, powered off and removed from the repo. A liveness check on a backup
server that nothing backs up to reports on nothing — and hide-errors: true
meant its status carried no detail on the public page either.
The opaque label keeps its number and is marked retired, the way Infrastructure
2 already records the Proxmox VE slot. Those labels are public; renumbering
would silently reassign what every other number means.
Also fills a gap in the maintainer legend: Web Service 10 (TRMNL) was serving
but undocumented, so the map stopped matching the endpoint list at 9.
Checked: the embedded config still parses, 12 endpoints remain, and the only
surviving pbs2 mention is the removal note.
No Proxmox remains. jupiter, saturn and neptune all run Talos on bare metal,
so everything that scraped, proxied, alerted on or rotated credentials for the
PVE API now points at hosts that answer nothing on :8006.
Two of these were not merely dead but actively failing on a schedule:
gatus an Infrastructure check on https://pve.renner.uno, every
60s, which can now only ever fail
openbao rotation a Class-B CronJob rolling the pve-exporter@pve API token
against pve.apps.svc.cluster.local:8006
Removed:
apps/pve/ Service, EndpointSlice (10.2.40.201-203),
IngressRoute for the PVE web UIs
monitoring/pve-exporter-* Deployment and its config ExternalSecret
prometheus scrape configs job_name pve and pve-node
dashboards/proxmox-cluster.json and its ConfigMap generator entry
image-automation/pve-exporter image policy for an image nothing runs
openbao/rotation/pve-exporter the rotation CronJob
openbao/policies/rotator-proxmox.hcl
crowdsec pve-ui-polling-whitelist, which suppressed false bans on PVE UI
API polling that no longer happens
Kept, with comments corrected: the insecure-backend ServersTransport, still
used by apps/file for the NAS web UI.
The gatus check leaves a numbering gap at "Infrastructure 2" rather than
renumbering 3 and 4. The labels are deliberately opaque and renumbering would
relabel the surviving checks, discarding their history for cosmetics.
Not touched: the opentofu Proxmox provider, VM and LXC modules,
vm_152_k8s_3.tf, sync_state_from_pve.py, and the ansible PVE inventory. Those
carry state implications and want a deliberate pass rather than folding into a
manifest cleanup.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase 1 of the TRMNL X dashboard plan: stand up the self-hosted Terminus
(BYOS) server the panel will poll. The Home Assistant dashboard, the e-ink
theme and the screenshot renderer are later phases and are not in here.
Phase 0 verification, all confirmed against upstream before writing any of
this:
* TRMNL X IS a first-class model in Terminus (id v2 / label "TRMNL X",
1872x1404, 16 colours, 4-bit) — the stop-and-report condition in the
plan does not trigger. Note it is NOT in config/db/seeds.rb: the model
list is populated by the MODEL_SYNCHRONIZER job pulling trmnl.com, so
the Models list is empty until that job has run once.
* The HA add-on can run standalone. Our HA is plain HA Core, not
Supervised, but the add-on reads env vars in preference to
/data/options.json and authenticates with an ordinary long-lived
token — SUPERVISOR_TOKEN only flips an isAddOn boolean. So the
fallback path in the plan (upstream container, not add-on) is viable.
That deployment lands with Phase 4.
Deviations from the repo's house patterns, each deliberate:
* tr-pg runs PostgreSQL 18, not the standard 16. Terminus ships a
committed config/db/structure.sql containing `SET transaction_timeout
= 0;`, a GUC that does not exist before PG 17, so schema load against
a 16 server fails outright. This is the first non-16 CNPG cluster
here. The existing 16 pin is inherited from 739a67e9 (Apr 2026) and
has no documented rationale.
* The device endpoint is cleartext HTTP on a Cilium BGP LoadBalancer
(trmnl-device.renner.uno) rather than an Ingress. The firmware cannot
be given the internal CA, and a TLS-less Ingress would be a new
pattern for this repo — a dedicated LB address is not. The admin UI
keeps a normal TLS Ingress at trmnl.renner.uno; both front the same
Service. Same split as home.renner.uno / ha.renner.uno.
* ghcr.io/usetrmnl/terminus is NOT enrolled in image automation despite
publishing a usable tag stream. It serves the firmware and
/api/display contract to a wall-mounted device, upstream is pre-1.0
with no releases or changelog, and an unattended bump that breaks
either leaves the panel dark. Bumps are manual and reviewed. The
Valkey sidecar is enrolled — it touches no device-facing contract.
* Egress is an FQDN allowlist (trmnl.com, plus the S3 host that serves
firmware binaries), not toEntities: world. The S3 host is returned by
the API at request time rather than being a source constant, so it can
drift; the policy comment says where to look when firmware sync
starts failing. This also means Terminus's Extensions/Poll feature,
which fetches admin-configured URLs, will not work — accepted, since
screen content is pushed in by the renderer.
* The CNPG pods are deliberately left OUT of that egress policy so the
barman WAL archive path is not behind a rule that fails closed.
Images verified to exist in their registries before pinning:
ghcr.io/usetrmnl/terminus:0.68.0, valkey/valkey:9.1-alpine,
ghcr.io/cloudnative-pg/postgresql:18.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
*.k.renner.uno was a temporary migration namespace and is banned (owner
directive 2026-07-14, documented in openbao/ingress.yaml) — it was never
the live naming. Removes the .k. hosts/SANs from ical-proxy, loki,
grafana and the mosquitto cert, renames syslog.k.renner.uno to
syslog.renner.uno (cert + external-dns annotation; any syslog sender
still configured with the k. name must be repointed), fixes the stale
comments, updates e2e-test.sh probes to the real names, and drops
k.renner.uno from the external-dns domainFilters so leftover k. records
get garbage-collected by policy: sync.
The migration script's historical docstring reference stays — it
documents what the migration did.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
The dns.renner.uno/public=true label on an Ingress marks it as publicly
registered (Cloudflare wildcard -> NetBird proxy); external-dns-unifi
now runs with labelFilter dns.renner.uno/public!=true and skips those
resources entirely. Visibility is a property of the app resource itself
— there is no central name list to keep in sync with the proxy.
Also removes the dead .k.renner.uno hosts from the public apps'
Ingresses (their TLS certs reissue without those SANs) and splits
photos.renner.uno onto its own unlabeled Ingress: the label filter is
per-resource and that name stays LAN-registered. Remaining .k. names on
LAN-only apps (loki, ical-proxy, syslog) are untouched here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014S5CLfizEtzHS51kxJFQgc
status.renner.uno / status.k.renner.uno is a public, unauthenticated gatus
page. Its endpoints named the internal service estate (Authentik, Gitea,
Immich, Grafana, HA, Paperless, Proxmox, PBS, UDM DNS, Mosquitto) and gatus
exposed each check's target hostname/URL/port — and on failure the error
text (e.g. mosquitto.home-assistant.svc.cluster.local, 10.2.40.1) — via the
public results and /api/v1/endpoints/statuses API (all hide-* default false).
Per owner decision keep the page public (not Authentik-gated) but scrub the
disclosure:
- rename every endpoint to an opaque label (Web Service N / Infrastructure N),
removing product/vendor/hostname identifiers from the public `name`
- add a per-endpoint ui block with hide-hostname/hide-url/hide-port/
hide-conditions/hide-errors: true so gatus scrubs targets server-side
All checks keep running unchanged (real targets stay in `url`, visible only
to git/cluster operators). Public page now shows group + opaque label +
up/down + response time only. Maintainer opaque->real map kept as a comment.
Fixes#206
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The renner.uno fallthrough forwarder still pointed at the Proxmox DNS HA VIP
10.2.40.53, which is decommissioned/unreachable. Every non-Ingress renner.uno
name (device/service hostnames like udm.renner.uno) therefore dead-ended in
SERVFAIL/timeout. A routine home-assistant pod restart on 2026-07-14 forced HA
to re-resolve its device hostnames and left ~1374 entities unavailable / ~30
config entries stuck (ZHA, Shelly, UniFi, Bluetooth, cloud integrations).
Point the fallthrough at the UDM (10.2.40.1 / fd00::1), which serves the
split-horizon renner.uno zone (verified: it answers udm.renner.uno). Also
scrub the stale 10.2.40.53 references in the CoreDNS comment, bootstrap.sh
DNS_SERVER, and the Gatus monitor (which was pointlessly probing the dead VIP).
A live hotfix has already been applied to the k8s-gateway ConfigMap to restore
resolution; this makes it durable so Flux does not revert it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PwcQMntG9y9XRkNMPDq2fe
image-automation commits tag bumps straight to master with no CI gate, and
the broker is a single pod — so an image that fails to come up would be
silent. Two changes make that safe:
- gatus TCP-checks the broker's ClusterIP on :1883, so a pod that never comes
up alerts within one interval. Reachability needed a scoped egress (apps ns)
+ ingress (home-assistant ns, gatus pod on :1883 only), mirroring the
existing evcc -> mosquitto:1883 allow.
- strategy Recreate -> RollingUpdate (maxUnavailable:0, maxSurge:1). With the
PVC gone there is no RWO volume to fence, so the new broker comes up and
passes its readiness probe before the old one is torn down — no full-down
window on a bump/reschedule; clients on the old pod just reconnect. The brief
two-pod overlap is two independent stateless brokers behind the LB, fine with
no shared state + self-healing discovery.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the external-dns/cloudflare label from the gatus Ingress and remove
the status.renner.uno cloudflared route, moving the public edge to the
NetBird reverse proxy. Gatus stays public (no proxy auth — it's the
status page). LAN/mesh via status.k.renner.uno -> Traefik is unaffected.
MERGE ONLY AFTER the status proxy service + DNS are live, then roll
cloudflared, or status.renner.uno goes dark in the gap.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
These four endpoints have been reporting failure every monitor cycle:
the three HTTP hostnames (media/wiki/influx.renner.uno) are NXDOMAIN
with no Ingress in any namespace, and tcp://10.2.40.135:5432 no longer
responds (no TCP, no ICMP). Removed from in-cluster gatus to stop the
continuous false-failure noise; tombstones left in place for context.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
pbs.obg.renner.uno resolves to 10.3.40.96 — an offsite Proxmox Backup
Server on a remote network reached over WAN (~50ms latency). Checking
it from inside the cluster doesn't measure what it should: it conflates
WAN reachability, in-cluster DNS path stability (CoreDNS → k8s-gateway
→ proxmox-dns LXC → UDM), and the PBS itself. The intermittent UDP
loss on the DNS path produced spammy i/o timeout logs and false-
positive gatus failures.
Monitor offsite PBS from the offsite side, or via a check that doesn't
loop through cluster DNS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two Alertmanager alerts cleared at the source:
1. KubeDeploymentRolloutStuck / KubeContainerWaiting / KubePodNotReady
on apps/gatus. The deployment's RWO PVC (gatus-data, ceph-block)
can't be mounted twice. Default RollingUpdate strategy tries to
start the new pod BEFORE terminating the old one — surge=25% of
1 replica rounds up to 1 surge pod. The new pod hangs in
ContainerCreating indefinitely (no event, just a stuck volume
attachment) and ProgressDeadlineExceeded fires.
Recreate strategy guarantees the old pod releases the volume first.
Same fix the home-assistant Deployment already uses for the same
reason.
2. TargetDown blackbox-tcp / monitoring. The only target was the
keepalived VIP for the legacy authentik-HA Postgres LXC pair at
10.2.40.135:5432. That LXC stack was decommissioned when authentik
moved to in-cluster CNPG (ak-pg). Probe deleted; the comment
block in the file documents what's no longer there.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
gatus (status), ical-proxy, kube-prometheus-stack (alerts/dash/prometheus)
have no nginx-only annotations, so the migration is pure
ingressClassName: nginx → traefik plus apps/allow-from-ingress now
permitting the traefik label as a source. cloudflared retargets
status.renner.uno (the only WAN-exposed name in this batch); the rest
are LAN-only via external-dns-unifi.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
RustFS was retired 2026-05-09 (per the comment in
opentofu-bootstrap/rustfs.tf). The S3 endpoint at s3.renner.uno:9000
has been unreachable since then, but a lot of references in code
remained — health checks pinged a dead endpoint, alerts fired
permanently, and the ansible role + playbook + tofu locals would have
broken the next time anyone tried to use them. Cleaning up:
DELETED (pure RustFS, no other consumers):
- opentofu/rustfs.tf.disabled
- opentofu/providers.tf.backup
- ansible/inventory/group_vars/proxmox_rustfs.yml
- ansible/plays/install-rustfs.yml
- ansible/plays/archived/migrate-rustfs-storage.yml
- ansible/roles/rustfs/ (whole role)
EDITED (RustFS sections removed):
- opentofu/variables.tf: drop rustfs_count
- opentofu/locals.tf: drop rustfs_instances local
- opentofu/backup_jobs.tf: drop the rustfs_count VMID-range
- kubernetes/apps/gatus/configmap.yaml: drop the dead s3.renner.uno
health check
- ansible/roles/gatus/templates/config.yaml.j2: same removal in the
ansible-managed gatus template
- ansible/roles/prometheus-server/templates/alert-rules.yml.j2:
drop the RustFSUnhealthy alert
- ansible/roles/grafana-observability/templates/alert-rules.yaml.j2:
drop the rustfs-unhealthy alert
- ansible/plays/check-chaos-testing.yml: drop the RustFS test block
and the rustfs_vip var
NOT TOUCHED (caddy-ha retirement scope, separate cleanup):
- ansible/roles/caddy-ha/defaults/main.yml (still has
caddy_ha_s3_endpoint = "s3.renner.uno:9000")
- ansible/plays/install-crowdsec-caddy.yml (still loads
vars/rustfs_vault.yml)
- ansible/plays/vars/rustfs_vault.yml (loaded by the play above)
- scripts/restore-authentik-db.sh (manual one-off restore script
that still references s3.renner.uno)
- opentofu-bootstrap/{rustfs.tf,variables.tf} (separate tofu
project with local state; cleanup needs manual apply)
The Woodpecker secrets tf_rfs_access and tf_rfs_secret are no longer
consumed by anything in the repo and can be removed manually from the
Woodpecker UI when convenient.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The annotation filter caused records on annotated ingresses to flap
once the UniFi external-dns instance also managed them: external-dns
turns annotations in its prefix namespace into ProviderSpecific on the
desired endpoint, the kashalls UniFi webhook does not implement
AdjustEndpoints to strip Cloudflare-only properties, and Records()
returns observed endpoints with empty ProviderSpecific — so plan
computed an Update every reconcile and the records cycled in UDM.
Labels are not translated into ProviderSpecific, so swapping the
selector from `external-dns.alpha.kubernetes.io/cloudflare-proxied`
to a plain `external-dns/cloudflare=true` label removes the round
trip. The Cloudflare instance still proxies all matched records
because of the existing `--cloudflare-proxied` extraArg default.
Touches the seven resources that use the selector: the cloudflare
external-dns HelmRelease and the six ingresses currently published
through the cargo tunnel (audiobookshelf, gatus, home-assistant,
immich, paperless, authentik-server).
Add the missing .k.renner.uno or .renner.uno host to loki, gatus,
and home-assistant Ingresses so every app is reachable on both the
internal LAN-side name and the public name. Drop the authentik test
Ingress; auth stays on auth.renner.uno only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a global cluster-settings ConfigMap (CLUSTER_IN_DEVELOPMENT,
ACME_SERVER) consumed via Flux postBuild substitution on the
infrastructure-crds Kustomization. A new letsencrypt-cluster
ClusterIssuer reads ACME_SERVER from the ConfigMap at reconcile time;
every Certificate / Ingress in the repo now references letsencrypt-
cluster instead of letsencrypt-{prod,staging} directly.
While CLUSTER_IN_DEVELOPMENT=true, ACME_SERVER points at acme-staging-
v02, so the only certs the cluster can issue are LE staging — sidesteps
the recurring 5-cert/168h prod rate-limit hit during back-to-back
rebuilds (latest occurrence: 2026-05-05 alloy-syslog + loki-ingress).
Flip both keys together when leaving development.
Per-app KSes intentionally do NOT enable postBuild — Grafana dashboards
and HA's deployment shell scripts contain unrelated \${VAR} markers
that would be clobbered. Substitution stays scoped to post-crds/ which
has no other variable expansions.
The standalone letsencrypt-prod / letsencrypt-staging ClusterIssuers
remain defined as escape hatches but are no longer referenced.
e2e-test.sh now also asserts letsencrypt-cluster Ready.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Flip all ingress cert-manager.io/cluster-issuer annotations from
letsencrypt-prod to letsencrypt-staging. Five rebuilds in a week
tripped LE's per-identifier rate limit (5 certs / 168h) and left 3
hostnames with the ingress-nginx fake cert. Staging has ~30k/week,
so rebuilds stop being rate-limited. ClusterIssuer definitions for
both remain in post-crds; flip back per-ingress when ready for prod.
- Rework e2e-test.sh TLS check to validate the cert *issuer* contains
"Let's Encrypt" rather than chain-validating against the public CA
bundle. Same check works for both staging and prod, and still
catches the real failure mode (fake cert = cert-manager broken).
- HTTP checks now use -k since staging certs aren't publicly trusted;
the separate TLS check is what asserts cert-manager worked.
- Chain the e2e step onto .woodpecker/rebuild-k8s.yml so every rebuild
validates the resulting cluster end-to-end before reporting success.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Move ExternalDNS from infrastructure to apps (not a prerequisite)
- Set --default-targets with tunnel ID so Ingresses only need
the annotation external-dns.alpha.kubernetes.io/cloudflare-proxied
- Simplify Gatus Ingress annotation (no tunnel ID needed)
- SOPS secret for Cloudflare token co-located with ExternalDNS
To expose via tunnel: add one annotation to Ingress.
To switch to NetBird-only: remove the annotation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ExternalDNS watches Ingress annotations and creates proxied CNAME
records in Cloudflare pointing to the tunnel endpoint.
- Deploy ExternalDNS with Cloudflare provider
- Annotate Gatus Ingress with tunnel target
- annotationFilter ensures only explicitly annotated Ingresses get
public DNS (opt-in, not opt-out)
- policy: upsert-only prevents deleting records not managed by K8s
To expose a service via Cloudflare tunnel, add to Ingress:
external-dns.alpha.kubernetes.io/target: <tunnel-id>.cfargotunnel.com
external-dns.alpha.kubernetes.io/cloudflare-proxied: "true"
To switch to NetBird-only: remove the annotations. No DNS record = VPN only.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
First app migration to Kubernetes. Gatus monitors all homelab services
via HTTP/TCP/DNS health checks. Deployed as a Deployment with ConfigMap
for endpoints config, 1Gi ceph-block PVC for SQLite, and Ingress on
status.renner.uno with cert-manager TLS.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
security-bot
claude-bot
k.rennerandClaude Opus 4.8