Files
homelab/kubernetes/apps/monitoring/loki-basic-auth.sops.yaml
T
k.rennerandClaude Opus 4.8 3ae01e9f6d edge-1: keep logs off the eMMC (volatile journal + Alloy->Loki) + flash tuning
edge-1 is a Dell Wyse 3040 on a 7.3 GB internal eMMC (not the N100/NVMe the
README assumed). Unbounded docker json logs (a single zwave-js log had grown
to 214 MB) plus a persistent journald were churning the flash. Move all
logging off local flash and ship it to Loki instead:

- journald Storage=volatile (RAM), size-capped; purge the on-disk store
- docker json-file -> journald log driver (no container json logs on disk)
- Grafana Alloy (Arch pkg) tails the RAM journal -> Loki, WAL on /run tmpfs

Loki endpoint: the unauthenticated srv-network Loki is firewalled off the IoT
VLAN, so Alloy pushes to the authenticated cluster ingress
(https://loki.renner.uno) with a dedicated 'edge-1' basic-auth user — its
htpasswd line is added to the loki-basic-auth cluster secret, plaintext in the
edge_host ansible vault. Delivery activates once Flux reconciles that secret;
until then Alloy 401s and buffers in RAM.

Flash longevity: noatime on root, weekly fstrim.timer, and a weekly
edge-emmc-health check of the JEDEC life-time / pre-EOL estimators.

The Alloy config is purpose-built in the edge-host role rather than reusing
the shared monitoring-agent role: that role is Debian-oriented (apt + GitHub
binary), force-enables a docker log source that would double-ship against the
journald driver, and always configures a Prometheus remote-write whose
endpoint is unreachable from the IoT VLAN.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 08:49:07 +02:00

24 lines
1.4 KiB
YAML

apiVersion: v1
kind: Secret
metadata:
name: loki-basic-auth
namespace: monitoring
type: Opaque
stringData:
users: ENC[AES256_GCM,data:LFh7/+lhYoyAwELQaG2EfMBCipoJrG9/QCrhUGM+k5+mpS2L2DjHBNE7mei80+sqVgmPZoVd2tQgzI9icwZdQ94UiJttGeT1cRqv1AyJxGqgVZ7wrKET11nxCeaNtymLj8kfI3oaSiNZ3BEvCPFqN6ZtPAYVD6uVPkkRect+aQQGvmTLAUKY+A==,iv:XWuHPXyOsNpBjaLC2OZxYeNBPkr4hOMEZMMAgBgBDq0=,tag:ih7dS4ifa5OQQzJfLQGQIg==,type:str]
sops:
age:
- recipient: age1yqjvqkwwwxdj7mr546zfs5ckzuwndp69ya2n2drd4vypwns8w4ys04qfy3
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsV2RVL0t3UkRmcmVJQmJp
TlJYZzEzc1hnd05zSWlnTk02UDJmVUZ6NWk4Cnd2WnZyQkZJVThaVE9xSS8xVlpX
NWNnK01wby9FbTc5Y1Rmdms5VVdIUGMKLS0tIHRhcmtjS3VzckpsUWNLZHhtUWpj
WlhndVNoeFJkMGdhY05RbjFuY2J5Z1EKCzi11fwIkATwPUHvAjuvF0LN/jZDHXGA
jGIVhngGk/61OKRm3+zuKoCsEHkdEuXRV+En15raeeWvUJIq9+Pv6g==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-07-07T06:40:10Z"
mac: ENC[AES256_GCM,data:n21Pljk9aWDoXtVFrWuLnNr6wiPWwfpx5VBPYIoyHnmTFMAd0kK8ztNK+yef9yULRChef1f0mfFK9pevR/oJ6W3RXlXd5Q4NhhvIOTCjpIrYSzdRg0Q8x9trXSLQPuVhh4MiDrOeRE1/ZnAgYqxHWQyXrpSEJm+4/TCy1u/xuFI=,iv:7GjRK1kJqpDtrBgLZH57DvdlhG0iixp+GN2OPZRpEu4=,tag:loWJMVcecEb4vxwBGLCGlQ==,type:str]
encrypted_regex: ^(data|stringData)$
version: 3.10.2