# Youssef's Homelab Context Document

Generated: 2026-04-05 | Last updated: 2026-09-17
Source: Perplexity Homelab Space (historical baseline) + ongoing YB Homelab maintenance

---

## 1. Demographics

| Field | Detail |
|-------|--------|
| Name | ******** |
| Email | ******** |
| Cloudflare Access Domain | youssefbouanani.com |
| Location | Casablanca, Morocco |
| ISP | IAM (Maroc Telecom) 200 Mbps Fiber (188–198 Mbps actual) |
| Vehicle | Tesla Model 3 "Tessy the beast" – 32,611 km, FW 2026.2.9.1 |
| Blog | blog.smarthomes.ma (Ghost) |
| Work | Corporate laptop HP EliteDesk with Zscaler; MeshCentral for remote access workaround |
| Subscriptions | Perplexity Max (Apr 2026), Google Gemini Pro (Sep 2026 expiry) |

### Family
- Brother: ********
- Family members: ********
- Parents: ********

-Family detail: ********

## 2. Active Projects & Interests

### Infrastructure
- **Hypervisor**: Proxmox cluster (2 nodes: pve and NAS). pve node: 2/4 VMs, 6/6 LXCs, 88% RAM; NAS node: 2/2 VMs + 1 LXC (TrueNAS + PBS + AI stack), ~85% RAM. GCP as Qdevice
- **Storage**: TrueNAS Scale (VM 102 on NAS node, boot disk on local-lvm), Pool Bolt 18% of 1.8TiB (set up currently as single drive — mirroring deferred, planned as part of full homelab revamp), Pool Tank 39% of 10.8TiB - this is a mirrored 2x12TB drives pool for storage and media. Pool Bolt = 1.8 TiB; Pool Tank = 10.8 TiB
- **Backup**: PBS local (VM-110) + 2 cloud instances: Tuxis (pbs005.tuxis.nl:8007), RemoteBackups (fra1-1.pbs-host.de:8007)
- **NVME wear**: 49% wear on KIOXIA NVMe (Zpool data storage, NOT boot drive) on NAS node – replacement drive on order. VM-102 already migrated off to local-lvm (MSI SSD). Only replication targets remain on KIOXIA.

### Hardware
- HARDWARE: HP EliteDesk G4 800 SFF with vPro for NAS proxmox node, Lenovo M920x tinnyPC with vPro for PVE proxmox Node

| Node | Hardware | CPU | RAM | Role |
|------|----------|-----|-----|------|
| M920x (PVE node) | Lenovo ThinkCentre M920x Tiny | i5-8500T 6c/6t 35W | 32 GB DDR4-2666 | Proxmox host: VMs & LXCs |
| HP G4 (NAS node) | HP EliteDesk 800 G4 SFF | i5-8500 6c/6t 65W | 48 GB DDR4-2666 | Proxmox host: TrueNAS VM & storage |
| GCP e2-micro | Google Cloud | Shared vCPU | 1 GB | Public-facing tunnel endpoint |
| M920q (spare) | Lenovo ThinkCentre M920q Tiny | i5-8500T 6c/6t 35W | 32 GB DDR4-2666 | Cold standby (not set up) |
| Brother PVE | ********'s node | — | — | 192.168.50.x |

### HP G4 NAS Node Exact Storage Layout

| Device | Drive | Size | Wear | Owner | Role |
|--------|-------|------|------|-------|------|
| /dev/sda | MSI S270 240GB | 240 GB SATA SSD | N/A | **Proxmox Host** | **BOOT DRIVE** (/boot/efi + pve-root ext4 + local-lvm thin pool). Hosts VM-102 (TrueNAS) boot disk + VM-104 (Zorin) disk + LXC 109 (ai-stack) disk. |
| /dev/nvme0n1 | KIOXIA KBG30ZMV256G | 256 GB | 49% | Proxmox Host | **Zpool ZFS data storage** (NOT boot). Hosts replication targets for LXCs 103/106/111 + VM-110 (PBS) disk. Replacement NVMe on order. |
| /dev/nvme1n1 | Samsung 990 EVO Plus | 2 TB | 0% | TrueNAS VM | Bolt pool (PCIe passthrough). TrueNAS system dataset + ALL Docker app volumes. |
| /dev/sdb | Seagate ST12000NE0008 | 12 TB HDD | N/A | TrueNAS VM | Tank pool drive 1 (passthrough). Mirrored with /dev/sdc. |
| /dev/sdc | Seagate ST12000NM001G | 12 TB HDD | N/A | TrueNAS VM | Tank pool drive 2 (passthrough). Mirrored with /dev/sdb. |

> **Key rule**: The Samsung 990 EVO Plus (Bolt) and both Seagate HDDs (Tank) are passed through directly to the TrueNAS VM. Proxmox has no visibility into them. New LXCs and VMs land on either the KIOXIA Zpool (being replaced) or the MSI 240 GB SATA SSD (local-lvm).

### HP G4 NAS Node Boot Drive Layout (MSI S270 240GB — /dev/sda)

sda (MSI S270 240GB) — PROXMOX BOOT DRIVE
├─ sda1 1007K BIOS boot
├─ sda2 1G vfat /boot/efi
└─ sda3 222.6G LVM2\_member
  ├─ pve-root 65.6G ext4 / ← Proxmox root
  ├─ pve-swap 8G swap
  ├─ pve-data 130.3G ← local-lvm thin pool
  ├─ pve-vm-102 32G ← TrueNAS boot disk (migrated from Zpool Apr 2026)
  ├─ pve-vm-104 32G ← Zorin OS disk
  └─ pve-vm-109 40G ← ai-stack LXC disk (Apr 2026)

### HP G4 NAS Node Zpool Contents (KIOXIA 256GB — /dev/nvme0n1)

| Dataset/Zvol | Size | Type | Status |
|---|---|---|---|
| Zpool/vm-110-disk-0 | 18.3 GB | ProxmoxBackup VM (local PBS) | **Running — local PBS instance** |
| Zpool/subvol-103-disk-1 | 2.35 GB | Jellyfin (LXC 103) | Replication target from PVE — auto-rebuilds |
| Zpool/subvol-106-disk-0 | 4.38 GB | MeshCentral (LXC 106) | Replication target from PVE — auto-rebuilds |
| Zpool/subvol-111-disk-1 | 7.75 GB | Docker LXC (LXC 111) | Replication target from PVE — auto-rebuilds |
| **Total used** | **~33 GB** | of 238 GB | |

### HP G4 TrueNAS ZFS Pools

| Pool | Drives | Usable | Redundancy | Contents |
|------|--------|--------|------------|----------|
| Bolt | 1× Samsung 990 EVO Plus 2 TB | 1.8 TiB | NONE (single drive — mirroring deferred to homelab revamp) | TrueNAS system dataset. ALL Docker app volumes (Immich DB/thumbs/ML/originals, Nextcloud, Radarr, Sonarr, NZBGet, Jellyseer, Audiobookshelf, Calibre, Kavita, Photodrop). |
| Tank | 2× Seagate 12 TB HDD (mirror) | 10.8 TiB | Mirror (survives 1 drive loss) | Bulk media storage. Snapshot replication targets. PBS backup storage. 39% used (~4.2 TiB). Target destination for Immich originals post-migration. |

### HP G4 Available Slots and Expansion

| Slot/Bay | Type | Status | Notes |
|----------|------|--------|-------|
| M.2 slot 1 | PCIe x4 2280 NVMe | Occupied | Samsung 990 EVO Plus 2 TB. The Bolt pool drive. Passed through to TrueNAS. |
| M.2 slot 2 | PCIe x4 2280 NVMe | **EMPTY — target for new NVMe** | Install new 512 GB–1 TB TLC NVMe here. Becomes new Zpool data storage (replacing KIOXIA). Replacement on order. |
| 2.5" SATA bay | SATA III | Occupied | MSI S270 240 GB SSD. **Proxmox boot drive** + local-lvm. |
| PCIe x16 (wired x4) | PCIe 3.0 | Empty | Available for GPU, HBA, or NVMe adapter if needed. |
| PCIe x1 (2 slots) | PCIe 3.0 x1 | Empty | Could host SATA expansion card (not recommended). |

> **Fikwot 512 GB SATA SSD note**: The HP G4's second M.2 slot is NVMe-only (PCIe), not SATA M.2. The Fikwot is a 2.5" SATA drive and the 2.5" bay is already taken by the MSI SSD. The Fikwot cannot be used inside the HP G4 without a PCIe-to-SATA card (not recommended). Keep as cold spare.

### M920x PVE Node Storage

| Storage | Type | Size | Role |
|---------|------|------|------|
| Zpool | ZFS (NVMe PCIe onboard) | ~238 GB usable | Proxmox Zpool – all VM/LXC disks |

> M920x Tiny has one M.2 slot only. No expansion bays. Both nodes use "Zpool" as the storage backend name — this enables seamless cross-node VM/LXC migration.

### Networking
- **Topology**: IAM ISP NOKIA Fiber router → Asus RT-AX58U v2 (Main Router with Gnuton's Merlin firmware + Tailmon on USB 3.0) → 2×5 ports GBE TP-Link switches → TP-Link Deco S4 APs (3 APs connected through Ethernet cables). Subnet 192.168.68.0/24. Brother (********) Subnet 192.168.50.0/24.
- ISP Router → Asus RT-AX58U v2 (Gnuton's Merlin) → TP-Link Deco S4 APs (AP mode)
- **Local Subnet**: 192.168.68.0/24
- **Brother Subnet**: 192.168.50.0/24
- **DNS**: AdGuard Home at 192.168.68.36 serves as network-wide DNS for 192.168.68.x subnet; admin UI on :84; Pi-hole decommissioned. After changing Cloudflare records from proxied to DNS-only, AdGuard may temporarily serve stale Cloudflare A/AAAA answers; compare against `1.1.1.1` and clear AdGuard cache when needed.
- **Reverse Proxy**: Nginx Proxy Manager at 192.168.68.36 (admin :81). Direct-public NPM services: Jellyfin, Immich, Nextcloud, PhotoDrop.
- **Public Access**: Cloudflare Tunnel + Zero Trust + Authentik OIDC for most services. Four large-file/streaming services intentionally bypass the Cloudflare reverse proxy via DNS-only records to `origin.youssefbouanani.com`.
- **Direct origin**: `origin.youssefbouanani.com` → A `********` (**DNS only**). `photos`, `photodrop`, `nextcloud`, and `media` are DNS-only CNAMEs to `origin`.
- **WAN/NAT**: Asus WAN IPv4 `********`; TCP 80 → `********:80`, TCP 443 → `********:443`.
- **Remote Access**: Tailscale mesh (all devices); GCP VPS (e2-micro, IP 35.211.62.198, Tailscale 100.86.11.23) as exit node
- **VPN**: Privado VPN on Asus router (speed issues noted ~10 Mbps)

### Backups and Replication
- **Brother replication**: Daily TrueNAS replication task (Youssef Truenas snapshots → ******** Truenas) via Tailscale.
- **Router backup**: Daily at 02:30 via custom script on Asus RT-AX58U v2. Backs up NVRAM, JFFS (scripts/addons/certs), and Entware (Tailscale/Tailmon/Skynet/packages) to TrueNAS Bolt/configs/router-backups via Dropbear SSH key auth (dbclient). 30-day retention with auto-cleanup. Persists across reboots via services-start. CIFS/SMB not available on Gnuton's Merlin build; SCP/SSH used instead.
- **Proxmox node config backup**: Daily at 08:00 on both pve and NAS nodes via root/nodebackupconfigs.sh. Pushes to Tuxis PBS (ns: ProxmoxNodes) with fixed archive names for deduplication. Auto-syncs to RemoteBackups PBS daily at 01:00. Gotify notifications on success/failure. Captures /etc/pve, /etc/network, /etc/fstab, /etc/modprobe.d (VFIO passthrough on pve, ZFS ARC tuning on NAS), /etc/sysctl.d, /etc/ssh, /etc/apt, /etc/systemd/system, root scripts + .ssh keys, plus metadata (dpkg list, lsmod, kernel cmdline, node config JSON, crontab).
- **Proxmox cross-node replication**: LXCs 103 (Jellyfin), 106 (MeshCentral), 111 (Docker) replicated from PVE node → NAS node Zpool via Proxmox replication jobs.
- **Docker DB backups to Backblaze B2**: Daily at 02:30 on both Docker LXCs. All dumps gzipped. B2 bucket: `youssef-docker-db-backups`. 7-day local retention + 8-day B2 retention. Gotify notifications on success/failure. Total B2 usage ~3.3 GB / 10 GB free tier.
- **.36 script** (`/root/backup-docker-dbs.sh`): Authentik (pg\_dump ~12MB), TeslaMate (pg\_dump ~97MB), NPM (SQLite ~73KB), AdGuard (YAML ~8.5KB), Uptime Kuma (SQLite ~325MB gzipped), Gotify (SQLite ~8.8MB) → rclone to B2.
- **.57 script** (`/root/scripts/backup-docker-dbs.sh`): Dawarich (pg\_dump ~5MB), Trilium (SQLite ~10MB), Mealie (SQLite ~1MB) → AWS CLI to B2.
- **Ghost backup**: Daily at 03:00 on .36 (`/root/backup-ghost.sh`). MySQL dump + content tar → rclone to B2 `youssef-ghost-backups`. 7d local + 8d B2 retention. Gotify alerting.
- **Spliit backup**: Daily at 03:00 on .57 (`/root/scripts/spliit\_backup.sh`). PostgreSQL dump → AWS CLI to B2 `spliit-backups`. 7 files in bucket. Gotify alerting.
- **Spliit receipts**: App-native upload on creation → B2 `spliit-receipts` (public). Keep all. No alerting configured. Intentionally left without alerting — low stakes.
- **Cloudflare export**: Daily at 01:00 on .36 (`/root/backup-cloudflare.sh`). Zone/DNS export → TrueNAS Bolt via SCP.
- **Home Assistant**: Google Drive backup addon — daily at 04:00, 21 generational retention, HA sensor alerting.
- **Immich photos**: rclone copy daily at 06:00 → OneDrive (1 TB, additive/never deletes). Gotify alerting. Also covered by ******** replication via TrueNAS snapshots.
- **Immich DB dumps**: Built-in periodic dumps to Bolt/configs/immich/backups/ — covered by Bolt snapshots + rclone. Via parent tasks.
- **TrueNAS Config**: Manual export before upgrades → Google Drive (Truenas\_Config\_Backup). 12 historical copies kept. Manual trigger only. Intentionally kept manual — upgrades are infrequent.
- **Local TrueNAS Snapshots**: Configured and documented. Snapshot schedules active on both Bolt and Tank pools. Used as source for brother replication.

### 3-2-1 Coverage Matrix

| ID | Workload | PBS Local | PBS NAS | PBS DE | PBS NL | Cold USB | App/Cloud | 3-2-1? |
|----|----------|-----------|---------|--------|--------|----------|-----------|--------|
| 100 | HomeAssistant | — | ✓ | — | ✓ | — | ✅ GDrive (21 gen) | ✅ Yes |
| 102 | TrueNAS | — | ✓ | ✓ | ✓ | — | ✅ GDrive config (manual) | ✅ Yes |
| 103 | Jellyfin | ✓ | ✓ | ✓ | ✓ | — | — | ✅ Yes |
| 104 | Zorin OS | — | ✓ | — | — | — | — | ⚠️ Single copy — intentional (test/disposable VM) |
| 105 | Win11 | — | ✓ | ✓ | ✓ | — | — | ✅ Yes |
| 106 | MeshCentral | — | ✓ | ✓ | ✓ | — | — | ✅ Yes |
| 107 | Vaultwarden | — | ✓ | ✓ | ✓ | — | — | ✅ Yes |
| 108 | rclone | — | ✓ | — | ✓ | — | — | ✅ Yes |
| 109 | ai-stack | ✓ | ✓ | ✓ | ✓ | — | — | ✅ Yes (added 2026-04-19) |
| 110 | PBS VM | — | — | — | ✓ | — | — | ✅ Yes |
| 111 | Docker (.36) | ✓ | ✓ | ✓ | ✓ | — | — | ✅ Yes |
| 112 | Ubuntu VM | — | ✓ | — | ✓ | — | — | ✅ Yes |
| 117 | Docker (.57) | ✓ | ✓ | ✓ | ✓ | — | — | ✅ Yes |

### Self-Hosted Services
- **Media**: Jellyfin, Radarr, Sonarr, Jellyseer, NZBGet/Usenet (UWEKA), Audiobookshelf, Calibre, Calibre-Web, Kavita
- **Photos**: Immich (main) + Immich Kiosk (parents' house)
- **Identity/Auth**: Authentik OIDC + Cloudflare Zero Trust
- **Passwords**: Vaultwarden (family)
- **Home Automation**: Home Assistant + TeslaMate/Grafana dashboards + HACS
- **Notes**: Trilium Notes
- **Recipes**: Mealie (recettes)
- **Location**: Dawarich (Google Timeline replacement)
- **Notifications**: Gotify
- **Local AI**: Open WebUI + Ollama + SearXNG on LXC 109 (192.168.68.40). Models: qwen3:14b (primary), gemma4:e4b (research, 32k ctx), qwen3:4b (fast), phi4-mini (coding), nomic-embed-text (RAG embeddings). Web search via SearXNG default ON. Admin hardened: sign-ups OFF, community sharing OFF, default role user.
- **Remote Access**: MeshCentral (replaced RustDesk, blocked by Zscaler). Installed via npm at /opt/meshcentral on LXC 106 (192.168.68.75). Updated to latest version on 2026-04-05 to fix Windows 11 24H2 agent install issue (WMIC removal). SSO enabled via Authentik OIDC. Device groups: AMT/vPro, Brother, HP G4 Backup, Desktops (BMAX-HOMEPC new desktop), Home Servers (980/108).
- **Code**: code-server (VS Code in browser) with SSH to Docker LXC, Docker2 LXC, PVE host
- **Cloud Storage**: Nextcloud (on TrueNAS)
- **Monitoring**: WatchYourLAN, Uptime Kuma (local + GCP) monitors health across local and GCP infrastructure
- **Utilities**: Stirling-PDF, Spliit, Zakat calculator, Book requests manager
- **DASHBOARD**: Homepage (gethomepage) v1.12.3 at home.youssefbouanani.com
- **Docker UI management**: Portainer at portainer.youssefbouanani.com
- **Docker compose DEVOPS**: Code Server at code.youssefbouanani.com – workspace: /config/homelab.code-workspace

### Active / Done
- **Local AI stack deployed** (2026-04-17): LXC 109 on NAS node (192.168.68.40, 4 cores / 16 GB RAM / 40 GB disk on local-lvm, nesting enabled). Docker Compose runs Ollama (:11434), Open WebUI (:3000), SearXNG (:8081). 4 chat models + 1 embedding model pulled and tuned with custom system prompts enforcing inline citations and preferring retrieved sources. Admin settings hardened (sign-ups OFF, community sharing OFF, user default role). Web search default ON via SearXNG (result count 8, concurrency 10, timeout 20). Document RAG via Ollama + nomic-embed-text (chunk 1500, overlap 150, top K 5).
- **Backup & DR plan fully resolved** (2026-04-19): All active workloads 3-2-1 compliant. LXC 109 added to PBS Local + Tuxis + RemoteBackups. Zorin OS (VM 104) intentionally left at single copy (test/disposable VM). Spliit receipts alerting intentionally skipped (low stakes). TrueNAS config backup intentionally kept manual (infrequent upgrades).

### Planned / In Evaluation
- Isolate Authentik in its own LXC if still co-hosted
- Cold PBS backup – Triggered on USB hard drive plug-in
- Tesla automations in HA – Off-peak charging, solar excess charging
- Xiaomi Mi Scale – Self hosting data
- Create `ai.youssefbouanani.com` Cloudflare route for Open WebUI

---

## 3. Infrastructure Detail

### Proxmox Node pve (192.168.68.122)

| ID | Name | Type | IP | Status |
|----|------|------|----|--------|
| 100 | HomeAssistant | VM | 192.168.68.147 | Running |
| 101 | Win11Template | VM | — | Template |
| 103 | Jellyfin | LXC | 192.168.68.52 | Running |
| 105 | Win11 | VM | — | Running |
| 106 | MeshCentral | LXC | 192.168.68.75 | Running |
| 107 | Vaultwarden | LXC | 192.168.68.107 | Running |
| 108 | rclone | LXC | — | Stopped |
| 111 | docker | LXC | 192.168.68.36 | Running |
| 112 | Ubuntu-VM | VM | — | Stopped |
| 117 | docker2 | LXC | 192.168.68.57 | Running |

pve-specific configs:
- /etc/fstab: NFS mounts to TrueNAS (immich data, spliit-minio), SSHFS mount for code-server (.57:root → /mnt/docker2-host)
- /etc/modprobe.d: VFIO passthrough (vfio\_iommu\_type1 allow\_unsafe\_interrupts=1), KVM nested virt, GPU blacklists (nouveau, nvidia, radeon, amdgpu, snd\_hda)

### Proxmox Node NAS (192.168.68.123)

| ID | Name | Type | IP | Notes |
|----|------|------|----|-------|
| 102 | TrueNAS | VM | 192.168.68.64 | 32 GiB RAM, hosts Immich/Nextcloud/arr stack. Boot disk on local-lvm (MSI SSD) — migrated from Zpool Apr 2026. |
| 104 | Zorin | VM | — | Disk on local-lvm (MSI). Stopped, no active purpose — test/disposable VM, intentionally single-copy backup only. |
| 109 | ai-stack | LXC | 192.168.68.40 | Running. Ollama + Open WebUI + SearXNG via Docker Compose. 4c/16GB/40GB on local-lvm (MSI SSD), nesting enabled. Deployed 2026-04-17. Added to PBS Local + Tuxis + RemoteBackups 2026-04-19. |
| 110 | ProxmoxBackup | VM | 192.168.68.71 | Running. Local PBS instance (192.168.68.71:8007). Disk on Zpool (KIOXIA). Stores local backups on NAS share. |

NAS-specific configs:
- /etc/modprobe.d/zfs.conf: zfs\_arc\_max=5027921920 (4.7 GiB ARC limit)
- storage.cfg: 7+ backends (local, local-lvm, Zpool, Tuxis PBS, backupsOnNas PBS, SMB\_Backup CIFS, RemoteBackups PBS, Cold-Backups PBS)
- Boot drive: MSI S270 240GB SATA SSD (/dev/sda) — pve-root ext4 + local-lvm

NAS node capacity: 2/2 VMs + 1 LXC (TrueNAS + PBS + AI stack), ~85% RAM.

### Docker LXC Container Inventory (Updated April 16, 2026)

#### LXC 111 — Docker (.36)

| Container | Image | DB Type | Backup |
|-----------|-------|---------|--------|
| authentik-postgresql-1 | postgres:16-alpine | PostgreSQL | backup-docker-dbs.sh → B2 |
| authentik-server-1 | goauthentik/server:2026.2.1 | — | Uses authentik-postgresql-1 |
| authentik-worker-1 | goauthentik/server:2026.2.1 | — | Uses authentik-postgresql-1 |
| authentik-redis-1 | redis:alpine | In-memory | Ephemeral cache |
| teslamate-database-1 | postgres:17 | PostgreSQL | backup-docker-dbs.sh → B2 |
| teslamate | teslamate/teslamate | — | Uses teslamate-database-1 |
| teslamate-grafana-1 | teslamate/grafana | — | Dashboards re-importable |
| teslamate-mosquitto-1 | eclipse-mosquitto:2 | — | MQTT broker, transient |
| nginx-proxy-manager | jc21/npm | SQLite | backup-docker-dbs.sh → B2 |
| adguardhome | adguard/adguardhome | YAML config | backup-docker-dbs.sh → B2 |
| uptime-kuma | louislam/uptime-kuma:1 | SQLite (~502MB raw) | backup-docker-dbs.sh → B2 (gzipped ~325MB) |
| gotify-gotify-1 | gotify/server | SQLite | backup-docker-dbs.sh → B2 |
| gotify-truenas-adapter | truenas-gotify-adapter | — | Stateless relay |
| ghost-official-ghost-1 | ghost:6-alpine | — | backup-ghost.sh → B2 |
| ghost-official-db-1 | mysql:8.0.44 | MySQL | backup-ghost.sh → B2 |
| ghost-official-traffic-analytics-1 | ghost/traffic-analytics | — | Regenerable analytics |
| trilium-relay | custom Python | — | Stateless bookmarklet relay |
| homepage | gethomepage | YAML | Config in compose |
| stirling-pdf | stirling-pdf | — | Stateless |
| cloudflareddns | cloudflare-ddns | — | Stateless |
| portainer | portainer-ce | BoltDB | Low value |
| keen\_bassi | cloudflared | — | Tunnel container |
| inspiring\_murdock | cloudflared | — | Tunnel container |
| watchtower | watchtower | — | Auto-updater |

#### LXC 117 — Docker2 (.57)

| Container | Image | DB Type | Backup |
|-----------|-------|---------|--------|
| dawarich\_db | postgis/postgis:17-3.5-alpine | PostgreSQL | backup-docker-dbs.sh → B2 |
| dawarich\_app | freikin/dawarich | — | Uses dawarich\_db |
| dawarich\_sidekiq | freikin/dawarich | — | Worker process |
| dawarich\_redis | redis:7.4-alpine | In-memory | Ephemeral cache |
| trillium-trillium-1 | triliumnext/notes | SQLite | backup-docker-dbs.sh → B2 |
| mealie | mealie:v2.6.0 | SQLite (1.1MB) | backup-docker-dbs.sh → B2 |
| spliit\_db | postgres:15 | PostgreSQL | spliit\_backup.sh → B2 |
| spliit\_app | spliit-mad | — | Uses spliit\_db |
| zakaty-zakaty-1 | cybrarist/zakaty | SQLite | Skip — re-enterable |
| homelab-hub | homelab-hub:latest | SQLite (Drizzle) | Skip — seeded, easy rebuild |
| immich\_kiosk | immich-kiosk | — | Stateless display |
| watchyourlan | aceberg/watchyourlan | SQLite | Skip — regenerates via scan |
| speedtest | speedtest-tracker | SQLite | Skip — historical nice-to-have |
| tugtainer | quenary/tugtainer | — | Docker pull UI |
| portainer | portainer-ce | BoltDB | Low value |
| portainer\_agent | portainer/agent | — | Agent, stateless |

#### LXC 109 — ai-stack (.40) — NAS node

| Container | Image | Port | Notes |
|-----------|-------|------|-------|
| ollama | ollama/ollama | 11434 | Models: qwen3:14b, gemma4:e4b, qwen3:4b, phi4-mini, nomic-embed-text |
| open-webui | ghcr.io/open-webui/open-webui | 3000 | Admin hardened: sign-ups OFF, community sharing OFF, default role user. RAG: Ollama + nomic-embed-text (chunk 1500 / overlap 150 / top K 5). Web search: SearXNG (count 8 / concurrency 10 / timeout 20), default ON. |
| searxng | searxng/searxng | 8081 | Local meta-search for Open WebUI web search feature |

Model tuning (Open WebUI):
- qwen3:14b — primary, temp 0.3, top_p 0.9, num_ctx 8192, web search ON, custom system prompt (inline citations, prefer retrieved sources)
- gemma4:e4b — research, temp 0.4, top_p 0.95, num_ctx 32768, web search ON
- qwen3:4b — fast/lightweight, temp 0.3, top_p 0.9, num_ctx 8192, web search ON
- phi4-mini — coding/reasoning, temp 0.3, top_p 0.9, num_ctx 8192, web search ON
- nomic-embed-text — embeddings for RAG/documents

### Service → IP Mapping

| Service | Subdomain | Destination |
|---------|-----------|-------------|
| AdGuard Home | adguard.youssefbouanani.com | 192.168.68.36:84 |
| Authentik SSO | authentik.youssefbouanani.com | Cloudflare Tunnel |
| Book Requests | bookrequests.youssefbouanani.com | 192.168.68.64:8084 |
| Books (audiobookshelf) | books.youssefbouanani.com | 192.168.68.64:30067 |
| Ebooks (Calibre-Web) | ebooks.youssefbouanani.com | 192.168.68.64:8083 |
| Dashboard (Home) | home.youssefbouanani.com | 192.168.68.36:2000 |
| Immich (Photos) | photos.youssefbouanani.com | 192.168.68.64:30041 (NPM direct-public; DNS-only via `origin`) |
| Immich Kiosk (Parents) | parents.youssefbouanani.com | 192.168.68.57:3000 |
| Jellyfin | media.youssefbouanani.com | 192.168.68.52:8096 (NPM direct-public; DNS-only via `origin`) |
| MeshCentral (Remote) | remote.youssefbouanani.com | 192.168.68.75:443 |
| Mealie (Recettes) | recettes.youssefbouanani.com | 192.168.68.57:9925 |
| Nextcloud | nextcloud.youssefbouanani.com | 192.168.68.64:30027 (NPM direct-public; DNS-only via `origin`) |
| Notes (Trilium) | notes.youssefbouanani.com | 192.168.68.57:8081 |
| Open WebUI (AI) | ai.youssefbouanani.com (pending) | 192.168.68.40:3000 |
| PBS (local) | pbs.youssefbouanani.com | 192.168.68.71:8007 |
| PDF (Stirling) | pdf.youssefbouanani.com | 192.168.68.36:8090 |
| Photodrop | photodrop.youssefbouanani.com | 192.168.68.64:9002 (NPM direct-public; DNS-only via `origin`) |
| Jellyseer (Requests) | requests.youssefbouanani.com | 192.168.68.64:30357 |
| Spliit | spliit.youssefbouanani.com | 192.168.68.57:3011 |
| Dawarich (Timeline) | timeline.youssefbouanani.com | 192.168.68.57:3091 |
| Vaultwarden (Vault) | vault.youssefbouanani.com | 192.168.68.53:8090 |
| Zakat Calculator | zakat.youssefbouanani.com | 192.168.68.57:8080 |
| Ghost Blog | blog.smarthomes.ma | — |
| Proxmox VE | proxmox.youssefbouanani.com | 192.168.68.122:8006 |
| TrueNAS Scale | truenas.youssefbouanani.com | 192.168.68.64 |
| Portainer | portainer.youssefbouanani.com | 192.168.68.36:9443 |
| Code Server | code.youssefbouanani.com | 192.168.68.36:8443 |
| Uptime Kuma (local) | monitoring.youssefbouanani.com | 192.168.68.36:3001 |
| Homelab Hub | hub.youssefbouanani.com | 192.168.68.57:3500 |
| Homelab Visualizer | homelab.youssefbouanani.com | Cloudflare Pages |

Additional internal services (no public subdomain):

| Service | Destination |
|---------|-------------|
| Uptime Kuma (GCP) | uptime.youssefbouanani.com → 172.17.0.1:3001 |
| Nginx Proxy Manager | 192.168.68.36:81 |
| Home Assistant | homeassistant.youssefbouanani.com → 192.168.68.147:8123 |
| Teslamate | teslamate.youssefbouanani.com → 192.168.68.36:3000 |
| Authentik IdP | authentik.youssefbouanani.com → 192.168.68.36:85 |
| Gotify | gotify.youssefbouanani.com → 192.168.68.36:8010 |
| WatchYourLAN | 192.168.68.57:8840 |
| OpenSpeedTest | 192.168.68.57:8765 |
| Ollama API | 192.168.68.40:11434 |
| SearXNG | 192.168.68.40:8081 |

---

## 4. Cloudflare Tunnels

| Tunnel | Runs On | Notes |
|--------|---------|-------|
| PVE node | 192.168.68.36 (Docker LXC) | Main tunnel – 2 cloudflared containers, 12+ routes |
| TrueNAS | 192.168.68.64 | Separate tunnel for TrueNAS node |
| GCP-US | 35.211.62.198 | Cloud tunnel (keep separate) |

---

## 4A. Cloudflare DNS, DDNS & Direct NPM Origin (Updated 2026-09-08)

### DNS model

- `youssefbouanani.com` → A `105.159.138.39` — **Proxied**
- `*.youssefbouanani.com` → CNAME `youssefbouanani.com` — **Proxied**
- `origin.youssefbouanani.com` → A `********` — **DNS only**
- `photos.youssefbouanani.com` → CNAME `origin.youssefbouanani.com` — **DNS only**
- `photodrop.youssefbouanani.com` → CNAME `origin.youssefbouanani.com` — **DNS only**
- `nextcloud.youssefbouanani.com` → CNAME `origin.youssefbouanani.com` — **DNS only**
- `media.youssefbouanani.com` → CNAME `origin.youssefbouanani.com` — **DNS only**

Purpose: the four direct services bypass Cloudflare's reverse proxy so large uploads and streaming are not constrained by Cloudflare's 100 MB request-body limit.

Expected path:

```text
client
  → <service>.youssefbouanani.com
  → origin.youssefbouanani.com
  → 105.159.138.39
  → Asus TCP 443
  → NPM 192.168.68.36:443
  → backend
```

### Cloudflare DDNS

Container: `favonia/cloudflare-ddns`

Logical environment:

```text
DOMAINS=youssefbouanani.com,origin.youssefbouanani.com
PROXIED=is(youssefbouanani.com)
IP6_PROVIDER=none
```

Behavior:
- root A record is maintained and remains proxied;
- `origin` A record is maintained and remains DNS-only;
- IPv6 DDNS is disabled;
- dedicated Cloudflare API token exists for DNS edit on the required zone only; token secret must never be stored in this context.

### ASUS firewall / port forwarding

Router: ASUS RT-AX58U_V2, ASUSWRT-Merlin / gnuton build.

Public WAN IPv4: `********`

Port forwards:

```text
WAN TCP 80  → 192.168.68.36:80
WAN TCP 443 → 192.168.68.36:443
```

A custom router-wide `CF_FILTER` iptables chain previously allowed only Cloudflare source CIDRs to NPM ports 80/443. This blocked all DNS-only/direct clients. The `CF_FILTER` rules and live chain were removed and must not be restored globally.

Current `/jffs/scripts/firewall-start`:

```sh
#!/bin/sh

/jffs/scripts/YazFi runnow & # YazFi Guest Networks
if [ -x /opt/bin/tailscale ]; then tailscale down; tailscale up; fi # Added by TAILMON
arp -s 192.168.68.123 ********
```

Security rule: Cloudflare-only restrictions must be applied per NPM virtual host (or via separate listeners/IPs/ports), not as a router-wide source-IP filter on 80/443.

### AdGuard DNS-cache behavior

Local resolver: `192.168.68.36`.

After changing a hostname from proxied to DNS-only, AdGuard can retain old Cloudflare A/AAAA answers for a short period. Diagnostic pattern:

```bash
nslookup <host> 1.1.1.1
nslookup <host> 192.168.68.36
```

If public DNS resolves through `origin.youssefbouanani.com → ********` while AdGuard still returns Cloudflare addresses (`104.21.x.x`, `172.67.x.x`, or `2606:4700:...`), clear the AdGuard DNS cache and retry.

### Direct NPM services

#### Immich

- Public hostname: `photos.youssefbouanani.com`
- Backend: `http://192.168.68.64:30041`
- WebSockets enabled
- Wildcard Let's Encrypt certificate
- Force SSL enabled

Advanced Nginx:

```nginx
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

client_max_body_size 0;
proxy_max_temp_file_size 16384m;

proxy_request_buffering off;
proxy_buffering off;
client_body_buffer_size 1024k;

proxy_read_timeout 600s;
proxy_send_timeout 600s;
send_timeout 600s;

location = /Portugal_Mototrip {
    return 302 https://photos.youssefbouanani.com/share/********;
}
```

Verified:
- backend healthy;
- NPM → Immich works;
- public direct route resolves to `105.159.138.39`;
- uploads through `photos.youssefbouanani.com` succeed without Cloudflare in the data path.

#### PhotoDrop / Immich Drop

- Public hostname: `photodrop.youssefbouanani.com`
- Backend: `http://192.168.68.64:9002`
- Container: `immich-drop` (`ghcr.io/nasogaa/immich-drop:latest`, host `9002` → container `8080`)
- Cache Assets off
- Block Common Exploits on
- WebSockets on
- Publicly Accessible

Advanced Nginx:

```nginx
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

client_max_body_size 0;

proxy_request_buffering off;
proxy_buffering off;

proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
send_timeout 3600s;
```

Verified:
- public direct GET reaches NPM/OpenResty and redirects to `/login`;
- a 140 MB video reached PhotoDrop and was identified as a duplicate with zero upload errors, proving the former Cloudflare 100 MB request-size path is bypassed.

#### Nextcloud

- Public hostname: `nextcloud.youssefbouanani.com`
- Backend: `http://192.168.68.64:30027`
- TrueNAS container: `ix-nextcloud-nextcloud-1`
- Cache Assets off
- Block Common Exploits on
- WebSockets on
- Publicly Accessible

Advanced Nginx:

```nginx
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

client_max_body_size 0;

proxy_request_buffering off;
proxy_buffering off;

proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
send_timeout 3600s;
```

PHP limits verified inside `ix-nextcloud-nextcloud-1`:

```text
max_execution_time = 0
max_input_time = -1
memory_limit = 512M
post_max_size = 3G
upload_max_filesize = 3G
```

Verified public direct route:
- resolves to `105.159.138.39`;
- wildcard Let's Encrypt certificate;
- `server: openresty`;
- HTTP/2 302 to `/login`;
- no Cloudflare `server` header or `cf-ray`.

#### Jellyfin

- Public hostname: `media.youssefbouanani.com`
- Backend: `http://192.168.68.52:8096`
- Backend server: Kestrel
- Cache Assets off
- Block Common Exploits on
- WebSockets on
- Publicly Accessible

Advanced Nginx:

```nginx
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_buffering off;

proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
send_timeout 3600s;
```

Verified public direct route:
- resolves to `105.159.138.39`;
- wildcard Let's Encrypt certificate;
- `server: openresty`;
- HTTP/2 302 to `web/`;
- no Cloudflare `server` header or `cf-ray`.

### Pending NPM security hardening

Because the router-wide Cloudflare-only filter was removed, NPM's public listener is directly reachable on forwarded TCP 80/443.

Next task:
1. inventory all NPM proxy hosts;
2. classify each as **direct-public**, **Cloudflare-only**, or **internal-only**;
3. leave `photos`, `photodrop`, `nextcloud`, and `media` unrestricted for direct public access;
4. restrict Cloudflare-only NPM virtual hosts to Cloudflare source ranges at Nginx/NPM level;
5. ensure internal-only hosts are not exposed through the public NPM listener.

Do not restore a global router-level Cloudflare source-IP filter on NPM ports 80/443.


---

## 5. code-server (Updated April 4, 2026)

Setup: Runs natively on .36 (LXC ID 111), NOT in Docker.

- Service: systemctl status code-server@root
- Config: ~root/.config/code-server/config.yaml — bind-addr: 0.0.0.0:8443, auth: password, cert: false
- Service override: /etc/systemd/system/code-server@root.service.d/override.conf
- Workspace: ~/root/homelab.code-workspace
- URL: https://code.youssefbouanani.com → Cloudflare tunnel → http://192.168.68.36:8443

Proxmox LXC 111 /etc/pve/lxc/111.conf:
- features: keyctl=1,nesting=1,fuse=1
- mp2: /mnt/docker2-host,mp=/mnt/docker2,ro=0

SSHFS mount on Proxmox host:
- .57:/root mounted at /mnt/docker2-host (persisted via /etc/fstab with x-systemd.automount)
- Bind-mounted into LXC 111 at /mnt/docker2

Workspace folders:
- Docker LXC (.36): /root
- Docker LXC (.57): /mnt/docker2

SSH config (~root/.ssh/config):
- All IdentityFile paths updated → /config/.ssh → /root/.ssh
- Hosts: docker2 (.57), pve (.122), nas (.123), gcp-us (35.211.62.198), asus-router (.1), homeassistant (.147)

Terminal profiles: one-click SSH → Docker2, PVE, NAS, HomeAssistant, Asus Router, GCP-US
Extensions: Container Tools, Docker, SSH FS (Kelvin Schoofs), YAML (Red Hat)
SSH FS configs: docker2, pve, nas, homeassistant, gcp-us, asus-router (password auth)

What works:
- Native terminal on .36 (full docker/docker compose access, no permission issues)
- Both LXC filesystems visible in Explorer tab
- One-click SSH terminal profiles to all servers
- SSH FS file browsing on all servers
- All Docker containers running normally
- Cloudflare tunnel healthy (HTTP to origin, not HTTPS)
- SSH keys intact at ~/root/.ssh

---

## 6. Router Backup (Added April 8, 2026)

Problem: Gnuton's Merlin firmware 3004.388.x lacks cifs.ko kernel module, so BACKUPMON cannot mount SMB shares.
Solution: Custom SCP-based backup script using Dropbear SSH (native to Merlin).

Setup:
- Script: /jffs/scripts/backup-router.sh
- SSH Key: /jffs/.ssh/id\_dropbear (Dropbear RSA 2048-bit key)
- TrueNAS User: youssef (local user with SSH Shell access enabled, home dir: /mnt/Bolt/configs/youssef)
- Remote Path: /mnt/Bolt/configs/router-backups/router-backup-YYYYMMDD
- Transfer Method: dbclient (Dropbear SSH client) with key auth (no password needed)
- Schedule: cru (cron) at 02:30 daily, persisted in /jffs/scripts/services-start
- Retention: 30 days auto-cleanup on TrueNAS

| File | Contents |
|------|----------|
| nvram\_YYYYMMDD.cfg | All router settings, DHCP, firewall rules |
| jffs\_YYYYMMDD.tar.gz | Scripts (services-start, YazFi, scMerlin, Skynet, SSL certs, addons) |
| usb\_entware\_YYYYMMDD.tar.gz | Entware packages (Tailscale, Tailmon, Skynet blocklists) ~39MB |

services-start contents:
#!/bin/sh
modprobe tun
/jffs/scripts/scmerlin startup scMerlin
/jffs/scripts/YazFi startup YazFi
cru a fixnasarp "\*/5 \* \* \* \*" "arp -s 192.168.68.123 ********"
cru a router-backup "30 2 \* \* \*" "/jffs/scripts/backup-router.sh"

---

## 7. Homelab Hub (NEW)

- **Stack:** React + Express + SQLite (Drizzle ORM + better-sqlite3)
- **Host:** lxc-117 (.57), port 3500 (internal), mapped from 5000
- **Docker dir:** `/mnt/docker2/root/docker/homelab-hub/`
- **Public URL:** https://hub.youssefbouanani.com (Cloudflare Tunnel, no Zero Trust auth — plain tunnel route)
- **Features:** Dashboard (Mission Control), Project detail with phase accordion + task toggle, Infrastructure reference, MD/HTML import engine, sidebar project delete (hover → confirm)
- **Seeded projects:** Infrastructure Restructuring (21 tasks), Backup Strategy (13 tasks), Homelab Visualizer (6 tasks)
- **iframe fix:** Express server removes `X-Frame-Options` and sets `frame-ancestors *.youssefbouanani.com` so it embeds cleanly in homelab-viz

---

## 8. Homelab Visualizer (homelab-viz)

- **Deployed to:** Cloudflare Pages at homelab.youssefbouanani.com
- **Files:** `index.html` + `homelab.json` + `dr-bible.md` (edit JSON locally, upload via Cloudflare UI)
- **Tabs:** Topology, Graph, List, Backups, **Project Hub** (iframe → hub.youssefbouanani.com), **DR Bible** (renders dr-bible.md with search, TOC, print)
- **DR Bible tab:** Added 2026-04-16. Loads `dr-bible.md` via fetch, renders with marked.js. Features: auto-generated clickable TOC, live search with highlighting, print/PDF button. Keyboard shortcut: `6`. Covers complete backup architecture, service-to-backup mapping, step-by-step recovery procedures for all VMs/LXCs/services, 5 major disaster playbooks, 6 DR exercises with schedule.
- **Maintenance:** Only `homelab.json` needs editing for infra changes; `index.html` only changes for new features; `dr-bible.md` updated when backup strategy changes
- **`\_headers`:** Custom CSP with `frame-src https://\*.youssefbouanani.com` to allow hub iframe
- **CDN deps:** D3.js v7 (graph view), marked.js (DR Bible markdown rendering)

---

## 9. Proxmox Node Config Backup (Updated April 8, 2026)

Script: root/nodebackupconfigs.sh on both pve and NAS nodes. Schedule: Daily at 08:00 via crontab. Logs to /var/log/nodebackupconfigs.log. Destination: Tuxis PBS (pbs005.tuxis.nl), ns: ProxmoxNodes → auto-syncs to RemoteBackups PBS (fra1-1.pbs-host.de) daily at 01:00 via sync job. Notifications: Gotify (token omitted from context) — success priority 2, failure priority 8.

| Archive | Contents |
|---------|----------|
| etc-pve.pxar | VM/LXC configs, storage.cfg, user.cfg, vzdump.cron, firewall |
| etc-network.pxar | Network interfaces |
| etc-modprobe.pxar | VFIO passthrough (pve), ZFS ARC tuning (NAS), GPU blacklists |
| etc-sysctl.pxar | Kernel params |
| etc-ssh.pxar | SSH host keys & server config |
| etc-apt.pxar | Repo sources & keys |
| root-home.pxar | Scripts (.sh), .ssh keys, .bashrc |
| etc-systemd.pxar | Custom units & overrides |
| hostfiles.pxar | fstab, hosts, resolv.conf, hostname, crontab, dpkg package list, lsmod, kernel cmdline, node config JSON |

Old script: node-backup.sh — Commented out on pve, can be deleted. Was backing up entire / to local PBS; replaced by targeted config backup.

---

## 10. Known Issues / Recurring Concerns

- **NPM direct-listener exposure**: OPEN 2026-09-08. Router-wide Cloudflare-only filtering was removed so direct-public services can work. Review all NPM hosts and apply Cloudflare source-IP restrictions per virtual host where appropriate; keep internal-only hosts off the public listener.

- **VM backup locks**: RESOLVED 2026-04-05. VMs/LXCs no longer remain locked after backup.
- **TrueNAS VM 102 backup**: RESOLVED 2026-04-05. Backups working reliably since fix.
- **Uptime Kuma DB bloat**: RESOLVED 2026-04-16. Retention set to 90 days in Settings → General. DB size will shrink over time.
- **GCP egress limit**: 1 GB/month free tier (running Portainer, Cloudflare, Tailscale exit node)
- **Privado VPN speed**: Only ~10 Mbps on Asus router (expected 200 Mbps)
- **KIOXIA NVMe degradation**: Zpool data storage drive at 49% wear. Replacement NVMe on order. VM-102 boot disk already migrated to local-lvm (MSI SSD). Only replication targets + VM-110 (PBS) remain on KIOXIA. When new NVMe arrives: install in M.2 slot 2, create new Zpool, move remaining data, rename pool, retire KIOXIA.
- **Gnuton's Merlin firmware**: No cifs.ko module → BACKUPMON network backup not possible; using custom Dropbear SCP script instead
- **MeshCentral Windows 11 agent install**: RESOLVED 2026-04-05. Agent GUI installer failed with "The specified procedure could not be found" on Windows 11 24H2 due to WMIC removal by Microsoft. Fixed by updating MeshCentral server to latest via npm install meshcentral@latest. CLI workaround: meshagent64-Desktops.exe -fullinstall.

---

## 11. Preferences & Conventions

- **Docker**: All services via Docker Compose files at root/docker/<service>/docker-compose.yml
- **Reverse proxy**: Cloudflare Tunnel + Authentik OIDC for most public services. NPM is used directly for four DNS-only services that must bypass the Cloudflare reverse proxy: Jellyfin (`media`), Immich (`photos`), Nextcloud, and PhotoDrop.
- **Backups**: PBS local (VM-110) + 2 cloud, Docker DB dumps to B2 (daily gzipped), rclone for offsite (immich to onedrive), router backup via Dropbear SCP to TrueNAS, node config backup via proxmox-backup-client to Tuxis PBS
- **AI tools**: Perplexity Max (primary), Google Gemini Pro, **self-hosted Open WebUI + Ollama on LXC 109** (qwen3:14b primary, gemma4:e4b research, qwen3:4b fast, phi4-mini coding, nomic-embed-text RAG; web search via SearXNG default ON; inline-citation system prompts)
- **Remote access**: MeshCentral (corporate workaround), Tailscale (homelab mesh)
- **Homelab context source of truth**: private Git repository `********/homelab-context`, canonical file `homelab-context.md`. Public sanitized publishing is automated via GitHub Actions → Wrangler v4 → Cloudflare Worker. Read endpoint: `https://homelab-context.youssef123.workers.dev/`. Automatic publishing verified working 2026-09-10. A confirmed infrastructure change is not considered complete until `homelab-context.md` is updated and pushed. Trilium receives the full private copy through the localhost-only relay. The public Worker receives only generated homelab-context-ai.md, never the canonical file.

---

## 12. Timeline / Changelog

- **2026-09-17**: Main Proxmox `pve` node maintenance completed after upgrade to **Proxmox VE 9.2.20**, running kernel `7.0.14-17-pve`. Removed obsolete accumulated 6.8 and older 6.17 kernels plus orphaned packages, cleaned the APT cache, and retained fallback kernels `7.0.12-1-pve` and `6.17.13-21-pve`. Root filesystem usage dropped from approximately **43 GB / 80 GB (57%)** to **30 GB / 80 GB (40%)**, reclaiming approximately **13 GB** without removing required data. `/boot` reduced from approximately 1.8 GB to ~500 MB. Required local vzdump backups for LXCs 103, 111 and 117 remain intentionally retained at approximately 22 GB. `local-lvm` is ~31.3% used. Main ZFS `Zpool` is ONLINE at ~78.1% usage with zero scrub errors and no known data errors. `Cold-Backups` remains intentionally inactive when not mounted. Final `systemctl --failed` reports **0 failed units**.

- **2026-09-17**: Removed obsolete Spliit/MinIO storage configuration. Confirmed Spliit runs in LXC 117 (`docker2`) as `spliit_app` + PostgreSQL `spliit_db`, with receipt/object storage now using **Backblaze B2 via S3** (`spliit-receipts`) rather than the old TrueNAS NFS/MinIO path. `/mnt/nas-spliit` was empty and had no active cron, systemd, Docker or application references. Removed CT 117 `mp0` bind mount and deleted the stale `192.168.68.64:/mnt/Tank/media/spliit-minio` entry from `/etc/fstab`; reloaded systemd and cleared the failed mount state. CT 117 retains only active Paperless mounts `mp1`-`mp4`. Security follow-up: rotate the Backblaze B2 S3 application key exposed during troubleshooting.



- **2026-09-10**: Improved public Worker verification reliability. `scripts/verify_public.py` now retries the `/` and `/homelab-context-ai.md` checks for up to approximately 60 seconds after deployment, allowing for brief Cloudflare Worker propagation differences before declaring a mismatch. Private/internal Worker paths continue to be checked for HTTP 404 exposure.
- **2026-09-10**: Work mode successfully edited the canonical homelab context file through code-server.

- **2026-09-10**: Hardened the Trilium context publishing path. Existing `trilium-relay` on Docker LXC 111 was changed from `0.0.0.0:5000` to localhost-only `127.0.0.1:5000`, removing LAN/Internet exposure of the relay write endpoint. Relay continues to update the fixed Trilium `Homelab Context` note via ETAPI on Docker2 LXC 117 (`192.168.68.57:8081`). Verified the canonical `homelab-context.md` can be pushed through the relay and retrieved from Trilium with an identical SHA256 hash. `publish.sh` now publishes the canonical context to private GitHub and Trilium, while also verifying the Cloudflare Worker deployment.

- **2026-09-10**: Proxmox PVE boot/storage cleanup completed. Extended logical volume `pve/root` by 12 GiB from <69.37 GiB to <81.37 GiB using previously unallocated VG space, then grew the ext4 root filesystem online. Root filesystem is now approximately 80 GiB with substantially improved free space, while 4 GiB remains unallocated in the `pve` VG as emergency headroom. `local-lvm` / the `pve-data` thin pool was intentionally left unchanged. APT cache and systemd journal cleanup reclaimed several GiB. Existing local vzdump job was verified to already use `keep-last=1` for VMIDs 103, 111 and 117, so no local backup retention change was required. `Cold-Backups` remains inactive/unmounted and Tuxis retention remains a separate follow-up item.

- **2026-09-10**: Completed Git-backed homelab context publishing workflow. Private repository `********/homelab-context` is now the canonical source of truth. `homelab-context.md` is automatically deployed on push to `main` using GitHub Actions, `cloudflare/wrangler-action@v4`, Wrangler v4, and scoped Cloudflare Actions secrets. Existing read endpoint `https://homelab-context.youssef123.workers.dev/` successfully serves the repository version as `text/markdown`. SSH authentication from Docker LXC 111 to GitHub configured for repository pushes. New operating rule: confirmed infrastructure changes are not complete until the canonical context is updated and pushed. Legacy Trilium update relay retained temporarily as fallback.

- **2026-09-08**: Reworked direct-public NPM routing for large-file/streaming services. Added DNS-only `origin.youssefbouanani.com` → `********`; root and wildcard remain Cloudflare-proxied. Moved `photos`, `photodrop`, `nextcloud`, and `media` to DNS-only CNAMEs pointing at `origin`. Cloudflare DDNS now maintains root + origin with `PROXIED=is(youssefbouanani.com)` and IPv6 DDNS disabled. Removed router-wide `CF_FILTER` Cloudflare-only iptables chain because it blocked direct clients; WAN 80/443 continue forwarding to NPM at 192.168.68.36. Verified direct NPM paths for Immich (********:30041), PhotoDrop (********:9002), Nextcloud (********:30027), and Jellyfin (********:8096), all terminating with Let's Encrypt/OpenResty rather than Cloudflare. PhotoDrop handled a 140 MB upload as a duplicate with zero errors. Nextcloud PHP limits verified at 3G upload/post, 512M memory, unlimited execution time. Documented AdGuard stale-DNS cache behavior after Cloudflare proxy-status changes. Created private Git repository `********/homelab-context` as the new canonical context source; Cloudflare Worker deployment automation was still pending at this point and was completed on 2026-09-10. Next security task: classify NPM hosts as direct-public / Cloudflare-only / internal-only and enforce Cloudflare source ranges per virtual host rather than globally at the router.

- **2026-04-19**: Backup & DR plan fully resolved. LXC 109 (ai-stack) added to PBS Local (VM-110), Tuxis (PBS NL), and RemoteBackups (PBS DE) — now 3-2-1 compliant. Zorin OS (VM 104) intentionally left at single-copy backup — test/disposable VM, no active purpose. Spliit receipts alerting intentionally skipped — low stakes. TrueNAS config backup intentionally kept manual — upgrades are infrequent. Only remaining open item: create `ai.youssefbouanani.com` Cloudflare tunnel route for Open WebUI.
- **2026-04-17**: Deployed local AI stack on NAS node as LXC 109 (ai-stack, 192.168.68.40, 4c/16GB/40GB on local-lvm MSI SSD, nesting enabled). Docker Compose stack: Ollama (:11434) + Open WebUI (:3000) + SearXNG (:8081). Pulled 4 chat models (qwen3:14b primary, gemma4:e4b research 32k ctx, qwen3:4b fast, phi4-mini coding) + nomic-embed-text for RAG embeddings. Each model given custom system prompt enforcing inline citations and preferring retrieved sources, with tuned temp/top_p/num_ctx and web search default ON. Open WebUI admin hardened: default role user, sign-ups OFF, community sharing OFF. Web search wired to SearXNG (result count 8, concurrency 10, timeout 20). Document RAG via Ollama + nomic-embed-text (chunk 1500, overlap 150, top K 5). NAS node now at 2/2 VMs + 1 LXC, ~85% RAM. Moved "Deploy local AI stack" from Planned → Active/Done. Added Service→IP entry `ai.youssefbouanani.com` (pending) → 192.168.68.40:3000. Follow-ups: create Cloudflare route for ai.youssefbouanani.com and add LXC 109 to backup/replication plan (currently not 3-2-1).
- **2026-04-16**: Created comprehensive Backup & DR Bible document (42K chars, 10 parts). Covers full backup architecture (4 layers), service-to-backup quick reference matrix, step-by-step recovery procedures for every VM/LXC/service, 5 major disaster playbooks (PVE loss, NAS loss, both nodes, internet outage, Cloudflare compromise), 6 DR exercises with quarterly/bi-annual schedule, 9 improvement recommendations, and printable emergency quick reference card. Integrated DR Bible as new tab in Homelab Visualizer — renders dr-bible.md with marked.js, auto-generated TOC, live search with highlighting, print/PDF support. Keyboard shortcut: 6. Updated Cloudflare Pages deployment to include dr-bible.md alongside index.html and homelab.json.
- **2026-04-16**: Backup strategy task list review and corrections. Confirmed Vaultwarden (LXC 107) and MeshCentral (LXC 106) fully covered by PBS local + remote — 3-2-1 compliant. Confirmed Home Assistant Google Drive backup addon running daily at 04:00 with 21 generational retention and HA sensor alerting. B2 storage monitoring deferred (nice-to-have, ~3.3 GB of 10 GB). Added full 3-2-1 coverage matrix and Layer 3 application-level backup details to context doc. Noted remaining gaps: Zorin OS (VM 104) not 3-2-1 compliant, Spliit receipts has no alerting, TrueNAS config backup is manual only. VM-102 boot disk confirmed migrated to local-lvm (MSI SSD) — no longer on KIOXIA Zpool. VM-110 (ProxmoxBackup) corrected to Running (local PBS instance, not stopped/to-delete). KIOXIA NVMe replacement drive on order — priority lowered. Bolt pool mirroring deferred to full homelab revamp — priority lowered. VM backup lock issue marked resolved (Apr 5). VM-102 backup reliability marked resolved (Apr 5). Uptime Kuma DB retention set to 90 days. TrueNAS snapshots confirmed documented. Removed orphaned cloudflared (distracted\_goldberg) from .57 LXC.
- **2026-04-11**: Implemented R3 Docker DB backups to Backblaze B2. Created B2 bucket `youssef-docker-db-backups`. Built backup scripts for .36 (Authentik, TeslaMate, NPM, AdGuard, Uptime Kuma, Gotify — 6 services, ~438MB/day gzipped) and .57 (Dawarich, Trilium, Mealie — 3 services, ~16MB/day). Both run daily at 02:30 via cron with 7-day rotation and Gotify alerts. Total B2 usage ~3.3GB/10GB free tier. Full container audit performed on both LXCs. Cleaned up: removed stale mealie from .36 (112KB test instance), removed teslamate-database-old-pg13 from .36 (PG13→PG17 migration leftover). Added Homelab Visualizer and Homelab Hub — both exposed via Cloudflare tunnel.
- **2026-04-09**: Corrected NAS node boot drive documentation — MSI S270 240GB (/dev/sda) is the Proxmox boot drive (ext4 + local-lvm), NOT the KIOXIA. KIOXIA (/dev/nvme0n1) hosts only the "Zpool" ZFS data storage (VM-102 TrueNAS boot disk 32.5GB + replication targets for LXCs 103/106/111 + VM-110). Confirmed both nodes use "Zpool" as storage backend name for seamless cross-node migration. Revised NVMe replacement plan: no Proxmox reinstall needed — just create new Zpool on new NVMe, move data, rename pool, retire KIOXIA. VM-104 (Zorin) confirmed on local-lvm, not Zpool.
- **2026-04-08**: Set up automated router backup (custom Dropbear SCP script replacing BACKUPMON — CIFS unavailable on Gnuton's Merlin). Backs up NVRAM + JFFS + Entware to TrueNAS Bolt/configs/router-backups daily at 02:30 with 30-day retention. Enabled SSH on TrueNAS for youssef user. Configured Dropbear RSA key auth. Persisted cron job in services-start.
- **2026-04-05**: Resolved VM backup lock issue. TrueNAS VM 102 backups now working reliably. MeshCentral updated to fix Windows 11 24H2 agent install issue.
## Public publishing architecture

- Canonical source: private GitHub repository, homelab-context.md.
- Full private mirror: Trilium through the localhost-only relay on Docker LXC 111, with SHA256 verification.
- Public AI bootstrap: homelab-context-ai.md preserves the full canonical document structure and operational infrastructure context while masking personal/family data, email and phone information, credential-like values, private-key material, MAC/device identifiers, credential-bearing URLs, and private repository ownership.
- Worker serves / and /homelab-context-ai.md only. The former /homelab-context.md route returns 404.
- 2026-09-10: Split private and public publication. GitHub Actions generates the sanitized copy and deploys from an isolated directory containing only Worker code, configuration, and sanitized text. Public SHA256 is checked against the generated file; Trilium SHA256 is checked against the canonical file.

---


## RustDesk - GCP VM

RustDesk is self-hosted on the Google Cloud Platform VM and is independent of the home Nginx Proxy Manager / Cloudflare reverse-proxy path.

### Server
- Current GCP public IPv4: `********`
- Previous IP `35.211.112.177` is obsolete.
- Docker Compose file: `/home/youssef123/rustdesk/compose.yml`
- Persistent data: `/home/youssef123/rustdesk/data`
- Both RustDesk containers mount `/home/youssef123/rustdesk/data` to `/root`
- Restart policy: `unless-stopped`

### hbbs
- Container: `hbbs`
- Image: `rustdesk/rustdesk-server:latest`
- Command: `hbbs -r 35.211.62.198:21117`
- TCP ports: `21115`, `21116`, `21118`
- UDP port: `21116`

### hbbr
- Container: `hbbr`
- Image: `rustdesk/rustdesk-server:latest`
- Command: `hbbr`
- TCP ports: `21117`, `21119`

### Clients
- ID server: `35.211.62.198`
- Relay server: `35.211.62.198`
- API server: blank
- Existing RustDesk server public key remains unchanged.
- BMAX Windows desktop and Android phone are configured against this server.

### Backup
Preserve:
- `/home/youssef123/rustdesk/compose.yml`
- `/home/youssef123/rustdesk/data`

The persistent data directory contains the RustDesk server identity and key material.