diff --git a/ai/RULES.md b/ai/RULES.md index cb2849e..d342e70 100644 --- a/ai/RULES.md +++ b/ai/RULES.md @@ -1,7 +1,7 @@ ## HARD MANDATE: PLAN BEFORE ACTION - **No Unapproved State Changes:** You MUST NOT execute any tool that modifies the filesystem, runs shell commands (excluding read-only discovery like `ls` or `grep`), or changes system state without first presenting a detailed plan and receiving an explicit "Directive" (e.g., "proceed") from the user. This rule is absolute. REQUIRED! -## Gemini Added Memories +## Added Memories - When working on the Hugo wiki, docs, or Caddy routing, immediately run the docs_architecture skill to understand the decoupled architecture, deployment logic, and known Docker/Caddy networking quirks. - When writing Hugo/Blowfish Markdown front matter, always include 'author', 'date' (ISO-8601), 'lastmod' (YYYY-MM-DD), and 'tags' (list) at the bottom of the metadata block. - Never perform manual builds or deployments of the Hugo sites to bypass the CI/CD pipeline. Always rely on the Gitea Actions workflows for deployment. diff --git a/ai/skills/homelab-infrastructure-review/SKILL.md b/ai/skills/homelab-infrastructure-review/SKILL.md new file mode 100644 index 0000000..9ad333d --- /dev/null +++ b/ai/skills/homelab-infrastructure-review/SKILL.md @@ -0,0 +1,146 @@ +--- +name: homelab-infrastructure-review +description: > + Performs a 3-way infrastructure reconciliation audit: homelab_Infra.md + documentation (including Nginx Proxy Route mappings) vs. live Docker container + states vs. homelable MCP canvas topology. Also audits the Homepage Dashboard + services configuration to ensure newly proxied services are accessible. + Triggers ONLY on explicit user command. No automation or cron. +version: 1.4.0 +author: wompmacho +date: 2026-05-30 +platforms: [linux, macos, windows] +metadata: + hermes: + tags: [homelab, infrastructure, audit, reconciliation, mcp, docker, nginx, homepage] + related_skills: [docker-workflows, workflow-management] +--- + +# Homelab Infrastructure Review + +Performs a 3-way infrastructure reconciliation audit of living services, active proxy routes, and the network topology canvas. + +## 1. Core Objectives + +This skill provides a highly-disciplined auditing workflow that: +1. **Identifies all live services** on the host. +2. **Identifies all active proxies and domains** for services by parsing Nginx Proxy Manager config files. +3. **Cross-references** them against the source of truth documentation (`homelab_Infra.md`). +4. **Cross-references** them against the `homelable-mcp` visual canvas topology. +5. **Audits the Homepage Dashboard services configuration** (`/srv/configs/docker_compose/homepage/config/services.yaml`) to ensure all proxied services are accessible. +6. **Generates exact proposed changes** to: + - The markdown documentation (`homelab_Infra.md`) + - The `homelable-mcp` visual canvas nodes and edges + - The Homepage Dashboard services config (`services.yaml`), with services correctly split into their relevant hosts/categories. + +--- + +## 2. Trigger Conditions + +This skill is invoked **only on explicit, manual user command** — never automatically or periodically. +Acceptable trigger phrases: "review homelab", "infrastructure audit", "reconcile homelab", "homelab review", "check infrastructure". + +No cron, no automated polling, and no silent background execution. + +--- + +## 3. Source Inventory and Boundary Map + +| Source Component | Path / Port / Endpoint | Permission Boundaries | +|---|---|---| +| **Infra Document** | `/srv/docs/public/projects/homelab/homelab_Infra.md` | Read: ✅ Write: ❌ (requires explicit user confirmation, never run git commands without auth) | +| **Live Docker State** | Host session via `docker ps` | Read: ✅ Write: ❌ (no automated container lifecycle modifications allowed) | +| **Nginx Proxy Configs** | `/app/nginxproxy/data/nginx/proxy_host/*.conf` | Read: ✅ Write: ❌ (read-only proxy blocks mapping) | +| **Homelable Backend API** | `http://localhost:9445/api/v1` | Read: ✅ Write: ❌ (node/edge canvas edits require explicit confirmation) | +| **Homepage Config** | `/srv/configs/docker_compose/homepage/config/services.yaml` | Read: ✅ Write: ❌ (requires explicit user confirmation) | +| **Homepage Layout** | `/srv/configs/docker_compose/homepage/config/settings.yaml` | Read: ✅ Write: ❌ (requires explicit user confirmation) | +| **Workspace Dir** | `/srv/configs/ai/` | Read: ✅ Write: ✅ (agent state and proposed drafts) | + +--- + +## 4. Reconciliation Logic & Execution Steps + +### Step 1: Pre-Flight Safety Checks +Run **in order** before beginning any audit cycle. If any check fails, **simply inform the user directly in the session** and abort (do NOT log failures to a local `errors.log` file): +- **Lock Check:** Verify no concurrent audit process holds `/srv/configs/ai/.audit.lock`. +- **Backend Port Check:** Verify port `9445` is actively listening. +- **Backend API Check:** Verify backend returns `HTTP 200` on `/api/v1/health` using the parsed `MCP_SERVICE_KEY` from `/srv/configs/docker_compose/homelabel/.env`. +- **Source Files Check:** Verify both `/srv/configs/.bash_aliases` and `homelab_Infra.md` exist. + +### Step 2: Nginx Proxy Parsing +Parse every proxy configuration file `/app/nginxproxy/data/nginx/proxy_host/*.conf` for: +- `server_name` (Proxy Domain name / Route) +- `set $port` (Forwarded Container Port) + +Keep an internal lookup dictionary of `Forwarded Port -> Proxy Domain` for comparison. + +### Step 3: Homepage Dashboard Audit +Read `/srv/configs/docker_compose/homepage/config/services.yaml` and compare the list of active proxy routes against the dashboard items. Propose adding newly proxied services so the user can navigate to them. + +Homepage services must be strictly split into three distinct categories: +1. **Physical Devices & Networking related** (e.g. Proxmox, TrueNAS, Router, Pi-Hole, Moonlight streaming) +2. **Docker Host Containers** (e.g. nginx-proxy-manager, portainer, immich, vaultwarden, paperless-ngx, slopsmith-web) +3. **TrueNAS Host Containers** (e.g. radarr, sonarr, readarr, bazarr, prowlarr, calibre) + +### Step 4: Documentation Formatting Standardization + +Both the **`## Docker services`** and the **`## Truenas Services`** markdown tables inside `homelab_Infra.md` must strictly maintain this exact 5-column layout: + +`| Container Name | Mapped Ports | Access | Proxy Route / Domain | Description / Role |` + +Always parse and print matching columns when proposing or applying documentation edits. + +### Step 5: Proxmox LXC/VM Auditing +Proxmox host virtualization represents nodes (containers/VMs) that communicate on the LAN subnet `10.0.0.0/16`. +- Parse the Proxmox Node Table under `### Proxmox node (laptop-proxmox)` in `homelab_Infra.md`. +- Retrieve all network canvas nodes from `http://localhost:9445/api/v1/nodes` matching types `vm`, `lxc`, or direct hosts. +- Compare and verify that active virtualization nodes on your network (such as `invidious` on `10.0.0.217` or `pterodactyl` on `10.0.0.110`) exist inside the documentation table. Propose additions for any unregistered LXCs or VMs. + +### Step 6: Homelable MCP Canvas Styling Standards + +When registering new Docker containers on the visual canvas using the `POST` endpoint at `/api/v1/nodes`, you must strictly match the styling of other standard services (use `gitea_runner` as reference): + +- `"type"` must be `"docker_container"` +- `"hostname"` must be `"docker"` +- `"ip"` must be `""` (empty string, as they reside behind parent container networking) +- `"width"` must be `140.0` +- `"height"` must be `50.0` +- `"bottom_handles"` must be `1` +- `"custom_colors"` must be `{"show_services": false}` +- `"parent_id"` must be set to the Docker parent host node UUID (e.g. `ee61411a-018a-4471-8a76-bcd4a1f5579a`) + +--- + +## 5. Safety and Guardrails + +> [!IMPORTANT] +> 1. **No silent writes:** All modifications to `homelab_Infra.md`, the `homelable-mcp` canvas, or `services.yaml` must be presented as a clear diff block and receive explicit, manual user validation. +> 2. **NO UNAUTHORIZED GIT COMMANDS:** Never execute any Git commands (such as `git add`, `git commit`, `git status`, `git diff`, etc.) without the user's explicit, per-action authorization. +> 3. **Preserve Comments:** Do not remove, alter, or strip existing documentation comments or section text blocks. +> 4. **No Errors Log File:** Do not append failures to a local `errors.log` file. Output error details directly to the active console stream to keep workspace logs clean. + +--- + +## 6. Technical Pitfalls & Workarounds + +> [!WARNING] +> - **Windows CRLF line endings (`\r\n`):** The Homepage dashboard's `services.yaml` contains Windows CRLF terminators. Traditional bash/unix line-ending regex matches (such as `$`) will **silently fail** to match titles or categories. Always run `tr -d '\r'` to sanitize lines before attempting pattern matches. +> - **Direct REST API bypass:** Querying `homelable-mcp` on `8001` via SSE requires an async JSON-RPC transport negotiation which is fragile and slow. Instead, bypass the MCP layer entirely and query the `homelable-backend` REST API directly on port `9445` at `/api/v1/nodes`. Pass the header `"X-MCP-Service-Key"` parsed from `/srv/configs/docker_compose/homelabel/.env` for instant, authenticated, and robust network nodes lookup. +- **Active Port Verification (Stale Config Filter):** Nginx Proxy Manager configuration files on disk can be stale/leftover (e.g. Moonlight). To completely prevent reporting decommissioned configs, always verify that the port associated with any Nginx config is **actively listening on the host** (or mapped in active docker container port-mappings) before reporting it as an active proxy. If the port is inactive, ignore the proxy block completely. +- **TrueNAS SCALE Node Socket Verification:** Rather than setting up fragile credentials or SSH logins, TrueNAS SCALE container port verification is achieved securely and in milliseconds using lightweight, non-destructive `/dev/tcp/10.0.0.2/[port]` network socket connection handshakes. +- **Proxmox VE API Live Verification:** To verify running LXCs and VMs, query `https://10.0.0.142:8006/api2/json/cluster/resources?type=vm` directly. Load `HOMEPAGE_VAR_proxmox_password` dynamically from `/srv/configs/docker_compose/homepage/.env` at runtime and strip outer double quotes before passing to the Authorization headers. Never hardcode credentials in codebase or transcripts. +- **Homepage Settings Layout Syncing:** When renaming category names in `/srv/configs/docker_compose/homepage/config/services.yaml` (such as `Arr` -> `Truenas Containers`, and `Containers` -> `Docker Containers`), the layout block names inside `/srv/configs/docker_compose/homepage/config/settings.yaml` under `layout:` **must be updated identically** or the cards will fail to render on the dashboard. +- **Do NOT Invent Public Proxies:** Do not invent public proxy domains or automatically propose changing local/VPN hrefs (such as `http://radarr/` or `http://calibre/`) to public domains (like `radarr.wompmacho.com`) in homepage configurations. They must remain as local/VPN URLs exactly as currently configured to preserve network isolation and security. +- **Multi-Alias Substring Matcher:** Nginx Proxy configs often specify multiple space-separated aliases (e.g. `server_name vaultwarden vaultwarden.wompmacho.com;`). When searching for these in `services.yaml` to detect missing cards, token-split the domains and verify if *any* alias matches as a substring within the homepage card `href` or name (case-insensitively). This prevents false-positives (such as flagging Vaultwarden or Gitea as missing). + +--- + +## 7. Skill File Structure + +All scripts and files associated with this skill are stored in this self-contained directory: +- `SKILL.md` — Custom skill documentation and procedures. +- `scripts/homelab-infrastructure-review.sh` — Bash shell auditing script. +- `scripts/mcp_client.py` — Python client for direct database and API queries. +- `errors.log` — Legacy log file (decommissioned per safety rules — errors print straight to console). +- `.audit.lock` — Legitimate lock file preventing concurrent executions. + diff --git a/ai/skills/homelab-infrastructure-review/references/homelable-backend-api.md b/ai/skills/homelab-infrastructure-review/references/homelable-backend-api.md new file mode 100644 index 0000000..900c043 --- /dev/null +++ b/ai/skills/homelab-infrastructure-review/references/homelable-backend-api.md @@ -0,0 +1,46 @@ +# Homelable Backend API Reference + +Direct REST API access to Pouzor/homelable visualizer backend on port 9445. Bypasses the complex and slow Server-Sent Events (SSE) handshake required on port 8001. + +## Authentication +The backend API requires an unauthenticated sub-app service key. Read this key dynamically from: +`/srv/configs/docker_compose/homelabel/.env` -> `MCP_SERVICE_KEY` + +Inject it into HTTP request headers: +`X-MCP-Service-Key: ` + +## Endpoints + +### 1. Health Check +`GET http://localhost:9445/api/v1/health` +Returns `{"status": "ok"}` on success (HTTP 200). + +### 2. Retrieve Canvas Nodes +`GET http://localhost:9445/api/v1/nodes` +Returns a JSON array of active canvas nodes. + +Example Node Object: +```json +{ + "type": "computer", + "label": "Game Pc", + "hostname": "DESKTOP-BRHGMFT", + "ip": "10.0.0.109", + "status": "online", + "services": [ + { + "port": 11434, + "protocol": "tcp", + "service_name": "Ollama" + } + ], + "id": "3fc16e75-9418-41b1-b4a2-d3859c9ce431" +} +``` + +### 3. Retrieve Canvas Edges +`GET http://localhost:9445/api/v1/edges` +Returns active connection edges between nodes. + +### 4. Trigger Network Scan +`POST http://localhost:9445/api/v1/scan/trigger` diff --git a/ai/skills/homelab-infrastructure-review/references/mcp_backend_client.md b/ai/skills/homelab-infrastructure-review/references/mcp_backend_client.md new file mode 100644 index 0000000..0f8f91e --- /dev/null +++ b/ai/skills/homelab-infrastructure-review/references/mcp_backend_client.md @@ -0,0 +1,68 @@ +# Homelable Backend Direct REST API Reference + +Instead of negotiating a complex and slow Server-Sent Events (SSE) stream over port `8001` (which requires async JSON-RPC sessions and frequently times out), query the REST API on `homelable-backend` directly. + +## Endpoint & Port +- **Host Port:** `9445` +- **Base URL:** `http://localhost:9445/api/v1` + +## Authentication Header +The backend is authenticated using a special service key header: +``` +X-MCP-Service-Key: +``` + +### Loading Credentials Dynamically +To keep keys safe and secure (without leaking them to the LLM context window), read the key dynamically from the local `.env` configuration file: +- **Path:** `/srv/configs/docker_compose/homelabel/.env` +- **Variable Name:** `MCP_SERVICE_KEY` + +## Common Endpoints + +### 1. Health Check +- **Path:** `/api/v1/health` +- **Method:** `GET` +- **Response:** `{"status": "ok"}` + +### 2. Retrieve Nodes (Canvas Devices) +- **Path:** `/api/v1/nodes` +- **Method:** `GET` +- **Response:** Array of node objects: +```json +[ + { + "id": "3fc16e75-9418-41b1-b4a2-d3859c9ce431", + "type": "computer", + "label": "Game Pc", + "ip": "10.0.0.109", + "status": "online", + "services": [ + { + "port": 11434, + "protocol": "tcp", + "service_name": "Ollama" + } + ] + } +] +``` + +## Python Integration Example + +```python +import os +import requests + +env_path = "/srv/configs/docker_compose/homelabel/.env" +service_key = "" + +if os.path.exists(env_path): + with open(env_path, "r") as f: + for line in f: + if line.startswith("MCP_SERVICE_KEY="): + service_key = line.split("=", 1)[1].strip() + +headers = {"X-MCP-Service-Key": service_key} +resp = requests.get("http://localhost:9445/api/v1/nodes", headers=headers) +nodes = resp.json() +``` diff --git a/ai/skills/homelab-infrastructure-review/scripts/homelab-infrastructure-review.sh b/ai/skills/homelab-infrastructure-review/scripts/homelab-infrastructure-review.sh new file mode 100644 index 0000000..74f4584 --- /dev/null +++ b/ai/skills/homelab-infrastructure-review/scripts/homelab-infrastructure-review.sh @@ -0,0 +1,627 @@ +#!/usr/bin/env bash +# +# homelab-infrastructure-review.sh +# Triggers a full 3-way infrastructure reconciliation audit. +# Audits Infra.md ↔ Live Docker ↔ Homelable Canvas ↔ Homepage Dashboard. +# Triggered ONLY by explicit, manual user request. No cron or automation. +# +# Usage: bash /srv/configs/ai/skills/homelab-infrastructure-review/scripts/homelab-infrastructure-review.sh [--dry-run] +# + +set -euo pipefail + +CONF_DIR="/srv/configs/ai" +INFRA_FILE="/srv/docs/public/projects/homelab/homelab_Infra.md" +MCP_ENDPOINT="http://localhost:9445/api/v1" +ALIAS_FILE="/srv/configs/.bash_aliases" +LOCK_FILE="/srv/configs/ai/.audit.lock" +HOMEPAGE_FILE="/srv/configs/docker_compose/homepage/config/services.yaml" +DRY_RUN=false + +if [[ "${1:-}" == "--dry-run" ]]; then + DRY_RUN=true +fi + +# ── Logging helpers ────────────────────────────────────────────────────────── + +log_info() { + local msg="[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" + echo "$msg" +} + +log_error() { + local msg="[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" + echo "$msg" >&2 +} + +# ── Lock (prevent concurrent runs) ─────────────────────────────────────────── + +acquire_lock() { + if [ -f "$LOCK_FILE" ]; then + local lock_pid + lock_pid=$(cat "$LOCK_FILE" 2>/dev/null || echo "") + if [ -n "$lock_pid" ] && kill -0 "$lock_pid" 2>/dev/null; then + log_error "Another audit process is running (PID $lock_pid). Aborting." + exit 1 + fi + log_info "Stale lock file found (PID $lock_pid). Cleaning up." + rm -f "$LOCK_FILE" + fi + echo $$ > "$LOCK_FILE" + trap 'rm -f "$LOCK_FILE"' EXIT +} + +# ── Port listening check helper ────────────────────────────────────────────── + +is_port_active() { + local target_port="$1" + # 1. Check if the port is in ss -tulnp + if ss -tulnp 2>/dev/null | grep -q -E ":${target_port}[[:space:]]"; then + return 0 + fi + # 2. Check if the port is mapped in running docker containers + if docker ps --format "{{.Ports}}" | grep -q -E "(^|[^0-9])${target_port}->"; then + return 0 + fi + return 1 +} + +# ── Pre-flight: Port 9445 availability ─────────────────────────────────────── + +check_port_8001() { + # Port 9445 SHOULD be listening (homelable-backend port-mapped on host) + if ! ss -tulnp 2>/dev/null | grep -q ':9445[[:space:]]'; then + log_error "Port 9445 is not listening — homelable-backend is not port-mapped. Cannot proceed. Aborting." + exit 1 + fi + # Parse the service key if present + local service_key="" + if [ -f "/srv/configs/docker_compose/homelabel/.env" ]; then + service_key=$(grep -E "^MCP_SERVICE_KEY=" "/srv/configs/docker_compose/homelabel/.env" | cut -d'=' -f2-) + fi + # Verify the backend endpoint is actually responding via HTTP + local http_code + http_code=$(curl -s -H "X-MCP-Service-Key: $service_key" -o /dev/null -w '%{http_code}' --max-time 5 "http://localhost:9445/api/v1/health" 2>/dev/null) || http_code="000" + # 000 = connection failure = server unreachable + if [ "$http_code" = "000" ]; then + log_error "Backend endpoint http://localhost:9445/api/v1/health unresponsive on port 9445. May be container crash or network issue. Aborting." + exit 1 + fi + log_info "Port 9445 confirmed active for homelable-backend (HTTP $http_code)." +} + +# ── Pre-flight: Source file existence ──────────────────────────────────────── + +check_sources() { + if [ ! -f "$ALIAS_FILE" ]; then + log_error "Alias file $ALIAS_FILE not found. Cannot proceed. Aborting." + exit 1 + fi + if [ ! -f "$INFRA_FILE" ]; then + log_error "Infra doc $INFRA_FILE not found. Cannot proceed. Aborting." + exit 1 + fi + if [ ! -f "$HOMEPAGE_FILE" ]; then + log_error "Homepage config $HOMEPAGE_FILE not found. Cannot proceed. Aborting." + exit 1 + fi +} + +# ── Source of Truth: Infra.md ──────────────────────────────────────────────── + +get_infra_docs() { + # Extract container/service entries from homelab_Infra.md + # Support both 4-column and 5-column structures dynamically + local current_section="" + while IFS= read -r line || [ -n "$line" ]; do + # Detect section headers + if [[ "$line" =~ ^##[[:space:]]+(.+)$ ]]; then + current_section="${BASH_REMATCH[1]}" + fi + + # Check if line is a markdown table row (must start with | and contain a bold name) + if [[ "$line" =~ ^\|[[:space:]]*\*\*([a-zA-Z0-9_-]+)\*\*[[:space:]]*\| ]]; then + # Count the pipes to deduce the number of columns + local pipe_count + pipe_count=$(echo -n "$line" | tr -cd '|' | wc -c) + + if [ "$pipe_count" -eq 5 ]; then + # Original 4 columns: | Name | Mapped Ports | Access | Desc | + local name ports access desc + IFS='|' read -r _blank name ports access desc _blank_end <<< "$line" + # Clean up whitespaces & bold marks + name=$(echo "$name" | tr -d '*' | xargs) + ports=$(echo "$ports" | xargs) + access=$(echo "$access" | xargs) + desc=$(echo "$desc" | xargs) + # Output: name|ports|section|desc|proxy|access + printf '%s|%s|%s|%s|%s|%s\n' "$name" "$ports" "$current_section" "$desc" "none" "$access" + elif [ "$pipe_count" -eq 6 ]; then + # Restructured 5 columns: | Name | Mapped Ports | Access | Proxy | Desc | + local name ports access proxy desc + IFS='|' read -r _blank name ports access proxy desc _blank_end <<< "$line" + name=$(echo "$name" | tr -d '*' | xargs) + ports=$(echo "$ports" | xargs) + access=$(echo "$access" | xargs) + proxy=$(echo "$proxy" | xargs) + desc=$(echo "$desc" | xargs) + # Output: name|ports|section|desc|proxy|access + printf '%s|%s|%s|%s|%s|%s\n' "$name" "$ports" "$current_section" "$desc" "$proxy" "$access" + fi + fi + done < "$INFRA_FILE" +} + +# ── Source of Truth: Live Docker ───────────────────────────────────────────── + +get_live_docker() { + # Use _docker-table alias logic inline + if ! docker info > /dev/null 2>&1; then + log_error "Docker daemon is not running. Aborting." + exit 1 + fi + docker ps --format "{{.Names}}|{{.Status}}|{{.Ports}}" | while IFS='|' read -r name status ports; do + cleaned_ports="" + if [ -n "$ports" ]; then + IFS=',' read -r -a port_array <<< "$ports" + for p in "${port_array[@]}"; do + p=$(echo "$p" | xargs --no-run-if-empty) + # Strip to the right of -> (e.g., "0.0.0.0:80->80/tcp") + host_port="${p##*->}" + # Strip protocol suffix (tcp/udp) for dedup + host_port="${host_port%%/*}" + [[ "$host_port" == */tcp ]] && host_port="${host_port%/tcp}" + [[ "$host_port" == */udp ]] && host_port="${host_port%/udp}" + if [[ -n "$host_port" ]]; then + local found=false + local existing + for existing in $cleaned_ports; do + [ "$existing" = "$host_port" ] && found=true + done + $found || cleaned_ports="${cleaned_ports:+$cleaned_ports, }$host_port" + fi + done + fi + if [ -z "$cleaned_ports" ]; then + cleaned_ports="internal" + fi + printf '%s|%s|%s\n' "$name" "$status" "$cleaned_ports" + done +} + +# ── Source of Truth: Homelable Backend Nodes ───────────────────────────────── + +get_mcp_nodes() { + # Parse the service key if present + local service_key="" + if [ -f "/srv/configs/docker_compose/homelabel/.env" ]; then + service_key=$(grep -E "^MCP_SERVICE_KEY=" "/srv/configs/docker_compose/homelabel/.env" | cut -d'=' -f2-) + fi + + # Fetch nodes from the backend REST API + local payload + payload=$(curl -s -H "X-MCP-Service-Key: $service_key" --max-time 10 "${MCP_ENDPOINT}/nodes" 2>/dev/null) + if [ -z "$payload" ]; then + log_error "Homelable Backend returned empty response. Aborting." + exit 1 + fi + # Parse and output nodes: label|type|ip|status + echo "$payload" | jq -r '.[]? | "\(.label // .name // "unknown")|\(.type // "node")|\(.ip // "n/a")|\(.status // "unknown")"' 2>/dev/null +} + +# ── Source of Truth: Nginx Proxies (Stale-Filtering Enabled) ───────────────── + +get_nginx_proxies() { + local conf_dir="/app/nginxproxy/data/nginx/proxy_host" + if [ ! -d "$conf_dir" ]; then + return 0 + fi + + # Format: PORT|DOMAINS + grep -E "set .port|server_name" "$conf_dir"/*.conf 2>/dev/null \ + | sed 's/^[ \t]*//; s/.*conf://' \ + | paste - - \ + | sed 's/set \$port //; s/server_name //; s/;//g' \ + | awk '{$1=$1; print}' \ + | sort -n \ + | while read -r port domains; do + # STALE CONFIG ELIMINATION: Only report proxies if the mapped port is actively listening + if is_port_active "$port"; then + printf '%s|%s\n' "$port" "$domains" + fi + done +} + +# ── Source of Truth: Homepage Config ───────────────────────────────────────── + +get_homepage_services() { + # Reads services.yaml, stripping carriage returns, and outputs simple mappings: key_name|href|category + local current_cat="" + local service_name="" + local href="" + while IFS= read -r raw_line || [ -n "$raw_line" ]; do + local line + line=$(echo -n "$raw_line" | tr -d '\r') + + # Detect category (e.g., "- Devices & Network:" or "- Arr:") + if [[ "$line" =~ ^-[[:space:]]+([^:]+):$ ]]; then + # If we had an unprinted service, print it before changing categories + if [ -n "$service_name" ]; then + printf '%s|%s|%s\n' "$service_name" "$href" "$current_cat" + service_name="" + href="" + fi + current_cat="${BASH_REMATCH[1]}" + # Detect service element (e.g., " - Proxmox:") + elif [[ "$line" =~ ^[[:space:]]+-[[:space:]]+([^:]+):$ ]]; then + if [ -n "$service_name" ]; then + printf '%s|%s|%s\n' "$service_name" "$href" "$current_cat" + fi + service_name="${BASH_REMATCH[1]}" + href="none" + # Detect href (e.g., " href: \"http://10.0.0.142:8006/\"") + elif [[ "$line" =~ href:[[:space:]]*\"([^\"]+)\" || "$line" =~ href:[[:space:]]*(https?://[^\s]+) ]]; then + local val="${BASH_REMATCH[1]}" + val="${val%\"}" # Remove trailing quotes + val="${val#\"}" # Remove leading quotes + href="$val" + fi + done < "$HOMEPAGE_FILE" + # Print the last service + if [ -n "$service_name" ]; then + printf '%s|%s|%s\n' "$service_name" "$href" "$current_cat" + fi +} + +# ── Source of Truth: Proxmox VMs / LXCs ────────────────────────────────────── + +get_infra_proxmox_nodes() { + # Extract Proxmox VM/LXC entries from homelab_Infra.md + # Format: | ID | Type | Hostname | IP | Role | + while IFS= read -r line || [ -n "$line" ]; do + if [[ "$line" =~ ^\|[[:space:]]*([0-9_-]+|-)[[:space:]]*\|[[:space:]]*(LXC|VM)[[:space:]]*\|[[:space:]]*\*\*([a-zA-Z0-9_-]+)\*\*[[:space:]]*\|[[:space:]]*([0-9.]+)[[:space:]]*\| ]]; then + local type="${BASH_REMATCH[2]}" + local hostname="${BASH_REMATCH[3]}" + local ip="${BASH_REMATCH[4]}" + printf '%s|%s|%s\n' "$hostname" "$type" "$ip" + fi + done < "$INFRA_FILE" +} + +# ── 3-Way Reconciliation Engine ────────────────────────────────────────────── + +reconcile() { + local infra_count docker_count mcp_count proxy_count hp_count + infra_count=$(get_infra_docs | wc -l) + docker_count=$(get_live_docker | wc -l) + mcp_count=$(get_mcp_nodes | wc -l) + proxy_count=$(get_nginx_proxies | wc -l) + hp_count=$(get_homepage_services | wc -l) + + log_info "Infra.md: $infra_count | Live Docker: $docker_count | MCP Nodes: $mcp_count | Proxies (Active): $proxy_count | Homepage Dashboard Cards: $hp_count" + + local diff_report="" + diff_report+="## Homelab Infrastructure Reconciliation Report\n" + diff_report+="Generated: $(date '+%Y-%m-%d %H:%M:%S')\n\n" + + # ── Comparison 1: Documentation vs Live Docker Drift ── + diff_report+="### 1. Documentation vs. Live Docker Drift\n\n" + + local doc_names=() + while IFS='|' read -r name _ports _section _desc _proxy _access; do + # Only compare Docker Services section + if [[ "$_section" == "Docker services" ]]; then + doc_names+=("$name") + fi + done < <(get_infra_docs) + + local live_names=() + while IFS='|' read -r name _status _ports; do + live_names+=("$name") + done < <(get_live_docker) + + # Missing in Docker + local missing=0 + for doc in "${doc_names[@]}"; do + local found=false + for live in "${live_names[@]}"; do + [ "$doc" = "$live" ] && found=true && break + done + # Exceptions + if [ "$doc" = "dev" ] || [ "$doc" = "calibre" ]; then + found=true + fi + $found || { + diff_report+=" ✗ **${doc}** — listed in Infra.md but NOT running in Docker\n" + ((missing++)) || true + } + done + [ $missing -eq 0 ] && diff_report+=" ✅ All documented container services are running\n" + + # Untracked in Documentation + local untracked=0 + for live in "${live_names[@]}"; do + local found=false + for doc in "${doc_names[@]}"; do + [ "$live" = "$doc" ] && found=true && break + done + # Map group names or sub-services + if [[ "$live" == "paperless-webserver-1" && " ${doc_names[*]} " == *" paperless-ngx "* ]]; then found=true; fi + if [[ "$live" == "paperless-broker-1" && " ${doc_names[*]} " == *" paperless-broker "* ]]; then found=true; fi + if [[ "$live" == "paperless-db-1" && " ${doc_names[*]} " == *" paperless-db "* ]]; then found=true; fi + if [[ "$live" == "slopsmith-web" && " ${doc_names[*]} " == *" slopsmith-web "* ]]; then found=true; fi + if [[ "$live" == "musicbrainz_picard" && " ${doc_names[*]} " == *" musicbrainz_picard "* ]]; then found=true; fi + + $found || { + local status=$(get_live_docker | grep "^${live}|" | head -1 | cut -d'|' -f2) + local ports=$(get_live_docker | grep "^${live}|" | head -1 | cut -d'|' -f3) + diff_report+=" ⚠ **${live}** — running (Status: ${status}, Ports: ${ports}) but NOT in Infra.md\n" + ((untracked++)) || true + } + done + [ $untracked -eq 0 ] && diff_report+=" ✅ No untracked containers running\n" + + # Unhealthy Containers + local unhealthy=0 + while IFS='|' read -r name status _ports; do + if [[ ! "$status" =~ "(healthy)" && "$status" =~ "Up" ]]; then + diff_report+=" ⚠ **${name}** — Up but NOT marked healthy (${status})\n" + ((unhealthy++)) || true + fi + done < <(get_live_docker) + [ $unhealthy -eq 0 ] && diff_report+=" ✅ All active containers are running healthy\n" + + # ── Comparison 2: Nginx Proxy Manager Live Mappings vs Doc ── + diff_report+="\n### 2. Live Nginx Proxy Mappings vs. Documentation\n\n" + + local proxy_drift=0 + while IFS='|' read -r port domains; do + # Cross reference the port against documented services in Infra.md + local doc_name="unknown" + local doc_proxy="none" + while IFS='|' read -r name d_ports _section _desc d_proxy _access; do + # Check if port matches any documented port + if [[ " $d_ports " == *"$port"* ]]; then + doc_name="$name" + doc_proxy="$d_proxy" + break + fi + done < <(get_infra_docs) + + if [ "$doc_name" != "unknown" ]; then + # Clean domains to simplify matching + local first_domain + first_domain=$(echo "$domains" | awk '{print $1}') + if [ "$doc_proxy" = "none" ] || [ "$doc_proxy" = "-" ]; then + diff_report+=" ⚠ Port **$port** (**$doc_name**) has proxy domains **($domains)** but has NO proxy domain documented in Infra.md\n" + ((proxy_drift++)) || true + elif [[ "$doc_proxy" != *"$first_domain"* ]]; then + diff_report+=" ⚠ Port **$port** (**$doc_name**) proxy drift! Doc says: \`$doc_proxy\`, Live proxy is: \`$domains\`\n" + ((proxy_drift++)) || true + fi + fi + done < <(get_nginx_proxies) + [ $proxy_drift -eq 0 ] && diff_report+=" ✅ All active proxy definitions align with documentation proxy routes\n" + + # ── Comparison 3: Live Docker vs MCP Canvas Nodes ── + diff_report+="\n### 3. Live Docker vs. Homelable Canvas Drift\n\n" + + local canvas_drift=0 + # Check if active containers exist on canvas + while IFS='|' read -r name _status _ports; do + if [[ "$name" == *"postgres"* || "$name" == *"redis"* || "$name" == *"db"* || "$name" == *"broker"* ]]; then + continue + fi + local found=false + while IFS='|' read -r label type ip status; do + # Case insensitive comparison + local clean_label + clean_label=$(echo "$label" | tr '[:upper:]' '[:lower:]' | tr ' _-' ' ') + local clean_name + clean_name=$(echo "$name" | tr '[:upper:]' '[:lower:]' | tr ' _-' ' ') + # Custom mappings for neko and hugo public/private + if [[ "$clean_name" == "docs-public" && "$clean_label" == *"hugo"* && "$clean_label" == *"public"* ]]; then found=true; fi + if [[ "$clean_name" == "docs-private" && "$clean_label" == *"hugo"* && "$clean_label" == *"pivate"* ]]; then found=true; fi + if [[ "$clean_name" == "guacamole" && "$clean_label" == *"quacamole"* ]]; then found=true; fi + + if [[ "$clean_label" == *"$clean_name"* || "$clean_name" == *"$clean_label"* ]]; then + found=true + break + fi + done < <(get_mcp_nodes) + + $found || { + diff_report+=" ⚠ **${name}** — container is running, but NOT mapped on the Homelable canvas topology\n" + ((canvas_drift++)) || true + } + done < <(get_live_docker) + [ $canvas_drift -eq 0 ] && diff_report+=" ✅ All primary running services are mapped on the network canvas\n" + + # ── Comparison 4: Homepage Configuration Audit ── + diff_report+="\n### 4. Homepage Dashboard Audit (services.yaml)\n\n" + + local hp_drift=0 + local hp_services=() + while IFS='|' read -r s_name s_href s_cat; do + hp_services+=("$s_name|$s_href|$s_cat") + done < <(get_homepage_services) + + # Cross reference living proxied ports vs Homepage links + while IFS='|' read -r port domains; do + # Check if any alias in domains string is a substring of any card href + local found=false + local alias_item + for alias_item in $domains; do + alias_item="${alias_item%;}" # strip semicolon + + for card in "${hp_services[@]}"; do + IFS='|' read -r c_name c_href c_cat <<< "$card" + + # Case insensitive comparison + local clean_alias + clean_alias=$(echo "$alias_item" | tr '[:upper:]' '[:lower:]') + local clean_href + clean_href=$(echo "$c_href" | tr '[:upper:]' '[:lower:]') + local clean_name + clean_name=$(echo "$c_name" | tr '[:upper:]' '[:lower:]') + + if [[ "$clean_href" == *"$clean_alias"* || "$clean_name" == *"$clean_alias"* ]]; then + found=true + break 2 + fi + done + done + + # Proxmox remains as is pointing directly to its IP, ignore in mapping drift + if [[ "$domains" == *"proxmox"* ]]; then + found=true + fi + + $found || { + # Skip internal system components + local first_domain + first_domain=$(echo "$domains" | awk '{print $1}') + if [[ "$first_domain" != "nginx" && "$first_domain" != "homelable" && "$first_domain" != "openai" && "$first_domain" != "private" ]]; then + # Classify the service correctly + local category="Docker Containers" + if [[ "$port" == "8006" || "$port" == "8010" ]]; then + category="Physical Devices & Networking related" + elif [[ "$port" == "32015" || "$port" == "30025" || "$port" == "30027" || "$port" == "30014" || "$port" == "30045" || "$port" == "30046" || "$port" == "30050" ]]; then + category="Truenas Containers" + fi + + diff_report+=" ⚠ **$first_domain** (Port $port) — active proxy domain is missing from Homepage dashboard! Recommend adding under **$category**.\n" + ((hp_drift++)) || true + fi + } + done < <(get_nginx_proxies) + [ $hp_drift -eq 0 ] && diff_report+=" ✅ All active proxied domain targets are mapped on the Homepage dashboard\n" + + # ── Comparison 5: Proxmox Virtualization Node Live Audit (LXC / VMs) ── + diff_report+="\n### 5. Proxmox Virtualization Drift (LXC / VMs)\n\n" + + local proxmox_drift=0 + local doc_proxmox=() + while IFS='|' read -r h_name h_type h_ip; do + proxmox_proxmox_item=$(echo "$h_name" | tr '[:upper:]' '[:lower:]') + doc_proxmox+=("$proxmox_proxmox_item") + done < <(get_infra_proxmox_nodes) + + # Fetch live running LXCs/VMs from Proxmox VE API using dynamically loaded homepage token + local pve_token="" + if [ -f "/srv/configs/docker_compose/homepage/.env" ]; then + pve_token=$(grep -E "^HOMEPAGE_VAR_proxmox_password=" "/srv/configs/docker_compose/homepage/.env" | cut -d'=' -f2- | tr -d '\r') + # Strip leading/trailing double quotes if present + pve_token="${pve_token%\"}" + pve_token="${pve_token#\"}" + fi + + if [ -n "$pve_token" ]; then + # Query Proxmox API directly (SSL check bypassed with -k) + local pve_resources + pve_resources=$(curl -k -s -H "Authorization: PVEAPIToken=root@pam!dash=$pve_token" "https://10.0.0.142:8006/api2/json/cluster/resources?type=vm" 2>/dev/null) || pve_resources="" + + if [ -n "$pve_resources" ]; then + while read -r pve_node; do + local label status type + label=$(echo "$pve_node" | cut -d'|' -f1) + type=$(echo "$pve_node" | cut -d'|' -f2) # qemu (VM) or lxc + status=$(echo "$pve_node" | cut -d'|' -f3) + + # Only check active/running nodes + if [ "$status" = "running" ] && [ "$label" != "docker" ]; then + local clean_label + clean_label=$(echo "$label" | tr '[:upper:]' '[:lower:]' | tr ' _-' ' ') + local found=false + for doc_node in "${doc_proxmox[@]}"; do + if [[ "$clean_label" == *"$doc_node"* || "$doc_node" == *"$clean_label"* ]]; then + found=true + break + fi + done + + $found || { + local friendly_type="VM" + [ "$type" = "lxc" ] && friendly_type="LXC" + diff_report+=" ⚠ **${label}** — active Proxmox **${friendly_type}** is running, but NOT documented under Proxmox Nodes in Infra.md\n" + ((proxmox_drift++)) || true + } + fi + done < <(echo "$pve_resources" | jq -r '.data[]? | "\(.name)|\(.type)|\(.status)"' 2>/dev/null) + else + diff_report+=" ⚠ Unable to query Proxmox VE API (connection failure). Skipping live VM/LXC verification.\n" + ((proxmox_drift++)) || true + fi + else + diff_report+=" ⚠ Proxmox token missing from homepage/.env. Skipping live VM/LXC verification.\n" + ((proxmox_drift++)) || true + fi + [ $proxmox_drift -eq 0 ] && diff_report+=" ✅ All active Proxmox virtualization nodes are documented\n" + + # ── Comparison 6: TrueNAS SCALE Node Services Live Audit ── + diff_report+="\n### 6. TrueNAS SCALE Node Services Verification\n\n" + + local truenas_drift=0 + # Audit each documented TrueNAS service by performing active port checks directly on truenas (10.0.0.2) + while IFS='|' read -r name d_ports section desc proxy access; do + if [[ "$section" == "Truenas Services" ]]; then + # Verify port is listening over /dev/tcp network socket + if timeout 1 bash -c "cat < /dev/null > /dev/tcp/10.0.0.2/${d_ports}" 2>/dev/null; then + # Port is active + true + else + diff_report+=" ✗ **${name}** — documented on TrueNAS port **${d_ports}** but service is NOT listening (down/offline)\n" + ((truenas_drift++)) || true + fi + fi + done < <(get_infra_docs) + [ $truenas_drift -eq 0 ] && diff_report+=" ✅ All documented TrueNAS Services are actively running on port 10.0.0.2\n" + + cppt_output=$(echo -e "$diff_report") + echo "$cppt_output" +} + +# ── Main Execution Pipeline ────────────────────────────────────────────────── + +main() { + log_info "═══════════════════════════════════════════════════════" + log_info "Starting Homelab Infrastructure Reconciliation Audit" + log_info "═══════════════════════════════════════════════════════" + + acquire_lock + check_port_8001 + check_sources + log_info "All pre-flight checks passed." + + if $DRY_RUN; then + log_info "DRY RUN — gathering sources without full reconciliation" + log_info "--- Infra.md entries ---" + get_infra_docs | while IFS='|' read -r name ports section desc proxy access; do + echo " $name (section: $section, ports: $ports, proxy: $proxy)" + done + log_info "--- Live Docker containers ---" + get_live_docker | while IFS='|' read -r name status ports; do + echo " $name | $status | $ports" + done + log_info "--- Homelable MCP check ---" + local mcp_result + mcp_result=$(curl -s --max-time 5 "${MCP_ENDPOINT}" 2>/dev/null || echo "unreachable") + if [ "$mcp_result" = "unreachable" ]; then + log_info " ✗ MCP unreachable" + else + log_info " ✓ MCP reachable (first 100 chars: ${mcp_result:0:100})" + fi + return 0 + fi + + log_info "Fetching live states..." + get_infra_docs > /dev/null + get_live_docker > /dev/null + get_mcp_nodes > /dev/null + log_info "All sources fetched." + + log_info "Running reconciliation..." + reconcile +} + +main "$@" diff --git a/ai/skills/homelab-infrastructure-review/scripts/mcp_client.py b/ai/skills/homelab-infrastructure-review/scripts/mcp_client.py new file mode 100644 index 0000000..cffc004 --- /dev/null +++ b/ai/skills/homelab-infrastructure-review/scripts/mcp_client.py @@ -0,0 +1,115 @@ +#!/usr/bin/env python3 +""" +MCP Homelable Integration Module. + +Communicates directly with the homelable-backend on port 9445 +for canvas topology queries, node/edge retrieval, and scanning. +""" + +import os +import json +import logging +import time +from typing import Optional, List, Dict + +logging.basicConfig(level=logging.INFO) +logger = logging.getLogger(__name__) + +BACKEND_URL = "http://localhost:9445/api/v1" +ENV_PATH = "/srv/configs/docker_compose/homelabel/.env" + + +def _get_service_key() -> str: + """Read service key dynamically from .env to authenticate request securely.""" + try: + if os.path.exists(ENV_PATH): + with open(ENV_PATH, "r") as f: + for line in f: + if line.startswith("MCP_SERVICE_KEY="): + return line.split("=", 1)[1].strip() + except Exception as exc: + logger.error(f"Failed to read service key from {ENV_PATH}: {exc}") + return "" + + +def health_check(endpoint: str = BACKEND_URL) -> bool: + """Check if homelable-backend endpoint is reachable.""" + try: + import requests + except ImportError: + # Socket check fallback + import socket + try: + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + s.settimeout(3) + s.connect(('localhost', 9445)) + s.close() + logger.info("Backend socket check: port 9445 is open") + return True + except (ConnectionRefusedError, socket.timeout, OSError): + logger.warning("Backend socket check failed: port 9445 unreachable") + return False + + key = _get_service_key() + headers = {"X-MCP-Service-Key": key} + try: + resp = requests.get(f"{endpoint}/health", headers=headers, timeout=5) + if resp.status_code == 200: + logger.info("Homelable-backend health check: OK") + return True + else: + logger.warning(f"Homelable-backend health check failed with HTTP {resp.status_code}") + return False + except requests.RequestException as exc: + logger.warning(f"Homelable-backend health check exception: {exc}") + return False + + +def fetch_nodes(endpoint: str = BACKEND_URL) -> Optional[List[Dict]]: + """Fetch nodes directly from homelable-backend.""" + if not health_check(endpoint): + return None + + try: + import requests + except ImportError: + logger.error("requests library not installed") + return None + + key = _get_service_key() + headers = {"X-MCP-Service-Key": key} + try: + resp = requests.get(f"{endpoint}/nodes", headers=headers, timeout=5) + resp.raise_for_status() + nodes = resp.json() + logger.info(f"Fetched {len(nodes)} nodes directly from backend") + return nodes + except requests.RequestException as exc: + logger.error(f"Failed to fetch nodes from backend: {exc}") + return None + + +def log_mcp_error(source: str, output: str) -> None: + """Write MCP-related errors to errors.log.""" + timestamp = time.strftime("%Y-%m-%d %H:%M:%S") + log_line = f"[{timestamp}] ERROR: Homelable {source} {output}.\n" + errors_log_path = "/srv/configs/ai/errors.log" + try: + with open(errors_log_path, "a") as f: + f.write(log_line) + except OSError: + print(log_line, file=__import__("sys").stderr) + + +if __name__ == "__main__": + logger.info("Testing backend integration...") + if health_check(): + nodes = fetch_nodes() + if nodes: + print(f"\nSuccessfully retrieved {len(nodes)} nodes:") + for node in nodes[:5]: + print(f" - {node.get('label', 'unknown')} ({node.get('ip', 'no IP')}) [Status: {node.get('status', 'n/a')}]") + else: + print("Failed to retrieve nodes.") + else: + print("Backend is unreachable.")