- deploy/compose/infra: Dienst litellm entfernt (D4), Router auf LITELLM_BASE_URL (appsrv08, Port noch offen), alle Images gepinnt, Grafana und pve-exporter ohne Profile, node-exporter für den LXC, Zustand über KNASTER_STATE_DIR - Prometheus: GPU-VMs per file_sd (node, gpu), Proxmox-Knoten per IP, infra-node - Grafana: Datasource und Dashboards provisioniert (grafana.com 10347, 1860, 14574 sowie knaster-Übersicht) - Alertmanager: n8n-Receiver nur vorbereitet (auskommentiert) - Ansible: Rollen infra-stack und gpu-exporters, Playbooks monitoring.yml und monitoring-targets.yml, Skript monitoring-targets.py (tofu output → file_sd) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
fd48fd1c7f
commit
bf7c137adf
@@ -0,0 +1,80 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Prometheus-file_sd-Ziele für die GPU-VMs erzeugen (läuft auf ops).
|
||||
|
||||
Die GPU-VMs bekommen ihre Adressen per DHCP (D5), deshalb stehen sie nicht fest in prometheus.yml.
|
||||
Eingabe ist `tofu output -json gpu_vms` (stdin), dazu die Registry für die Slot-Labels des Healers.
|
||||
|
||||
tofu output -json gpu_vms | python3 monitoring-targets.py --registry sites/knaster/registry.yaml --out DIR
|
||||
|
||||
Schreibt DIR/node.json (node_exporter, Port 9100) und DIR/gpu.json (nvidia_gpu_exporter, Port 9835).
|
||||
|
||||
Labels je Ziel: instance (VM-Name, bleibt bei IP-Wechsel gleich), vm, pve_node, vmid, slot, slots.
|
||||
`slot` ist der erste Slot der Registry mit dieser vmid (Reihenfolge der Registry), `slots` alle,
|
||||
kommagetrennt. VMs ohne Slot bekommen slot="none" (der Healer markiert dann keinen echten Slot).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
PORTS = {"node": 9100, "gpu": 9835}
|
||||
|
||||
|
||||
def slots_by_vmid(registry: dict) -> dict[str, list[str]]:
|
||||
out: dict[str, list[str]] = {}
|
||||
for name, slot in (registry.get("slots") or {}).items():
|
||||
vmid = str((slot.get("ops") or {}).get("vmid", ""))
|
||||
if vmid:
|
||||
out.setdefault(vmid, []).append(name)
|
||||
return out
|
||||
|
||||
|
||||
def build_targets(gpu_vms: dict, registry: dict) -> dict[str, list[dict]]:
|
||||
by_vmid = slots_by_vmid(registry)
|
||||
result: dict[str, list[dict]] = {job: [] for job in PORTS}
|
||||
for name in sorted(gpu_vms):
|
||||
vm = gpu_vms[name]
|
||||
if not vm.get("ipv4"):
|
||||
print(f"Warnung: {name} hat keine IPv4 (Gast-Agent?), übersprungen", file=sys.stderr)
|
||||
continue
|
||||
vmid = str(vm["vmid"])
|
||||
slots = by_vmid.get(vmid, [])
|
||||
labels = {
|
||||
"instance": name,
|
||||
"vm": name,
|
||||
"pve_node": vm["node"],
|
||||
"vmid": vmid,
|
||||
"slot": slots[0] if slots else "none",
|
||||
"slots": ",".join(slots),
|
||||
}
|
||||
for job, port in PORTS.items():
|
||||
result[job].append({"targets": [f"{vm['ipv4']}:{port}"], "labels": labels})
|
||||
return result
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
ap.add_argument("--registry", required=True, type=Path)
|
||||
ap.add_argument("--out", required=True, type=Path)
|
||||
args = ap.parse_args()
|
||||
|
||||
gpu_vms = json.load(sys.stdin)
|
||||
registry = yaml.safe_load(args.registry.read_text(encoding="utf-8"))
|
||||
targets = build_targets(gpu_vms, registry)
|
||||
args.out.mkdir(parents=True, exist_ok=True)
|
||||
for job, entries in targets.items():
|
||||
path = args.out / f"{job}.json"
|
||||
tmp = path.with_suffix(".tmp")
|
||||
tmp.write_text(json.dumps(entries, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||
tmp.replace(path)
|
||||
print(f"{path}: {len(entries)} Ziele")
|
||||
return 0 if targets["node"] else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in New Issue
Block a user