sites: Inventar und Zielbild an echte Knoten (pve44–pve61) angepasst; ADR-0001/0002/0006 ergänzt
ci / test (push) Successful in 25s
ci / test (push) Successful in 25s
Aufbauplan Task 0.4: - inventory.yaml: pve44/46/48/52/61 und GODE-PC-05/06 (Ada, Windows, cpu_roles: false) - proxmox.yaml: eine GPU-VM pro Knoten mit allen Karten (D2), local-lvm/tank (D1 B), 192.168.100.0/24 ohne VLAN, Gast-IPs .220–.239 (ANNAHME bis Task 0.2) - registry.yaml/prometheus.yml: ops-Labels auf gpu-pve44/46/48, Ada ohne pve_node/vmid - tofu: dynamische hostpci, Speicher je Gast; Ansible: pve_gpu_hosts/pve_kiosk, Replikation auf mehrere Ziele - Placement: cpu_roles=false schließt Arbeitsplätze/Kiosk von CPU-Rollen aus - PLAN §1.3/§1.4/§9 und ADR-Nachträge (D1 entschieden, D2/D3 Bestätigung offen) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
@@ -7,8 +7,8 @@ Proxmox-spezifische Schicht (austauschbar, siehe PLAN §7.2). Eingabe ist immer
|
||||
## Ablauf
|
||||
|
||||
1. **Inventar:** auf jedem Node `knaster inventory collect` ausführen und `sites/knaster/inventory.yaml` ersetzen. Danach `knaster placement plan --inventory sites/knaster/inventory.yaml` und die Ergebnisse in `proxmox.yaml` übertragen (PCI-Pfade über `lspci -nn | grep -i nvidia`, IOMMU-Gruppen).
|
||||
2. **Hosts vorbereiten:** `ansible-playbook playbooks/site.yml --limit pve_hosts`. Setzt IOMMU und VFIO und sperrt die NVIDIA-Treiber auf dem Host. Danach die Hosts neu starten. Der ZFS-Pool `tank` wird **nicht** automatisch angelegt (Datenverlustgefahr), die Rolle gibt dafür den Befehl aus.
|
||||
3. **Template:** pro Node ein Ubuntu-24.04-Cloud-Image als Template mit VMID 9000.
|
||||
2. **Hosts vorbereiten:** `ansible-playbook playbooks/site.yml --limit pve_gpu_hosts`. Setzt IOMMU und VFIO und sperrt die NVIDIA-Treiber, nur auf GPU-Knoten (pve61 bleibt unberührt, Kiosk). Danach die Hosts neu starten. Der ZFS-Pool `tank` wird **nicht** automatisch angelegt (Datenverlustgefahr), die Rolle gibt dafür den Befehl aus.
|
||||
3. **Template:** pro GPU-Knoten ein Ubuntu-24.04-Cloud-Image als Template mit VMID 9000.
|
||||
4. **VMs/LXC:** `cd tofu && tofu init && tofu plan -var proxmox_api_token=...` und danach `tofu apply`.
|
||||
5. **Gäste:** `ansible-playbook playbooks/site.yml --limit gpu_vms,infra`. Installiert die Treiber pro Generation gepinnt (legacy/modern), das Container-Toolkit, Docker und den Xid-Exporter.
|
||||
6. **Dienste:** `deploy/compose/<rolle>` auf den jeweiligen Gästen starten.
|
||||
|
||||
@@ -1,29 +1,40 @@
|
||||
# ANNAHME bis Deployment – Hosts und IPs aus sites/knaster/proxmox.yaml
|
||||
# Hosts und IPs aus sites/knaster/proxmox.yaml (Aufbauplan Task 0.4). Ansible läuft im ops-LXC (Task 3.1).
|
||||
# Gast-IPs: ANNAHME bis Task 0.2 (DHCP-Ausschluss .220–.239).
|
||||
all:
|
||||
vars:
|
||||
knaster_site: knaster
|
||||
children:
|
||||
pve_hosts:
|
||||
hosts:
|
||||
pve-ada: {}
|
||||
pve1: {}
|
||||
pve2: {}
|
||||
pve3: {}
|
||||
pve4: {}
|
||||
pve_all:
|
||||
children:
|
||||
pve_gpu_hosts:
|
||||
hosts:
|
||||
pve44: {ansible_host: 192.168.100.210}
|
||||
pve46: {ansible_host: 192.168.100.209}
|
||||
pve48: {ansible_host: 192.168.100.212}
|
||||
pve52: {ansible_host: 192.168.100.207}
|
||||
pve_kiosk: # keine VFIO-/NVIDIA-Sperre, iGPU bleibt beim Host (Kiosk)
|
||||
hosts:
|
||||
pve61: {ansible_host: 192.168.100.214}
|
||||
vars:
|
||||
ansible_user: root
|
||||
gpu_vms:
|
||||
hosts:
|
||||
gpu-main: {ansible_host: 10.20.0.201, driver_generation: modern, knaster_role: gpu-vllm}
|
||||
gpu-flex: {ansible_host: 10.20.0.202, driver_generation: modern, knaster_role: gpu-vllm}
|
||||
gpu-embed-1: {ansible_host: 10.20.0.221, driver_generation: legacy, knaster_role: gpu-llamacpp, llamacpp_profile: embed}
|
||||
gpu-embed-2: {ansible_host: 10.20.0.222, driver_generation: legacy, knaster_role: gpu-llamacpp, llamacpp_profile: embed}
|
||||
gpu-small: {ansible_host: 10.20.0.231, driver_generation: legacy, knaster_role: gpu-llamacpp, llamacpp_profile: chat}
|
||||
gpu-pve44: {ansible_host: 192.168.100.224}
|
||||
gpu-pve46: {ansible_host: 192.168.100.226}
|
||||
gpu-pve48: {ansible_host: 192.168.100.228}
|
||||
gpu-pve52: {ansible_host: 192.168.100.232}
|
||||
vars:
|
||||
ansible_user: knaster
|
||||
ansible_become: true
|
||||
driver_generation: legacy # R580, Maxwell/Pascal
|
||||
knaster_role: gpu-llamacpp
|
||||
infra:
|
||||
hosts:
|
||||
infra: {ansible_host: 10.20.0.101}
|
||||
infra: {ansible_host: 192.168.100.221}
|
||||
vars:
|
||||
ansible_user: root
|
||||
ops:
|
||||
hosts:
|
||||
ops: {ansible_host: 192.168.100.220}
|
||||
vars:
|
||||
ansible_user: root
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# Reihenfolge: Hosts vorbereiten → (tofu apply) → GPU-VMs → Infra
|
||||
- name: Proxmox-Hosts (IOMMU, VFIO, ZFS-Replikation)
|
||||
hosts: pve_hosts
|
||||
# Reihenfolge: GPU-Knoten vorbereiten → (tofu apply) → GPU-VMs → Infra
|
||||
# pve61 (pve_kiosk) bekommt bewusst keine Host-Rolle: keine VFIO-Bindung, keine NVIDIA-Sperre (Review Focus 5).
|
||||
- name: Proxmox-GPU-Knoten (IOMMU, VFIO, ZFS-Replikation)
|
||||
hosts: pve_gpu_hosts
|
||||
roles: [pve-host]
|
||||
|
||||
- name: GPU-VMs (Treiber gepinnt, Container-Toolkit, Docker, Xid-Exporter)
|
||||
|
||||
@@ -50,21 +50,28 @@
|
||||
|
||||
- name: Hinweis, falls tank fehlt (Pool wird bewusst NICHT automatisch angelegt – Datenverlustgefahr)
|
||||
ansible.builtin.debug:
|
||||
msg: "Auf {{ inventory_hostname }} fehlt der ZFS-Pool 'tank'. Anlegen z. B.: zpool create -o ashift=12 tank mirror /dev/disk/by-id/<ssd2> /dev/disk/by-id/<ssd3> && pvesm add zfspool tank -pool tank"
|
||||
when: tank.rc != 0
|
||||
msg: >-
|
||||
Auf {{ inventory_hostname }} fehlt der ZFS-Pool 'tank'. Nach Freigabe (D1) und Prüfung mit
|
||||
`lsblk -o NAME,SIZE,FSTYPE,LABEL /dev/sdb`: zpool create -o ashift=12 -O compression=lz4 -O atime=off
|
||||
tank /dev/disk/by-id/<ssd2> && pvesm add zfspool tank --pool tank --content images,rootdir
|
||||
when:
|
||||
- tank.rc != 0
|
||||
- knaster_pve.nodes[inventory_hostname].zfs_tank | default(false)
|
||||
|
||||
- name: Bestehende Replikationsjobs lesen
|
||||
ansible.builtin.command: pvesr list
|
||||
register: pvesr_jobs
|
||||
changed_when: false
|
||||
|
||||
- name: ZFS-Replikation für Infra-Gäste
|
||||
# Ein Job pro Ziel: <vmid>-0, <vmid>-1, … (Reihenfolge aus replicate_to)
|
||||
- name: ZFS-Replikation für HA-Gäste
|
||||
ansible.builtin.command: >
|
||||
pvesr create-local-job {{ item.value.vmid }}-0 {{ item.value.replicate_to }}
|
||||
--schedule '{{ knaster_pve.replication_schedule }}' --comment 'knaster {{ item.key }}'
|
||||
loop: "{{ knaster_pve.containers | dict2items }}"
|
||||
pvesr create-local-job {{ item.0.value.vmid }}-{{ idx }} {{ item.1 }}
|
||||
--schedule '{{ knaster_pve.replication_schedule }}' --comment 'knaster {{ item.0.key }}'
|
||||
loop: "{{ knaster_pve.containers | dict2items | subelements('value.replicate_to', skip_missing=true) }}"
|
||||
vars:
|
||||
idx: "{{ item.0.value.replicate_to.index(item.1) }}"
|
||||
when:
|
||||
- item.value.node == inventory_hostname
|
||||
- item.value.replicate_to is defined
|
||||
- (item.value.vmid | string) + '-0' not in pvesr_jobs.stdout
|
||||
- item.0.value.node == inventory_hostname
|
||||
- ((item.0.value.vmid | string) + '-' + (idx | string)) not in pvesr_jobs.stdout
|
||||
changed_when: true
|
||||
|
||||
@@ -41,12 +41,12 @@ resource "proxmox_virtual_environment_vm" "gpu" {
|
||||
}
|
||||
|
||||
efi_disk {
|
||||
datastore_id = local.site.cluster.storage_vm
|
||||
datastore_id = try(each.value.storage, local.site.cluster.storage_default)
|
||||
type = "4m"
|
||||
}
|
||||
|
||||
disk {
|
||||
datastore_id = local.site.cluster.storage_vm
|
||||
datastore_id = try(each.value.storage, local.site.cluster.storage_default)
|
||||
interface = "scsi0"
|
||||
size = each.value.disk_gb
|
||||
discard = "on"
|
||||
@@ -55,7 +55,7 @@ resource "proxmox_virtual_environment_vm" "gpu" {
|
||||
|
||||
# Modelle: eigenes Disk-Image, nicht repliziert, nicht gesichert (ADR-0002)
|
||||
disk {
|
||||
datastore_id = local.site.cluster.storage_vm
|
||||
datastore_id = try(each.value.storage, local.site.cluster.storage_default)
|
||||
interface = "scsi1"
|
||||
size = each.value.models_disk_gb
|
||||
discard = "on"
|
||||
@@ -64,20 +64,24 @@ resource "proxmox_virtual_environment_vm" "gpu" {
|
||||
replicate = false
|
||||
}
|
||||
|
||||
hostpci {
|
||||
device = "hostpci0"
|
||||
mapping = proxmox_virtual_environment_hardware_mapping_pci.gpu[each.value.gpu].name
|
||||
pcie = true
|
||||
rombar = true
|
||||
# Alle Karten des Knotens in eine VM (D2); Reihenfolge = Index in der VM
|
||||
dynamic "hostpci" {
|
||||
for_each = each.value.gpus
|
||||
content {
|
||||
device = "hostpci${hostpci.key}"
|
||||
mapping = proxmox_virtual_environment_hardware_mapping_pci.gpu[hostpci.value].name
|
||||
pcie = true
|
||||
rombar = true
|
||||
}
|
||||
}
|
||||
|
||||
network_device {
|
||||
bridge = local.site.cluster.bridge
|
||||
vlan_id = local.site.cluster.vlan_svc
|
||||
vlan_id = local.site.cluster.vlan_svc # null = flaches Netz
|
||||
}
|
||||
|
||||
initialization {
|
||||
datastore_id = local.site.cluster.storage_vm
|
||||
datastore_id = try(each.value.storage, local.site.cluster.storage_default)
|
||||
ip_config {
|
||||
ipv4 {
|
||||
address = each.value.ip
|
||||
@@ -94,16 +98,16 @@ resource "proxmox_virtual_environment_vm" "gpu" {
|
||||
}
|
||||
}
|
||||
|
||||
resource "proxmox_virtual_environment_container" "infra" {
|
||||
resource "proxmox_virtual_environment_container" "lxc" {
|
||||
for_each = local.site.containers
|
||||
node_name = each.value.node
|
||||
vm_id = each.value.vmid
|
||||
tags = ["knaster", "infra"]
|
||||
tags = ["knaster", each.key]
|
||||
unprivileged = true
|
||||
start_on_boot = true
|
||||
|
||||
features {
|
||||
nesting = true # Docker im LXC
|
||||
nesting = try(each.value.nesting, true) # Docker im LXC
|
||||
}
|
||||
|
||||
cpu {
|
||||
@@ -115,12 +119,12 @@ resource "proxmox_virtual_environment_container" "infra" {
|
||||
}
|
||||
|
||||
disk {
|
||||
datastore_id = local.site.cluster.storage_vm
|
||||
datastore_id = try(each.value.storage, local.site.cluster.storage_default)
|
||||
size = each.value.disk_gb
|
||||
}
|
||||
|
||||
operating_system {
|
||||
template_file_id = "${local.site.cluster.storage_iso}:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst"
|
||||
template_file_id = local.site.cluster.lxc_template
|
||||
type = "debian"
|
||||
}
|
||||
|
||||
@@ -132,6 +136,9 @@ resource "proxmox_virtual_environment_container" "infra" {
|
||||
|
||||
initialization {
|
||||
hostname = each.key
|
||||
dns {
|
||||
servers = [local.site.cluster.dns]
|
||||
}
|
||||
ip_config {
|
||||
ipv4 {
|
||||
address = each.value.ip
|
||||
@@ -149,5 +156,5 @@ resource "proxmox_virtual_environment_haresource" "infra" {
|
||||
resource_id = "ct:${each.value.vmid}"
|
||||
state = "started"
|
||||
comment = "knaster ${each.key}"
|
||||
depends_on = [proxmox_virtual_environment_container.infra]
|
||||
depends_on = [proxmox_virtual_environment_container.lxc]
|
||||
}
|
||||
|
||||
@@ -2,6 +2,6 @@ output "gpu_vms" {
|
||||
value = { for k, v in proxmox_virtual_environment_vm.gpu : k => { node = v.node_name, vmid = v.vm_id } }
|
||||
}
|
||||
|
||||
output "infra_containers" {
|
||||
value = { for k, v in proxmox_virtual_environment_container.infra : k => { node = v.node_name, vmid = v.vm_id } }
|
||||
output "containers" {
|
||||
value = { for k, v in proxmox_virtual_environment_container.lxc : k => { node = v.node_name, vmid = v.vm_id } }
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user