我想在 rook ceph 上配置一个 LVM LV 设备。
我正在使用 rook rook-ceph 和 rook-ceph-cluster 的官方 helm 图表。
我已经在 rook-ceph-cluster 的 values.yaml 中配置了我的 LVM LV 设备：
...
cephClusterSpec:
...
storage:
useAllNodes: false
useAllDevices: false
nodes:
- name: "jay"
deviceFilter: "^dm."
devices: # specific devices to use for storage can be specified for each node
- name: /dev/disk/by-id/dm-name-fedora_v2202102141697143664-rook_ceph_osd
- name: "sam"
deviceFilter: "^dm."
devices:
- name: /dev/disk/by-id/dm-name-fedora_v2202201141697174721-rook_ceph_osd
...
但是 OSD 准备作业没有根据它们的日志文件找到卷：
2022-01-15 07:43:21.070783 I | cephosd: discovering hardware
99
2022-01-15 07:43:21.070804 D | exec: Running command: lsblk --all --noheadings --list --output KNAME
98
2022-01-15 07:43:21.074809 D | exec: Running command: lsblk /dev/sda --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
97
2022-01-15 07:43:21.078027 D | exec: Running command: sgdisk --print /dev/sda
96
2022-01-15 07:43:21.080645 D | exec: Running command: udevadm info --query=property /dev/sda
95
2022-01-15 07:43:21.086668 D | exec: Running command: lsblk --noheadings --path --list --output NAME /dev/sda
94
2022-01-15 07:43:21.090210 I | inventory: skipping device "sda" because it has child, considering the child instead.
93
2022-01-15 07:43:21.091515 D | exec: Running command: lsblk /dev/sda1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
92
2022-01-15 07:43:21.097514 D | exec: Running command: udevadm info --query=property /dev/sda1
91
2022-01-15 07:43:21.116469 D | exec: Running command: lsblk /dev/sda2 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
90
2022-01-15 07:43:21.118684 D | exec: Running command: udevadm info --query=property /dev/sda2
89
2022-01-15 07:43:21.123381 D | exec: Running command: lsblk /dev/sr0 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
88
2022-01-15 07:43:21.125113 W | inventory: skipping device "sr0". unsupported diskType rom
87
2022-01-15 07:43:21.125133 D | exec: Running command: lsblk /dev/nbd0 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
86
2022-01-15 07:43:21.126627 E | sys: failed to execute lsblk. output: .
85
2022-01-15 07:43:21.126644 W | inventory: skipping device "nbd0". exit status 32
84
2022-01-15 07:43:21.126649 D | exec: Running command: lsblk /dev/nbd1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
83
2022-01-15 07:43:21.128075 E | sys: failed to execute lsblk. output: .
82
2022-01-15 07:43:21.128088 W | inventory: skipping device "nbd1". exit status 32
81
2022-01-15 07:43:21.128093 D | exec: Running command: lsblk /dev/nbd2 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
80
2022-01-15 07:43:21.129559 E | sys: failed to execute lsblk. output: .
79
2022-01-15 07:43:21.129572 W | inventory: skipping device "nbd2". exit status 32
78
2022-01-15 07:43:21.129578 D | exec: Running command: lsblk /dev/nbd3 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
77
2022-01-15 07:43:21.131124 E | sys: failed to execute lsblk. output: .
76
2022-01-15 07:43:21.131140 W | inventory: skipping device "nbd3". exit status 32
75
2022-01-15 07:43:21.131146 D | exec: Running command: lsblk /dev/nbd4 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
74
2022-01-15 07:43:21.133046 E | sys: failed to execute lsblk. output: .
73
2022-01-15 07:43:21.133069 W | inventory: skipping device "nbd4". exit status 32
72
2022-01-15 07:43:21.133149 D | exec: Running command: lsblk /dev/nbd5 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
71
2022-01-15 07:43:21.135040 E | sys: failed to execute lsblk. output: .
70
2022-01-15 07:43:21.135143 W | inventory: skipping device "nbd5". exit status 32
69
2022-01-15 07:43:21.135220 D | exec: Running command: lsblk /dev/nbd6 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
68
2022-01-15 07:43:21.136804 E | sys: failed to execute lsblk. output: .
67
2022-01-15 07:43:21.136837 W | inventory: skipping device "nbd6". exit status 32
66
2022-01-15 07:43:21.136848 D | exec: Running command: lsblk /dev/nbd7 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
65
2022-01-15 07:43:21.139179 E | sys: failed to execute lsblk. output: .
64
2022-01-15 07:43:21.139199 W | inventory: skipping device "nbd7". exit status 32
63
2022-01-15 07:43:21.139206 D | exec: Running command: lsblk /dev/dm-0 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
62
2022-01-15 07:43:21.140993 D | exec: Running command: sgdisk --print /dev/dm-0
61
2022-01-15 07:43:21.142915 D | exec: Running command: udevadm info --query=property /dev/dm-0
60
2022-01-15 07:43:21.147636 D | exec: Running command: lsblk /dev/dm-1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
59
2022-01-15 07:43:21.149838 D | exec: Running command: sgdisk --print /dev/dm-1
58
2022-01-15 07:43:21.152494 D | exec: Running command: udevadm info --query=property /dev/dm-1
57
2022-01-15 07:43:21.157126 D | exec: Running command: lsblk /dev/dm-3 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
56
2022-01-15 07:43:21.159051 D | exec: Running command: sgdisk --print /dev/dm-3
55
2022-01-15 07:43:21.162884 D | exec: Running command: udevadm info --query=property /dev/dm-3
54
2022-01-15 07:43:21.169529 D | exec: Running command: lsblk /dev/nbd8 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
53
2022-01-15 07:43:21.172039 E | sys: failed to execute lsblk. output: .
52
2022-01-15 07:43:21.172060 W | inventory: skipping device "nbd8". exit status 32
51
2022-01-15 07:43:21.172066 D | exec: Running command: lsblk /dev/nbd9 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
50
2022-01-15 07:43:21.173975 E | sys: failed to execute lsblk. output: .
49
2022-01-15 07:43:21.174067 W | inventory: skipping device "nbd9". exit status 32
48
2022-01-15 07:43:21.174126 D | exec: Running command: lsblk /dev/nbd10 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
47
2022-01-15 07:43:21.175679 E | sys: failed to execute lsblk. output: .
46
2022-01-15 07:43:21.175762 W | inventory: skipping device "nbd10". exit status 32
45
2022-01-15 07:43:21.175817 D | exec: Running command: lsblk /dev/nbd11 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
44
2022-01-15 07:43:21.177453 E | sys: failed to execute lsblk. output: .
43
2022-01-15 07:43:21.177469 W | inventory: skipping device "nbd11". exit status 32
42
2022-01-15 07:43:21.177475 D | exec: Running command: lsblk /dev/nbd12 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
41
2022-01-15 07:43:21.179143 E | sys: failed to execute lsblk. output: .
40
2022-01-15 07:43:21.179356 W | inventory: skipping device "nbd12". exit status 32
39
2022-01-15 07:43:21.179449 D | exec: Running command: lsblk /dev/nbd13 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
38
2022-01-15 07:43:21.181421 E | sys: failed to execute lsblk. output: .
37
2022-01-15 07:43:21.181440 W | inventory: skipping device "nbd13". exit status 32
36
2022-01-15 07:43:21.181446 D | exec: Running command: lsblk /dev/nbd14 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
35
2022-01-15 07:43:21.183644 E | sys: failed to execute lsblk. output: .
34
2022-01-15 07:43:21.183725 W | inventory: skipping device "nbd14". exit status 32
33
2022-01-15 07:43:21.183736 D | exec: Running command: lsblk /dev/nbd15 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
32
2022-01-15 07:43:21.185612 E | sys: failed to execute lsblk. output: .
31
2022-01-15 07:43:21.185624 W | inventory: skipping device "nbd15". exit status 32
30
2022-01-15 07:43:21.185626 D | inventory: discovered disks are:
29
2022-01-15 07:43:21.185716 D | inventory: &{Name:sda1 Parent:sda HasChildren:false DevLinks:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0-part1 /dev/disk/by-uuid/aaee3811-80bc-4c89-82a8-b3966295cf22 /dev/disk/by-partuuid/f50baa8d-01 /dev/disk/by-path/pci-0000:00:1d.0-scsi-0:0:0:0-part1 Size:1073741824 UUID: Serial:0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0 Type:part Rotational:true Readonly:false Partitions:[] Filesystem:xfs Vendor:QEMU Model:QEMU_HARDDISK WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/sda1 KernelName:sda1 Encrypted:false}
28
2022-01-15 07:43:21.185732 D | inventory: &{Name:sda2 Parent:sda HasChildren:false DevLinks:/dev/disk/by-partuuid/f50baa8d-02 /dev/disk/by-path/pci-0000:00:1d.0-scsi-0:0:0:0-part2 /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0-part2 /dev/disk/by-id/lvm-pv-uuid-lI35mV-Ccyr-h1AG-3h1x-H3a0-w8ml-xnUkml Size:342522593280 UUID: Serial:0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0 Type:part Rotational:true Readonly:false Partitions:[] Filesystem:LVM2_member Vendor:QEMU Model:QEMU_HARDDISK WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/sda2 KernelName:sda2 Encrypted:false}
27
2022-01-15 07:43:21.185775 D | inventory: &{Name:dm-0 Parent: HasChildren:false DevLinks:/dev/mapper/fedora_v2202102141697143664-root /dev/disk/by-id/dm-name-fedora_v2202102141697143664-root /dev/disk/by-uuid/6c6ee237-f833-4f84-a98a-2403db411749 /dev/fedora_v2202102141697143664/root /dev/disk/by-id/dm-uuid-LVM-s3l5by1Je0DPnSs1kJdPsCt4Kyqds9efvhVGsekVynCQaCDo1z2CDJjQhEJq8Y4v Size:187367948288 UUID:9a777c10-b588-4b09-bf1d-1a2f0a21bfda Serial: Type:lvm Rotational:true Readonly:false Partitions:[] Filesystem:xfs Vendor: Model: WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/mapper/fedora_v2202102141697143664-root KernelName:dm-0 Encrypted:false}
26
2022-01-15 07:43:21.185788 D | inventory: &{Name:dm-1 Parent: HasChildren:false DevLinks:/dev/disk/by-id/dm-uuid-LVM-s3l5by1Je0DPnSs1kJdPsCt4Kyqds9efAgWqTUpCoc5FMqDBPSKcOCx2RyCsp7K6 /dev/fedora_v2202102141697143664/rook_ceph_osd /dev/disk/by-id/dm-name-fedora_v2202102141697143664-rook_ceph_osd /dev/mapper/fedora_v2202102141697143664-rook_ceph_osd Size:53687091200 UUID:fc9a2fd5-b68d-4337-8698-f2a837e81579 Serial: Type:lvm Rotational:true Readonly:false Partitions:[] Filesystem: Vendor: Model: WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/mapper/fedora_v2202102141697143664-rook_ceph_osd KernelName:dm-1 Encrypted:false}
25
2022-01-15 07:43:21.185829 D | inventory: &{Name:dm-3 Parent: HasChildren:false DevLinks:/dev/mapper/fedora_v2202102141697143664-swap /dev/disk/by-uuid/a43e162e-45a2-4b32-a29a-c4985ec23eed /dev/fedora_v2202102141697143664/swap /dev/disk/by-id/dm-name-fedora_v2202102141697143664-swap /dev/disk/by-id/dm-uuid-LVM-s3l5by1Je0DPnSs1kJdPsCt4Kyqds9efFnFkg4i3GaZC8bDNDWHx8N3Uk5PH7vNN Size:532676608 UUID:0da8f4ef-b125-4c3c-8f0e-8627a1efdf03 Serial: Type:lvm Rotational:true Readonly:false Partitions:[] Filesystem:swap Vendor: Model: WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/mapper/fedora_v2202102141697143664-swap KernelName:dm-3 Encrypted:false}
24
2022-01-15 07:43:21.185835 I | cephosd: creating and starting the osds
23
2022-01-15 07:43:21.185887 D | cephosd: desiredDevices are [{Name:/dev/disk/by-id/dm-name-fedora_v2202102141697143664-rook_ceph_osd OSDsPerDevice:1 MetadataDevice: DatabaseSizeMB:0 DeviceClass: InitialWeight: IsFilter:false IsDevicePathFilter:false}]
22
2022-01-15 07:43:21.185894 D | cephosd: context.Devices are:
21
2022-01-15 07:43:21.185926 D | cephosd: &{Name:sda1 Parent:sda HasChildren:false DevLinks:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0-part1 /dev/disk/by-uuid/aaee3811-80bc-4c89-82a8-b3966295cf22 /dev/disk/by-partuuid/f50baa8d-01 /dev/disk/by-path/pci-0000:00:1d.0-scsi-0:0:0:0-part1 Size:1073741824 UUID: Serial:0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0 Type:part Rotational:true Readonly:false Partitions:[] Filesystem:xfs Vendor:QEMU Model:QEMU_HARDDISK WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/sda1 KernelName:sda1 Encrypted:false}
20
2022-01-15 07:43:21.185940 D | cephosd: &{Name:sda2 Parent:sda HasChildren:false DevLinks:/dev/disk/by-partuuid/f50baa8d-02 /dev/disk/by-path/pci-0000:00:1d.0-scsi-0:0:0:0-part2 /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0-part2 /dev/disk/by-id/lvm-pv-uuid-lI35mV-Ccyr-h1AG-3h1x-H3a0-w8ml-xnUkml Size:342522593280 UUID: Serial:0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0 Type:part Rotational:true Readonly:false Partitions:[] Filesystem:LVM2_member Vendor:QEMU Model:QEMU_HARDDISK WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/sda2 KernelName:sda2 Encrypted:false}
19
2022-01-15 07:43:21.185977 D | cephosd: &{Name:dm-0 Parent: HasChildren:false DevLinks:/dev/mapper/fedora_v2202102141697143664-root /dev/disk/by-id/dm-name-fedora_v2202102141697143664-root /dev/disk/by-uuid/6c6ee237-f833-4f84-a98a-2403db411749 /dev/fedora_v2202102141697143664/root /dev/disk/by-id/dm-uuid-LVM-s3l5by1Je0DPnSs1kJdPsCt4Kyqds9efvhVGsekVynCQaCDo1z2CDJjQhEJq8Y4v Size:187367948288 UUID:9a777c10-b588-4b09-bf1d-1a2f0a21bfda Serial: Type:lvm Rotational:true Readonly:false Partitions:[] Filesystem:xfs Vendor: Model: WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/mapper/fedora_v2202102141697143664-root KernelName:dm-0 Encrypted:false}
18
2022-01-15 07:43:21.185990 D | cephosd: &{Name:dm-1 Parent: HasChildren:false DevLinks:/dev/disk/by-id/dm-uuid-LVM-s3l5by1Je0DPnSs1kJdPsCt4Kyqds9efAgWqTUpCoc5FMqDBPSKcOCx2RyCsp7K6 /dev/fedora_v2202102141697143664/rook_ceph_osd /dev/disk/by-id/dm-name-fedora_v2202102141697143664-rook_ceph_osd /dev/mapper/fedora_v2202102141697143664-rook_ceph_osd Size:53687091200 UUID:fc9a2fd5-b68d-4337-8698-f2a837e81579 Serial: Type:lvm Rotational:true Readonly:false Partitions:[] Filesystem: Vendor: Model: WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/mapper/fedora_v2202102141697143664-rook_ceph_osd KernelName:dm-1 Encrypted:false}
17
2022-01-15 07:43:21.185999 D | cephosd: &{Name:dm-3 Parent: HasChildren:false DevLinks:/dev/mapper/fedora_v2202102141697143664-swap /dev/disk/by-uuid/a43e162e-45a2-4b32-a29a-c4985ec23eed /dev/fedora_v2202102141697143664/swap /dev/disk/by-id/dm-name-fedora_v2202102141697143664-swap /dev/disk/by-id/dm-uuid-LVM-s3l5by1Je0DPnSs1kJdPsCt4Kyqds9efFnFkg4i3GaZC8bDNDWHx8N3Uk5PH7vNN Size:532676608 UUID:0da8f4ef-b125-4c3c-8f0e-8627a1efdf03 Serial: Type:lvm Rotational:true Readonly:false Partitions:[] Filesystem:swap Vendor: Model: WWN: WWNVendorExtension: Empty:false CephVolumeData: RealPath:/dev/mapper/fedora_v2202102141697143664-swap KernelName:dm-3 Encrypted:false}
16
2022-01-15 07:43:21.186031 I | cephosd: skipping device "sda1" because it contains a filesystem "xfs"
15
2022-01-15 07:43:21.186047 I | cephosd: skipping device "sda2" because it contains a filesystem "LVM2_member"
14
2022-01-15 07:43:21.186054 I | cephosd: skipping 'dm' device "dm-0"
13
2022-01-15 07:43:21.186056 I | cephosd: skipping 'dm' device "dm-1"
12
2022-01-15 07:43:21.186058 I | cephosd: skipping 'dm' device "dm-3"
11
2022-01-15 07:43:21.192599 I | cephosd: configuring osd devices: {"Entries":{}}
10
2022-01-15 07:43:21.192622 I | cephosd: no new devices to configure. returning devices already configured with ceph-volume.
9
2022-01-15 07:43:21.192773 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log lvm list --format json
8
2022-01-15 07:43:21.536515 D | cephosd: {}
7
2022-01-15 07:43:21.536552 I | cephosd: 0 ceph-volume lvm osd devices configured on this node
6
2022-01-15 07:43:21.536561 D | exec: Running command: cryptsetup luksDump
5
2022-01-15 07:43:21.542683 E | cephosd: failed to determine if the encrypted block "" is from our cluster. failed to dump LUKS header for disk "". Device does not exist or access denied.: exit status 4
4
2022-01-15 07:43:21.542758 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log raw list --format json
3
2022-01-15 07:43:22.157161 D | cephosd: {}
2
2022-01-15 07:43:22.157218 I | cephosd: 0 ceph-volume raw osd devices configured on this node
1
2022-01-15 07:43:22.157227 W | cephosd: skipping OSD configuration as no devices matched the storage settings for this node "jay"
我想知道的日志行是
2022-01-15 07:43:21.186056 I | cephosd: skipping 'dm' device "dm-1"
为什么它会跳过我的 dm-1 设备？我的 helm 配置有什么问题？