0

我需要帮助以使 XFS 配额在 DigitalOcean 上的 Kubernetes 中工作。

我的问题本质上是该xfs_quota工具似乎仅在一个人还可以访问磁盘设备时才起作用,而不仅仅是对已安装的卷。但是,无论我尝试什么,我似乎都无法同时访问设备和安装。

我尝试了卷安装和原始块卷。

卷安装

这是我的存储类:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: block-storage-retain-xfs-prjquota
provisioner: dobs.csi.digitalocean.com
parameters:
  fsType: xfs
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
mountOptions:
  - prjquota

然后我声明一个新卷并将其添加到这样的 pod 中:

volumeClaimTemplates:
  - metadata:
      name: my-storage
      namespace: my-namespace
    spec:
      accessModes:
        - ReadWriteOnce
      storageClassName: block-storage-retain-xfs-prjquota
      resources:
        requests:
          storage: 1Gi

并安装它:

volumeMounts:
  - name: my-storage
    mountPath: "/var/www"

在 pod 中,一切都正确安装,我可以访问卷(我可以在其中创建内容)并且安装标志设置正确:

$ mount | grep -i www
/dev/disk/by-id/scsi-0DO_Volume_pvc-650ccba6-3177-45b5-9ffb-0ac2a931fddc on /var/www type xfs (rw,relatime,attr2,inode64,prjquota)

但是,磁盘设备在 pod 中不可用:

$ ls -la /dev/disk/by-id/scsi-0DO_Volume_pvc-650ccba6-3177-45b5-9ffb-0ac2a931fddc
ls: cannot access '/dev/disk/by-id/scsi-0DO_Volume_pvc-650ccba6-3177-45b5-9ffb-0ac2a931fddc': No such file or directory

(其实整个/dev/disk/目录都不可用)

根据我的调查,无法访问设备是 XFS 工具失败的原因:

$ xfs_quota -x -c 'report -h' /var/www
xfs_quota: cannot setup path for mount /var/www: No such device or address

原始块卷

我还尝试改用原始块卷:

volumeClaimTemplates:
  - metadata:
      name: my-storage
      namespace: my-namespace
    spec:
      accessModes:
        - ReadWriteOnce
      volumeMode: Block
      storageClassName: block-storage-retain-xfs-prjquota
      resources:
        requests:
          storage: 1Gi

并将其添加为:

volumeDevices:
  - name: my-storage
    devicePath: /dev/my-storage

这给了我设备,但由于某种原因我无法格式化/安装它(实际上既不是 XFS 也不是 ext4):

$ mkfs.xfs /dev/my-storage
mkfs.xfs: error - cannot set blocksize 512 on block device /dev/my-storage: Permission denied
$ mkfs.ext4 /dev/my-storage
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 262144 4k blocks and 65536 inodes
Filesystem UUID: 18f07181-737c-4b68-a5fe-ccd7f2c50ff8
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376

Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

$ mount /dev/my-storage /var/www
mount: /var/www: cannot mount /dev/my-storage read-only.

使用SYS_ADMINLinux 功能,我实际上可以格式化它,但我仍然无法挂载它:

$ mkfs.xfs -f /dev/my-storage
meta-data=/dev/my-storage        isize=512    agcount=4, agsize=65536 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

$ mount /dev/my-storage /var/www
mount: /var/www: cannot mount /dev/my-storage read-only.

(为什么磁盘设备是只读的?)

原始块卷 - 带分区

好的,所以我尝试创建一个分区并对其进行格式化。分区创建成功,但我无权访问分区设备:

$ fdisk -l /dev/my-storage
Disk /dev/my-storage: 1 GiB, 1073741824 bytes, 2097152 sectors
Disk model: Volume
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xb4a24142

Device            Boot Start     End Sectors  Size Id Type
/dev/my-storage1        2048 2097151 2095104 1023M 83 Linux

但是,/dev/my-storage1不存在:

$ ls -la /dev/my-storage*
brw-rw---- 1 root disk 8, 48 Oct 25 14:42 /dev/my-storage

我尝试运行容器,因为privileged它使我可以访问更多设备/dev,但是我根本没有看到我的原始块卷设备。

接下来是什么?

As I see that, any of those would work for me:

  1. Getting access to the underlying block device for volume mounts.
  2. Access to the partition device so that I can mount it.
  3. Ability to mount the raw block volume (e.g. by making it not read-only, whatever it means?).
  4. Making the xfs_quota tool NOT require the underlying device.

I believe I made it work a few months ago using raw block volumes with partitions, but either I forgot how or something changed on DigitalOcean and I can't seem to be able to create and access partitions anymore.

Any help is hugely appreciated, thank you!

4

1 回答 1

2

Timo here from the Managed Kubernetes (DOKS) team at DigitalOcean.

What you are missing is the host system mount of the /dev directory. If you add both

 volumes:
 - name: device-dir
   hostPath:
     path: /dev

and

volumeMounts:
  - name: device-dir
    mountPath: /dev

to the manifest at the right places, things should work as expected.

于 2020-11-02T14:44:36.220 回答