docker - COS 的 /var/lib/docker 卷的 inode 不足

Question

我正在尝试让我们在 GCP 上运行一些服务。

我目前看到的一个问题是，我很快启动的虚拟机似乎用完了 /var/lib/docker 文件系统的 inode。我原以为这是在容器优化操作系统中调整的事情之一？

wouter@nbwm-cron ~ $ df -hi Filesystem Inodes IUsed IFree IUse% Mounted on /dev/root 78K 13K 65K 17% / devtmpfs 463K 204 463K 1% /dev tmpfs 464K 1 464K 1% /dev/shm tmpfs 464K 500 463K 1% /run tmpfs 464K 13 464K 1% /sys/fs/cgroup tmpfs 464K 9 464K 1% /mnt/disks tmpfs 464K 16K 448K 4% /tmp /dev/sda8 4.0K 11 4.0K 1% /usr/share/oem /dev/sda1 1013K 998K 15K 99% /var tmpfs 464K 45 464K 1% /var/lib/cloud overlayfs 464K 39 464K 1% /etc

wouter@nbwm-cron ~ $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE <name>/stackdriver-agent latest 0c4b075e7550 3 days ago 1.423 GB <none> <none> 96d027d3feea 4 days ago 905.2 MB gcr.io/<project>/nbwm-ops/docker-php5 latest 5d2c59c7dd7a 2 weeks ago 1.788 GB

nbwm-cron wouter # tune2fs -l /dev/sda1 tune2fs 1.43.3 (04-Sep-2016) Filesystem volume name: STATE Last mounted on: /var Filesystem UUID: ca44779b-ffd5-405a-bd3e-528071b45f73 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Remount read-only Filesystem OS type: Linux Inode count: 1036320 Block count: 4158971 Reserved block count: 0 Free blocks: 4062454 Free inodes: 1030756 First block: 0 Block size: 4096 Fragment size: 4096 Group descriptor size: 64 Reserved GDT blocks: 747 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8160 Inode blocks per group: 510 Flex block group size: 16 Filesystem created: Thu Jun 15 22:39:33 2017 Last mount time: Wed Jun 28 13:51:31 2017 Last write time: Wed Jun 28 13:51:31 2017 Mount count: 5 Maximum mount count: -1 Last checked: Thu Nov 19 19:00:00 2009 Check interval: 0 (<none>) Lifetime writes: 67 MB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 32 Desired extra isize: 32 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: 66aa0e7f-57da-41d0-86f7-d93270e53030 Journal backup: inode blocks

如何调整文件系统以获得更多可用的 inode？

Answer 1

I have witnessed the same issue with all COS versions from 57.9202.64.0 (docker 1.11.2) on GKE 1.5 to 65.10323.85.0 (docker 17.03.2) on GKE 1.8.12-gke.1. Older version were certainly affected too.

Those all use the overlay driver:

pdecat@gke-cluster-test-pdecat-default-pool-e8945081-xhj6 ~ $ docker info 2>&1 | grep "Storage Driver"
Storage Driver: overlay

pdecat@gke-cluster-test-pdecat-default-pool-e8945081-xhj6 ~ $ grep "\(CHROMEOS_RELEASE_VERSION\|CHROMEOS_RELEASE_CHROME_MILESTONE\)" /etc/lsb-release
CHROMEOS_RELEASE_CHROME_MILESTONE=65
CHROMEOS_RELEASE_VERSION=10323.85.0

The overlay2 driver is only used for GKE 1.9+ clusters (fresh or upgraded) with the same COS version:

pdecat@gke-cluster-test-pdecat-default-pool-e8945081-xhj6 ~ $ docker info 2>&1 | grep "Storage Driver"
Storage Driver: overlay2

pdecat@gke-cluster-test-pdecat-default-pool-e8945081-xhj6 ~ $ grep "\(CHROMEOS_RELEASE_VERSION\|CHROMEOS_RELEASE_CHROME_MILESTONE\)" /etc/lsb-release
CHROMEOS_RELEASE_CHROME_MILESTONE=65
CHROMEOS_RELEASE_VERSION=10323.85.0

When the free space/inodes issue occurs with the overlay driver, I resolve it using spotify's docker-gc:

# docker run --rm --userns host -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc spotify/docker-gc

Before:

# df -hi /var/lib/docker/
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/sda1        6.0M  5.0M  1.1M   83% /var

# df -h /var/lib/docker/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        95G   84G   11G  89% /var

# du --inodes -s /var/lib/docker/*
180     /var/lib/docker/containers
4093    /var/lib/docker/image
4       /var/lib/docker/network
4906733 /var/lib/docker/overlay
1       /var/lib/docker/tmp
1       /var/lib/docker/trust
25      /var/lib/docker/volumes

After:

# df -hi /var/lib/docker/
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/sda1        6.0M  327K  5.7M    6% /var/lib/docker

# df -h /var/lib/docker/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        95G  6.6G   88G   7% /var/lib/docker

# du --inodes -s /var/lib/docker/*
218     /var/lib/docker/containers
1792    /var/lib/docker/image
4       /var/lib/docker/network
279002  /var/lib/docker/overlay
1       /var/lib/docker/tmp
1       /var/lib/docker/trust
25      /var/lib/docker/volumes

Note: using the usual docker rmi $(docker images --filter "dangling=true" -q --no-trunc) and docker rm $(docker ps -qa --no-trunc --filter "status=exited") did not help to recover resources in /var/lib/docker/overlay.

Answer 2

overlay这是docker中存储驱动程序的一个已知问题，驱动程序已解决overlay2。新的 cos-61版本使用带有overlay2存储驱动程序的 docker 17.03。请您试一试，看看问题是否再次出现？

谢谢！