0

我目前正在运行两个堆栈，一个在 EBS 上，一个在 VirtualBox 上。

我的 VirtualBox 设置能够毫无问题地创建和访问卷，但我无法使用rexray/ebs插件达到等效状态，我可以在其中创建卷并将卷附加到容器。

在我的实例（Ubuntu 18.04 - T2.small us-west-1）上，我正在运行以下版本的 docker 和 rexray

1. docker version 19.03.4, build 9013bf583a
2. docker-machine version 0.16.0, build 702c267f
3. docker plugin ls

ID                  NAME                DESCRIPTION              ENABLED
e3b496c6a066        rexray/ebs:latest   REX-Ray for Amazon EBS   true

我的基于 EBS 的 rexray 配置看起来像这样。code-sample1

libstorage:
  logging:
    level: debug
  service: ebs
  integration:
    volume:
      operations:
        mount:
          preempt: true
docker:
  size: 1

ebs:
  accessKey: valid_access_key
  secretKey: valid_secret
  region: us-west-1
  maxRetries: 10

在 VirtualBox 上，我明确地为 libstorage 提供了与 VirtualBox 连接的路径，VirtualBox 通过 API 充当卷驱动程序。在这种情况下http://10.0.0.6:18083。

我的 rexray 的 VirtualBox 配置/etc/config/rexray.yml

code-sample2

rexray:
  loglevel: warn
  storageDrivers:
    - virtualbox
  volumeDrivers:
    - docker

  docker:
    size: 1

libstorage:
  service: virtualbox
  integration:
    volume:
      operations:
        mount:
          preempt: true

virtualbox:
  endpoint: http://10.0.0.6:18083
  volumePath: "/home/sidravic/VirtualBox VMs"
  controllerName: SATA
  tls: false

但是，对于 EBS，我的假设是插件根据区域确定 AWS 的连接位置以操作卷，因此我将它简单地作为服务运行，而不在 yml 中提供服务器配置

然而问题是当我尝试通过 rexray 创建一个新卷时，日志似乎正在调用一个 unix 套接字

Nov 10 10:16:10 ip-172-31-1-66 dockerd[3128]: time="2019-11-10T10:16:10Z" level=error msg="time=\"2019-11-10T10:16:10Z\" level=error msg=\"error creating volume\" availabilityZone=0xc420194200 host=\"unix:///var/run/rexray/156329133.sock\" instanceID=\"ebs=i-03656f91f34f2d9f4,availabilityZone=us-west-1b&region=us-west-1\" iops=0xc420194208 route=volumeCreate server=snapdragon-paw-eg service=ebs size=0xc420194210 storageDriver=ebs task=0 time=1573380970129 tls=false txCR=1573380969 txID=eb4f7412-ca73-4c69-6977-a68c0aa90744 type=0xc420194218 volumeName=test-vol1 " plugin=e3b496c6a06643ecc9babd1e1648606fef2cc964946d9ba003e73868847ef8c7
Nov 10 10:16:10 ip-172-31-1-66 dockerd[3128]: time="2019-11-10T10:16:10Z" level=error msg="time=\"2019-11-10T10:16:10Z\" level=error msg=\"error creating volume with EC2 API call\" host=\"unix:///var/run/rexray/156329133.sock\" instanceID=\"ebs=i-03656f91f34f2d9f4,availabilityZone=us-west-1b&region=us-west-1\" route=volumeCreate server=snapdragon-paw-eg service=ebs storageDriver=ebs task=0 time=1573380970130 tls=false txCR=1573380969 txID=eb4f7412-ca73-4c69-6977-a68c0aa90744 " plugin=e3b496c6a06643ecc9babd1e1648606fef2cc964946d9ba003e73868847ef8c7
Nov 10 10:16:10 ip-172-31-1-66 dockerd[3128]: time="2019-11-10T10:16:10Z" level=error msg="time=\"2019-11-10T10:16:10Z\" level=error msg=\"error: api call failed\" host=\"unix:///var/run/rexray/156329133.sock\" route=volumeCreate server=snapdragon-paw-eg time=1573380970131 tls=false txCR=1573380969 txID=eb4f7412-ca73-4c69-6977-a68c0aa90744 " plugin=e3b496c6a06643ecc9babd1e1648606fef2cc964946d9ba003e73868847ef8c7
Nov 10 10:16:10 ip-172-31-1-66 dockerd[3128]: time="2019-11-10T10:16:10Z" level=error msg="time=\"2019-11-10T10:16:10Z\" level=error msg=\"docker-legacy: Create: test-vol1: failed: error creating volume with EC2 API call\" host=\"unix:///var/run/rexray/156329133.sock\" integrationDriver=linux osDriver=linux service=ebs storageDriver=libstorage time=1573380970133 " plugin=e3b496c6a06643ecc9babd1e1648606fef2cc964946d9ba003e73868847ef8c7
Nov 10 10:16:10 ip-172-31-1-66 dockerd[3128]: time="2019-11-10T10:16:10.134279110Z" level=error msg="Handler for POST /v1.40/volumes/create returned error: create test-vol1: VolumeDriver.Create: docker-legacy: Create: test-vol1: failed: error creating volume with EC2 API call"

rexray 命令本身返回此错误

$ sudo rexray volume create test-vol1 --size=2 -c /etc/rexray/config.yml
...
INFO[0000] DockerLegacyMode                              enabled=true time=1573381404622
INFO[0000] loaded custom config                          path=/etc/rexray/config.yml time=1573381404622
DEBU[0000] trapped signals                               time=1573381404622
DEBU[0000] usage template path                           path=/root/.rexray/usage.template time=1573381404624
DEBU[0000] loaded usage template                         source=UsageTemplate time=1573381404624
DEBU[0000] created scoped scope                          new=rexray.cli parentScopes=rexray,
DEBU[0000] updating log level                            level=debug time=1573381404628
INFO[0000] updated log level                             logLevel=debug
DEBU[0000] os.args                                       time=1573381404634 val=[rexray volume create test-vol1 --size=2 -c /etc/rexray/config.yml]
DEBU[0000] activating libStorage                         cmd=create time=1573381404634
DEBU[0000] parseSafeHost - no change                     postParse=unix:///var/run/rexray/157173546.sock preParse=unix:///var/run/rexray/157173546.sock time=1573381404642
DEBU[0000] read spec file                                host=unix:///var/run/rexray/157173546.sock path=/var/run/rexray/rexray.spec time=1573381404642
DEBU[0000] is local unix server active                   sock=/var/run/rexray/157173546.sock time=1573381404642
DEBU[0000] set host in context                           host=unix:///var/run/rexray/157173546.sock time=1573381404643
DEBU[0000] set host in config                            host=unix:///var/run/rexray/157173546.sock time=1573381404643
DEBU[0000] not starting embedded server; already running  host=unix:///var/run/rexray/157173546.sock time=1573381404643
DEBU[0000] creating libStorage client                    cmd=create host=unix:///var/run/rexray/157173546.sock time=1573381404643
DEBU[0000] created scoped scope                          new=libstorage.client parentScopes=rexray.cli,rexray,
INFO[0000] configured logging                            host=unix:///var/run/rexray/157173546.sock libstorage.logging.httpRequests=false libstorage.logging.httpResponses=false libstorage.logging.level=debug time=1573381404783
INFO[0000] set client service name                       host=unix:///var/run/rexray/157173546.sock service=ebs serviceName=ebs time=1573381404787
DEBU[0000] got configured host address                   host=unix:///var/run/rexray/157173546.sock service=ebs storageDriver=libstorage time=1573381404795
DEBU[0000] disabling tls for unix sockets                host=unix:///var/run/rexray/157173546.sock service=ebs storageDriver=libstorage time=1573381404799
DEBU[0000] getHost proto == "unix"                       getHost=libstorage-server host=unix:///var/run/rexray/157173546.sock service=ebs storageDriver=libstorage time=1573381404799
INFO[0000] created libStorage client                     clientType=integration disableKeepAlive=false enableInstanceIDHeaders=true enableLocalDevicesHeaders=true host=unix:///var/run/rexray/157173546.sock lAddr=libstorage-server logRequests=false logResponses=false lsxPath= service=ebs storageDriver=libstorage time=1573381404829
INFO[0000] registered custom context key                 externalID=Libstorage-Tx internalID=2 keyBitmask=2
INFO[0000] registered custom context key                 externalID=Libstorage-Instanceid internalID=3 keyBitmask=2
INFO[0000] registered custom context key                 externalID=Libstorage-Localdevices internalID=4 keyBitmask=2
INFO[0000] registered custom context key                 externalID=Authorization internalID=5 keyBitmask=2
DEBU[0000] successful connection                         host=unix:///var/run/rexray/157173546.sock service=ebs storageDriver=libstorage time=1573381404830
INFO[0000] initializing supported cache                  host=unix:///var/run/rexray/157173546.sock server=wheat-heron-vi service=ebs storageDriver=libstorage time=1573381404833
DEBU[0000] executor using large device range             host=unix:///var/run/rexray/157173546.sock server=wheat-heron-vi service=ebs storageDriver=libstorage time=1573381404926 txCR=1573381404 txID=a36adacb-decc-435c-5089-df382ad39d6d
DEBU[0000] cached supported flag                         host=unix:///var/run/rexray/157173546.sock server=wheat-heron-vi service=ebs storageDriver=libstorage supported=15 time=1573381404932 txCR=1573381404 txID=a36adacb-decc-435c-5089-df382ad39d6d
INFO[0000] initializing instance ID cache                host=unix:///var/run/rexray/157173546.sock server=wheat-heron-vi service=ebs storageDriver=libstorage time=1573381404932
DEBU[0000] executor using large device range             host=unix:///var/run/rexray/157173546.sock server=wheat-heron-vi service=ebs storageDriver=libstorage time=1573381405023 txCR=1573381404 txID=ec3f824e-dd49-4789-572c-511efb5bb972
DEBU[0000] cached instanceID                             host=unix:///var/run/rexray/157173546.sock instanceID=ebs=i-03656f91f34f2d9f4,availabilityZone=us-west-1b&region=us-west-1 server=wheat-heron-vi service=ebs storageDriver=libstorage time=1573381405032 txCR=1573381404 txID=ec3f824e-dd49-4789-572c-511efb5bb972
DEBU[0000] xli instanceID success                        host=unix:///var/run/rexray/157173546.sock instanceID=ebs=i-03656f91f34f2d9f4,availabilityZone=us-west-1b&region=us-west-1 server=wheat-heron-vi service=ebs storageDriver=libstorage time=1573381405033 txCR=1573381404 txID=ec3f824e-dd49-4789-572c-511efb5bb972
INFO[0000] successefully dialed libStorage server        host=unix:///var/run/rexray/157173546.sock server=wheat-heron-vi service=ebs storageDriver=libstorage time=1573381405033
INFO[0000] storage driver initialized                    host=unix:///var/run/rexray/157173546.sock service=ebs storageDriver=libstorage time=1573381405033
INFO[0000] os driver initialized                         host=unix:///var/run/rexray/157173546.sock osDriver=linux service=ebs storageDriver=libstorage time=1573381405037
INFO[0000] linux integration driver successfully initialized  host=unix:///var/run/rexray/157173546.sock integrationDriver=linux libstorage.integration.volume.operations.create.default.IOPS=0 libstorage.integration.volume.operations.create.default.availabilityZone= libstorage.integration.volume.operations.create.default.fsType=ext4 libstorage.integration.volume.operations.create.default.size=16 libstorage.integration.volume.operations.create.default.type= libstorage.integration.volume.operations.create.implicit=true libstorage.integration.volume.operations.mount.path=/var/lib/rexray/volumes libstorage.integration.volume.operations.mount.rootPath=/data osDriver=linux service=ebs storageDriver=libstorage time=1573381405080
INFO[0000] path cache initializion disabled              host=unix:///var/run/rexray/157173546.sock integrationDriver=linux osDriver=linux service=ebs storageDriver=libstorage time=1573381405092
INFO[0000] libStorage integration driver successfully initialized  host=unix:///var/run/rexray/157173546.sock integrationDriver=linux libstorage.integration.volume.operations.create.disable=false libstorage.integration.volume.operations.mount.preempt=true libstorage.integration.volume.operations.path.cache.async=true libstorage.integration.volume.operations.path.cache.enabled=false libstorage.integration.volume.operations.remove.disable=false libstorage.integration.volume.operations.unmount.ignoreusedcount=false osDriver=linux service=ebs storageDriver=libstorage time=1573381405122
INFO[0000] integration driver initialized                host=unix:///var/run/rexray/157173546.sock integrationDriver=linux osDriver=linux service=ebs storageDriver=libstorage time=1573381405122
INFO[0000] created libStorage client                     host=unix:///var/run/rexray/157173546.sock integrationDriver=linux osDriver=linux service=ebs storageDriver=libstorage time=1573381405122
DEBU[0000] executor using large device range             host=unix:///var/run/rexray/157173546.sock instanceID=ebs=i-03656f91f34f2d9f4,availabilityZone=us-west-1b&region=us-west-1 integrationDriver=linux osDriver=linux server=wheat-heron-vi service=ebs storageDriver=libstorage time=1573381405213 txCR=1573381405 txID=0b934f6e-10af-48c0-6236-e12964e11416
WARN[0000] device does not match                         deviceName=name deviceRX=^xvd[f-p]$ host=unix:///var/run/rexray/157173546.sock instanceID=ebs=i-03656f91f34f2d9f4,availabilityZone=us-west-1b&region=us-west-1 integrationDriver=linux osDriver=linux server=wheat-heron-vi service=ebs storageDriver=libstorage time=1573381405218 txCR=1573381405 txID=0b934f6e-10af-48c0-6236-e12964e11416
WARN[0000] device does not match                         deviceName=loop0 deviceRX=^xvd[f-p]$ host=unix:///var/run/rexray/157173546.sock instanceID=ebs=i-03656f91f34f2d9f4,availabilityZone=us-west-1b&region=us-west-1 integrationDriver=linux osDriver=linux server=wheat-heron-vi service=ebs storageDriver=libstorage time=1573381405219 txCR=1573381405 txID=0b934f6e-10af-48c0-6236-e12964e11416
WARN[0000] device does not match                         deviceName=loop1 deviceRX=^xvd[f-p]$ host=unix:///var/run/rexray/157173546.sock instanceID=ebs=i-03656f91f34f2d9f4,availabilityZone=us-west-1b&region=us-west-1 integrationDriver=linux osDriver=linux server=wheat-heron-vi service=ebs storageDriver=libstorage time=1573381405219 txCR=1573381405 txID=0b934f6e-10af-48c0-6236-e12964e11416
WARN[0000] device does not match                         deviceName=xvda deviceRX=^xvd[f-p]$ host=unix:///var/run/rexray/157173546.sock instanceID=ebs=i-03656f91f34f2d9f4,availabilityZone=us-west-1b&region=us-west-1 integrationDriver=linux osDriver=linux server=wheat-heron-vi service=ebs storageDriver=libstorage time=1573381405219 txCR=1573381405 txID=0b934f6e-10af-48c0-6236-e12964e11416
WARN[0000] device does not match                         deviceName=xvda1 deviceRX=^xvd[f-p]$ host=unix:///var/run/rexray/157173546.sock instanceID=ebs=i-03656f91f34f2d9f4,availabilityZone=us-west-1b&region=us-west-1 integrationDriver=linux osDriver=linux server=wheat-heron-vi service=ebs storageDriver=libstorage time=1573381405219 txCR=1573381405 txID=0b934f6e-10af-48c0-6236-e12964e11416
DEBU[0000] xli localdevices success                      host=unix:///var/run/rexray/157173546.sock instanceID=ebs=i-03656f91f34f2d9f4,availabilityZone=us-west-1b&region=us-west-1 integrationDriver=linux osDriver=linux server=wheat-heron-vi service=ebs storageDriver=libstorage time=1573381405219 txCR=1573381405 txID=0b934f6e-10af-48c0-6236-e12964e11416
DEBU[0000] built output template                         host=unix:///var/run/rexray/157173546.sock template={{define "printVolumeID"}}{{.ID}}{{end}}{{define "printAuthTokenFields"}}{{.Subject}}  {{.IssuedAt}}   {{.NotBefore}}  {{.Expires}}{{end}}{{define "printEncVolumeWithPathFields"}}{{.ID}} {{.Name}}   {{.Volume.AttachmentState | printAttState}} {{.Size}}   {{.Encrypted}}  {{.Path}}{{end}}{{define "printInstanceFields"}}{{.InstanceID.ID}}  {{.Name}}   {{.ProviderName}}   {{.Region}}{{end}}{{define "printServiceFields"}}{{.Name}}  {{.Driver.Name}}{{end}}{{define "printScriptFields"}}{{.Path}}  {{.Installed}}  {{.Modified}}{{end}}{{define "printObject"}}{{printf "%v" .}}{{end}}{{define "printStringSlice"}}{{.}}{{end}}{{define "printJSON"}}{{. | json}}{{end}}{{define "printPrettyJSON"}}{{. | jsonp}}{{end}}{{define "printEncVolumeFields"}}{{.ID}}  {{.Name}}   {{.AttachmentState | printAttState}}    {{.Size}}   {{.Encrypted}}{{end}}{{define "printSnapshotFields"}}{{.ID}}    {{.Name}}   {{.Status}} {{.VolumeID}}{{end}}{{define "printMountFields"}}{{.ID}}    {{.Source}} {{.MountPoint}}{{end}}{{define "printVolumeFields"}}{{.ID}} {{.Name}}   {{.AttachmentState | printAttState}}    {{.Size}}{{end}}{{define "printVolumeWithPathFields"}}{{.ID}}   {{.Name}}   {{.Volume.AttachmentState | printAttState}} {{.Size}}   {{.Path}}{{end}}{{range .D }}{{template "printObject" .}}
{{end}} time=1573381405385
FATA[0000] error creating volume                         error.status=500 error.msg=error creating volume with EC2 API call volume=test-vol1
ubuntu@ip-172-31-1-66:/etc/rexray$ sudo rexray volume create test-vol1 --size=2 -c /etc/rexray/config.yml

我觉得我在这里遗漏了一些非常基本的东西。帮助！

4

1 回答 1

0

所以我设法找出问题所在。

  1. 主要问题是使用了错误的凭据EBS_ACCESS，导致无法识别节点的 instanceID 错误。

  2. 该插件启动它自己的 rexray 实例，可在此处获得

~$ ps aux | grep rexray
root     14931  0.0  1.2  38624 24800 ?        Ssl  13:26   0:02 rexray start -f --nopid
ubuntu   15373  0.0  0.0  14856  1108 pts/0    S+   17:33   0:00 grep rexray

所以我理解的缺陷是没有必要像我对/etc/rexray/config.yml

这对于 Virtualbox 是必不可少的，但对于 docker 管理的插件则不然，它会自动启动一个客户端，该客户端根据用于初始化插件的 EBS_REGION 属性访问卷。

使用 VirtualBox，需要明确告知 rexray 插件将请求发送到何处，因此在 yaml 中描述了此处描述的已启动 Web 服务端点

virtualbox:
  endpoint: http://10.0.0.6:18083
  volumePath: "/home/sidravic/VirtualBox VMs"
  controllerName: SATA
  tls: false

另一个需要记住的有用步骤是在初始化插件时传递参数，如下所示

docker plugin install rexray/ebs:latest REXRAY_PREEMPT=true EBS_REGION=us-west-1 EBS_ACCESSKEY=valid_key EBS_SECRETKEY=valid_Secret --grant-all-permissions EBS_ENDPOINT=ec2.us-west-1.amazonaws.com EBS_USELARGEDEVICERANGE=true

虽然 EBS_ENDPOINT 的规范不是必需的，因为它是从区域推断的。

于 2019-11-10T17:37:29.207 回答