0

我有一个 Docker 容器在 AWS 上“成功”运行 R 脚本（即脚本已完成），但返回以下错误：

Error: ignoring SIGPIPE signal  
Execution halted

这是 R 脚本（在我缩小了问题发生的范围后策划）：

library(arrow)
library(naniar)
library(tidyr)
library(dplyr)

df <- arrow::read_parquet("s3://mybucket/myfile.parquet")

start_time <- Sys.time()

print("Calculate missing-shadow")
df %>%
  # slice(1:1000) %>%
  # Use Naniar to bind a shadow matrix (ie. a T/F matrix for if a varible is NA or not)
  bind_shadow() %>%
  # Keep only ID and the variables that are shadows
  select(bet_cus_id,ends_with("_NA")) %>%
  # Gather so we can combine into one giant string
  gather(var,isna,-bet_cus_id) %>%
  # Change to true/false, becuase 1/0 is easier to read than !NA and NA
  mutate(isna = isna == "NA") %>%
  unite("Merged",var:isna,remove=TRUE) %>%
  # Sort so every  string has the same sorting
  group_by(bet_cus_id) %>%
  arrange(Merged, .by_group = TRUE) %>%
  {.} -> df.player

print("Finished missing-shadow")
print(df.player %>% slice(1:3))
print(length(unique(df.player$bet_cus_id)))
print(length(df.player$bet_cus_id))

Sys.time()-start_time

和 Dockerfile

FROM rstudio/r-base:4.0.4-focal

RUN apt-get update

RUN apt-get install -y --no-install-recommends git cmake

# arrow
RUN apt-get install -y libcurl4-openssl-dev
RUN apt-get install -y libssl-dev

ENV ARROW_S3=ON

RUN apt-get update

# h2o
RUN apt-get install -y default-jdk
RUN R CMD javareconf

ENV RENV_VERSION 0.13.2
RUN R -e "install.packages('remotes', repos = c(CRAN = 'https://cloud.r-project.org'))"
RUN R -e "remotes::install_github('rstudio/renv@${RENV_VERSION}')"

RUN Rscript -e "install.packages('devtools', repos='https://packagemanager.rstudio.com/all/__linux__/focal/latest')"
RUN Rscript -e "devtools::install_version('h2o', version = '3.30.0.1', repos = 'https://packagemanager.rstudio.com/all/__linux__/focal/latest')"

WORKDIR /project
COPY renv.lock renv.lock
RUN R -e 'renv::restore(repos=c("https://packagemanager.rstudio.com/all/__linux__/focal/latest"))'

RUN rm -rf /tmp/* \
  && apt-get remove --purge -y $BUILDDEPS \
  && apt-get autoremove -y \

EXPOSE 3840
  
COPY test.R test.R

CMD ["Rscript","test.R"]

这是日志文件的最后一行（来自journalctl -u docker.service）

Jul 06 08:59:53 ip-XXX.eu-north-1.compute.internal 131e033f5fe2[4359]: Time difference of 3.678913 mins
Jul 06 08:59:53 ip-XXX.eu-north-1.compute.internal 131e033f5fe2[4359]: Error: ignoring SIGPIPE signal
Jul 06 08:59:53 ip-XXX.eu-north-1.compute.internal 131e033f5fe2[4359]: Execution halted
Jul 06 08:59:53 ip-XXX.eu-north-1.compute.internal dockerd[4359]: time="2021-07-06T08:59:53.867394818Z" level=info msg="ignoring event" container=131e033f5fe2cf5f0854ce97115544889a59f97a959509bdc7fef24d8ba08cd4 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
    enter code here

我无法从上面的日志中发现任何有用的东西。然而，这似乎与内存有关，因为如果我将数据帧的大小减少到 1000 行，则不会发生SIGPIPE/执行停止（输入上的数据帧相当大：~45M 行/3 列） . 可能是由于数据帧的大尺寸因此导致 SIGPIPE 问题，在脚本结束之前日志记录无法跟上。只是一个猜测......而且我不知道如何解决这个问题。

06/07 更新：我应该提到上面的 R 脚本本身运行良好（即在 docker 容器之外）。

任何帮助将非常感激。

4

0 回答 0