2

我有一个 bash 脚本，它想要并行执行一些工作，我通过将每个作业放在一个在后台运行的子外壳中来做到这一点。虽然同时运行的作业数量应该在一定范围内，但我首先将一些行放入 FIFO 中，然后在分叉子 shell 之前，父脚本需要从该 FIFO 中读取一行。只有在它得到一行之后，它才能分叉子shell。到目前为止，一切正常。但是当我试图从子shell中的FIFO读取一行时，似乎只有一个子shell可以得到一行，即使FIFO中显然有更多的行。所以我想知道为什么即使 FIFO 中有更多行，其他子外壳也无法读取一行。
我的测试代码如下所示： 


#!/bin/sh

fifo_path="/tmp/fy_u_test2.fifo"
mkfifo $fifo_path
#open fifo for r/w at fd 6
exec 6<> $fifo_path

process_num=5
#put $process_num lines in the FIFO

for ((i=0; i<${process_num}; i++)); do
    echo "$i"
done >&6

delay_some(){
    local index="$1"
    echo "This is what u can see. $index \n"
    sleep 20;
}

#In each iteration, try to read 2 lines from FIFO, one from this shell,
#the other from the subshell
for i in 1 2
do
    date >>/tmp/fy_date
#If a line can be read from FIFO, run a subshell in bk, otherwise, block.
    read -u6
    echo " $$ Read --- $REPLY  --- from 6 \n" >> /tmp/fy_date
    {
        delay_some $i
#Try to read a line from FIFO, __ only one subshell succeeds the following line. __
        read -u6
        echo " $$ This is in child # $i, read --- $REPLY --- from 6 \n" >> /tmp/fy_date
    } &
done


输出文件 /tmp/fy_date 的内容为： 


Mon Apr 26 16:02:18 CST 2010
 32561 Read --- 0  --- from 6 \n
Mon Apr 26 16:02:18 CST 2010
 32561 Read --- 1  --- from 6 \n
 32561 This is in child # 1, read --- 2 --- from 6 \n

在那里，我期待这样的一行：


 32561 This is in child # 2, read --- 3 --- from 6 \n

但它永远不会出现，并且子#2 进程在那里被阻塞，直到我发出：
echo something > /tmp/fy_u_test2.fifo

4

7 回答 7

1

Keep in mind that a FIFO on POSIX systems is essentially a named pipe. In order to move data around on a pipe, one side needs a reader and the other side needs a writer, and when one is closed the other loses usefulness.

In other words, you cannot cat on a fifo after some other reader has exited, because the contents of the FIFO will be gone.

You may want to see about using a normal file (and use file locking to ensure that you are synchronizing your access to that normal file), or use a directory with multiple files in it, or even use shared memory or something similar to that (perhaps not in a shell script, though). It all depends on what your end-goal is, really, what the best way to go about it would be.

于 2010-04-26T15:54:38.623 回答
1

您对fifo的写入是否可能有一些缓冲？如果您有可用的 unbuffer，您可以尝试在 echo 前面加上它吗？我真的不知道它怎么会在这里发生，但症状很合适，所以值得一试。

于 2010-04-26T15:44:59.140 回答
1

似乎与'read -u6' shell 调用有关。如果我关闭了 shell 的 STDIN，当发出“read -u6”时，它会尝试从 fd 6 中读取 128 个字节。但如果 STDIN 保持不变，当发出“read -u6”时，它会读取一个字节一个直到遇到'\n'。我从“strace”中发现了这个奇怪的动作，在第一种情况下，“read -u6”调用导致了以下系统调用：

read(6, "0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n"..., 128) = 50

在后一种情况下，“read -u6”调用会导致以下系统调用：

30371 16:27:15 read(6, "0", 1)          = 1
30371 16:27:15 read(6, "\n", 1)         = 1

测试代码如下：


#!/bin/bash

fifo_path="/tmp/fy_u_test2.fifo"
mkfifo $fifo_path
#open fifo for r/w at fd 6
exec 6<> $fifo_path

#comment or decomment the following line makes difference
exec 0>&-

process_num=20
#put $process_num lines in the FIFO
for ((i=0;i<${process_num};i++));do
    echo "$i"
done >&6

delay_some(){
    local index="$1"
    echo "This is what u can see. $index \n"
    sleep 10;
}

#In each iteration, try to read 2 lines from FIFO, one from this shell,
#the other from the subshell
for i in 1 2 3
do
    date >>/tmp/fy_date
#If a line can be read from FIFO, run a subshell in bk, otherwise, block.
    read -u6
    echo " $$ Read --- $REPLY  --- from 6 \n" >> /tmp/fy_date
    {
        delay_some $i
#Try to read a line from FIFO
#   read -u6
        echo " $$ This is in child # $i, read --- $REPLY --- from 6 \n" >> /tmp/fy_date
        echo " $$ Again this is in child # $i, read --- $REPLY --- from 6 \n" >> /tmp/fy_date
        echo "$i xx" >&6
#       echo xx >&6
    } &
done

#sleep 13
#wait
#read -u6
echo "$$ After fork, in parent, read --- $REPLY --- from 6 \n" >> /tmp/fy_date
于 2010-04-28T08:47:04.577 回答
0

当父 shell 退出时，我发现 FIFO 中未读取的数据在父 shell 退出时丢失了。
如果我有以下代码：


#!/bin/sh

fifo_path="/tmp/fy_u_test2.fifo"
mkfifo $fifo_path
#open fifo for r/w at fd 6
exec 6<> $fifo_path

process_num=9
#put $process_num lines in the FIFO

for ((i=0;i<${process_num};i++));do
echo "$i"
done >&6

for i in 1 2 3;
do
 read -u6
done

这段代码结束后，命令 'cat /tmp/fy_u_test2.fifo' 什么也没有。
但是，如果我有以下代码。


#!/bin/sh

fifo_path="/tmp/fy_u_test2.fifo"
mkfifo $fifo_path
#open fifo for r/w at fd 6
exec 6<> $fifo_path

process_num=9
#put $process_num lines in the FIFO

for ((i=0;i<${process_num};i++));do
echo "$i"
done >&6

for i in 1 2 3;
do
 read -u6
done
#__ notice this line __
sleep 60

发出此代码运行后，在其休眠 60 秒期间，命令“cat /tmp/fy_u_test2.fifo”给出以下输出：

$猫/tmp/fy_u_test2.fifo
3
4
5
6
7
8
于 2010-04-26T12:23:33.543 回答
0

当我运行它时，我得到了日志文件中的所有四行。如果您将 shebang 更改为 会发生什么#!/bin/bash？

于 2010-04-26T10:49:36.497 回答
0

这可能是一个并发问题，两个子 shell 都试图同时从同一个 fifo 中读取。它总是发生吗？

您可以尝试添加一个flock -x 6语句或更改两个子shell 的延迟，看看会发生什么。

顺便说一句，我可以确认使用 bash 3.2 和内核 2.6.28 您的代码可以正常工作。

于 2010-04-26T11:38:30.750 回答
0

由于此处其他答案中解释的原因，除非您可以同时从管道读取和写入，否则您不需要管道。

因此，建议使用另一种 IPC 方法或重组您对 fifos 的使用，以便异步进程填满管道，而主进程创建工作进程（或相反）。

这是一种使用简单文件作为一种队列来获取所需内容的方法：

#!/usr/bin/env bash

stack=/tmp/stack
> "$stack"

# Create an initial 5 spots on the stack
for i in {1..5}; do
    echo >> "$stack"
done

for i in {1..10}; do
    # Wait for a spot on the stack.
    until read; do sleep 1; done

    {
        echo "Starting process #$i"
        sleep $((5 + $i)) # Do something productive
        echo "Ending process #$i"

        # We're done, free our spot on the stack.
        echo >> "$stack"
    } &
done < "$stack"

旁注：此方法不适合无限工作，因为它为调用的每个进程向堆栈文件添加一个字节，这意味着堆栈文件增长缓慢。

于 2010-08-05T11:02:59.707 回答