2

我有一个应用程序可以将数据复制到外部磁盘上。复制请求存储在 MySQL 数据库中,将从多个复制机器读取,这些复制机器运行 bash 脚本来获取请求。一旦接收到请求,它就会在数据库中设置为“进行中”。但是,我试图避免多台机器一起读取请求并开始复制相同数据的情况。

我打算使用表锁定来执行此操作,但我很挣扎,因为会话过期时表锁也会过期,所以如果我这样做:

mysql="mysql -h dbhost -u user -pPassword diskcopydb"
echo "LOCK tables diskcopy WRITE;" | $mysql
echo "SELECT SQL_NO_CACHE * FROM diskcopy WHERE status=\"request\"" | $mysql

锁在获取请求时实际上已经过期,因此竞争条件仍然存在。这里有一个问题:

命令行中的 MySQL - 我实际上可以使用 LOCK 吗?

SQL 命令被打包到一个块中,它们一起通过管道传输到 MySQL,但我需要从 MySQL 中途获取输出以获取请求。有人有这个食谱吗?看起来它应该是一个相当常见的用例......

4

3 回答 3

1

这个问题实际上在这里得到了回答:

从 bash 批量插入 MySQL 相关表

换句话说,使用 Bash 与 MySQL 进行双向通信是很棘手的,使用 Perl 或 Python 等“适当的”编程语言更容易,您可以在其中打开和保持连接。

于 2012-09-18T08:37:35.403 回答
1

不可能是可能的!

>但我需要从 MySQL 中途获取输出以获取请求。有人有这个食谱吗?

这是我的解决方案演示:

#!/bin/bash
################################################################################
#                  Bash synchronous client-server                              #
#                              or                                              #
#     Bash parallel processes with synchronized rendezvous points              #
#                              or                                              #
#          Bash interprocess communication with named pipes                    #
################################################################################
#Declare your session 'globals'
args=    #command line arguments are stored here
DIR=     #script's current path is stored here

#Early store your bash args before you lose them
for arg in "$@"; do
   args[i]=$arg
   (( i+=1 ))
done
################################################################################
## Initialize constants                                                       ##
#------------------------------------------------------------------------------#
# Explain more...                                                              #
################################################################################
function init_vars() {
   #absolute path of this script
   DIR=$( cd "$( dirname "$0" )" && pwd )
   #global MYSQL_PASS is defined in ~/.profile
   #this is the file where we store MySQL root password
   if [[ ! $MYSQL_PASS ]]; then
      MYSQL_PASS='.mysqlpass'
   fi
   #do some other stuff here...
}

################################################################################
## Rotate over passed arguments                                               ##
#------------------------------------------------------------------------------#
#Parse array 'args' based on the logic of your command line argument syntax    #
#Say, '--db <database>' denotes the processing of one database, '--db all' of  #
#all databases etc.                                                            #
#Linux utilities have contradicting rules for such kind of argument processing #
#(full of crap!), i.e. see tar: tar -xvf or tar --xfv or tar xvf, mysql -p     #
#<pass> or mysql -p=<pass> or mysql -p'<pass>' (???)                           #
################################################################################
function parse_args() {
   #your logic goes here
}

################################################################################
## Opens a MySQL client session                                               ##
#------------------------------------------------------------------------------#
# For security reasons password is kept in a file given by MYSQL_PASS.         #
################################################################################
function mysql_client() {
   mysql --host=127.0.0.1 --port=3306 --default-character-set=utf8 -u root -p"$( cd $DIR; cat $MYSQL_PASS )"
}

################################################################################
## A job dispatcher                                                           ##
#------------------------------------------------------------------------------#
# This thread initiates the workers and also reads intermediate results from   #
# the spawned jobs. We don't want to parallelize the whole part of a job but   #
# rather to put some parts to run in parallel with the dispatcher and some     #
# parts in a sequential order determined by what we call rendezvous points.    #
# See the diagram: A, B, C, D, E and F are events or actions during the life of#
# a program.                                                                   #
#                                                                              #
#  dispatcher                                                                  #
#     |         start                                                          #
#     A     ---------------->    worker                                        #
#     |                            |                                           #
#     B (read blocks)              C                                           #
#     |                   signals  |                                           #
#     B (unblocks)<--------------- D (write blocks)                            #
#     |                            |                                           #
#     E (1st set of results)       F                                           #
#     |                            |                                           #
# We can only guarantee that E comes after D or, D->E in time but E->F or F->E #
# and B->C or C->B (we don't care)                                             #
################################################################################
function dispatcher() {
   #go to the directory of this script
   cd $DIR
   #make a temporary directory secured in 'time & space'
   TMPDIR=$( mktemp -d XXXXXXXXXX )
   #catch exit of this script and delete temporary folder
   trap 'cd $DIR && rm -rf "$TMPDIR"' EXIT
   #move into the temporary folder
   cd ${TMPDIR}
   #create named pipes-global to all functions of this script
   TMPSQL=mysql-$RANDOM.$RANDOM.$RANDOM.$$
   TMPSCRIPT=myscript-$RANDOM.$RANDOM.$RANDOM.$$
   mkfifo $TMPSQL
   mkfifo $TMPSCRIPT
   #exec 3<> ${DIR}/${TMPDIR}/$TMPSQL || (echo 'error'; exit 1)
   #exec 4<> ${DIR}/${TMPDIR}/$TMPSCRIPT || ( echo 'error'; exit 1)
   echo '===1.PARENT=== Starting background jobs...'
   echo 'We can guarantee the succession only of 2->3,4 and 5->6,7 but NOT 3->4 or 6->7!'
   worker &
   ##################################
   cat $TMPSQL #make a blocking read!
   ##################################
   echo '===4.PARENT=== ...1st query has been read'
   ##################################
   cat $TMPSQL #make a blocking read!
   ##################################
   echo '===7.PARENT=== ...2nd query has been read'
   #don't you dare to exit!
   ##################################
   cat $TMPSQL #make a blocking read!
   ##################################
}

################################################################################
## A job spawned by the dispatcher                                            ##
#------------------------------------------------------------------------------#
################################################################################
function worker() {
   echo "===2.CHILD=== Executing 1st query......writing to $TMPSQL"
   #the dash symbol '-' makes tabs at the beginning of line to be
   #ignored inside the here-doc but improves formation!
   mysql_client <<- QUERY
\! tee $TMPSQL
use test;
select * from customers;
\! echo '===3.CHILD=== End 1st query'
#####################################
#\. $TMPSCRIPT
#####################################
QUERY

   echo "===5.CHILD=== Executing 2nd query......writing to $TMPSQL"
   mysql_client <<- QUERY
\! tee $TMPSQL
use test;
select * from cars;
\! echo '===6.CHILD=== End 2nd query'
#####################################
#\. $TMPSCRIPT
#####################################
QUERY

   echo '===8.CHILD=== End of background jobs'
   echo > $TMPSQL
}

   #parse command line arguments
   parse_args

   #initialize variables
   init_vars

   #call dispatcher
   dispatcher

假设您有一个“测试”数据库,那么您可以看到以下输出:

===1.PARENT=== Starting background jobs...
We can guarantee the succession only of 2->3,4 and 5->6,7 but NOT 3->4 or 6->7!
===2.CHILD=== Executing 1st query......writing to mysql-30627.1495.5394.7533
===4.PARENT=== ...1st query has been read
id_customer firstname   secondname
1   John    Pincolo
2   Mark    Denonto
3   Ann Curtis
4   Jeny    Wirth
===3.CHILD=== End 1st query
===5.CHILD=== Executing 2nd query......writing to mysql-30627.1495.5394.7533
===7.PARENT=== ...2nd query has been read
id_car  type    plate   date_rent   date_returned
1   fiat    BG-457  2012-07-18 00:00:00 2012-07-20 00:00:00
2   renault AS-1234 2012-07-20 00:00:00 2012-07-25 00:00:00
3   fiat    JYB-2856    2012-06-23 00:00:00 2012-06-24 00:00:00
===6.CHILD=== End 2nd query
===8.CHILD=== End of background jobs

如您所见,我的第一个数据库结果集位于 2 到 5 之间!您想要查询和“主”程序之间的中间结果,并且您拥有它们!

您可能想知道:“嘿,我想获得一个数据库锁,在我的'主'程序执行其他操作时停止会话以冻结!” 嗯,也有答案。函数 'worker' 有 2 个注释命令:

#####################################
#\. $TMPSCRIPT
#####################################

通过启用他们你的执行挂起!而已!现在,您可以开始在调度程序中执行操作,当您完成发送一个“回声”时,如下所示:

echo > $TMPSCRIPT

通过将这行代码添加到我们的工作程序中,我们将调度程序和工作程序都转换为阻塞线程:现在两者都互相监听。示意图:

  dispatcher
     |         start
     A     ---------------->    worker
     |                            |
     B (read blocks)              C
     |                   signals  |
     B (unblocks)<--------------- D (write blocks)
     |                            |
     E (1st set of results)       F
     |                            |
     |                            G (read blocks)
     |  signals                   |
     H ---------------->          G (unblocks)
     | (write blocks)             |

如果你需要更多的兴奋,发送什么

echo 'set @x=@x+1;' > $TMPSCRIPT

其中“@x”是用户定义的变量!我们这里有什么?好吧,我们只是将实时代码“注入”到我们的 sql 作业中!非常棒!!!

现在,可以获取数据库的 LOCk,进行逻辑备份,停止并继续 mysql 客户端,并在数据库 LOCK 仍处于活动状态时为所欲为!

它可以更好:创建二进制备份怎么样?(每年只需为您节省 2,000-10,000 美元,请参阅:https ://shop.oracle.com )

总比更好:一堆说 10 个数据库可以被锁定并开始备份,第一个完成发送确认到调度程序,调度程序在列表中添加另一个数据库,直到它变空!只记得每个工人都需要两把锁来进行双向锁定。

我们甚至可以同时启动 3 个调度程序,因为它们在空间上是独一无二的,因此不可能面对它们的命名管道的冲突。

这是我的 2,000 美元建议

甜的!

(百人队)

于 2012-11-27T23:38:40.417 回答
0

在同一连接中运行您的 SQL:

mysql="mysql -h dbhost -u user -pPassword diskcopydb"
echo "LOCK tables diskcopy WRITE;SELECT SQL_NO_CACHE * FROM diskcopy WHERE status=\"request\"" | $mysql

否则,锁定是针对在您第二次运行 mysql 时已经失效的会话。

于 2012-09-17T14:29:36.173 回答