1

我目前正在将网页的视频转换部分(有点像 youtube,用户上传视频,我们将它们转换为 flv/mp4)到不同的服务器。我已经让系统在同一台机器上与gearman一起运行。因此,当用户将视频文件上传到服务器 A 时,同一服务器 A 上的齿轮工会选择其中的内容。

现在我将worker移动到服务器B。所以服务器B上的worker需要访问服务器A上上传的文件。目前我使用SCP将文件从A复制到B然后处理它。这种方法有效,但我觉得应该有一种更干净的方法,但我还没有找到任何关于将文件(或大文件)发送给 gearman 工作人员的信息。你会如何处理这个问题?

最好客户端将视频文件作为命令的一部分发送以启动后台作业,因此我不必担心文件实际上来自工作人员内部的哪个位置。这样我就可以毫不费力地添加更多的转换服务器。

我的网页和工作人员都在使用 PHP(带有 Gearman扩展)。

4

1 回答 1

5

As was suggested in the comments, having a shared FS is the (usual) way to implement this, and simply pass the path around in the job request from gearman. Gearman is not well-suited for passing around large blobs of data, as it has to keep all of the information for a job in memory. It was never designed for handling the transfer and distribution of large files. Since MogileFS was also initially developed at Danga, there simply was no need to also incorporate file transfer and handling in Gearman (and that's a good thing, there's quite a few technologies that solve that problem better than Gearman would ever do).

We're using NFS for handling distributed workers when videos arrive, and the encoder puts the encoded video back onto the NFS share that's available to the public when it's done. Haven't had a serious issue yet, NFS is stable and it's problems are well known and already solved for the kind of loads you'll see.

于 2012-07-26T09:42:58.073 回答