mysql - 如何编写用于将数据从 MySQL 加载到 Redshift 的 Luigi 工作流程？

Question

Luigi 是否支持我执行整个流程？

流程是这样的：将 MySQL 数据转储到 S3，然后使用 Copy 命令将数据移动到 Redshift。

我可以使用 Luigi 执行上述工作流程吗？

Answer 1

Basically you can do anything that you can do with a python script, in Luigi, since you can write that python code in the run() method of Luigi tasks, which means there is no real limits actually.

See the Luigi docs for tasks, and in particular the run() method, for more information: http://luigi.readthedocs.org/en/stable/tasks.html#task-run

So, the remaining question is then how to access RedShift and MySQL from python itself, but that is a separate question, for which I think googling around for a good guide or tutorial is the best option.