apache-pig - Load multiple files in pig

Question

I am new to pig.

In pig, I want to load multiple files with timestamps at their names.

If I load files between 2012/02/12 to 2012/02/19, the following works

$START = "12"
$END = "19"
raw_data = load '/table/status/2012/02/{$START,$END}' using Loader()

Suppose the start date is 2011/12/29 and end date is 2012/01/04, how do I change the line of code?

Thanks for any help!

Answer 1

The range you need can be loaded by using the following globs:

raw_data = load '/table/status/{2011/12/{29,3{0,1}},2012/01/0[1-4]}' using Loader()

Note, that Pig uses Hadoop's globbing so what you can use depends on the version of the underlying HDFS. This works under 0.20.

If a more complex range is needed then this way of directory enumeration might be a bit inflexible. Therefore consider using a shell script to gather the directories you need as suggested here