hadoop - Exception in thread "main" org.apache.hadoop.mapred.InvalidJobConfException: Output directory not set

Question

Hey All can u please help me in clearing following error ? I am getting this when I am running Mapreduce job fopr inserting data into hbase tables from hdfs files. using HFileOutputFormat.class , earlier i ran the same using MultiTableOutputFormat.class it was working fine , but it was taking so much time while inserting data into hbase tables .

So can u please help me ... :) *

job.setMapOutputKeyClass(ImmutableBytesWritable.class); job.setMapOutputValueClass(Put.class); job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(HFileOutputFormat.class); job.setMapperClass(Map.class); HTable hTable = new HTable(conf,"ARK_3000"); HFileOutputFormat.configureIncrementalLoad(job, hTable);

ERROR security.UserGroupInformation: PriviledgedActionException as:reddym (auth:SIMPLE) cause:org.apache.hadoop.mapred.InvalidJobConfException: Output directory not set. Exception in thread "main" org.apache.hadoop.mapred.InvalidJobConfException: Output directory not set.

Thanks Madhusudhana Reddy

Answer 1

Make sure the output directory is not write protected and try again Don't use directories like the root of a drive. hope this helps Regards

EDIT : try adding this where it asks you for the output directory

HFileOutputFormat.setOutputPath(conf, new Path(<YOUR_LOCATION>));

or FileOutputFormat.setOutputPath(conf, new Path(<YOUR_LOCATION>));

And you need to mention the output directory so it knows where to write the data to.

Hope this helps you further with your problem, let me know if you need any more help. as i am always happy to help.

EDIT 2 : You need to mention an output directory because the tables you are writing to is stored in the memory and when you end the program all of the data in the memory is lost. For example if the program crashes all of the data is lost unless it is written to disk. So the data saved to the output directory is a backup in-case anything goes wrong.

I hope this answers your question clearly for you, if you need any more assistance leave another comment Regards Tom