java - Can I write custom java methods inside map-reduce?

Question

Can I write my own custom static method in JAVA inside map-reduce that I may want to call for every key-value pair that enters my mapper?

Answer 1

You can do such a thing and I do it all the time. The methods you call can be either static or not.

Basically:

If you call the outside code from code that you've written yourself thats included in your project, you need to bundle that up with your code when you export the project to a .jar file.

If you are referencing methods and such from a .jar file that you've included in the build path for your project, you need to place that .jar file on every node of your cluster where the job would run, and edit your HADOOP_CLASSPATH variable on your servers in the file $HADOOP_HOME/conf/hadoop-env.sh file and restart your cluster. Otherwise you will get a class not found error.

An example is:

export HBASE_CLASSPATH=/path/to/one/file.jar:/path/to/another/file.jar

Answer 2

Yes, you can do that. (I have to enter additional characters so I can submit this answer, so just ignore this sentence)