Thursday, June 26, 2014

Running Oozie workflow examples on Hartonworks HDP 2.1:



Hi,
This blog is regarding resolving the NoClassDefFoundError while running the Oozie example on Hortonworks HDP 2.1 VM:
1. After downloading the HDP 2.1, you can download the oozie exampels from oozie sit, https://oozie.apache.org/docs/3.1.3-incubating/DG_Examples.html
as instruction given in that site if you run the oozie examples you may get below errors:

  1. java.lang.NoClassDefFoundError: org/apache/oozie/action/hadoop/JavaMainException
  2. java.lang.NoClassDefFoundError: org/apache/oozie/action/hadoop/JavaMainException
  3. java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.JavaMain not found

     
    Solution:
    The issue here is, oozie client is not able to find all required oozie libraries to run the oozie workflow, so you need to place the oozie libraries in hdfs file system.

    You  can see folder '/usr/lib/oozie/ oozie-sharelib.tar.gz/' untar it, you can see folder 'share' now you need to move this to the hdfs folder 'share'. One thing you need to make sure that if you are running the oozie workflow job with the user 'root' then you have to move the 'share' fore to hdfs using the same user (root) then only it works.
    here is the command to move share folder to hdfs - 'hadoop fs -put share share'.

    Also make sure that in '/usr/lib/oozie/conf/oozie-default.xml' the property 'use.system.libpath.for.mapreduce.and.pig.jobs' value is 'true', also add the another proper on the same file 'oozie.use.system.libpath' with value 'true'.

    The reason why we need to move the 'share' folder with the user with which we start the oozie service and workflow is, the oozie client looks for the share libraries using the folder - '/user/${user.name}/share/lib'.

    After all these setting, restart the oozie siervce (with root user)
       [root@sandbox bin]# ls
    oozie  ooziedb.sh  oozied.sh  oozie-env.sh  oozie-run.sh  oozie-setup.sh  oozie-start.sh  oozie-stop.sh  oozie-sys.sh
    [root@sandbox bin]# ./oozie-stop.sh
     [root@sandbox bin]# ./oozie-start.sh

    Now go toe examples folder, run the examples using the commad:
     /usr/lib/oozie/bin/oozie job -oozie http://localhost:11000/oozie -run -config apps/map-reduce/job.properties

    You can track the oozie status using UI:
    http://127.0.0.1:11000/oozie/


No comments:

Post a Comment