Oozie hive action job stuck in PREP state.

Problem:

Oozie triggered workflow which has a hive action stuck in PREP state:

[gpadmin@pccadmin ~]$ oozie job -oozie http://pccadmin.phd.local:11000/oozie -info 0000000-140619225354896-oozie-oozi-W
Job ID : 0000000-140619225354896-oozie-oozi-W
------------------------------------------------------------------------------------------------------------------------------------
Workflow Name : hive-wf
App Path : hdfs://test/user/gpadmin/examples/apps/hive
Status : RUNNING
Run : 0
User : gpadmin
Group : -
Created : 2014-06-20 05:54 GMT
Started : 2014-06-20 05:54 GMT
Last Modified : 2014-06-20 05:54 GMT
Ended : -
CoordAction ID: -
 
Actions
------------------------------------------------------------------------------------------------------------------------------------
ID                                           Status      Ext ID     Ext Status      Err Code
------------------------------------------------------------------------------------------------------------------------------------
0000001-140619235104359-oozie-oozi-W@:start:   OK           -          OK             -
------------------------------------------------------------------------------------------------------------------------------------
0000001-140619235104359-oozie-oozi-W@hive-node PREP         -           -              -
------------------------------------------------------------------------------------------------------------------------------------

Cause: If oozie job is stuck in PREP state, one must start looking into the logs to identify the cause of such a behavior. You can start with oozie.log, and in the instance described in this article, the execution of the oozie hive action threw an exception NoClassDefFoundError for "HiveConf" class after which it was stuck.

0:24:52,497 ERROR ActionStartXCommand:536 - USER[nweissi] GROUP[-] TOKEN[] APP[hive-wf] JOB[0000000-140619202307001-oozie-oozi-W] ACTION[0000000-140619202307001-oozie-oozi-W@hive-node] Error,
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
 at org.apache.oozie.action.hadoop.HCatCredentialHelper.getHCatClient(HCatCredentialHelper.java:79)
 at org.apache.oozie.action.hadoop.HCatCredentialHelper.set(HCatCredentialHelper.java:52)
 at 
..
..
 at java.lang.Thread.run(Thread.java:744)
Exception in thread "pool-2-thread-9" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
 at org.apache.oozie.action.hadoop.HCatCredentialHelper.getHCatClient(HCatCredentialHelper.java:79)

Fix:

common.loader variable in catalog.properties sets the classpath for the Oozie launcher jobs. It is located under $CATALINA_BASE/conf directory, use /etc/default/oozie to identify the CATALINA_BASE path in PHD 1.x / 2.x or the files which set this configuration based on your install.

Ensure that catalina.properties file has specified the required libraries. In this case, it was HiveConf.class which is packaged in hive-common.jar and hive-exec.jar, but hive library directory was missing from common.loader variable, due to which the launcher job was failing to retrieve the hcat credentials used in workflow.xml on secured cluster.

Another way to verify is to open Oozie Web Console UI at http://<oozie-server-host>:11000/oozie/. Browser over to the "System Info" -> "Java System Props" tab and look for the parameter common.loaders. It must have the required jars.