Monday, May 25, 2015

Null pointer exception while running Hadoop commands

Symptom:
When executing hadoop read / write commands, a Null Pointer Exception is seen. 
[hive@aoabdlp00004 ~]$ hdfs dfs -get /app-logs/hive/logs/application_1430501760753_0003
get: java.lang.NullPointerException

Troubleshooting:
Namenode logs can give insights into the probable cause of the error. In this specific instance, while running hadoop commands to pull data from HDFS, rack script was not resolving the client IP to any of the rack. The snippet below lists the reason "/etc/hadoop/conf/CentralPerkRack.pl returned 0 values when 1 were expected" when the rack resolve script was fed with an input IP 192.168.255.4
2015-05-01 14:03:42,299 ERROR net.ScriptBasedMapping (ScriptBasedMapping.java:resolve(198)) - Script /etc/hadoop/conf/CentralPerkRack.pl returned 0 values when 1 were expected.
2015-05-01 14:03:42,300 WARN  ipc.Server (Server.java:run(2058)) - IPC Server handler 31 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 192.168.255.4:52208 Call#
2 Retry#0
java.lang.NullPointerException
        at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.sortLocatedBlocks(DatanodeManager.java:359)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1799)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:543)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:364)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)

Fix:
Rack resolving script must resolve the incoming client request to a rack. Either you can include the client to a specific rack or you can add a default rack to resolve any client IP to it if it does not matches any of the other configured rack IP ranges.

No comments:

Post a Comment