hadoop - ImportError: No module named hdfs.ext.kerberos -


i able find full yarn log job. seems passed arguments oozie fine. when run script itself, kerberos works fine. when run through oozie i've got import error. did kinit , added kerberos oozie run script.

part of workflow

<action name="forceloadfromlocal2hdfs"> <shell xmlns="uri:oozie:shell-action:0.1">   <job-tracker>${jobtracker}</job-tracker>   <name-node>${namenode}</name-node>   <configuration>     <property>       <name>mapred.job.queue.name</name>       <value>${queuename}</value>     </property>   </configuration>   <exec>driver-script.sh</exec>   <argument>s</argument>   <argument>00_load_local_2_hdfs.py</argument>   <argument>${localfilepath}v</argument>   <argument>${hdfspath}</argument>     <file>driver-script.sh#driver-script.sh</file>   <file>00_load_local_2_hdfs.py#00_load_local_2_hdfs.py</file> </shell> <ok to="createinttable"/> <error to="killaction"/> </action>

log output

invoking main class >>>  setting [tez.application.tags] tag: oozie-65aba8c86f895c9d1c4566bff4aff2f3 current working dir     /srv/data4/hadoop/yarn/local/usercache/me/appcache/application_1501083332770_0061/container_e69_1501083332770_0061_01_000002  ------------------------- 0:driver-script.sh: 1:s: 2:load_local_2_hdfs.py: 3:/melocal/teplota.csv: 4:/mehdp/dataframe/data/: list of passing environment  invoking shell command line >>  ... mess ...  stdoutput traceback (most recent call last): stdoutput   file "/srv/data4/hadoop/yarn/local/usercache/me/appcache/application_1501083332770_0061/container_e69_1501083332770_0061_01_000002/load_local_2_hdfs.py", line 2, in <module> stdoutput     hdfs.ext.kerberos import kerberosclient stdoutput importerror: no module named hdfs.ext.kerberos stdoutput     ============================================================================== stdoutput zacatek       1501229981 stdoutput konec 1501229982 stdoutput celkem        1 exit code of shell command 0 invocation of shell command completed <<< invocation of main class completed <<<

part of oozie run script

oozie job -config ${job_properties} -run -verbose \  -oozie ${oozie_url} \  -d oozie.wf.application.path=${hdfs_uri} \  -d oozie.use.system.libpath=true \  -auth kerberos

authorization part in workflow hive

<credentials> <credential name="hs2-creds" type="hive2">   <property>     <name>hive2.server.principal</name>       <value>hive/namenode@domain.had</value>   </property>   <property>    <name>hive2.jdbc.url</name>      <value>jdbc:hive2://namenode:10000/default</value>   </property> </credential>

thx lot!

edit: think need move hdfs.ext.kerberos libs /user/oozie/share/lib/spark , set property oozie.action.sharelib.for.spark path.


Comments

Post a Comment

Popular posts from this blog

php - Vagrant up error - Uncaught Reflection Exception: Class DOMDocument does not exist -

i've set bento/ubuntu-16.04 vagrant box using php 5.6, , getting following error quite few times when try , run shell script: "uncaught reflection exception: class domdocument not exist". i've tried adding following top of script (not together), each no success: "sudo apt-get install php5.6-xml", "sudo apt-get install php5-xml", "sudo apt-get install php-dom" i've tried resetting server "sudo service apache2 restart". does have suggestions?
Read more

vue.js - Create hooks for automated testing -

Image
qa guys complained can't automate frontend testing. because our html looks same outside. lazy hack did this <template> <div :role="$options.name"> ... </div> </template> <script> export default { name: 'vmcomponentname' } </script> which takes name script , applies html. in browser get: <div role="vmcomponentname"> ... </div> line :role="$options.name" goes in every component. is there dry er solutions? please share. additional details i'll explain in details looking for. imagine have vmusercreate form creates user. test case can create user . to automatically test without role, have use following selector: .wrapper > .wrapper > .wrapper > .submit-button the test extremely brittle (will break lot). if use roles can use [role="vmusercreateform"] .submit-button selector. many folds less brittle. so basically, i'm ...
Read more

Add new key value to json node in java -

i have json file : { "id":1, "name":"abc", "addressdetails": { "city":"newyork" } } i wanted add 1 more key-value ("pincode" : "414141") node 'addressdetails'. i tried using : objectmapper mapper = new objectmapper(); jsonnode root = mapper.readtree(new file("d://test.json")); objectnode node = mapper.createobjectnode(); node.with("addressdetails").put("pincode", "414141"); but it's not getting added, there way can ? you can try you should modify file path objectmapper mapper = new objectmapper(); objectnode nodes = mapper.readvalue(new file("d:\\test.txt"), objectnode.class); nodes.with("addressdetails").put("pincode", "414141"); mapper.writer().writevalue(new file("d:\\test.txt"), nodes);
Read more