hadoop - How can I add external python libraries into HDFS? -


is there way, how add external libraries this one hdfs? seems pyspark needs external libs have them in shared folder on hdfs. byt since using shellscript, runs pyspark script external libraries, fails importing them.

see post here importerror.

you can add external lib --py-files option. can provide either .py file or .zip.

for exemple, using spark submit : 

spark-submit --master yarn --py-files ./hdfs.zip myjob.py

check corresponding documentation : submitting applications


Comments

Post a Comment

Popular posts from this blog

service - Android MediaPlayer calls onCompletion before it already finished -

guys, trying implement android foreground service play music url. mediaplayer starts , play song after while calls oncompletion() when song not finished. set onerrorlistener it's not called... what have done: checked connection url > ok checked song duration > ok implemented partial wake lock > ? app doesn't crash, mediaplayer call oncompletion , don't know do. here have service , activity code. in advance: service: public class musicservice extends service implements mediaplayer.onpreparedlistener, mediaplayer.onerrorlistener, mediaplayer.oncompletionlistener{ //filter broadcasts public final static string action_filter = "action_hhgroups"; //media actions: public final static string action_play_one = "action_play_one"; //foreground: public static boolean is_foreground = false; public final static int foreground_service = 101; public final static string action_start_foreground = "action_start_foreground"; pu...
Read more

javascript - Training Neural Network to play flappy bird with genetic algorithm - Why can't it learn? -

Image
i have been learning neural networks , genetic algorithms, , test learning, have tried make ai learns play flappy bird: i have left running @ least 10 hours (overnight , longer), fittest member still fails show significant advancements in intelligence when began simulation apart avoiding floor , ceilings. inputs rays (as can see above) act sight lines, , network fed in lengths, , birds vertical velocity. seems best bird ignoring sight lines except horizontal one, , when short, jumping. output number between 0 , 1, if output larger 0.5, bird jumps. there 4 hidden layers, 15 neurons each, input layer feeding forward first hidden layers, 1st hidden layer feeding forward 2nd 1 ... , final hidden layer feeding forward output, dna of bird array of real numbers representing weights of neural networks, have made project using same style of neural network, , genetic algorithm, in ants had travel food, , worked perfectly. here code: https://github.com/karan0110/flappy-bird-ai please ...
Read more

javascript - Create a stacked percentage column -

i have collection of data : var data = [ {"p301a":"10","p301b":"7","p301c":"7","p301d":"3","p301e":"8","p301f":"8","p301g":"8","p301h":"8","p301i":"8","p301j":"8","p301k":"8","p301l":"8","p301m":"8","p301n":"8","p301o":"8","age":"31-40 years","profesion":"2","position":"2"}, {"p301a":"5","p301b":"4","p301c":"4","p301d":"4","p301e":"4","p301f":"4","p301g":"4","p301h":"4","p301i":"4","p301j":"4","p301k":"4...
Read more