java - Spark fails to serialize on RDD.count(); -


so relatively new spark, guess understanding bit more everyday. have methods running, not. trying rdd.count() gives me error of :

exception in thread "main" `org.apache.spark.sparkexception`: job aborted due stage failure: failed serialize task 5, not attempting retry it.  exception during serialization: `java.io.notserializableexception`:  serialization stack: - object not serializable  element of array (index: 0) - array (class [ljava.lang.object;, size 224) - field (class: scala.collection.mutable.wrappedarray$ofref, name: array, type: class [ljava.lang.object;) - object (class scala.collection.mutable.wrappedarray$ofref, wrappedarray - writeobject data (class: org.apache.spark.rdd.parallelcollectionpartition) - object (class org.apache.spark.rdd.parallelcollectionpartition, org.apache.spark.rdd.parallelcollectionpartition@82b) - field (class: org.apache.spark.scheduler.resulttask, name: partition, type: interface org.apache.spark.partition) - object (class org.apache.spark.scheduler.resulttask, resulttask(4, 0))

the object in question made of strings, booleans, , other classes transient sets. have tried implementing serializable classes error throws, becomes way recursive, , feels there no end implementing serializable everything, , feel there more efficient way of doing this. here code: 

javasparkcontext sc = new javasparkcontext(spark.sparkcontext());     list<objectinquestion> rowsindba = new arraylist<objectinquestion>();     try     {         dba.readallrows(objectinquestion.class, new dbentitylistener<objectinquestion>(){              @override             public void entityreceived(objectinquestion entity)             {                 rowsindba.add(entity);              }         });     }     catch (exception e)     {         // todo auto-generated catch block         e.printstacktrace();     }     javardd<objectinquestion> schemardd = sc.parallelize(rowsindba, 2);     long count = schemardd.count();     return  (int) count;

after debugging , looking @ values method reaches sc.parallelize rdd contains entities given rowsindba. removed .count() before, , there no issues running. still looking myself , trying find answers, feel may know error caused way before come solution. thanks!


Comments

Post a Comment

Popular posts from this blog

php - Vagrant up error - Uncaught Reflection Exception: Class DOMDocument does not exist -

i've set bento/ubuntu-16.04 vagrant box using php 5.6, , getting following error quite few times when try , run shell script: "uncaught reflection exception: class domdocument not exist". i've tried adding following top of script (not together), each no success: "sudo apt-get install php5.6-xml", "sudo apt-get install php5-xml", "sudo apt-get install php-dom" i've tried resetting server "sudo service apache2 restart". does have suggestions?
Read more

vue.js - Create hooks for automated testing -

Image
qa guys complained can't automate frontend testing. because our html looks same outside. lazy hack did this <template> <div :role="$options.name"> ... </div> </template> <script> export default { name: 'vmcomponentname' } </script> which takes name script , applies html. in browser get: <div role="vmcomponentname"> ... </div> line :role="$options.name" goes in every component. is there dry er solutions? please share. additional details i'll explain in details looking for. imagine have vmusercreate form creates user. test case can create user . to automatically test without role, have use following selector: .wrapper > .wrapper > .wrapper > .submit-button the test extremely brittle (will break lot). if use roles can use [role="vmusercreateform"] .submit-button selector. many folds less brittle. so basically, i'm ...
Read more

Add new key value to json node in java -

i have json file : { "id":1, "name":"abc", "addressdetails": { "city":"newyork" } } i wanted add 1 more key-value ("pincode" : "414141") node 'addressdetails'. i tried using : objectmapper mapper = new objectmapper(); jsonnode root = mapper.readtree(new file("d://test.json")); objectnode node = mapper.createobjectnode(); node.with("addressdetails").put("pincode", "414141"); but it's not getting added, there way can ? you can try you should modify file path objectmapper mapper = new objectmapper(); objectnode nodes = mapper.readvalue(new file("d:\\test.txt"), objectnode.class); nodes.with("addressdetails").put("pincode", "414141"); mapper.writer().writevalue(new file("d:\\test.txt"), nodes);
Read more