scala - Load S3 files in parallel Spark -


i loading files spark, s3, through following code. it's working, noticing there delay between 1 file , another, , loaded sequentially. improve loading in parallel. 

        // load files loaded firehose on day     var s3files = spark.sqlcontext.read.schema(schema).json("s3n://" + job.awsaccesskey + ":" + job.awssecretkey + "@" + job.bucketname + "/" + job.awss3rawfileexpression + "/" + year + "/" + monthcheck + "/" + daycheck + "/*/").rdd      // apply schema rdd, here have duplicates     val usersdataframe = spark.createdataframe(s3files, schema)      usersdataframe.createorreplacetempview("results")      // clean , use partition keys eliminate duplicates , latest record     var results = spark.sql(buildcleaningquery(job, "results"))     results.createorreplacetempview("filteredresults")     val records = spark.sql("select count(*) filteredresults")

i have tried loading through textfile() method, having problems converting rdd[string] rdd[row] because afterwards need move on use spark sql. using in following manner;

        var s3files = sparkcontext.textfile("s3n://" + job.awsaccesskey + ":" + job.awssecretkey + "@" + job.bucketname + "/" + job.awss3rawfileexpression + "/" + year + "/" + monthcheck + "/" + daycheck + "/*/").tojavardd()

what ideal manner load json files (multiple files around 50mb each) spark? validate properties against schema, later on able spark sql queries clean data.

what's going on dataframe being converted rdd , dataframe again, loses partitioning information.

var s3files = spark   .sqlcontext   .read.schema(schema)   .json(...)   .createorrepla‌​cetempview("results"‌​)

should sufficient, , partitioning information should still present, allowing json files loaded concurrently.


Comments

Post a Comment

Popular posts from this blog

javascript - Create a stacked percentage column -

i have collection of data : var data = [ {"p301a":"10","p301b":"7","p301c":"7","p301d":"3","p301e":"8","p301f":"8","p301g":"8","p301h":"8","p301i":"8","p301j":"8","p301k":"8","p301l":"8","p301m":"8","p301n":"8","p301o":"8","age":"31-40 years","profesion":"2","position":"2"}, {"p301a":"5","p301b":"4","p301c":"4","p301d":"4","p301e":"4","p301f":"4","p301g":"4","p301h":"4","p301i":"4","p301j":"4","p301k":"4...
Read more

Optimising Firebase database by automatically overwriting data -

in firebase app have following structure: posts latest all users can write posts > latest . want limit entries 20 posts save space in database. , posts beyond 20 unnecessary never shown on homepage. how can limit posts 20, when user writes posts > latest posts @ end drop off (be deleted) automatically? this sample should you. you need : const max_log_count = 20; exports.removeold = functions.database.ref('/posts/latest/{postid}').oncreate(event => { const parentref = event.data.ref.parent; return parentref.once('value').then(snapshot => { if (snapshot.numchildren() >= max_log_count) { let childcount = 0; const updates = {}; snapshot.foreach(function(child) { if (++childcount <= snapshot.numchildren() - max_log_count) { updates[child.key] = null; } }); // update parent. removes children. ...
Read more

javascript - Angular UI-Grid customTemplate directive causing rows to load slowly/? -

Image
each grid cell can have more 1 customtemplate loaded. have directive holds watcher on model_col_field listen cell updates on scroll when dom rows updated , recompile celltemplate . without watcher , $compile rows repeat on scroll, showing same set of rows. is there more angular way of approaching or way improve performance? relevant controller code: create gridoptions.data & gridoptions.columndefs : var customtemp = '<div ng-repeat="opt in grid.appscope.colvalueoptionsmodel" ng-init="tempstring=model_col_field.clazz+opt.key;"><comparison-directive dbo="model_col_field" attobj="opt" fff="grid.appscope.getattributetemplate(tempstring)" /></div>'; $scope.gridoptions.columndefs = [ { name:'yindex',pinnedleft:true,celltemplate: customtemp, displayname:'index'}, { name:'col1',celltemplate: customtemp}, { name:'col2',celltemplate: c...
Read more