java - Spark does not find Scala specific methods -


the problem every job fails following exception:

exception in thread "main" java.lang.nosuchmethoderror: scala.predef$.refarrayops([ljava/lang/object;)[ljava/lang/object; @ ps.sparkapp.classification$.main(classification.scala:35) @ ps.sparkapp.classification.main(classification.scala) @ sun.reflect.nativemethodaccessorimpl.invoke0(native method) @ sun.reflect.nativemethodaccessorimpl.invoke(nativemethodaccessorimpl.java:62) @ sun.reflect.delegatingmethodaccessorimpl.invoke(delegatingmethodaccessorimpl.java:43) @ java.lang.reflect.method.invoke(method.java:498) @ org.apache.spark.deploy.sparksubmit$.org$apache$spark$deploy$sparksubmit$$runmain(sparksubmit.scala:743) @ org.apache.spark.deploy.sparksubmit$.dorunmain$1(sparksubmit.scala:187) @ org.apache.spark.deploy.sparksubmit$.submit(sparksubmit.scala:212) @ org.apache.spark.deploy.sparksubmit$.main(sparksubmit.scala:126) @ org.apache.spark.deploy.sparksubmit.main(sparksubmit.scala)

this exception meas task can not find method. develop using intelij community edition. have no problems compiling package. dependencies packaged correctly. here build.sbt:

name := "sparkapp" version := "1.0"  scalaversion := "2.11.6"  librarydependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.1" librarydependencies += "org.apache.spark" % "spark-mllib_2.11" % "2.1.1"    scala -version   scala code runner version 2.11.6 -- copyright 2002-2013, lamp/epfl

i found out error has somehow scala because happens when use functionality native scala, e.g scala loop, .map or .drop(2). class , still written in scala, if avoid functionality .map or drop(2) works fine.

import org.apache.spark.ml.feature.vectorassembler import org.apache.spark.sql.sparksession import org.apache.spark.ml.linalg.vector  object classification {    def main(args: array[string]) {     ...     //df.printschema()     var dataset = df.groupby("user_id","measurement_date").pivot("rank").min()      val col = dataset.schema.fieldnames.drop(2) // <- here error happens      // take features , put them 1 vector     val assembler = new vectorassembler()       .setinputcols(col)       .setoutputcol("features")      val data = assembler.transform(dataset)     data.printschema()     data.show()      sc.stop()   }  }

as said if not use .drop(2) works perfectly, avoiding these methods no option since painful..

i not find solution on web, ideas?

btw: can use these methods within spark-shell, find strange.

thanks in advance.

note 1)

i use: spark version 2.1.1

using scala version 2.11.8 (openjdk 64-bit server vm, java 1.8.0_131)

try adding actual scala libraries etc project dependency. e.g.:

librarydependencies += "org.scala-lang" % "scala-library" % "2.11.6"


Comments

Post a Comment

Popular posts from this blog

javascript - Create a stacked percentage column -

i have collection of data : var data = [ {"p301a":"10","p301b":"7","p301c":"7","p301d":"3","p301e":"8","p301f":"8","p301g":"8","p301h":"8","p301i":"8","p301j":"8","p301k":"8","p301l":"8","p301m":"8","p301n":"8","p301o":"8","age":"31-40 years","profesion":"2","position":"2"}, {"p301a":"5","p301b":"4","p301c":"4","p301d":"4","p301e":"4","p301f":"4","p301g":"4","p301h":"4","p301i":"4","p301j":"4","p301k":"4...
Read more

Optimising Firebase database by automatically overwriting data -

in firebase app have following structure: posts latest all users can write posts > latest . want limit entries 20 posts save space in database. , posts beyond 20 unnecessary never shown on homepage. how can limit posts 20, when user writes posts > latest posts @ end drop off (be deleted) automatically? this sample should you. you need : const max_log_count = 20; exports.removeold = functions.database.ref('/posts/latest/{postid}').oncreate(event => { const parentref = event.data.ref.parent; return parentref.once('value').then(snapshot => { if (snapshot.numchildren() >= max_log_count) { let childcount = 0; const updates = {}; snapshot.foreach(function(child) { if (++childcount <= snapshot.numchildren() - max_log_count) { updates[child.key] = null; } }); // update parent. removes children. ...
Read more

javascript - Angular UI-Grid customTemplate directive causing rows to load slowly/? -

Image
each grid cell can have more 1 customtemplate loaded. have directive holds watcher on model_col_field listen cell updates on scroll when dom rows updated , recompile celltemplate . without watcher , $compile rows repeat on scroll, showing same set of rows. is there more angular way of approaching or way improve performance? relevant controller code: create gridoptions.data & gridoptions.columndefs : var customtemp = '<div ng-repeat="opt in grid.appscope.colvalueoptionsmodel" ng-init="tempstring=model_col_field.clazz+opt.key;"><comparison-directive dbo="model_col_field" attobj="opt" fff="grid.appscope.getattributetemplate(tempstring)" /></div>'; $scope.gridoptions.columndefs = [ { name:'yindex',pinnedleft:true,celltemplate: customtemp, displayname:'index'}, { name:'col1',celltemplate: customtemp}, { name:'col2',celltemplate: c...
Read more