amazon s3 - Create external table from S3 on Spark Beeline -
made below change /etc/dse/spark/hive-site.xml on each node in 4 node cluster.
<property> <name>fs.s3.awsaccesskeyid</name> <value>****</value> </property> <property> <name>fs.s3.awssecretaccesskey</name> <value>****</value> </property> <property> <name>fs.s3n.awsaccesskeyid</name> <value>****</value> </property> <property> <name>fs.s3n.awssecretaccesskey</name> <value>****</value> </property> <property> <name>fs.s3a.awsaccesskeyid</name> <value>****</value> </property> <property> <name>fs.s3a.awssecretaccesskey</name> <value>****</value> </property>
set below env variables on node spark thrift server , spark-beeline client runs export aws_secret_access_key=****
export aws_access_key_id=*****
started spark thrift server below
dse -u cassandra -p ***** spark-sql-thriftserver start --conf spark.cores.max=2 --conf spark.executor.memory=2g --conf spark.driver.maxresultsize=1g --conf spark.kryoserializer.buffer.max=512m --conf spark.sql.thriftserver.incrementalcollect=true
created table spark beeline s3 bucket source
dse -u cassandra -p ***** spark-beeline --total-executor-cores 2 --executor-memory 2g log file @ /home/ubuntu/.spark-beeline.log beeline version 1.2.1.2_dse_spark apache hive beeline> !connect jdbc:hive2://localhost:10000 cassandra connecting jdbc:hive2://localhost:10000 enter password jdbc:hive2://localhost:10000: **************** connected to: spark sql (version 1.6.3) driver: hive jdbc (version 1.2.1.2_dse_spark) transaction isolation: transaction_repeatable_read 0: jdbc:hive2://localhost:10000> create external table test_table (name string,phone string) partitioned by(day date) row format delimited fields terminated ',' location 's3a://hive-getsimpl/test';
i below error
error: org.apache.spark.sql.execution.queryexecutionexception: failed: execution error, return code 1 org.apache.hadoop.hive.ql.exec.ddltask.metaexception (message:com.amazonaws.services.s3.model.amazons3exception: status code: 403, aws service: amazon s3, aws request id: 29991e2338cc6b49, aws error code: null, aws error message: forbidden, s3 extended request id: kidxznqi73pbslugolqlb4+vedix0t82y/j/q69na18k8mnsileyo5ricuj3qceigifrqb4rabc=) (state=,code=0)
note: aws keys valid , have been working other python scripts
the config names used s3a different; need set fs.s3a.access.key
, fs.s3a.secret.key
. don't touch environment vars, add confusion.
Comments
Post a Comment