hbase - MapReduce for Loading TSV -


i trying load data cloud bigtable using mapreduce job such values of first column set hbase row key.here code , sample tsv file.

1011    v1  v2  v3  v4 1012    c1  c2  c3  c4 1013    k1  k2  k3  k4 1014    s1  s2  s3  s4 1015    r1  r3  r2  r4 1016    p1  p2  p7  p9

here code sample :-

    public static class tokenizermapper extends             mapper<text, text, text,text> {         @override         public void map(text key, text value, context context) throws ioexception,                 interruptedexception {              string fields[] = null;             csvparser csvparser = new csvparser('\t');             fields = csvparser.parseline(value.tostring());             log.info(fields[0]);             context.write(new text(fields[0]), value);         }     }      public static class mytablereducer extends             tablereducer<text, text, text> {         @override         public void reduce(text key, iterable<text> values, context context)                 throws ioexception, interruptedexception {             string[] fields = null;             csvparser csvparser = new csvparser('\t');             try {                 for(text value: values) {                     fields = csvparser.parseline(value.tostring());                     (int = 1; < fields.length; ++i) {                          put put = new put(bytes.tobytes(fields[0]));                         put.addcolumn(column_family, bytes.tobytes(cols[i]), bytes.tobytes(fields[i]));                         context.write(key, put);                      }                 }             } catch (exception ex) {                 context.getcounter("hbasekvmapper", "parse_errors").increment(1);                 return;             }         }     }      public static void main(string[] args) throws exception {         configuration conf = hbaseconfiguration.create(); conf.set("mapreduce.input.keyvaluelinerecordreader.key.value.separator", "\t");         job job = job.getinstance(conf, "bigtableloader");         job.setinputformatclass(keyvaluetextinputformat.class);         keyvaluetextinputformat.addinputpath(job,new path(args[0]));          tablename tablename = tablename.valueof(args[1]);         job.setjarbyclass(bigtableloader.class);         job.setmapperclass(tokenizermapper.class);         job.setmapoutputvalueclass(text.class);         job.setmapoutputkeyclass(text.class);         tablemapreduceutil.inittablereducerjob(tablename.getnameasstring(), mytablereducer.class, job);         system.exit(job.waitforcompletion(true) ? 0 : 1);     } }

the problem getting instead of first column, second column has been set key in table, , first column ignored i.e. doesn't exist in table. there missing??


Comments

Post a Comment

Popular posts from this blog

php - Vagrant up error - Uncaught Reflection Exception: Class DOMDocument does not exist -

i've set bento/ubuntu-16.04 vagrant box using php 5.6, , getting following error quite few times when try , run shell script: "uncaught reflection exception: class domdocument not exist". i've tried adding following top of script (not together), each no success: "sudo apt-get install php5.6-xml", "sudo apt-get install php5-xml", "sudo apt-get install php-dom" i've tried resetting server "sudo service apache2 restart". does have suggestions?
Read more

vue.js - Create hooks for automated testing -

Image
qa guys complained can't automate frontend testing. because our html looks same outside. lazy hack did this <template> <div :role="$options.name"> ... </div> </template> <script> export default { name: 'vmcomponentname' } </script> which takes name script , applies html. in browser get: <div role="vmcomponentname"> ... </div> line :role="$options.name" goes in every component. is there dry er solutions? please share. additional details i'll explain in details looking for. imagine have vmusercreate form creates user. test case can create user . to automatically test without role, have use following selector: .wrapper > .wrapper > .wrapper > .submit-button the test extremely brittle (will break lot). if use roles can use [role="vmusercreateform"] .submit-button selector. many folds less brittle. so basically, i'm ...
Read more

Add new key value to json node in java -

i have json file : { "id":1, "name":"abc", "addressdetails": { "city":"newyork" } } i wanted add 1 more key-value ("pincode" : "414141") node 'addressdetails'. i tried using : objectmapper mapper = new objectmapper(); jsonnode root = mapper.readtree(new file("d://test.json")); objectnode node = mapper.createobjectnode(); node.with("addressdetails").put("pincode", "414141"); but it's not getting added, there way can ? you can try you should modify file path objectmapper mapper = new objectmapper(); objectnode nodes = mapper.readvalue(new file("d:\\test.txt"), objectnode.class); nodes.with("addressdetails").put("pincode", "414141"); mapper.writer().writevalue(new file("d:\\test.txt"), nodes);
Read more