r - Still struggling with handling large data set -


i have been reading around on website , haven't been able find exact answer. if exists, apologize repost.

i working data sets extremely large (600 million rows, 64 columns on computer 32 gb of ram). need smaller subsets of data, struggling perform functions besides importing 1 data set in fread, , selecting 5 columns need. after that, try overwrite dataset specific conditions need, hit ram cap , message "error: cannot allocate vector size of 4.5 gb. looked @ ff , bigmemory packages alternatives, seems can't subset before importing in packages? there solution problem besides upgrading ram on computer?

tasks trying perform: 

>sampletable<-fread("my.csv", header = t, sep = ",", select=c("column1", "column2", "column7", "column12", "column15"))  >sampletable2<-sampletable[sampletable[,column1=="6" & column7=="1"]]

at point, hit memory cap. better try , use package import 64 columns of 600 million rows? don't want spend hours upon hours perform 1 import.

what read csv file in chunks:

# define subset of columns csv <- "my.csv" colnames <- names(read.csv(csv, header = true, nrows = 1)) colclasses <- rep(list(null), length(colnames)) ind <- c(1, 2, 7, 12, 15) colclasses[ind] <- "double"  # read header , first line library(dplyr) l_df <- list() con <- file(csv, "rt") df <- read.csv(con, header = true, nrows = 1, colclasses = colclasses) %>%   filter(v1 == 6, v7 == 1) names(df) <- paste0("v", ind) l_df[[i <- 1]] <- df  # read other lines , combine repeat {   <- + 1   df <- read.csv(con, header = false, nrows = 9973, colclasses = colclasses)   l_df[[i]] <- filter(df, v1 == 6, v7 == 1)   if (nrow(df) < 9973) break } df <- do.call("rbind", l_df)

9973 arbitrary prime number has few chance divisor of nlines - 1.


Comments

Post a Comment

Popular posts from this blog

php - Vagrant up error - Uncaught Reflection Exception: Class DOMDocument does not exist -

i've set bento/ubuntu-16.04 vagrant box using php 5.6, , getting following error quite few times when try , run shell script: "uncaught reflection exception: class domdocument not exist". i've tried adding following top of script (not together), each no success: "sudo apt-get install php5.6-xml", "sudo apt-get install php5-xml", "sudo apt-get install php-dom" i've tried resetting server "sudo service apache2 restart". does have suggestions?
Read more

vue.js - Create hooks for automated testing -

Image
qa guys complained can't automate frontend testing. because our html looks same outside. lazy hack did this <template> <div :role="$options.name"> ... </div> </template> <script> export default { name: 'vmcomponentname' } </script> which takes name script , applies html. in browser get: <div role="vmcomponentname"> ... </div> line :role="$options.name" goes in every component. is there dry er solutions? please share. additional details i'll explain in details looking for. imagine have vmusercreate form creates user. test case can create user . to automatically test without role, have use following selector: .wrapper > .wrapper > .wrapper > .submit-button the test extremely brittle (will break lot). if use roles can use [role="vmusercreateform"] .submit-button selector. many folds less brittle. so basically, i'm ...
Read more

Add new key value to json node in java -

i have json file : { "id":1, "name":"abc", "addressdetails": { "city":"newyork" } } i wanted add 1 more key-value ("pincode" : "414141") node 'addressdetails'. i tried using : objectmapper mapper = new objectmapper(); jsonnode root = mapper.readtree(new file("d://test.json")); objectnode node = mapper.createobjectnode(); node.with("addressdetails").put("pincode", "414141"); but it's not getting added, there way can ? you can try you should modify file path objectmapper mapper = new objectmapper(); objectnode nodes = mapper.readvalue(new file("d:\\test.txt"), objectnode.class); nodes.with("addressdetails").put("pincode", "414141"); mapper.writer().writevalue(new file("d:\\test.txt"), nodes);
Read more