multithreading - Concurrency: Java Map -

what best way push 20 million entities java map object?

1. without multi-threading taking ~40 seconds.
2. using forkjoinpool taking ~25 seconds, have created 2 tasks , each of these tasks pushing 10 million entities

i believe both these tasks running in 2 different cores. question: when create 1 task pushes 10 million data, takes ~9 seconds, when running 2 tasks each of these tasks pushes 10 million data, why take ~26 seconds ? doing wrong ?

is there different solution inserting 20 m data takes less 10 seconds ?

without seeing code, probable cause of these bad performance results due garbage collection activity. demonstrate it, wrote following program:

import java.lang.management.managementfactory; import java.util.*; import java.util.concurrent.*;  public class testmap {   // assume nb_entities divisible nb_tasks   static final int nb_entities = 20_000_000, nb_tasks = 2;   static map<string, string> map = new concurrenthashmap<>();    public static void main(string[] args) {     try {       system.out.printf("running nb entities = %,d, nb tasks = %,d, vm args = %s%n", nb_entities, nb_tasks, managementfactory.getruntimemxbean().getinputarguments());       executorservice executor = executors.newfixedthreadpool(nb_tasks);       int entitiespertask = nb_entities / nb_tasks;       list<future<?>> futures = new arraylist<>(nb_tasks);       long starttime = system.nanotime();       (int i=0; i<nb_tasks; i++) {         mytask task = new mytask(i * entitiespertask, (i + 1) * entitiespertask - 1);         futures.add(executor.submit(task));       }       (future<?> f: futures) {         f.get();       }       long elapsed = system.nanotime() - starttime;       executor.shutdownnow();       system.gc();       runtime rt = runtime.getruntime();       long usedmemory = rt.maxmemory() - rt.freememory();       system.out.printf("processing completed in %,d ms, usedmemory after gc = %,d bytes%n", elapsed/1_000_000l, usedmemory);     } catch (exception e) {       e.printstacktrace();     }   }    static class mytask implements runnable {     private final int startidx, endidx;      public mytask(final int startidx, final int endidx) {       this.startidx = startidx;       this.endidx = endidx;     }      @override     public void run() {       long starttime = system.nanotime();       (int i=startidx; i<=endidx; i++) {         map.put("sambit:rout:" + i, "c:\\images\\provision_images");       }       long elapsed = system.nanotime() - starttime;       system.out.printf("task[%,d - %,d], completed in %,d ms%n", startidx, endidx, elapsed/1_000_000l);     }   } } 

at end of processing, code computes approximation of used memory doing system.gc() followed runtime.maxmemory() - runtime.freememory(). shows map 20 million entries takes approximately under 2.2 gb, considerable. have run 1 , 2 threads, various values of -xmx , -xms jvm arguments, here resulting outputs (just clear: 2560m = 2.5g):

running nb entities = 20,000,000, nb tasks = 1, vm args = [-xms2560m, -xmx2560m] task[0 - 19,999,999], completed in 11,781 ms processing completed in 11,782 ms, usedmemory after gc = 2,379,068,760 bytes  running nb entities = 20,000,000, nb tasks = 2, vm args = [-xms2560m, -xmx2560m] task[0 - 9,999,999], completed in 8,269 ms task[10,000,000 - 19,999,999], completed in 12,385 ms processing completed in 12,386 ms, usedmemory after gc = 2,379,069,480 bytes  running nb entities = 20,000,000, nb tasks = 1, vm args = [-xms3g, -xmx3g] task[0 - 19,999,999], completed in 12,525 ms processing completed in 12,527 ms, usedmemory after gc = 2,398,339,944 bytes  running nb entities = 20,000,000, nb tasks = 2, vm args = [-xms3g, -xmx3g] task[0 - 9,999,999], completed in 12,220 ms task[10,000,000 - 19,999,999], completed in 12,264 ms processing completed in 12,265 ms, usedmemory after gc = 2,382,777,776 bytes  running nb entities = 20,000,000, nb tasks = 1, vm args = [-xms4g, -xmx4g] task[0 - 19,999,999], completed in 7,363 ms processing completed in 7,364 ms, usedmemory after gc = 2,402,467,040 bytes  running nb entities = 20,000,000, nb tasks = 2, vm args = [-xms4g, -xmx4g] task[0 - 9,999,999], completed in 5,466 ms task[10,000,000 - 19,999,999], completed in 5,511 ms processing completed in 5,512 ms, usedmemory after gc = 2,381,821,576 bytes  running nb entities = 20,000,000, nb tasks = 1, vm args = [-xms8g, -xmx8g] task[0 - 19,999,999], completed in 7,778 ms processing completed in 7,779 ms, usedmemory after gc = 2,438,159,312 bytes  running nb entities = 20,000,000, nb tasks = 2, vm args = [-xms8g, -xmx8g] task[0 - 9,999,999], completed in 5,739 ms task[10,000,000 - 19,999,999], completed in 5,784 ms processing completed in 5,785 ms, usedmemory after gc = 2,396,478,680 bytes 

these results can summarized in following table:

-------------------------------- heap      | exec time (ms) for:  size (gb) | 1 thread | 2 threads -------------------------------- 2.5       |    11782 |     12386 3.0       |    12527 |     12265 4.0       |     7364 |      5512 8.0       |     7779 |      5785 -------------------------------- 

i observed that, 2.5g , 3g heap sizes, there high cpu activity, spikes @ 100% during whole processing time, due gc activity, whereas 4g , 8g observed @ end due system.gc() call.

to conclude:

1. if heap sized inappropriately, garbage collection kill performance gain hope obtain. should make large enough avoid side effects of long gc pauses.

2. you must aware using concurrent collection such concurrenthashmap has significant performance overhead. illustrate this, modified code each task uses own hashmap, @ end maps aggregated (with map.putall()) in map of first task. processing time fell around 3200 ms