python - Modifying a class instance variable in an external function using a multiprocessing.Pool of workers -


i trying populate dictionary, class instance variable called self.diags, using pool of workers, reason dictionary empty after pool.map line executes. have 3 lists: dt, idx1, , ivals. looping through entries of dt, , adding each entry key self.diags dictionary, value of [idx1[i], ivals[i], none], i index of current entry in dt. here relevant code:

class eventcloudextractor:     def __init__(self, dl, dw):         # other stuff         self.diags = defaultdict(list)         self.idx1 = []         self.ivals = []      def p_triplet_to_diags(self, dt, idx1, ivals, dl = none, dt_min = 0, dt_max = none, ivals_thresh = 0, min_eventid = 0):         # other stuff         self.idx1 = idx1         self.ivals = ivals         num_cores = min(multiprocessing.cpu_count(), 24)         pool = multiprocessing.pool(num_cores)         pool.map(populate_diags, [(i, dtval, self) i, dtval in izip(count(), dt)]) 

i'm passing in 3 values (i, dtval, self) in tuple 1 argument. instead of passing idx1 , ivals lists arguments too, made them instance variables , passed in self reference avoid copying them on many times in list comprehension. here populate_diags:

def populate_diags(x):     = x[0]     dtval = x[1]     obj = x[2]     obj.diags[dtval].append([obj.idx1[i], obj.ivals[i], none]) 

i know tuple immutable, tuple elements lists or dictionaries aren't (and confirmed tuple isn't problem because got same result when used list).

my understanding when pass self this, i'm passing reference class instance, means here pool of workers share reference , modify self.diags. however, after pool.map done, self.diags empty. how can modify instance variable using pool of workers? tried using initializer pool.map, , tried making diags global variable, got same result.

i printed id(obj) inside populate_diags , id(self) inside p_triplet_to_diags, , different. in fact, different calls id(obj) inside calls populate_diags pool.map showed different object id values (though -- of calls had repeated values). i'm guessing has how memory managed across different processes spawned pool, since don't understand how done, don't know how go solving this. appreciate or insights.


Comments

Popular posts from this blog

php - Vagrant up error - Uncaught Reflection Exception: Class DOMDocument does not exist -

vue.js - Create hooks for automated testing -

Add new key value to json node in java -