python - Modifying a class instance variable in an external function using a multiprocessing.Pool of workers -
i trying populate dictionary, class instance variable called self.diags
, using pool of workers, reason dictionary empty after pool.map
line executes. have 3 lists: dt
, idx1
, , ivals
. looping through entries of dt
, , adding each entry key self.diags
dictionary, value of [idx1[i], ivals[i], none]
, i
index of current entry in dt
. here relevant code:
class eventcloudextractor: def __init__(self, dl, dw): # other stuff self.diags = defaultdict(list) self.idx1 = [] self.ivals = [] def p_triplet_to_diags(self, dt, idx1, ivals, dl = none, dt_min = 0, dt_max = none, ivals_thresh = 0, min_eventid = 0): # other stuff self.idx1 = idx1 self.ivals = ivals num_cores = min(multiprocessing.cpu_count(), 24) pool = multiprocessing.pool(num_cores) pool.map(populate_diags, [(i, dtval, self) i, dtval in izip(count(), dt)])
i'm passing in 3 values (i, dtval, self)
in tuple 1 argument. instead of passing idx1
, ivals
lists arguments too, made them instance variables , passed in self
reference avoid copying them on many times in list comprehension. here populate_diags
:
def populate_diags(x): = x[0] dtval = x[1] obj = x[2] obj.diags[dtval].append([obj.idx1[i], obj.ivals[i], none])
i know tuple immutable, tuple elements lists or dictionaries aren't (and confirmed tuple isn't problem because got same result when used list).
my understanding when pass self
this, i'm passing reference class instance, means here pool of workers share reference , modify self.diags
. however, after pool.map
done, self.diags
empty. how can modify instance variable using pool of workers? tried using initializer pool.map
, , tried making diags
global variable, got same result.
i printed id(obj)
inside populate_diags
, id(self)
inside p_triplet_to_diags
, , different. in fact, different calls id(obj)
inside calls populate_diags
pool.map
showed different object id
values (though -- of calls had repeated values). i'm guessing has how memory managed across different processes spawned pool
, since don't understand how done, don't know how go solving this. appreciate or insights.
Comments
Post a Comment