python - TensorFlow CNN on multiple GPUs -
i trying parallelize code have tensorflow model run on multiple gpus. reason, code wrote parallelize training works standard deep neural net, throws errors when using convolutional neural net. here code compute average gradients: def average_gradients(tower_grads): average_grads = [] grad_and_vars in zip(*tower_grads): # note each grad_and_vars looks following: # ((grad0_gpu0, var0_gpu0), ... , (grad0_gpun, var0_gpun)) grads = [] g, _ in grad_and_vars: # add 0 dimension gradients represent tower. expanded_g = tf.expand_dims(g, 0) # append on 'tower' dimension average on below. grads.append(expanded_g) # average on 'tower' dimension. grad = tf.concat(axis=0, values=grads) grad = tf.reduce_mean(grad, 0) # keep in mind variables redundant because shared # across towers. .. return first tower's pointer # variable. v = grad_and_vars[0][1] grad_and_var = (grad, v) average_grads.a...