Rendering a large number of figures in parallel on a multicore machine

Long time matplotlib user. I’m looking to create and save to disk a large number of figures (10k-1m). These will be analyzed and subsets, and the entirety will be composited into animations with ffmpeg. The challenge I have is that this takes about 12 hours. I have access to a multicore workstation/server, and I was wondering if there’s a way to run this across multiple cores so that each is responsible for rendering and saving one figure at a time. The hope being that I get some sort of near linear speed up, since all of these calculations are independent. Is there a template of guidance for doing something like this. A schematic code snippet is shown below.

for i in range(100000):
   plotting_function(i)
   plt.savefig(...)

I wonder if you could use Joblib for that use case: Embarrassingly parallel for loops — joblib 1.4.2 documentation.
Which could look something like that (haven’t tried it though):

from joblib import Parallel, delayed

def plotting_function(idx):
    [...]
    plt.savefig(...)
    return

n_jobs_parallel_max = 10
Parallel(n_jobs=n_jobs_parallel_max)(delayed(plotting_function)(idx) for idx in range(100000))
1 Like