How to use mutliprocessing in my matplotlib code, i tried not working

Community
1

Hi,
my code is below

def run(strt):

     print("Plotting Script Started Running.")
     import pandas as pd
     import matplotlib.pyplot as plt
     import matplotlib.tri as mtri
     import numpy as np

   from io import StringIO as StringIO

   from datetime import date, datetime, timedelta
   from scipy import stats
   from  math import sqrt
   ,,,,,,,
   ,,,,,,,
   tri_sub = tri_new.apply(lambda x: x - 1)

   triang = mtri.Triangulation(x1, y1, triangles=tri_sub)
   cmap = plt.cm.jet

   pp = plt.tripcolor(triang, req_elevation, cmap=cmap, 
   vmin= 0.0, vmax="2.0")

  cmap.set_under('white', 0.0)
  pp.cmap.set_under('white', 0.0)
  ....
  plt.savefig("p_"+str(count)+".png")
if __name__=="__main__":
  
  try:
      strt="20200717" #datastamp
      run(strt)
  finally:
         print("End")

How i can make this code to work as multiprocessor , so that all my plots save faster,
as currently its taking time. :frowning:
Thank you.
I will really appreciate your this help.

2

You can’t use multiple processors on the Matplotlib triangulation code. You can do the triangulation using other tools and then pass it into matplotlib, perhaps after massaging the form of the data. Googling "https://www.google.com/search?client=safari&rls=en&q=parallel+triangulation+python&ie=UTF-8&oe=UTF-8 gives a few hits.

Gridding data is hard and expensive computationally. You may look at other approaches than just triangulation. i.e. the various approaches in https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.griddata.html may work for you. In general if you have so many data points that triangulation is slow, you might consider an approach that does some spatial averaging.

3

@jklymak
Thanks for reply
currently i am generating 900 plots and it is taking 6 hours to make total 900 plots and even it takes lots of memory space also.
Isnt there any ways i can have 900 plots fast with less memory space ?
Thank you.

4

No idea without knowing your situation. But at 24s/plot my guess would be you are trying to keep them all in memory and the disk is swapping virtual memory. Don’t keep 900 plots in memory. Save them as something. Then your easy way to multiprocessing is to just divide the 900 plots into ncore jobs and run the jobs from separate python processes.

But this is just general computing. Figure out what your bottle neck is and try to work around it.

5

@jklymak
Thanks for reply ,
how i can achieve this:- Don’t keep 900 plots in memory. Save them as something .
i am using plt.savefig(“filename”+str(count)+".png")
i am saving all the 900 png plots then at last making one gif.
one more thing at beginning minutes my plot script it make 10 plots in one minute slowly the number of plots generate per minutes get reduce and at last it generate 1 plot per 3 minute ,can you please guide me how to have this same at least 10 plots generate per minute from start till end.
I thought that at beginning there are lots of free memory space so the plots are coming 10 per mintue slowly free space is getting reduce and more time it is taking to generate the plots, so after saving the plot i even use the del variable name & gc collect () so that variable will get deleted from memory but still there is no effect on generating number of plots, :frowning:

         count+=1   
        plt.savefig("PXE_" + str(count) + '.png')
  
        print("###################Plotting is working#############")
        del req_elevation
        del lons,x
        del lats,y
        del labels
      
   
        del e
        del parms1
        del parms2
        del newZ
        del legend_elements,legend_properties
        del pp,cmap,cbar,temp_elevation,zvaluefile

        gc.collect()

Thank you.
I will really appreciate your this help.

6

Are you closing the figures?

7

hi,
wat do u mean by closing the figure.
i am not closing the plot.
do u mean this plt.close(‘all’)
is there any benefit of using plt.close(‘all’) aftersavefig() ?
I am generating the plot for each timestamps and when i got all 900 plots i execute another function to make gif of all those plots.

8

Well, thats certainly your problem. Don’t try and have 900 plots open at once. If you are already saving all the pngs just do plt.close() and then create the gif from the pngs (presumably using imagemagick). Then you can parallelize by dividing the 900 plots up among processes.

9

hi,
i added plt.close(“all”)
modified code as below , hope this will make all plot generation fast.

            count+=1
        #plt.savefig("PXE_" + str(pngname) + '.png')
        plt.savefig("PXE_" + str(count) + '.png')
        #plt.show()
        plt.close('all')
        print("###################Plotting is working#############")
        del req_elevation
10

Hi,
i have included clear & close statements into my plot code still no luck, still the code is taking long time.
Actually i am using loop - one loop iteration will create one plot then second time loop iterative it will generate second plot, :frowning:

count+=1
#plt.savefig("PXE_" + str(pngname) + '.png')
plt.savefig("PXE_" + str(count) + '.png')
matplotlib.pyplot.figure().clear()
plt.close(plt.gcf())
plt.close('all')
print("###################Plotting is working#############")
del req_elevation

not working plt.close(‘all’),matplotlib.pyplot.figure().clear(),plt.close(plt.gcf())
Number of generating plot is reducing as per time :frowning: , kindly let me know any other way
Any more suggestion for clearing memory in matplotlib.
Thank you.