matplotlib slow compared to gnuplot?

Tom_Leys · November 11, 2009, 7:33pm

It looks like you are storing your source data in a python list. Matplotlib runs much faster if you store your data using a numpy array instead.

I'm no expert, but it certianly sped up my graph drawing.

-Tom

Message: 5

···

Date: Wed, 11 Nov 2009 08:53:58 -0600
From: Mike Anderson <mbanderson@...150...>
Subject: [Matplotlib-users] matplotlib slow compared to gnuplot?
To: matplotlib-users@lists.sourceforge.net
Message-ID: <AE4D4739-44BE-43F2-9E56-DAEDBE990BFB@...150...>
Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes

Hi all,

Previously I was a user of gnuplot but have been giving matplotlib a try. One thing I've run in to right away is that matplotlib appears to be significantly slower.

A script to produce a dozen plots was taking me ~1 second with gnuplot, and now takes me ~18 seconds with matplotlib.

I'm curious if anyone knows how to speed things up. To figure out what is taking most of the time, I've used cProfile and pstats and below is the top 15 functions taking the most time.
(note: "plotStackedJobsVsTime" is my function that uses matplotlib.)

My script, for the curious, is at
http://www.hep.wisc.edu/cms/comp/routerqMonitor/prodJobMonitorPlots_matplotlib.py
and produces these plots:
http://www.hep.wisc.edu/cms/comp/routerqMonitor/index.html

Any hints at what I can do to speed up my script? Or is it out of my hands because it's all in matplotlib?

Thanks for any help,
Mike

Jeff_Klukas · February 24, 2010, 9:07pm

Tom Leys wrote:

It looks like you are storing your source data in a python list.
Matplotlib runs much faster if you store your data using a numpy array
instead.

I'm no expert, but it certianly sped up my graph drawing.

I am trying something similar to what Mike was describing. I have converted
all of the stored data to numpy arrays rather than python lists, as per your
suggestion. But the plots I'm making don't have more than 50 entries
apiece, and I'm not seeing any appreciable speed-up. It still takes ~15
seconds to produce each plot (using PDF backend to save to files).

Does anybody have other thoughts on what I might be able to try to bring the
lag down to something more reasonable?

My use case is that I'm building a library to dump the content of histograms
from ROOT (a data analysis package used in physics) into a convenient Hist
object that has members to directly plot the data as a matplotlib errorbar
or bar plot. You can see the 100 lines I've written here:
http://www.hep.wisc.edu/~jklukas/public/root2matplot.py

I've tried running a profiler like Mike did, but I'm afraid I'm not able to
parse any useful information from it either.

I'd appreciate any advice from the community.

Thanks,
Jeff

Output of profiler for producing one single plot:

···

----------------------------------------------------

Wed Feb 24 15:01:20 2010 timing

6995086 function calls (6983887 primitive calls) in 22.301 CPU
seconds

Ordered by: cumulative time
List reduced from 1080 to 25 due to restriction <25>

   ncalls tottime percall cumtime percall filename:lineno(function)
        1 0.004 0.004 22.301 22.301 ../overlayHists.py:367(main)
     20/1 0.139 0.007 21.669 21.669
../overlayHists.py:123(process_directory)
        1 0.001 0.001 21.529 21.529
../overlayHists.py:231(process_hist_matplotlib)
        2 0.000 0.000 21.384 10.692
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/pyplot.py:354(savefig)
        2 0.000 0.000 21.384 10.692
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/figure.py:959(savefig)
        2 0.000 0.000 21.384 10.692
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/backend_bases.py:1372(print_figure)
        2 0.000 0.000 21.373 10.687
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/backends/backend_pdf.py:2012(print_pdf)
    126/2 0.001 0.000 12.314 6.157
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/artist.py:44(draw_wrapper)
        2 0.000 0.000 12.314 6.157
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/figure.py:729(draw)
        2 0.000 0.000 12.310 6.155
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/axes.py:1632(draw)
        4 0.002 0.000 11.940 2.985
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/axis.py:727(draw)
       40 0.005 0.000 11.927 0.298
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/text.py:504(draw)
       58 0.023 0.000 10.349 0.178
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/dviread.py:805(find_tex_file)
       58 0.008 0.000 9.904 0.171
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/subprocess.py:662(communicate)
    15059 9.899 0.001 9.899 0.001 {method 'read' of 'file'
objects}
        2 0.000 0.000 9.051 4.526
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/backends/backend_pdf.py:447(close)
        2 0.001 0.001 9.043 4.521
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/backends/backend_pdf.py:520(writeFonts)
        8 0.008 0.001 9.041 1.130
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/backends/backend_pdf.py:554(embedTeXFont)
        8 0.000 0.000 8.637 1.080
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/type1font.py:47(__init__)
      304 0.001 0.000 7.796 0.026
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/dviread.py:50(__iter__)
  157/152 0.017 0.000 7.780 0.051
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/dviread.py:114(_read)
5788/3854 0.025 0.000 7.765 0.002
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/dviread.py:146(_dispatch)
      116 0.004 0.000 7.751 0.067
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/backends/backend_pdf.py:1643(get_text_width_height_descent)
      116 0.002 0.000 7.711 0.066
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/texmanager.py:576(get_text_width_height_descent)
       82 0.007 0.000 7.690 0.094
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-packages/matplotlib/text.py:250(_get_layout)

--
View this message in context: http://old.nabble.com/matplotlib-slow-compared-to-gnuplot--tp26302608p27714298.html
Sent from the matplotlib - users mailing list archive at Nabble.com.

Sebastien_Binet · February 25, 2010, 7:10pm

hi Jeff,

I was wondering if I could reap out your little root2matplot script.
I am planning to release a few helper functions as part of a tutorial I'll be
giving here at Orsay (how to use python/numpy/.... as an analysis foundation
in Atlas), and it seems it would be quite interesting to help people switch to
matplotlib and still interact with the rest of the HEP crowd...

cheers,
sebastien.

···

On Wednesday 24 February 2010 22:07:03 klukas wrote:

Tom Leys wrote:
> It looks like you are storing your source data in a python list.
> Matplotlib runs much faster if you store your data using a numpy array
> instead.
>
> I'm no expert, but it certianly sped up my graph drawing.

I am trying something similar to what Mike was describing. I have
converted all of the stored data to numpy arrays rather than python lists,
as per your suggestion. But the plots I'm making don't have more than 50
entries apiece, and I'm not seeing any appreciable speed-up. It still
takes ~15 seconds to produce each plot (using PDF backend to save to
files).

Does anybody have other thoughts on what I might be able to try to bring
the lag down to something more reasonable?

My use case is that I'm building a library to dump the content of
histograms from ROOT (a data analysis package used in physics) into a
convenient Hist object that has members to directly plot the data as a
matplotlib errorbar or bar plot. You can see the 100 lines I've written
here:
http://www.hep.wisc.edu/~jklukas/public/root2matplot.py

I've tried running a profiler like Mike did, but I'm afraid I'm not able to
parse any useful information from it either.

I'd appreciate any advice from the community.

Thanks,
Jeff

Output of profiler for producing one single plot:
----------------------------------------------------

Wed Feb 24 15:01:20 2010 timing

         6995086 function calls (6983887 primitive calls) in 22.301 CPU
seconds

   Ordered by: cumulative time
   List reduced from 1080 to 25 due to restriction <25>

   ncalls tottime percall cumtime percall filename:lineno(function)
        1 0.004 0.004 22.301 22.301 ../overlayHists.py:367(main)
     20/1 0.139 0.007 21.669 21.669
../overlayHists.py:123(process_directory)
        1 0.001 0.001 21.529 21.529
../overlayHists.py:231(process_hist_matplotlib)
        2 0.000 0.000 21.384 10.692
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/pyplot.py:354(savefig) 2 0.000 0.000 21.384
10.692
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/figure.py:959(savefig) 2 0.000 0.000 21.384
10.692
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/backend_bases.py:1372(print_figure) 2 0.000 0.000
21.373 10.687
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/backends/backend_pdf.py:2012(print_pdf) 126/2 0.001
0.000 12.314 6.157
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/artist.py:44(draw_wrapper) 2 0.000 0.000 12.314
6.157
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/figure.py:729(draw) 2 0.000 0.000 12.310 6.155
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/axes.py:1632(draw) 4 0.002 0.000 11.940 2.985
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/axis.py:727(draw) 40 0.005 0.000 11.927 0.298
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/text.py:504(draw) 58 0.023 0.000 10.349 0.178
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/dviread.py:805(find_tex_file) 58 0.008 0.000 9.904
   0.171
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/subproces
s.py:662(communicate) 15059 9.899 0.001 9.899 0.001 {method
'read' of 'file' objects}
        2 0.000 0.000 9.051 4.526
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/backends/backend_pdf.py:447(close) 2 0.001 0.001
9.043 4.521
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/backends/backend_pdf.py:520(writeFonts) 8 0.008 0.001
   9.041 1.130
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/backends/backend_pdf.py:554(embedTeXFont) 8 0.000
0.000 8.637 1.080
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/type1font.py:47(__init__) 304 0.001 0.000 7.796
0.026
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/dviread.py:50(__iter__) 157/152 0.017 0.000 7.780
  0.051
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/dviread.py:114(_read) 5788/3854 0.025 0.000 7.765
  0.002
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/dviread.py:146(_dispatch) 116 0.004 0.000 7.751
0.067
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/backends/backend_pdf.py:1643(get_text_width_height_descent)
116 0.002 0.000 7.711 0.066
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/texmanager.py:576(get_text_width_height_descent) 82
0.007 0.000 7.690 0.094
/Library/Frameworks/Python.framework/Versions/6.0.4/lib/python2.6/site-pack
ages/matplotlib/text.py:250(_get_layout)

--
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################