Wind Barbs Full Patch

Development matplotlib-devel
1

Hi,

As promised, here's a patch to add wind barbs to matplotlib. This should addresses all of the comments I've received as well as all of the issues I previously mentioned. A synopsis of some of the updates:

1) Based on a conversation with Eric Firing, I've gone ahead and added the Barbs collection to quiver.py, which seems like a logical place.

2) Method added to set the data components.

3) Masked arrays are handled by saving all of the original arrays and using copies fed through delete_masked_points whenever the U/V are updated.

4) Setting the color of the edges/line portion of the barbs was easy, thanks to the setting of edgecolor='face' that Eric Firing pointed out. I guess this is a post 0.98.1 addition, since I don't have it in my system copy, only SVN. I also, for some reason, don't remember seeing the patch either. Anyhow, colormapping of the *whole* barb works now.

5) I added an empty circle marker for low wind speeds (vector magnitudes). Accomplishing having the unfilled circle while having the barbs filled involved a bit of a "elegant hack". Using the set of vertices that draws the CirclePolygon, I add an additional copy of these vertices, basically drawing the circle back the other way. This is basically tricking the drawing algorithm into drawing a really thin annulus with a very small gap, but it works perfectly as far as I can tell. It's also somewhat consistent with the way the lines on the barb are drawn. It is *far* simpler than any other solution, which would have required somehow mapping a color to each polygon *before* calling
draw_path_collection(). None of the backends I test had a problem, including PS, PDF, and SVG (tested with Evince, Firefox, and Acroread).

6) The relative sizes of components of the barb can be controlled by
passing a dictionary in through the sizes keyword parameter. The dictionary has keys 'spacing', 'height', 'width', 'emptybarb', which
map to values which are the size of these components relative to the length of the barb. I've also adjust some of the defaults a little so that tweaking shouldn't be needed (such as the issue Jeff raised). This seemed a lot better than passing around all of these parameters separately.

7)Driver demo code was moved to barb_demo.py under the pylab_examples. I loved Jeff's demo, but obviously that can only go in basemap.

The only issue I've seen is that scaling with PS is way too big. I've attached ps and pdf files from the same run to show the problem.

It should apply fine to SVN, but there are commented lines that should be switched with the ones there when set/get_offsets() are added to Collections.

Comments and Suggestions?

How do you guys manage committing only parts of your working copy, especially when you want to commit part of a file? I figure there's got to be a better way than multiple SVN checkouts and manually editing diffs.

Ryan

barbs_patch.diff (18.7 KB)

barbs.pdf (32.9 KB)

barbs.ps (45.5 KB)

···

--
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma

2

The only issue I've seen is that scaling with PS is way too big. I've
attached ps and pdf files from the same run to show the problem.

The only thing I can think of is since you are using a identity
transform and drawing in pixels, you are seeing the effect of the
savefig dpi in pdf and png but not in ps, which hardcodes the dpi to
72. If this is correct, you should not see the effect if you pass
dpi=72 to savefig when saving the PS and PDF. You will probably want
to modify the patch to make sure your barbs scales are dpi
independent. I have only looked briefly at the barbs code so I could
be missing something obvious, but this is the first thing that comes
to mind.

It should apply fine to SVN, but there are commented lines that should be
switched with the ones there when set/get_offsets() are added to
Collections.

Comments and Suggestions?

How do you guys manage committing only parts of your working copy,
especially when you want to commit part of a file? I figure there's got to
be a better way than multiple SVN checkouts and manually editing diffs.

svn should do this automagically; it only commits the diff from your
current working version and the svn HEAD.

svn up

# do some work

···

On Sat, Jul 19, 2008 at 11:09 PM, Ryan May <rmay31@...149...> wrote:

svn diff # these are the changes that will be committed, just preview them
svn commit -m 'my log message' # the diff will be committed

3

John Hunter wrote:

The only issue I've seen is that scaling with PS is way too big. I've
attached ps and pdf files from the same run to show the problem.

The only thing I can think of is since you are using a identity
transform and drawing in pixels, you are seeing the effect of the
savefig dpi in pdf and png but not in ps, which hardcodes the dpi to
72. If this is correct, you should not see the effect if you pass
dpi=72 to savefig when saving the PS and PDF. You will probably want
to modify the patch to make sure your barbs scales are dpi
independent. I have only looked briefly at the barbs code so I could
be missing something obvious, but this is the first thing that comes
to mind.

I don't think an IdentityTransform() implies drawing in pixels. My length 9 barb is a lot longer than 9 pixels. It looks more like ~50. I really was looking for (and thought I found) a way to draw in a resolution independant fashion. Axes coordinates are close, but unfortunately, as you stretch a figure, this can distort things.

It should apply fine to SVN, but there are commented lines that should be
switched with the ones there when set/get_offsets() are added to
Collections.

Comments and Suggestions?

How do you guys manage committing only parts of your working copy,
especially when you want to commit part of a file? I figure there's got to
be a better way than multiple SVN checkouts and manually editing diffs.

svn should do this automagically; it only commits the diff from your
current working version and the svn HEAD.

svn up

# do some work

svn diff # these are the changes that will be committed, just preview them
svn commit -m 'my log message' # the diff will be committed

I'm more interested how you guys handle having multiple lines of development going on in a single working copy, like working on multiple separate additions to axes.py. Trying to commit only a subset of those changes is difficult as far as I can tell. Or is the advice "don't do that" and use separate working copies? What if I'm working on something big and then have a small bug fix to do on the same file? Additional working copies wouldn't be a big deal, but it seems to take forever to do a fresh checkout from sourceforge.

Ryan

···

On Sat, Jul 19, 2008 at 11:09 PM, Ryan May <rmay31@...149...> wrote:

--
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma

4

Ryan May wrote:

John Hunter wrote:

The only issue I've seen is that scaling with PS is way too big. I've
attached ps and pdf files from the same run to show the problem.

The only thing I can think of is since you are using a identity
transform and drawing in pixels, you are seeing the effect of the
savefig dpi in pdf and png but not in ps, which hardcodes the dpi to
72. If this is correct, you should not see the effect if you pass
dpi=72 to savefig when saving the PS and PDF. You will probably want
to modify the patch to make sure your barbs scales are dpi
independent. I have only looked briefly at the barbs code so I could
be missing something obvious, but this is the first thing that comes
to mind.

I don't think an IdentityTransform() implies drawing in pixels. My length 9 barb is a lot longer than 9 pixels. It looks more like ~50. I really was looking for (and thought I found) a way to draw in a resolution independant fashion. Axes coordinates are close, but unfortunately, as you stretch a figure, this can distort things.

It should apply fine to SVN, but there are commented lines that should be
switched with the ones there when set/get_offsets() are added to
Collections.

Comments and Suggestions?

How do you guys manage committing only parts of your working copy,
especially when you want to commit part of a file? I figure there's got to
be a better way than multiple SVN checkouts and manually editing diffs.

svn should do this automagically; it only commits the diff from your
current working version and the svn HEAD.

svn up

# do some work

svn diff # these are the changes that will be committed, just preview them
svn commit -m 'my log message' # the diff will be committed

I'm more interested how you guys handle having multiple lines of development going on in a single working copy, like working on multiple separate additions to axes.py. Trying to commit only a subset of those changes is difficult as far as I can tell. Or is the advice "don't do that" and use separate working copies? What if I'm working on something big and then have a small bug fix to do on the same file? Additional working copies wouldn't be a big deal, but it seems to take forever to do a fresh checkout from sourceforge.

Ryan

I think you could have a master checkout, and then use a local rsync to make copies of it for hacking around on different parts. (This is the sort of thing that is made very fast and easy with mercurial, but the mercurial-svn interface mechanisms seem to be a bit clumsy, unfortunately. Mike recently mentioned doing this sort of thing with git. I haven't looked into git much; it has the reputation of being rather hard to understand, and I have been happily using mercurial for my local work for quite a long time, so I am not eager to start getting confused by an alternative.)

Eric

···

On Sat, Jul 19, 2008 at 11:09 PM, Ryan May <rmay31@...149...> wrote:

5

Ryan May wrote:

5) I added an empty circle marker for low wind speeds (vector magnitudes). Accomplishing having the unfilled circle while having the barbs filled involved a bit of a "elegant hack". Using the set of vertices that draws the CirclePolygon, I add an additional copy of these vertices, basically drawing the circle back the other way. This is basically tricking the drawing algorithm into drawing a really thin annulus with a very small gap, but it works perfectly as far as I can tell. It's also somewhat consistent with the way the lines on the barb are drawn. It is *far* simpler than any other solution, which would have required somehow mapping a color to each polygon *before* calling
draw_path_collection(). None of the backends I test had a problem, including PS, PDF, and SVG (tested with Evince, Firefox, and Acroread).

Having replied before reading all my e-mail, I see you arrived at a similar solution to the one I suggested. Great to hear that it worked.

Cheers,
Mike

6

Michael Droettboom wrote:

Ryan May wrote:

5) I added an empty circle marker for low wind speeds (vector magnitudes). Accomplishing having the unfilled circle while having the barbs filled involved a bit of a "elegant hack". Using the set of vertices that draws the CirclePolygon, I add an additional copy of these vertices, basically drawing the circle back the other way. This is basically tricking the drawing algorithm into drawing a really thin annulus with a very small gap, but it works perfectly as far as I can tell. It's also somewhat consistent with the way the lines on the barb are drawn. It is *far* simpler than any other solution, which would have required somehow mapping a color to each polygon *before* calling
draw_path_collection(). None of the backends I test had a problem, including PS, PDF, and SVG (tested with Evince, Firefox, and Acroread).

Having replied before reading all my e-mail, I see you arrived at a similar solution to the one I suggested. Great to hear that it worked.

I'm just glad to know that it's an accepted hack. :slight_smile:

Ryan

···

--
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma

7

How do you guys manage committing only parts of your working copy,
especially when you want to commit part of a file? I figure there's got to
be a better way than multiple SVN checkouts and manually editing diffs.

svn should do this automagically; it only commits the diff from your
current working version and the svn HEAD.

svn up

# do some work

svn diff # these are the changes that will be committed, just preview them
svn commit -m 'my log message' # the diff will be committed

I'm more interested how you guys handle having multiple lines of development going on in a single working copy, like working on multiple separate additions to axes.py. Trying to commit only a subset of those changes is difficult as far as I can tell. Or is the advice "don't do that" and use separate working copies? What if I'm working on something big and then have a small bug fix to do on the same file? Additional working copies wouldn't be a big deal, but it seems to take forever to do a fresh checkout from sourceforge.

Ryan

I think you could have a master checkout, and then use a local rsync to make copies of it for hacking around on different parts. (This is the sort of thing that is made very fast and easy with mercurial, but the mercurial-svn interface mechanisms seem to be a bit clumsy, unfortunately. Mike recently mentioned doing this sort of thing with git. I haven't looked into git much; it has the reputation of being rather hard to understand, and I have been happily using mercurial for my local work for quite a long time, so I am not eager to start getting confused by an alternative.)

This (and another comment I got in private) was pretty much what I expected. I just wanted to make sure I wasn't missing anything. I've already got two separate checkouts at the moment, so I think I'll try to just keep them up to date and keep one pristine for small stuff and testing. I'll keep rsync in mind however when I need a fresh one, thanks,

Ryan

···

--
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma

8

John Hunter wrote:

The only issue I've seen is that scaling with PS is way too big. I've
attached ps and pdf files from the same run to show the problem.

The only thing I can think of is since you are using a identity
transform and drawing in pixels, you are seeing the effect of the
savefig dpi in pdf and png but not in ps, which hardcodes the dpi to
72. If this is correct, you should not see the effect if you pass
dpi=72 to savefig when saving the PS and PDF. You will probably want
to modify the patch to make sure your barbs scales are dpi
independent. I have only looked briefly at the barbs code so I could
be missing something obvious, but this is the first thing that comes
to mind.

<grumble> Ok, it fixes the problem if I pass dpi=72 to savefig(). Curiously, passing dpi=72 to Figure() does not have the same effect. So now how do I fix it? I'm really not sure what's going wrong here. If I had to guess, it's a problem between figure size being in inches while I'm drawing in pixels (still don't know how that works, because there's no way those barbs are 9 pixels long).

Ideas?

Ryan

···

On Sat, Jul 19, 2008 at 11:09 PM, Ryan May <rmay31@...149...> wrote:

--
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma

9

<grumble> Ok, it fixes the problem if I pass dpi=72 to savefig(). Curiously,
passing dpi=72 to Figure() does not have the same effect. So now how do I

That is because "savefig" has its own dpi, which overrides the figure
dpi. Tee ideas is that you typically want a different dpi for the UI
window and for the harcopy

fix it? I'm really not sure what's going wrong here. If I had to guess,
it's a problem between figure size being in inches while I'm drawing in
pixels (still don't know how that works, because there's no way those barbs
are 9 pixels long).

I'm not familiar enough with the windbarb code to know where the "9
pixel" thing that is bothering you is creeping in, I'm just saying
that using an identity transform means you are drawing in canvas
(pixel) space and not accounting for dpi. The Figure instance has a
"dpi_scale_transform" that is designed to handle dpi scaling, and
updates itself when the figure dpi is changed so you don't have to
handle the callbacks. Take a look at this and see if you can
incorporate it. If you have troubles, Michael or I can advise
further.

If you clarify the "9 pixel" problem that is bothering you, I may be
able to help more sooner...

JDH

···

On Wed, Jul 23, 2008 at 10:05 PM, Ryan May <rmay31@...149...> wrote:

10

John Hunter wrote:

<grumble> Ok, it fixes the problem if I pass dpi=72 to savefig(). Curiously,
passing dpi=72 to Figure() does not have the same effect. So now how do I

That is because "savefig" has its own dpi, which overrides the figure
dpi. Tee ideas is that you typically want a different dpi for the UI
window and for the harcopy

fix it? I'm really not sure what's going wrong here. If I had to guess,
it's a problem between figure size being in inches while I'm drawing in
pixels (still don't know how that works, because there's no way those barbs
are 9 pixels long).

I'm not familiar enough with the windbarb code to know where the "9
pixel" thing that is bothering you is creeping in, I'm just saying
that using an identity transform means you are drawing in canvas
(pixel) space and not accounting for dpi. The Figure instance has a
"dpi_scale_transform" that is designed to handle dpi scaling, and
updates itself when the figure dpi is changed so you don't have to
handle the callbacks. Take a look at this and see if you can
incorporate it. If you have troubles, Michael or I can advise
further.

If you clarify the "9 pixel" problem that is bothering you, I may be
able to help more sooner...

Part of the problem is the horrible and misleading sizes arg in PolyCollections, probably a hangover (yes, a headache) from supporting a Matlab-compatible argument in scatter. I can try to straighten this out and clarify the situation after the release, not before. I will add an alternative kwarg to scatter at the same time, and hope the old usage gradually dies out.

Eric

···

On Wed, Jul 23, 2008 at 10:05 PM, Ryan May <rmay31@...149...> wrote:

JDH

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
matplotlib-devel List Signup and Options

11

Eric Firing wrote:

John Hunter wrote:

<grumble> Ok, it fixes the problem if I pass dpi=72 to savefig(). Curiously,
passing dpi=72 to Figure() does not have the same effect. So now how do I

That is because "savefig" has its own dpi, which overrides the figure
dpi. Tee ideas is that you typically want a different dpi for the UI
window and for the harcopy

fix it? I'm really not sure what's going wrong here. If I had to guess,
it's a problem between figure size being in inches while I'm drawing in
pixels (still don't know how that works, because there's no way those barbs
are 9 pixels long).

I'm not familiar enough with the windbarb code to know where the "9
pixel" thing that is bothering you is creeping in, I'm just saying
that using an identity transform means you are drawing in canvas
(pixel) space and not accounting for dpi. The Figure instance has a
"dpi_scale_transform" that is designed to handle dpi scaling, and
updates itself when the figure dpi is changed so you don't have to
handle the callbacks. Take a look at this and see if you can
incorporate it. If you have troubles, Michael or I can advise
further.

If you clarify the "9 pixel" problem that is bothering you, I may be
able to help more sooner...

Part of the problem is the horrible and misleading sizes arg in PolyCollections, probably a hangover (yes, a headache) from supporting a Matlab-compatible argument in scatter. I can try to straighten this out and clarify the situation after the release, not before. I will add an alternative kwarg to scatter at the same time, and hope the old usage gradually dies out.

That might be part of it, I'm not sure. My problem is that I have length set to 7 in the code. That implies that when I draw, the distance along the y-axis between where I start and where I end is 7 units (which is then rotated). What gets shown on my screen (when the BarbCollection transform is set to IdentityTransform()) is not possibly 7 pixels long. My gnome-panel bar is 22 pixels high, and that looks like a pretty good fit to how long things are actually in the figure I see on screen. This is at my default figure dpi of 72.

What else is confusing is how that relates to DPI. When I change the figure's dpi, using set_dpi, (and redraw), I get physically *bigger* barbs. To me, if I'm actually specifying pixels, there's no way that they should get bigger when I change the DPI.

Then I also can't figure out what the PS backend is doing. If PS is hardcoded to 72 DPI, why does passing dpi=72 to savefig() have any effect?

I found the fig.dpi_scale_trans. However, this then makes things too big on the on screen figure when I use this as the tranform for the BarbCollection.

Thoughts?

Ryan

···

On Wed, Jul 23, 2008 at 10:05 PM, Ryan May <rmay31@...149...> wrote:

--
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma

12

What else is confusing is how that relates to DPI. When I change the
figure's dpi, using set_dpi, (and redraw), I get physically *bigger* barbs.
To me, if I'm actually specifying pixels, there's no way that they should
get bigger when I change the DPI.

When you increase the dpi, the canvas gets bigger (inches*dpi equals
canvas size in pixels). If you are drawing in pixels, and not
scaling, the barbs should look smaller, since they are a smaller
proportion of the canvas size. So if this explanation is right, the
barbs will look smaller with larger dpi.

Then I also can't figure out what the PS backend is doing. If PS is
hardcoded to 72 DPI, why does passing dpi=72 to savefig() have any effect?

ps should be unaffected, but dpi dependent backends will. By setting
the dpi to be 72, it should make the *other* backend look like the ps
backend and the ps backend should be unaffected.

JDH

···

On Thu, Jul 24, 2008 at 10:00 PM, Ryan May <rmay31@...149...> wrote:

13

John Hunter wrote:

What else is confusing is how that relates to DPI. When I change the
figure's dpi, using set_dpi, (and redraw), I get physically *bigger* barbs.
To me, if I'm actually specifying pixels, there's no way that they should
get bigger when I change the DPI.

Ryan, I think you are giving the length in points (sort of, because of the bizarre size kwarg), not pixels. So you are right--the barbs are not 7 pixels long. The translation from points to pixels for the Agg backend depends on the dpi, as John says below.

When you increase the dpi, the canvas gets bigger (inches*dpi equals
canvas size in pixels). If you are drawing in pixels, and not
scaling, the barbs should look smaller, since they are a smaller
proportion of the canvas size. So if this explanation is right, the
barbs will look smaller with larger dpi.

Then I also can't figure out what the PS backend is doing. If PS is
hardcoded to 72 DPI, why does passing dpi=72 to savefig() have any effect?

ps should be unaffected, but dpi dependent backends will. By setting
the dpi to be 72, it should make the *other* backend look like the ps
backend and the ps backend should be unaffected.

I don't think so, unless I am misunderstanding your phrase "make the *other* backend look like the ps backend". If you pass 72 dpi to the Agg backend, you will get a relatively small number of pixels, and when you display it at natural size on the screen, it will be tiny--only a little more than half-size on my laptop screen, for example. Whereas, if the ps renderer knows the screen dpi, then it will display full-size.

···

On Thu, Jul 24, 2008 at 10:00 PM, Ryan May <rmay31@...149...> wrote:

JDH

14

Eric Firing wrote:

John Hunter wrote:

What else is confusing is how that relates to DPI. When I change the
figure's dpi, using set_dpi, (and redraw), I get physically *bigger* barbs.
To me, if I'm actually specifying pixels, there's no way that they should
get bigger when I change the DPI.

Ryan, I think you are giving the length in points (sort of, because of the bizarre size kwarg), not pixels. So you are right--the barbs are not 7 pixels long. The translation from points to pixels for the Agg backend depends on the dpi, as John says below.

Now I know what you were saying, I was too braindead to understand last night. I am drawing in points (or whatever scatter uses for size), because I modeled my code off of scatter. Scatter also uses an Identity Transform() for it's Collection objects (which is where I got the idea). Not coincidentally, it shows a similar issue. While it's not as bad as my barbs issue (for reasons unknown to me), you can change the size of the scatter object relative to the size of the canvas just by changing the dpi passed to savefig. Again, it takes passing in dpi=72 to get a figure to match what's on screen, even though 72 is supposedly what's hard coded.

Ryan

···

On Thu, Jul 24, 2008 at 10:00 PM, Ryan May <rmay31@...149...> wrote:

--
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma

15

John Hunter wrote:

What else is confusing is how that relates to DPI. When I change the
figure's dpi, using set_dpi, (and redraw), I get physically *bigger* barbs.
To me, if I'm actually specifying pixels, there's no way that they should
get bigger when I change the DPI.

When you increase the dpi, the canvas gets bigger (inches*dpi equals
canvas size in pixels). If you are drawing in pixels, and not
scaling, the barbs should look smaller, since they are a smaller
proportion of the canvas size. So if this explanation is right, the
barbs will look smaller with larger dpi.

Then I also can't figure out what the PS backend is doing. If PS is
hardcoded to 72 DPI, why does passing dpi=72 to savefig() have any effect?

ps should be unaffected, but dpi dependent backends will. By setting
the dpi to be 72, it should make the *other* backend look like the ps
backend and the ps backend should be unaffected.

Ok, I think I found our problem, at line 859 of backend_ps.py (inside _print_ps()):

         self.figure.set_dpi(72) # Override the dpi kwarg
         dpi = kwargs.get("dpi", 72)

The problem here is that while it sets the figure dpi here to 72, it's using the dpi that's passed in down the chain. Since I don't give it a dpi explicity, it grabs the default dpi from my matplotlibrc, which has it set to 300 dpi. So 300 is getting passed down into the chain and I believe the drawing commands are using 300 dpi. If I change the second line above to dpi = 72, I get the proper results.

So what's the proper fix in this case, or is this it? It'd probably be good to get this in for a release.

There were a couple other things I noticed:

1) The canvas.print_figure() method goes to the trouble of saving the facecolor and edgecolor before resetting them to what's passed in for purposes of writing out the image. Then later, backend_ps._print_figure() does the same thing a few calls down the stack. Is one of these redundant, or is there a reason for the duplication?

2) backend_ps._print_figure() uses the md5 module to create a temporary filename. This module is deprecated in python 2.5 and removed (I believe) in 3.0, replaced by hashlib. Is there any opposition to changing the direct use of md5.md5() to using a try...except to import md5() from it's proper place?

Ryan

···

On Thu, Jul 24, 2008 at 10:00 PM, Ryan May <rmay31@...149...> wrote:

--
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma

16

Ok, I think I found our problem, at line 859 of backend_ps.py (inside
_print_ps()):

       self.figure.set_dpi(72) # Override the dpi kwarg
       dpi = kwargs.get("dpi", 72)

The problem here is that while it sets the figure dpi here to 72, it's using
the dpi that's passed in down the chain. Since I don't give it a dpi
explicity, it grabs the default dpi from my matplotlibrc, which has it set
to 300 dpi. So 300 is getting passed down into the chain and I believe the
drawing commands are using 300 dpi. If I change the second line above to
dpi = 72, I get the proper results.

Yes, we definitely need to get this fixed before any release, but we
need to be careful here. I had forgotten that Nicholas Young had
submitted a patch to make the ps backend respect higher resolutions
for embedded images

  [matplotlib-devel] Using selected dpi for PS output

so we cannot simply revert this to 72. I also see the problem you
pointed out in the collections module too. While I don't have a quick
fix at hand, at least we are starting to converge onto where the
problems lie. Note that in the collections module the
IdentityTransform is bit of a red herring in the set_transforms
method, since the collection maintains a list of transforms in
_transforms that incorporates a dpi a draw time so I don't thing the
standard artist transform property is used.

2) backend_ps._print_figure() uses the md5 module to create a temporary
filename. This module is deprecated in python 2.5 and removed (I believe)
in 3.0, replaced by hashlib. Is there any opposition to changing the direct
use of md5.md5() to using a try...except to import md5() from it's proper
place?

I wasn't aware of the deprecation so I don't have any strong opinion,
except that it wold be better if you can find a non-deprecated
equivalent which is python 2.4 and 2.5 compatible.

JDH

···

On Fri, Jul 25, 2008 at 5:47 PM, Ryan May <rmay31@...149...> wrote:

17

Ryan May wrote:

John Hunter wrote:

What else is confusing is how that relates to DPI. When I change the
figure's dpi, using set_dpi, (and redraw), I get physically *bigger* barbs.
To me, if I'm actually specifying pixels, there's no way that they should
get bigger when I change the DPI.

When you increase the dpi, the canvas gets bigger (inches*dpi equals
canvas size in pixels). If you are drawing in pixels, and not
scaling, the barbs should look smaller, since they are a smaller
proportion of the canvas size. So if this explanation is right, the
barbs will look smaller with larger dpi.

Then I also can't figure out what the PS backend is doing. If PS is
hardcoded to 72 DPI, why does passing dpi=72 to savefig() have any effect?

ps should be unaffected, but dpi dependent backends will. By setting
the dpi to be 72, it should make the *other* backend look like the ps
backend and the ps backend should be unaffected.

Ok, I think I found our problem, at line 859 of backend_ps.py (inside _print_ps()):

        self.figure.set_dpi(72) # Override the dpi kwarg
        dpi = kwargs.get("dpi", 72)

The problem here is that while it sets the figure dpi here to 72, it's using the dpi that's passed in down the chain. Since I don't give it a dpi explicity, it grabs the default dpi from my matplotlibrc, which has it set to 300 dpi. So 300 is getting passed down into the chain and I believe the drawing commands are using 300 dpi. If I change the second line above to dpi = 72, I get the proper results.

So what's the proper fix in this case, or is this it? It'd probably be good to get this in for a release.

This is the kind of thing I would want to look at and test *very* carefully, or know that someone like John or Mike has done so--which perhaps one of them can do quickly. I have looked at this dpi business with puzzlement before; I don't have it all straight in my head; and I would need a chunk of time to review it, which I might get in the next day or so, but can't guarantee.

It would be nice to not only fix the bug, if such there is, but also clarify the whole mess for all backends, maybe with some additional variable names. I think that for pixel backends, there is a single dpi per rendered figure, so the situation is simple; but for vector backends (ps, pdf, svg), there are fundamentally two: one that is used for translating dots to inches as measures of length, and another that sets the dpi (hence resolution, not size) for raster images that are contained within the figure. (I don't know where the cairo backend sits, since it can render either raster or vector file types.)

Another aspect of the problem is that at least for use with vector backends, specifying lengths in dots is unnatural; and it seems to me like something to be avoided when possible even for raster backends. I would argue that pad variables should be in physical or relative units, where relative could mean relative to the figure size, or to a font em, for example. Specifying lengths in dots is just asking for trouble except when the plot is not intended to be scaled; when making a small png for the web, precise control via lengths in dots may be helpful.

Eric

···

On Thu, Jul 24, 2008 at 10:00 PM, Ryan May <rmay31@...149...> wrote:

There were a couple other things I noticed:

1) The canvas.print_figure() method goes to the trouble of saving the facecolor and edgecolor before resetting them to what's passed in for purposes of writing out the image. Then later, backend_ps._print_figure() does the same thing a few calls down the stack. Is one of these redundant, or is there a reason for the duplication?

2) backend_ps._print_figure() uses the md5 module to create a temporary filename. This module is deprecated in python 2.5 and removed (I believe) in 3.0, replaced by hashlib. Is there any opposition to changing the direct use of md5.md5() to using a try...except to import md5() from it's proper place?

Ryan

18

John Hunter wrote:

Ok, I think I found our problem, at line 859 of backend_ps.py (inside
_print_ps()):

       self.figure.set_dpi(72) # Override the dpi kwarg
       dpi = kwargs.get("dpi", 72)

The problem here is that while it sets the figure dpi here to 72, it's using
the dpi that's passed in down the chain. Since I don't give it a dpi
explicity, it grabs the default dpi from my matplotlibrc, which has it set
to 300 dpi. So 300 is getting passed down into the chain and I believe the
drawing commands are using 300 dpi. If I change the second line above to
dpi = 72, I get the proper results.

Yes, we definitely need to get this fixed before any release, but we
need to be careful here. I had forgotten that Nicholas Young had
submitted a patch to make the ps backend respect higher resolutions
for embedded images

  [matplotlib-devel] Using selected dpi for PS output

so we cannot simply revert this to 72. I also see the problem you
pointed out in the collections module too. While I don't have a quick
fix at hand, at least we are starting to converge onto where the
problems lie. Note that in the collections module the
IdentityTransform is bit of a red herring in the set_transforms
method, since the collection maintains a list of transforms in
_transforms that incorporates a dpi a draw time so I don't thing the
standard artist transform property is used.

I was alluding to this in my earlier rants about scatter and PolyCollection. The mechanism of the list of transforms is fine, but the kwarg name and API are bad, and additional comments are needed. I was going to hold off until after the release, but maybe I should go ahead and try to improve this, in a 100% backwards-compatible way, as soon as I can get to it.

2) backend_ps._print_figure() uses the md5 module to create a temporary
filename. This module is deprecated in python 2.5 and removed (I believe)
in 3.0, replaced by hashlib. Is there any opposition to changing the direct
use of md5.md5() to using a try...except to import md5() from it's proper
place?

I wasn't aware of the deprecation so I don't have any strong opinion,
except that it wold be better if you can find a non-deprecated
equivalent which is python 2.4 and 2.5 compatible.

If it is a temporary file, why not use one of the functions in the tempfile module? I have not looked at the code to see whether one of them would be appropriate, but it seems odd to have to use some other module.

Eric

···

On Fri, Jul 25, 2008 at 5:47 PM, Ryan May <rmay31@...149...> wrote:

19

This is the kind of thing I would want to look at and test *very* carefully,
or know that someone like John or Mike has done so--which perhaps one of
them can do quickly. I have looked at this dpi business with puzzlement
before; I don't have it all straight in my head; and I would need a chunk of
time to review it, which I might get in the next day or so, but can't
guarantee.

I just committed a fix here -- I renamed the renderer dpi instance to
imagedpi to avoid confusion, and modified the various places in the
artist code (collections, text) which were using renderer.dpi (they
now use figure.dpi). Right now the only backend that was using the
image magnification was backend ps.

Another aspect of the problem is that at least for use with vector backends,
specifying lengths in dots is unnatural; and it seems to me like something
to be avoided when possible even for raster backends. I would argue that
pad variables should be in physical or relative units, where relative could
mean relative to the figure size, or to a font em, for example. Specifying
lengths in dots is just asking for trouble except when the plot is not
intended to be scaled; when making a small png for the web, precise control
via lengths in dots may be helpful.

Yes, certainly points or some other physical dimension is the right
way to specify pads. Where in the code are dots/pixels used?

JDH

···

On Fri, Jul 25, 2008 at 6:42 PM, Eric Firing <efiring@...229...> wrote:

20

John Hunter wrote:

Ok, I think I found our problem, at line 859 of backend_ps.py (inside
_print_ps()):

       self.figure.set_dpi(72) # Override the dpi kwarg
       dpi = kwargs.get("dpi", 72)

The problem here is that while it sets the figure dpi here to 72, it's using
the dpi that's passed in down the chain. Since I don't give it a dpi
explicity, it grabs the default dpi from my matplotlibrc, which has it set
to 300 dpi. So 300 is getting passed down into the chain and I believe the
drawing commands are using 300 dpi. If I change the second line above to
dpi = 72, I get the proper results.

Yes, we definitely need to get this fixed before any release, but we
need to be careful here. I had forgotten that Nicholas Young had
submitted a patch to make the ps backend respect higher resolutions
for embedded images

  [matplotlib-devel] Using selected dpi for PS output

so we cannot simply revert this to 72. I also see the problem you
pointed out in the collections module too. While I don't have a quick
fix at hand, at least we are starting to converge onto where the
problems lie. Note that in the collections module the
IdentityTransform is bit of a red herring in the set_transforms
method, since the collection maintains a list of transforms in
_transforms that incorporates a dpi a draw time so I don't thing the
standard artist transform property is used.

Yeah, I figured it was more complex thatn that. While the IdentityTransform() might be a bit of a red herring, the fact remains that changing the dpi still changes the size of the of the marker relative to the canvas in a scatter plot saved with backend_ps, which doesn't occur with the image backends (at least I tested for png).

2) backend_ps._print_figure() uses the md5 module to create a temporary
filename. This module is deprecated in python 2.5 and removed (I believe)
in 3.0, replaced by hashlib. Is there any opposition to changing the direct
use of md5.md5() to using a try...except to import md5() from it's proper
place?

I wasn't aware of the deprecation so I don't have any strong opinion,
except that it wold be better if you can find a non-deprecated
equivalent which is python 2.4 and 2.5 compatible.

Therein lies the problem, 2.5 deprecated md5 in favor of hashlib, which was added in 2.5. So the options are:

1)Do nothing now. Go back and fix the problem when we get around to
   supporting 3.x. md5 has officially been removed from SVN for python
   3.0.
2)Move completely over to hashlib, drop support for <2.5
3)try to import md5() from hashlib, if that fails fall back to importing
   md5() from the md5 module.

Option 2 is not really an option, so never mind it. Option 1 is the status quo, and it just means we need to keep it in mind as one of the issues that will have to be handled later when we move to 3.x. I just read that in 2.6, md5 will issue a DeprecationWarning. That probably kicks us to option 3.

Option 3 isn't too bad. Both the md5 objects from hashlib and from the md5 module support the same API. Hashlib lacks a module level function new() which md5 has, but this is just an alternative to using a class constructor, so code that uses new() (backend_svg) is easily moved over to code that works for both.

I'll volunteer to do the ports, which will mostly consist of:

try:
     from hashlib import md5
except ImportError:
     from md5 import md5

Thoughts?

Ryan

···

On Fri, Jul 25, 2008 at 5:47 PM, Ryan May <rmay31@...149...> wrote:

--
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma