Bug in Agg-based backends when yscale == "log"?

Hi,

The simple code snippet at the end of this mail should plot a single line.

Unfortunately, depending on
- the backend
- the windowsize
- and the pan/zoom position inside the plot
one or more additional lines appear.

Under windows it looks like this:

http://img217.imageshack.us/my.php?image=matplotlibproblemei6.png
(disable Adblock Plus on Firefox, otherwise the image might not be visible)

When I pan the plot, the additional lines jump around randomly and sometimes dis- and reappear at their own will.

I get this problem for all the AGG-based backends (one additional line) and the GTK backend (several additional lines). GTKCairo does not show this behavior.

The problem appears under an up-to-date Fedora 64-bit with matplotlib 0.98.3 and 0.89.6 SVN (from today)

The output of "python setup.py" is this:

···

------------------
BUILDING MATPLOTLIB
matplotlib: 0.98.6svn
python: 2.5.2 (r252:60911, Sep 30 2008, 15:42:03) [GCC 4.3.2 20080917 (Red Hat 4.3.2-4)]
platform: linux2

REQUIRED DEPENDENCIES
numpy: 1.2.0
freetype2: 9.18.3

OPTIONAL BACKEND DEPENDENCIES
libpng: 1.2.33
Tkinter: no
wxPython: 2.8.9.1
      * WxAgg extension not required for wxPython >= 2.8
Gtk+: gtk+: 2.14.5, glib: 2.18.3, pygtk: 2.13.0,
      pygobject: 2.15.4
Qt4: Qt: 4.4.3, PyQt4: 4.4.4
Cairo: 1.4.12

OPTIONAL DATE/TIMEZONE DEPENDENCIES
datetime: present, version unknown
dateutil: 1.4
pytz: 2008i

OPTIONAL USETEX DEPENDENCIES
dvipng: no
ghostscript: 8.63
latex: no
------------------

Additionally I see this problem under 32-bit Windows XP using the Enthought EPD Py25 v4.1.30101 distribution which uses matplotlib 0.98.3

To reproduce the error:
- switch to GTKAgg
- run the script below
- enlarge or maximize the window (something larger than 1280*1024 should be fine)
- play around with zoom/pan

Expected result:
- a single line is plotted

Actually result:
- two or more lines are plotted

If you need more information just contact me,
Jan

-----------------------------------------

import numpy as np
import matplotlib.pyplot as plt

E = np.array((
    1.00000000e+00, 1.50000000e+00, 2.00000000e+00, 3.00000000e+00,
    4.00000000e+00, 5.00000000e+00, 6.00000000e+00, 8.00000000e+00,
    1.00000000e+01, 1.50000000e+01, 2.00000000e+01, 3.00000000e+01,
    4.00000000e+01, 5.00000000e+01, 6.00000000e+01, 8.00000000e+01,
    1.00000000e+02, 1.50000000e+02, 2.00000000e+02, 3.00000000e+02,
    
    4.00000000e+02, 5.00000000e+02, 6.00000000e+02, 8.00000000e+02,
    1.00000000e+03, 1.02200000e+03, 1.25000000e+03, 1.50000000e+03,
    2.00000000e+03, 2.04400000e+03, 3.00000000e+03, 4.00000000e+03,
    5.00000000e+03, 6.00000000e+03, 7.00000000e+03, 8.00000000e+03,
    9.00000000e+03, 1.00000000e+04, 1.10000000e+04, 1.20000000e+04,
    
    1.30000000e+04, 1.40000000e+04, 1.50000000e+04, 1.60000000e+04,
    1.80000000e+04, 2.00000000e+04, 2.20000000e+04, 2.40000000e+04,
    2.60000000e+04, 2.80000000e+04, 3.00000000e+04, 4.00000000e+04,
    5.00000000e+04, 6.00000000e+04, 8.00000000e+04, 1.00000000e+05,
    1.50000000e+05, 2.00000000e+05, 3.00000000e+05, 4.00000000e+05,
    
    5.00000000e+05, 6.00000000e+05, 8.00000000e+05, 1.00000000e+06,
    1.50000000e+06, 2.00000000e+06, 3.00000000e+06, 4.00000000e+06,
    5.00000000e+06, 6.00000000e+06, 8.00000000e+06, 1.00000000e+07,
    1.50000000e+07, 2.00000000e+07, 3.00000000e+07, 4.00000000e+07,
    5.00000000e+07, 6.00000000e+07, 8.00000000e+07, 1.00000000e+08))
   
att = np.array((
    6.81740051e+00, 1.75185086e+00, 6.63815247e-01, 1.67656668e-01,
    6.29160626e-02, 2.93190047e-02, 1.56961535e-02, 5.86499592e-03,
    2.72337524e-03, 6.73972636e-04, 2.49871770e-04, 6.16613263e-05,
    2.28421754e-05, 1.05816094e-05, 5.64930077e-06, 2.10496950e-06,
    9.82279267e-07, 2.49513274e-07, 9.62561983e-08, 2.63733618e-08,
    
    1.11014287e-08, 5.93131769e-09, 3.67936480e-09, 1.86298464e-09,
    1.17168470e-09, 1.12328773e-09, 7.79131487e-10, 5.81480647e-10,
    3.70505702e-10, 3.58735080e-10, 2.10556699e-10, 1.46027404e-10,
    1.11492282e-10, 9.01020155e-11, 7.55231748e-11, 6.50072897e-11,
    5.70367268e-11, 5.08048699e-11, 4.57978746e-11, 4.16871195e-11,
    
    3.82455571e-11, 3.53357639e-11, 3.28322663e-11, 3.06573900e-11,
    2.70724292e-11, 2.42403101e-11, 2.19399603e-11, 2.00399310e-11,
    1.84446235e-11, 1.70823384e-11, 1.59052762e-11, 1.18363457e-11,
    9.42247205e-12, 7.82716448e-12, 5.84766861e-12, 4.66702151e-12,
    3.10158861e-12, 2.32245712e-12, 1.54571561e-12, 1.15853984e-12,
    
    9.26114881e-13, 7.71364072e-13, 5.78493179e-13, 4.62698944e-13,
    3.08366381e-13, 2.31229973e-13, 1.54153316e-13, 1.15555237e-13,
    9.24919894e-14, 7.70766578e-14, 5.77835936e-14, 4.62280699e-14,
    3.08187133e-14, 2.31110475e-14, 1.54093566e-14, 1.15555237e-14,
    9.24322401e-15, 7.70169085e-15, 5.77776187e-15, 4.62220950e-15))

plt.figure()
plt.plot(E,att)

plt.yscale("log")
plt.xscale("linear")

plt.xlim(xmin=np.log(20), xmax=np.log(500))

plt.ylim(ymin=-18,ymax=5)

plt.show()

Jan M�ller wrote:

Hi,

The simple code snippet at the end of this mail should plot a single line.

I can confirm this bug on Ubuntu running matplotlib svn revision 6827. However I think it doesn't have to do with the log-scale but with the big variations on the x-scale and the custom xscale. I've reproduced a similar behavior with the following script (pan and zoom to see the buggy behavior).

···

--------------------

import numpy as np
import matplotlib.pyplot as plt

x = np.array([1.0,2.0,3.0,1.0E5,2.0E5])
y = np.arange(len(x))
plt.plot(x,y)
plt.xlim(xmin=2,xmax=6)
plt.show()

--------------------

Best Regards,
Jo�o Silva

Thanks for this.

I believe both of these examples illustrate a shortcoming in Agg when the distance between two points on either end of a line is too great.

I'll do some digging around and see what may be causing this and if any limits can be adjusted -- I may not get to this today, however.

Mike

Jo�o Lu�s Silva wrote:

···

Jan M�ller wrote:
  

Hi,

The simple code snippet at the end of this mail should plot a single line.

I can confirm this bug on Ubuntu running matplotlib svn revision 6827. However I think it doesn't have to do with the log-scale but with the big variations on the x-scale and the custom xscale. I've reproduced a similar behavior with the following script (pan and zoom to see the buggy behavior).
--------------------

import numpy as np
import matplotlib.pyplot as plt

x = np.array([1.0,2.0,3.0,1.0E5,2.0E5])
y = np.arange(len(x))
plt.plot(x,y)
plt.xlim(xmin=2,xmax=6)
plt.show()

--------------------

Best Regards,
Jo�o Silva

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
  
--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

Okay -- I think I've at least narrowed it down to a cause. Agg uses fixed-point arithmetic to render at the low-level -- by default it uses 24.8 (i.e. 24 integer bits and 8 fractional bits). Therefore, it can only handle pixel coordinates in the range -2^23 to 2^23. Both of the provided examples, after the data has been scaled, draw outside of this range, which results in integer overflow, hence things going in the wrong direction etc.

We could change the fixed point in agg_basics.h, but I hesitate to do so, as it's at the expense of fine detail. We could possibly move to 64-bits, but I'm not sure how easy that would be, or what the impact might be on 32-bit platforms.

    //----------------------------------------------------poly_subpixel_scale_e
    // These constants determine the subpixel accuracy, to be more precise,
    // the number of bits of the fractional part of the coordinates.
    // The possible coordinate capacity in bits can be calculated by formula:
    // sizeof(int) * 8 - poly_subpixel_shift, i.e, for 32-bit integers and
    // 8-bits fractional part the capacity is 24 bits.
    enum poly_subpixel_scale_e
    {
        poly_subpixel_shift = 8, //----poly_subpixel_shift
        poly_subpixel_scale = 1<<poly_subpixel_shift, //----poly_subpixel_scale
        poly_subpixel_mask = poly_subpixel_scale-1, //----poly_subpixel_mask
    };

One thing I will look into is whether the line simplification algorithm can be extended to actually clip the lines when they go outside of the image range. At the moment, it does some work to reduce the number of points outside of the image, but it always draws at least one point outside at its original location. It looks like Agg has some of the pieces necessary to do this -- whether it's feasible to integrate that into our existing line simplification algorithm remains to be seen.

Mike

Michael Droettboom wrote:

···

Thanks for this.

I believe both of these examples illustrate a shortcoming in Agg when the distance between two points on either end of a line is too great.

I'll do some digging around and see what may be causing this and if any limits can be adjusted -- I may not get to this today, however.

Mike

Jo�o Lu�s Silva wrote:
  

Jan M�ller wrote:
  

Hi,

The simple code snippet at the end of this mail should plot a single line.

I can confirm this bug on Ubuntu running matplotlib svn revision 6827. However I think it doesn't have to do with the log-scale but with the big variations on the x-scale and the custom xscale. I've reproduced a similar behavior with the following script (pan and zoom to see the buggy behavior).
--------------------

import numpy as np
import matplotlib.pyplot as plt

x = np.array([1.0,2.0,3.0,1.0E5,2.0E5])
y = np.arange(len(x))
plt.plot(x,y)
plt.xlim(xmin=2,xmax=6)
plt.show()

--------------------

Best Regards,
Jo�o Silva

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
  
--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA

I just completed some open-heart surgery on the path simplification code to resolve this and other issues. Though the integer overflow bug is on the maintenance branch, the proper solution was large enough and risky enough that I have only committed it to the development trunk.

Before, the path simplification code jumbled a number of discrete steps together in the code:

   1. Nan (nonfinite) handling
   2. Clipping
   3. Quantization
   4. Simplification

This has been rewritten (see path_converters.h) so each step is handled by a separate iterator class that can be independently switched on-and-off.

This refactoring made it much easier to rewrite the clipping algorithm to actually bisect line segments at the figure boundary, rather than (as before) simply removing vertices after crossing outside the boundary.

Additionally, nan-value-handling was formerly handled in both Python and C++, which was both double the maintenance and slower in Python. Now, the C++ version of the entire pipeline has been exposed to Python, so we have a single (and faster) code path to debug. (Note that whereas it behaves as an iterator in C++, it actually writes to a new array in the Python-wrapped version to avoid lots of tiny Python function calls). All backends now use this pipeline (through the use of Path.iter_segments).

A side effect of this is that whereas before Python backends were forced to get clipping and simplification together, they now have independent control. This fixes two long-standing bugs in non-Agg backends:

   1. large values would cause integer overflow (causes the lines to
      appear to go in the wrong direction)
   2. clipping should be turned off for filled regions, but that
      wasn't previously possible without also losing simplification

Lastly, the threshold of angular similarity below which simplification will start removing vertices has been exposed to the user as an rcParam (path.simplify_threshold). It can also be set on an individual basis to any Path object.

The one remaining major piece is to get the native Cocoa backend to support this infrastructure. Right now, it reimplements its own nan-handling and doesn't perform any of the other steps. This will probably require some code compiled in C++ but exported as C to make it accessible to Objective-C. The general roadmap is in "cleanup_path" in _path.cpp. I'm happy to help with this, but without a Mac, it will be hard to compile and test these changes.

I've looked through all the backend_driver images, and everything seems ok, but let me know if you see any strangely drawn paths etc.

Cheers,
Mike

Michael Droettboom wrote:

···

Okay -- I think I've at least narrowed it down to a cause. Agg uses fixed-point arithmetic to render at the low-level -- by default it uses 24.8 (i.e. 24 integer bits and 8 fractional bits). Therefore, it can only handle pixel coordinates in the range -2^23 to 2^23. Both of the provided examples, after the data has been scaled, draw outside of this range, which results in integer overflow, hence things going in the wrong direction etc.

We could change the fixed point in agg_basics.h, but I hesitate to do so, as it's at the expense of fine detail. We could possibly move to 64-bits, but I'm not sure how easy that would be, or what the impact might be on 32-bit platforms.

    //----------------------------------------------------poly_subpixel_scale_e
    // These constants determine the subpixel accuracy, to be more precise,
    // the number of bits of the fractional part of the coordinates.
    // The possible coordinate capacity in bits can be calculated by formula:
    // sizeof(int) * 8 - poly_subpixel_shift, i.e, for 32-bit integers and
    // 8-bits fractional part the capacity is 24 bits.
    enum poly_subpixel_scale_e
    {
        poly_subpixel_shift = 8, //----poly_subpixel_shift
        poly_subpixel_scale = 1<<poly_subpixel_shift, //----poly_subpixel_scale
        poly_subpixel_mask = poly_subpixel_scale-1, //----poly_subpixel_mask
    };

One thing I will look into is whether the line simplification algorithm can be extended to actually clip the lines when they go outside of the image range. At the moment, it does some work to reduce the number of points outside of the image, but it always draws at least one point outside at its original location. It looks like Agg has some of the pieces necessary to do this -- whether it's feasible to integrate that into our existing line simplification algorithm remains to be seen.

Mike

Michael Droettboom wrote:
  

Thanks for this.

I believe both of these examples illustrate a shortcoming in Agg when the distance between two points on either end of a line is too great.

I'll do some digging around and see what may be causing this and if any limits can be adjusted -- I may not get to this today, however.

Mike

Jo�o Lu�s Silva wrote:
  

Jan M�ller wrote:
  

Hi,

The simple code snippet at the end of this mail should plot a single line.

I can confirm this bug on Ubuntu running matplotlib svn revision 6827. However I think it doesn't have to do with the log-scale but with the big variations on the x-scale and the custom xscale. I've reproduced a similar behavior with the following script (pan and zoom to see the buggy behavior).
--------------------

import numpy as np
import matplotlib.pyplot as plt

x = np.array([1.0,2.0,3.0,1.0E5,2.0E5])
y = np.arange(len(x))
plt.plot(x,y)
plt.xlim(xmin=2,xmax=6)
plt.show()

--------------------

Best Regards,
Jo�o Silva

------------------------------------------------------------------------------
This SF.net email is sponsored by:
SourcForge Community
SourceForge wants to tell your story.
http://p.sf.net/sfu/sf-spreadtheword
_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
  
--
Michael Droettboom
Science Software Branch
Operations and Engineering Division
Space Telescope Science Institute
Operated by AURA for NASA