Weird KnownFailure problem in IPython test suite with mpl master

Hi folks,

I'm wondering if the following rings any bells for you... Right now,
on an ubuntu 13.04 machine, if I install mpl master (say to my home
directory), the IPython test suite fails here:

iptest -vx IPython.core.tests.test_run

...

···

======================================================================
ERROR: Test that namespace cleanup is not too aggressive GH-238
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/numpy/testing/decorators.py",
line 213, in knownfailer
    raise KnownFailureTest(msg)
KnownFailureTest: This test is known to fail

If I uninstall mpl from master and revert to the system version
(1.2.1), then it's all OK. We'd seen this problem a few years ago,
but in that case it was the opposite: mpl master was OK and the issue
was mis-packaging by Ubuntu:

https://github.com/ipython/ipython/issues/823
https://bugs.launchpad.net/ubuntu/+source/matplotlib/+bug/871176

But now the problem appears to be caused by mpl master. Can you think
of any recent changes to mpl in how you load your nose plugins that
could be causing this?

Since SciPy'13 is coming and I know Mike will be there, I'm happy to
try to debug this face to face in Austin. I just wanted to put it on
your radar, in case it's an easy fix (esp. if it's one you can apply
before 1.3.0 goes out).

Cheers,

f
--
Fernando Perez (@fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail

Fernando,

Does the same thing happen with the v1.3.x branch? You said you tested
master, but that isn't exactly the same as v1.3.x.

Cheers!
Ben Root

···

On Tue, Jun 18, 2013 at 4:19 PM, Fernando Perez <fperez.net@...149...>wrote:

Hi folks,

<snip>

Since SciPy'13 is coming and I know Mike will be there, I'm happy to
try to debug this face to face in Austin. I just wanted to put it on
your radar, in case it's an easy fix (esp. if it's one you can apply
before 1.3.0 goes out).

Cheers,

Hi Ben,

Does the same thing happen with the v1.3.x branch? You said you tested
master, but that isn't exactly the same as v1.3.x.

I just tested with the v1.3.0rc3, and the problem is present there:

((v1.3.0rc3))longs[matplotlib]> iptest -vx IPython.core.tests.test_run
Check that %run doesn't damage __builtins__ ... ok
Check that the type of __builtins__ doesn't change with %run. ... ok
Test that prompts correctly generate after %run ... ok
Test that the option -p, which invokes the profiler, do not ... ok
Test that namespace cleanup is not too aggressive GH-238 ... ERROR

···

On Tue, Jun 18, 2013 at 1:29 PM, Benjamin Root <ben.root@...553...> wrote:

======================================================================
ERROR: Test that namespace cleanup is not too aggressive GH-238
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/usr/lib/python2.7/dist-packages/numpy/testing/decorators.py",
line 213, in knownfailer
    raise KnownFailureTest(msg)
KnownFailureTest: This test is known to fail

----------------------------------------------------------------------
Ran 5 tests in 0.006s

FAILED (errors=1)

Cheers,

f

This was an attempt to fix a bug that mpl's KnownFailure plugin wouldn't load when running tests directly using the nosetests commandline script. I see IPython has a testing wrapper script (iptest) -- is that in part to solve that problem?

In any case, the revert should be simple -- can you try commenting out the "entry_points" kwarg at the bottom of the setup.py script? (You'll probably need to blitz the matplotlib installation directory and `build` for good measure). I can't actually reproduce the bug myself, but I suspect that's because this is somewhat dependent on the order in which things are installed into the virtualenv or the phases of the moon...

If that works for you, we can just take that out and require testers to use our testing script (and unfortunately will have to make another release candidate).

Mike

···

On 06/18/2013 06:13 PM, Fernando Perez wrote:

Hi Ben,

On Tue, Jun 18, 2013 at 1:29 PM, Benjamin Root <ben.root@...553...> wrote:

Does the same thing happen with the v1.3.x branch? You said you tested
master, but that isn't exactly the same as v1.3.x.

I just tested with the v1.3.0rc3, and the problem is present there:

((v1.3.0rc3))longs[matplotlib]> iptest -vx IPython.core.tests.test_run
Check that %run doesn't damage __builtins__ ... ok
Check that the type of __builtins__ doesn't change with %run. ... ok
Test that prompts correctly generate after %run ... ok
Test that the option -p, which invokes the profiler, do not ... ok
Test that namespace cleanup is not too aggressive GH-238 ... ERROR

======================================================================
ERROR: Test that namespace cleanup is not too aggressive GH-238
----------------------------------------------------------------------
Traceback (most recent call last):
   File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
     self.test(*self.arg)
   File "/usr/lib/python2.7/dist-packages/numpy/testing/decorators.py",
line 213, in knownfailer
     raise KnownFailureTest(msg)
KnownFailureTest: This test is known to fail

----------------------------------------------------------------------
Ran 5 tests in 0.006s

FAILED (errors=1)

Cheers,

f

------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:

Build for Windows Store.

http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Hi Mike,

This was an attempt to fix a bug that mpl's KnownFailure plugin wouldn't
load when running tests directly using the nosetests commandline
script. I see IPython has a testing wrapper script (iptest) -- is that
in part to solve that problem?

Only in part. We wrote iptest because we need to start nose multiple
times in different subprocesses for each chunk of IPython, as trying
to load all of IPython into a single python process ends up producing
tears (conflicts between things that don't like to live together in
sys.modules like multiple gui toolkits, etc).

In any case, the revert should be simple -- can you try commenting out
the "entry_points" kwarg at the bottom of the setup.py script? (You'll
probably need to blitz the matplotlib installation directory and `build`
for good measure). I can't actually reproduce the bug myself, but I
suspect that's because this is somewhat dependent on the order in which
things are installed into the virtualenv or the phases of the moon...

Yup, problem gone. With this change:

((v1.3.0rc3))longs[matplotlib]> git diff
diff --git a/setup.py b/setup.py
index 5f1b561..b4d1763 100644
--- a/setup.py
+++ b/setup.py
@@ -230,9 +230,9 @@ if __name__ == '__main__':
           zip_safe=False,

           # Install our nose plugin so it will always be found
- entry_points={
- 'nose.plugins.0.10': [
- 'KnownFailure = matplotlib.testing.noseclasses:KnownFailure'
- ]
- },
+ # entry_points={
+ # 'nose.plugins.0.10': [
+ # 'KnownFailure = matplotlib.testing.noseclasses:KnownFailure'
+ # ]
+ # },
          )

I get as expected:

((v1.3.0rc3))longs[matplotlib]> iptest -vx IPython.core.tests.test_run

[...]

···

On Tue, Jun 18, 2013 at 3:37 PM, Michael Droettboom <mdroe@...31...> wrote:

----------------------------------------------------------------------
Ran 23 tests in 2.277s

OK (KNOWNFAIL=1)

If that works for you, we can just take that out and require testers to
use our testing script (and unfortunately will have to make another
release candidate).

Well, I wouldn't want to force mpl to have to ship a custom testing
script, that's kind of an ugly kludge that we live with but that is
really sub-optimal. I wish we could just fix this plugin issue. The
problem, I suspect, is the presence of multiple KnownFailure classes
in a way that trips an isisnstance() check somewhere.

Cheers,

f

When we drop support for Python 2.6, I think we can use the expectedFailure
mechanism included in unittest from 2.7 onwards. So long as nose recognises
that, we should be able to drop our copy of the KnownFailure plugin.

Thomas

···

On 19 June 2013 00:09, Fernando Perez <fperez.net@...149...> wrote:

I wish we could just fix this plugin issue.

Good point, I didn't know about that new mechanism.

I think we should keep 2.6 support for IPython 1.0, but drop it
afterwards. We can discuss that during the dev meeting...

Cheers,

f

···

On Tue, Jun 18, 2013 at 4:16 PM, Thomas Kluyver <thomas@...1071...> wrote:

On 19 June 2013 00:09, Fernando Perez <fperez.net@...149...> wrote:

I wish we could just fix this plugin issue.

When we drop support for Python 2.6, I think we can use the expectedFailure
mechanism included in unittest from 2.7 onwards. So long as nose recognises
that, we should be able to drop our copy of the KnownFailure plugin.

Thomas

--
Fernando Perez (@fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail

Hi Mike,

This was an attempt to fix a bug that mpl's KnownFailure plugin wouldn't
load when running tests directly using the nosetests commandline
script. I see IPython has a testing wrapper script (iptest) -- is that
in part to solve that problem?

Only in part. We wrote iptest because we need to start nose multiple
times in different subprocesses for each chunk of IPython, as trying
to load all of IPython into a single python process ends up producing
tears (conflicts between things that don't like to live together in
sys.modules like multiple gui toolkits, etc).

We are likely to have to do the same thing in the near future -- we already have to spawn subprocesses in the setup script because we can't, for example, import pygtk (classic) and pygi (the new hotness) in the same process.

In any case, the revert should be simple -- can you try commenting out
the "entry_points" kwarg at the bottom of the setup.py script? (You'll
probably need to blitz the matplotlib installation directory and `build`
for good measure). I can't actually reproduce the bug myself, but I
suspect that's because this is somewhat dependent on the order in which
things are installed into the virtualenv or the phases of the moon...

Yup, problem gone. With this change:

((v1.3.0rc3))longs[matplotlib]> git diff
diff --git a/setup.py b/setup.py
index 5f1b561..b4d1763 100644
--- a/setup.py
+++ b/setup.py
@@ -230,9 +230,9 @@ if __name__ == '__main__':
            zip_safe=False,

            # Install our nose plugin so it will always be found
- entry_points={
- 'nose.plugins.0.10': [
- 'KnownFailure = matplotlib.testing.noseclasses:KnownFailure'
- ]
- },
+ # entry_points={
+ # 'nose.plugins.0.10': [
+ # 'KnownFailure = matplotlib.testing.noseclasses:KnownFailure'
+ # ]
+ # },
           )

I get as expected:

((v1.3.0rc3))longs[matplotlib]> iptest -vx IPython.core.tests.test_run

[...]

----------------------------------------------------------------------
Ran 23 tests in 2.277s

OK (KNOWNFAIL=1)

If that works for you, we can just take that out and require testers to
use our testing script (and unfortunately will have to make another
release candidate).

Well, I wouldn't want to force mpl to have to ship a custom testing
script, that's kind of an ugly kludge that we live with but that is
really sub-optimal. I wish we could just fix this plugin issue. The
problem, I suspect, is the presence of multiple KnownFailure classes
in a way that trips an isisnstance() check somewhere.

Well, this wouldn't be anything new. The 1.2 series (and I think as far back as we've had nosetests) requires this already, I was just hoping this would be a solution to that.

I think I'm fine with just taking this out for 1.3.0 and, as Thomas Kluyver suggested, dealing with this properly once we drop support for Python 2.6.

Cheers,
Mike

···

On 06/18/2013 07:09 PM, Fernando Perez wrote:

On Tue, Jun 18, 2013 at 3:37 PM, Michael Droettboom <mdroe@...31...> wrote:

Cheers,

f

Just to close the loop on this, I have created:

https://github.com/matplotlib/matplotlib/pull/2139

···

On 06/18/2013 07:18 PM, Fernando Perez wrote:

Good point, I didn't know about that new mechanism.

I think we should keep 2.6 support for IPython 1.0, but drop it
afterwards. We can discuss that during the dev meeting...

Cheers,

f

On Tue, Jun 18, 2013 at 4:16 PM, Thomas Kluyver <thomas@...1071...> wrote:

On 19 June 2013 00:09, Fernando Perez <fperez.net@...149...> wrote:

I wish we could just fix this plugin issue.

When we drop support for Python 2.6, I think we can use the expectedFailure
mechanism included in unittest from 2.7 onwards. So long as nose recognises
that, we should be able to drop our copy of the KnownFailure plugin.

Thomas

Thanks a ton, Mike! Great not to have to worry about this on our side.

cheers,

f

···

On Wed, Jun 19, 2013 at 10:02 AM, Michael Droettboom <mdroe@...31...> wrote:

Just to close the loop on this, I have created:

https://github.com/matplotlib/matplotlib/pull/2139

On 06/18/2013 07:18 PM, Fernando Perez wrote:

Good point, I didn't know about that new mechanism.

I think we should keep 2.6 support for IPython 1.0, but drop it
afterwards. We can discuss that during the dev meeting...

Cheers,

f

On Tue, Jun 18, 2013 at 4:16 PM, Thomas Kluyver <thomas@...1071...> > wrote:

On 19 June 2013 00:09, Fernando Perez <fperez.net@...149...> wrote:

I wish we could just fix this plugin issue.

When we drop support for Python 2.6, I think we can use the expectedFailure
mechanism included in unittest from 2.7 onwards. So long as nose recognises
that, we should be able to drop our copy of the KnownFailure plugin.

Thomas

--
Fernando Perez (@fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail