Recently, it has become more and more common that the travis builds
fail. They fail not because the test suite fails but because of some
inherent issues with travis itself. Usually, either the git clone
fails or some other simple shell command fails. I consider these
'failures' as being false negatives. While a false negative once or
twice a month would not be anything to worry about, I feel like this
is happening often enough to render the pull request status extremely
unhelpful.
I realise the above is merely my opinion, but I feel that I would be
surprised if others did not find the Travis' false negative hit rate
tiresome.
If practical, I would like to move away from travis and have our test
suite run on a more reliable service. How do others feel about this?
···
--
Damon McDougall http://www.damon-is-a-geek.com
Institute for Computational Engineering Sciences
201 E. 24th St.
Stop C0200
The University of Texas at Austin
Austin, TX 78712-1229
Recently, it has become more and more common that the travis builds
fail. They fail not because the test suite fails but because of some
inherent issues with travis itself. Usually, either the git clone
fails or some other simple shell command fails. I consider these
'failures' as being false negatives. While a false negative once or
twice a month would not be anything to worry about, I feel like this
is happening often enough to render the pull request status extremely
unhelpful.
I realise the above is merely my opinion, but I feel that I would be
surprised if others did not find the Travis' false negative hit rate
tiresome.
Agreed.
If practical, I would like to move away from travis and have our test
suite run on a more reliable service. How do others feel about this?
Good--but what is the superior alternative?
I know nothing about who offers such services, why they do so, or the quality of the service.
Recently, it has become more and more common that the travis builds
fail. They fail not because the test suite fails but because of some
inherent issues with travis itself. Usually, either the git clone
fails or some other simple shell command fails. I consider these
'failures' as being false negatives. While a false negative once or
twice a month would not be anything to worry about, I feel like this
is happening often enough to render the pull request status extremely
unhelpful.
I realise the above is merely my opinion, but I feel that I would be
surprised if others did not find the Travis' false negative hit rate
tiresome.
Agreed.
If practical, I would like to move away from travis and have our test
suite run on a more reliable service. How do others feel about this?
Good--but what is the superior alternative?
There's shiningpanda: this is what scikit-learn now uses (with
travis-ci): https://jenkins.shiningpanda.com/scikit-learn/
It's quite flexible, but I think you are limited to only a couple of
builds per day on master. It doesn't really replace travis, as travis
is supposed to run against PR but it is better than nothing.
Previously, scikit-learn used buildbot, hosted on one of the french
python association. I've got access to the server, and I can ask the
board members if we can set up a buildbot there. I'm sure that's not a
problem. But, once again, this will run the tests on master, not on
the PR.
Last but not least, maybe we can see what numfocus has to offer.
···
I know nothing about who offers such services, why they do so, or the
quality of the service.
Eric
------------------------------------------------------------------------------
Master Java SE, Java EE, Eclipse, Spring, Hibernate, JavaScript, jQuery
and much more. Keep your Java skills current with LearnJavaNow -
200+ hours of step-by-step video tutorials by Java experts.
SALE $49.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122612
_______________________________________________
Matplotlib-devel mailing list
Matplotlib-devel@lists.sourceforge.net matplotlib-devel List Signup and Options
Absolutely! I'll be offline for two weeks, but others on the list can
certainly propose this to numfocus on the list and we can look into
what can be done, esp. in a way that could also help other projects as
well.
Also, there's snakebite: http://www.snakebite.net. The project seemed
very dormant for a long time, but there's been some activity since: http://www.snakebite.net/network. I'd ping Titus Brown on Twitter
(@ctitusbrown) for info...
Cheers,
f
···
On Wed, Jan 16, 2013 at 12:10 AM, Nelle Varoquaux <nelle.varoquaux@...149...> wrote:
Last but not least, maybe we can see what numfocus has to offer.
When this has come up on the numfocus list before the status was, sure, they can provide resources/funding if you can find someone to do the actual work :-). If anyone is interested in putting in the time to solve this problem properly then a lot of projects would be grateful I think…
-n
···
On 16 Jan 2013 09:30, “Fernando Perez” <fperez.net@…149…> wrote:
On Wed, Jan 16, 2013 at 12:10 AM, Nelle Varoquaux > <nelle.varoquaux@…322…9…> wrote:
Last but not least, maybe we can see what numfocus has to offer.
Absolutely! I’ll be offline for two weeks, but others on the list can
certainly propose this to numfocus on the list and we can look into
what can be done, esp. in a way that could also help other projects as
well.
We'd certainly be interested in a better solution for IPython - I think we
see more Travis builds failing because of Travis than because of the actual
code.
Before Travis started testing pull requests, we wrote our own test_pr.py
script [1], which fetches a PR, merges it into master, tests on several
versions of Python, and (optionally) posts a comment to Github with the
results. This should be pretty easy to adapt to other projects - NetworkX
already decide to use it [2]. With a little more code, I imagine we could
configure a server to regularly scan pull requests for changes and test
them.
Sympy also have a system for automatically testing pull requests [3]. I
think this is somewhat more advanced, with a queue on Google Appengine, and
workers which pull jobs from the queue.
Finally, we're keen users of ShiningPanda [4]. It doesn't test pull
requests, but it's reliable, and can handle things like coverage metrics
nicely.
On 16 January 2013 08:35, Nathaniel Smith <njs@...503...> wrote:
When this has come up on the numfocus list before the status was, sure,
they can provide resources/funding if you can find someone to do the actual
work :-). If anyone is interested in putting in the time to solve this
problem properly then a lot of projects would be grateful I think...
This definitely would be worth investigating. At least in my
institution, finding CPU time isn’t terribly difficult. Hosting our
own network services is – so something that does builds locally and
then publishes the results somewhere else (i.e. Github) may make a
lot of sense.
The matplotlib test suite is really CPU-intensive, and the last time
I did the math, the ShiningPanda free plan was too tight – it’s
about an hour of CPU time (on ShiningPanda’s machines) per test for
4 Numpy/Python combinations. It’s about $0.50/hr, and you get $6
free per month.
I really like ShiningPanda, mostly for how flexible the setup of
prerequisites are, but it will cost some money to use.
Cheers,
Mike
···
On 01/16/2013 07:15 AM, Thomas Kluyver
wrote:
On 16 January 2013 08:35, Nathaniel
Smith <njs@…503…>
wrote:
When this has come up on the numfocus list before the
status was, sure, they can provide resources/funding if
you can find someone to do the actual work :-). If anyone
is interested in putting in the time to solve this problem
properly then a lot of projects would be grateful I
think…
We'd certainly be interested in a
better solution for IPython - I think we see more Travis
builds failing because of Travis than because of the actual
code.
Before Travis started testing pull requests, we wrote our own
test_pr.py script [1], which fetches a PR, merges it into
master, tests on several versions of Python, and (optionally)
posts a comment to Github with the results. This should be
pretty easy to adapt to other projects - NetworkX already
decide to use it [2]. With a little more code, I imagine we
could configure a server to regularly scan pull requests for
changes and test them.
Sympy also have a system for
automatically testing pull requests [3]. I think this is
somewhat more advanced, with a queue on Google Appengine, and
workers which pull jobs from the queue.
Finally, we're keen users of
ShiningPanda [4]. It doesn’t test pull requests, but it’s
reliable, and can handle things like coverage metrics nicely.
------------------------------------------------------------------------------
Master Java SE, Java EE, Eclipse, Spring, Hibernate, JavaScript, jQuery
and much more. Keep your Java skills current with LearnJavaNow -
200+ hours of step-by-step video tutorials by Java experts.
SALE $49.99 this month only -- learn more at:
_______________________________________________
Matplotlib-devel mailing list
I haven't been active enough lately to have a good feel overall on the
false negatives. What I can say right now is that it would help if we would
disable python 3.1 (and optionally replace it with 3.3). I believe (based
on numpy commit I saw the other day) that Travis no longer supports 3.1.
That would solve the failures I've seen with my PR's.
Ryan
···
On Tue, Jan 15, 2013 at 6:20 PM, Damon McDougall <damon.mcdougall@...149...>wrote:
Hi all,
Recently, it has become more and more common that the travis builds
fail. They fail not because the test suite fails but because of some
inherent issues with travis itself. Usually, either the git clone
fails or some other simple shell command fails. I consider these
'failures' as being false negatives. While a false negative once or
twice a month would not be anything to worry about, I feel like this
is happening often enough to render the pull request status extremely
unhelpful.
--
Ryan May
Graduate Research Assistant
School of Meteorology
University of Oklahoma
There is also the open build service, which is more of a build farm but can be set up pull, build, test, and publish git snapshots for most common Linux distributions to your own online software repository simultaneously with one click on a website or one commandline call.
They provide hosting, building, and distribution for free. You can probably set up a script to automatically rebuild master on a change, or daily. However, setting it up to test individual commits would be overly difficult and probably be seen as an abuse of the system. Using it to always build, test, and release offer the latest master to most linux distros, on the other hand, would be fine. If someone contacts them they can probably set up a repository just for you, or if this sort of thing is useful a more general repository you can share with others (there is already devel:languages:python, maybe devel:languages:python:unstable).
You can also use it to build release rpms and debs for various distros. It is already being used to build the packages discussed so far for openSUSE, but if someone is willing to maintain them they can be built for other distros as well.
···
On Jan 16, 2013 9:30 AM, “Fernando Perez” <fperez.net@…149…> wrote:
On Wed, Jan 16, 2013 at 12:10 AM, Nelle Varoquaux > <nelle.varoquaux@…149…> wrote:
Last but not least, maybe we can see what numfocus has to offer.
Absolutely! I’ll be offline for two weeks, but others on the list can
certainly propose this to numfocus on the list and we can look into
what can be done, esp. in a way that could also help other projects as
well.
Also, there’s snakebite: http://www.snakebite.net. The project seemed
very dormant for a long time, but there’s been some activity since: http://www.snakebite.net/network. I’d ping Titus Brown on Twitter
(@ctitusbrown) for info…
GitHub allow for a custom service-hook. If, as Mike says, it's not too
hard to garner compute cycles, it shouldn't be too hard to write a
small script to execute the test suite when the github repo receives a
push.
···
On Wed, Jan 16, 2013 at 10:54 AM, Todd <toddrjen@...149...> wrote:
On Jan 16, 2013 9:30 AM, "Fernando Perez" <fperez.net@...149...> wrote:
On Wed, Jan 16, 2013 at 12:10 AM, Nelle Varoquaux >> <nelle.varoquaux@...149...> wrote:
> Last but not least, maybe we can see what numfocus has to offer.
Absolutely! I'll be offline for two weeks, but others on the list can
certainly propose this to numfocus on the list and we can look into
what can be done, esp. in a way that could also help other projects as
well.
Also, there's snakebite: http://www.snakebite.net. The project seemed
very dormant for a long time, but there's been some activity since: http://www.snakebite.net/network. I'd ping Titus Brown on Twitter
(@ctitusbrown) for info...
Cheers,
There is also the open build service, which is more of a build farm but can
be set up pull, build, test, and publish git snapshots for most common Linux
distributions to your own online software repository simultaneously with one
click on a website or one commandline call.
They provide hosting, building, and distribution for free. You can probably
set up a script to automatically rebuild master on a change, or daily.
However, setting it up to test individual commits would be overly difficult
and probably be seen as an abuse of the system. Using it to always build,
test, and release offer the latest master to most linux distros, on the
other hand, would be fine. If someone contacts them they can probably set
up a repository just for you, or if this sort of thing is useful a more
general repository you can share with others (there is already
devel:languages:python, maybe devel:languages:python:unstable).
You can also use it to build release rpms and debs for various distros. It
is already being used to build the packages discussed so far for openSUSE,
but if someone is willing to maintain them they can be built for other
distros as well.
--
Damon McDougall http://www.damon-is-a-geek.com
Institute for Computational Engineering Sciences
201 E. 24th St.
Stop C0200
The University of Texas at Austin
Austin, TX 78712-1229