In discussions with Perry Greenfield at STScI, it seems we are in the position of being able to get some financial support to pay for a continuous integration system.
Travis has shown itself to be rather glitchy for matplotlib. The tight integration with Github PRs is really convenient. However, matplotlib has longer test runs and more dependencies than many projects. The frequent false positives start to wear down at a certain point. And we still aren't testing everything because we can't aren't installing Inkscape and other things in the Travis VM.
So we're looking to add another paid, hosted continuous integration service that will better meet our needs. A hosted service is nice because it's easy to give multiple developers access to the system so anyone can tweak it and keep it going -- the bottleneck of a single developer responsible for maintenance of the build machine was a problem years ago when we were using buildbot. This may remain "in addition to" rather than "instead of" Travis for some time.
An obvious first choice to me is ShiningPanda, as I have experience using it for astropy. Jenkins (the software Shining Panda is based on), is a really easy-to-use system, for those not familiar with it. And we can store the output of the tests (i.e. the result_images) for later inspection. I think this feature alone is huge for matplotlib. They also offer Windows build slaves. There is no OS-X (probably because Apple licensing doesn't really allow for use in a VM), but results can be "published" to their Jenkins instance.
Are there any other similar alternatives we should consider before we move forward?
Mike
I think hosted infrastructure is the right choice. I was initially
going to suggest that we try and work with a bespoke solution. That
way we could roll our own build architectures.
On reflection I think the maintenance headache of managing our own
build slaves outweighs the convenience of having it hosted, as you
point out.
···
On Wed, Jan 30, 2013 at 8:44 AM, Michael Droettboom <mdroe@...31...> wrote:
In discussions with Perry Greenfield at STScI, it seems we are in the
position of being able to get some financial support to pay for a
continuous integration system.
Travis has shown itself to be rather glitchy for matplotlib. The tight
integration with Github PRs is really convenient. However, matplotlib
has longer test runs and more dependencies than many projects. The
frequent false positives start to wear down at a certain point. And we
still aren't testing everything because we can't aren't installing
Inkscape and other things in the Travis VM.
So we're looking to add another paid, hosted continuous integration
service that will better meet our needs. A hosted service is nice
because it's easy to give multiple developers access to the system so
anyone can tweak it and keep it going -- the bottleneck of a single
developer responsible for maintenance of the build machine was a problem
years ago when we were using buildbot. This may remain "in addition to"
rather than "instead of" Travis for some time.
An obvious first choice to me is ShiningPanda, as I have experience
using it for astropy. Jenkins (the software Shining Panda is based on),
is a really easy-to-use system, for those not familiar with it. And we
can store the output of the tests (i.e. the result_images) for later
inspection. I think this feature alone is huge for matplotlib. They
also offer Windows build slaves. There is no OS-X (probably because
Apple licensing doesn't really allow for use in a VM), but results can
be "published" to their Jenkins instance.
Are there any other similar alternatives we should consider before we
move forward?
Mike
--
Damon McDougall
http://www.damon-is-a-geek.com
Institute for Computational Engineering Sciences
201 E. 24th St.
Stop C0200
The University of Texas at Austin
Austin, TX 78712-1229
Of course, you're still going to need people who are willing/able to
maintain OSX build slaves if you want to do testing on OSX (which
obviously you should). It's easy with Jenkins to run your own
instances and have it push results to the Shining Panda instance or
any other hosted service. But otherwise you're back to square one in
terms of having to rely on the people running the OSX builds. We've
had this problem on Astropy in that I'm maintaining our current only
Windows machine (ShingingPanda does have Windows support but a a cost)
and so whenever the build bot itself breaks on Windows it's up to me
to do anything about it. It also makes it hard for other admins to
make tweaks to the build configuration.
Unfortunately you're probably going to have that problem with OSX and
probably Windows with any other hosted service as well.
Erik
···
On Wed, Jan 30, 2013 at 9:27 PM, Damon McDougall <damon.mcdougall@...149...> wrote:
On Wed, Jan 30, 2013 at 8:44 AM, Michael Droettboom <mdroe@...31...> wrote:
In discussions with Perry Greenfield at STScI, it seems we are in the
position of being able to get some financial support to pay for a
continuous integration system.
Travis has shown itself to be rather glitchy for matplotlib. The tight
integration with Github PRs is really convenient. However, matplotlib
has longer test runs and more dependencies than many projects. The
frequent false positives start to wear down at a certain point. And we
still aren't testing everything because we can't aren't installing
Inkscape and other things in the Travis VM.
So we're looking to add another paid, hosted continuous integration
service that will better meet our needs. A hosted service is nice
because it's easy to give multiple developers access to the system so
anyone can tweak it and keep it going -- the bottleneck of a single
developer responsible for maintenance of the build machine was a problem
years ago when we were using buildbot. This may remain "in addition to"
rather than "instead of" Travis for some time.
An obvious first choice to me is ShiningPanda, as I have experience
using it for astropy. Jenkins (the software Shining Panda is based on),
is a really easy-to-use system, for those not familiar with it. And we
can store the output of the tests (i.e. the result_images) for later
inspection. I think this feature alone is huge for matplotlib. They
also offer Windows build slaves. There is no OS-X (probably because
Apple licensing doesn't really allow for use in a VM), but results can
be "published" to their Jenkins instance.
Are there any other similar alternatives we should consider before we
move forward?
Mike
I think hosted infrastructure is the right choice. I was initially
going to suggest that we try and work with a bespoke solution. That
way we could roll our own build architectures.
On reflection I think the maintenance headache of managing our own
build slaves outweighs the convenience of having it hosted, as you
point out.