Announcing toydist, improving distribution and packaging situation

(warning, long post)

Hi there,

   As some of you already know, the packaging and distributions of
scientific python packages have been a constant source of frustration.
Open source is about making it easy for anyone to use software how
they see fit, and I think python packaging infrastructure has not been
very successfull for people not intimately familiar with python. A few
weeks ago, after Guido visited Berkeley and was told how those issues
were still there for the scientific community, he wrote an email
asking whether current efforts on distutils-sig will be enough (see
http://aspn.activestate.com/ASPN/Mail/Message/distutils-sig/3775972).

Several of us have been participating to this discussion, but I feel
like the divide between current efforts on distutils-sig and us (the
SciPy community) is not getting smaller. At best, their efforts will
be more work for us to track the new distribute fork, and more likely,
it will be all for nothing as it won't solve any deep issue. To be
honest, most of what is considered on distutils-sig sounds like
anti-goals to me.

Instead of keeping up with the frustrating process of "improving"
distutils, I think we have enough smart people and manpower in the
scientific community to go with our own solution. I am convinced it is
doable because R or haskell, with a much smaller community than
python, managed to pull out something with is miles ahead compared to
pypi. The SciPy community is hopefully big enough so that a
SciPy-specific solution may reach critical mass. Ideally, I wish we
had something with the following capabilities:

  - easy to understand tools
  - http-based package repository ala CRAN, which would be easy to
mirror and backup (through rsync-like tools)
  - decoupling the building, packaging and distribution of code and data
  - reliable install/uninstall/query of what is installed locally
  - facilities for building windows/max os x binaries
  - making the life of OS vendors (Linux, *BSD, etc...) easier

The packaging part

···

==============

Speaking is easy, so I started coding part of this toolset, called
toydist (temporary name), which I presented at Scipy India a few days
ago:

http://github.com/cournape/toydist/

Toydist is more or less a rip off of cabal
(http://www.haskell.org/cabal/), and consist of three parts:
  - a core which builds a package description from a declarative file
similar to cabal files. The file is almost purely declarative, and can
be parsed so that no arbitrary code is executed, thus making it easy
to sandbox packages builds (e.g. on a build farm).
  - a set of command line tools to configure, build, install, build
installers (egg only for now) etc... from the declarative file
  - backward compatibility tools: a tool to convert existing setup.py
to the new format has been written, and a tool to use distutils
through the new format for backward compatibility with complex
distutils extensions should be relatively easy.

The core idea is to make the format just rich enough to describe most
packages out there, but simple enough so interfacing it with external
tools is possible and reliable. As a regular contributor to scons, I
am all too aware that a build tool is a very complex beast to get
right, and repeating their efforts does not make sense. Typically, I
envision that complex packages such as numpy, scipy or matplotlib
would use make/waf/scons for the build - in a sense, toydist is
written so that writing something like numscons would be easier. OTOH,
most if not all scikits should be buildable from a purely declarative
file.

To give you a feel of the format, here is a snippet for the grin
package from Robert K. (automatically converted):

Name: grin
Version: 1.1.1
Summary: A grep program configured the way I like it.
Description:
    ====
    grin
    ====

    I wrote grin to help me search directories full of source code.
The venerable
    GNU grep_ and find_ are great tools, but they fall just a little
short for my
    normal use cases.

    <snip>
License: BSD
Platforms: UNKNOWN
Classifiers:
    License :: OSI Approved :: BSD License,
    Development Status :: 5 - Production/Stable,
    Environment :: Console,
    Intended Audience :: Developers,
    Operating System :: OS Independent,
    Programming Language :: Python,
    Topic :: Utilities,

ExtraSourceFiles:
    README.txt,
    setup.cfg,
    setup.py,

Library:
    InstallDepends:
        argparse,
    Modules:
        grin,

Executable: grin
    module: grin
    function: grin_main

Executable: grind
    module: grin
    function: grind_main

Although still very much experimental at this point, toydist already
makes some things much easier than with distutils/setuptools:
  - path customization for any target can be done easily: you can
easily add an option in the file so that configure --mynewdir=value
works and is accessible at every step.
- making packages FHS compliant is not a PITA anymore, and the scheme
can be adapted to any OS, be it traditional FHS-like unix, mac os x,
windows, etc...
  - All the options are accessible at every step (no more distutils
commands nonsense)
  - data files can finally be handled correctly and consistently,
instead of the 5 or 6 magics methods currently available in
distutils/setuptools/numpy.distutils
  - building eggs does not involve setuptools anymore
  - not much coupling between package description and build
infrastructure (building extensions is actually done through distutils
ATM).

Repository

The goal here is to have something like CRAN
(http://cran.r-project.org/web/views/), ideally with a build farm so
that whenever anyone submits a package to our repository, it would
automatically be checked, and built for windows/mac os x and maybe a
few major linux distributions. One could investigate the build service
from open suse to that end (http://en.opensuse.org/Build_Service),
which is based on xen VM to build installers in a reproducible way.

Installed package db

I believe that the current open source enstaller package from
Enthought can be a good starting point. It is based on eggs, but eggs
are only used as a distribution format (eggs are never installed as
eggs AFAIK). You can easily remove packages, query installed versions,
etc... Since toydist produces eggs, interoperation between toydist and
enstaller should not be too difficult.

What's next ?

At this point, I would like to ask for help and comments, in particular:
- Does all this make sense, or hopelessly intractable ?
- Besides the points I have mentioned, what else do you think is needed ?
- There has already been some work for the scikits webportal, but I
think we should bypass pypi entirely (the current philosophy of not
enforcing consistent metadata does not make much sense to me, and is
at the opposite of most other similar system out there).
- I think a build farm for at least windows packages would be a
killer feature, and enough incentive to push some people to use our
new infrastructure. It would be good to have a windows guy familiar
with windows sandboxing/virtualization to do something there. The
people working on the opensuse build service have started working on
windows support
- I think being able to automatically convert most of scientific
packages is a significant feature, and needs to be more robust - so
anyone is welcomed to try converting existing setup.py with toydist
(see toydist readme).

thanks,

David

Hi,
I would like to add a comment from the user perspective:

  • the main reason why I’m not satisfied with pypi/distutils/etc. and why I will not be satisfied with toydist (with the features you listed), is that they break my installation (debian/ubuntu). The main advantage of these kinds of linux is their packaging system. So even if there is a way to query if a python package is installed, I will need three or more commands to check if it really is there (toydist, distutils, aptitude and other packages installed by hand).

I know you don’t want to hear this, but my suggestion (as a user) would be: use the debian package system. If you introduce a new system anyway, why not use the best there is and remove at least one incompatibility with at least one system?
It shouldn’t make a big difference on windows, because they would have to install/learn/use your new system anyway. The same is true for OS X, BSDs and RPM based linuxes. So they won’t have any disadvantage when debs are used. But deb based Linuxes would have a big advantage: their system wouldn’t be broken.

And with debs, you would have:

  • http-based package repository ala debian/ubuntu, which would be easy to
    mirror and backup (through rsync-like tools)
  • decoupling the building, packaging and distribution of code and data
  • reliable install/uninstall/query of what is installed locally
  • making the life of OS vendors (Linux, *BSD, etc…) easier

You wouldn’t get automatic compiling for windows and I’m not sure if the tools are as easy to understand as your toydist would be. But they are mighty, ready to use, heavily tested, and maintained even if you don’t find the time to contribute.

I’m aware, that there might be issues that I did not mention here, and this is meant as an idea and not as a complete solution…

Cheers,
Stefan

···

On Mon, Dec 28, 2009 at 15:03, David Cournapeau <cournape@…149…> wrote:

(warning, long post)

Hi there,

As some of you already know, the packaging and distributions of

scientific python packages have been a constant source of frustration.

Open source is about making it easy for anyone to use software how

they see fit, and I think python packaging infrastructure has not been

very successfull for people not intimately familiar with python. A few

weeks ago, after Guido visited Berkeley and was told how those issues

were still there for the scientific community, he wrote an email

asking whether current efforts on distutils-sig will be enough (see

http://aspn.activestate.com/ASPN/Mail/Message/distutils-sig/3775972).

Several of us have been participating to this discussion, but I feel

like the divide between current efforts on distutils-sig and us (the

SciPy community) is not getting smaller. At best, their efforts will

be more work for us to track the new distribute fork, and more likely,

it will be all for nothing as it won’t solve any deep issue. To be

honest, most of what is considered on distutils-sig sounds like

anti-goals to me.

Instead of keeping up with the frustrating process of “improving”

distutils, I think we have enough smart people and manpower in the

scientific community to go with our own solution. I am convinced it is

doable because R or haskell, with a much smaller community than

python, managed to pull out something with is miles ahead compared to

pypi. The SciPy community is hopefully big enough so that a

SciPy-specific solution may reach critical mass. Ideally, I wish we

had something with the following capabilities:

  • easy to understand tools

  • http-based package repository ala CRAN, which would be easy to

mirror and backup (through rsync-like tools)

  • decoupling the building, packaging and distribution of code and data

  • reliable install/uninstall/query of what is installed locally

  • facilities for building windows/max os x binaries

  • making the life of OS vendors (Linux, *BSD, etc…) easier

The packaging part

==============

Speaking is easy, so I started coding part of this toolset, called

toydist (temporary name), which I presented at Scipy India a few days

ago:

http://github.com/cournape/toydist/

Toydist is more or less a rip off of cabal

(http://www.haskell.org/cabal/), and consist of three parts:

  • a core which builds a package description from a declarative file

similar to cabal files. The file is almost purely declarative, and can

be parsed so that no arbitrary code is executed, thus making it easy

to sandbox packages builds (e.g. on a build farm).

  • a set of command line tools to configure, build, install, build

installers (egg only for now) etc… from the declarative file

  • backward compatibility tools: a tool to convert existing setup.py

to the new format has been written, and a tool to use distutils

through the new format for backward compatibility with complex

distutils extensions should be relatively easy.

The core idea is to make the format just rich enough to describe most

packages out there, but simple enough so interfacing it with external

tools is possible and reliable. As a regular contributor to scons, I

am all too aware that a build tool is a very complex beast to get

right, and repeating their efforts does not make sense. Typically, I

envision that complex packages such as numpy, scipy or matplotlib

would use make/waf/scons for the build - in a sense, toydist is

written so that writing something like numscons would be easier. OTOH,

most if not all scikits should be buildable from a purely declarative

file.

To give you a feel of the format, here is a snippet for the grin

package from Robert K. (automatically converted):

Name: grin

Version: 1.1.1

Summary: A grep program configured the way I like it.

Description:

====

grin

====



I wrote grin to help me search directories full of source code.

The venerable

GNU grep_ and find_ are great tools, but they fall just a little

short for my

normal use cases.



<snip>

License: BSD

Platforms: UNKNOWN

Classifiers:

License :: OSI Approved :: BSD License,

Development Status :: 5 - Production/Stable,

Environment :: Console,

Intended Audience :: Developers,

Operating System :: OS Independent,

Programming Language :: Python,

Topic :: Utilities,

ExtraSourceFiles:

README.txt,

setup.cfg,

setup.py,

Library:

InstallDepends:

    argparse,

Modules:

    grin,

Executable: grin

module: grin

function: grin_main

Executable: grind

module: grin

function: grind_main

Although still very much experimental at this point, toydist already

makes some things much easier than with distutils/setuptools:

  • path customization for any target can be done easily: you can

easily add an option in the file so that configure --mynewdir=value

works and is accessible at every step.

  • making packages FHS compliant is not a PITA anymore, and the scheme

can be adapted to any OS, be it traditional FHS-like unix, mac os x,

windows, etc…

  • All the options are accessible at every step (no more distutils

commands nonsense)

  • data files can finally be handled correctly and consistently,

instead of the 5 or 6 magics methods currently available in

distutils/setuptools/numpy.distutils

  • building eggs does not involve setuptools anymore

  • not much coupling between package description and build

infrastructure (building extensions is actually done through distutils

ATM).

Repository

========

The goal here is to have something like CRAN

(http://cran.r-project.org/web/views/), ideally with a build farm so

that whenever anyone submits a package to our repository, it would

automatically be checked, and built for windows/mac os x and maybe a

few major linux distributions. One could investigate the build service

from open suse to that end (http://en.opensuse.org/Build_Service),

which is based on xen VM to build installers in a reproducible way.

Installed package db

===============

I believe that the current open source enstaller package from

Enthought can be a good starting point. It is based on eggs, but eggs

are only used as a distribution format (eggs are never installed as

eggs AFAIK). You can easily remove packages, query installed versions,

etc… Since toydist produces eggs, interoperation between toydist and

enstaller should not be too difficult.

What’s next ?

==========

At this point, I would like to ask for help and comments, in particular:

  • Does all this make sense, or hopelessly intractable ?

  • Besides the points I have mentioned, what else do you think is needed ?

  • There has already been some work for the scikits webportal, but I

think we should bypass pypi entirely (the current philosophy of not

enforcing consistent metadata does not make much sense to me, and is

at the opposite of most other similar system out there).

  • I think a build farm for at least windows packages would be a

killer feature, and enough incentive to push some people to use our

new infrastructure. It would be good to have a windows guy familiar

with windows sandboxing/virtualization to do something there. The

people working on the opensuse build service have started working on

windows support

  • I think being able to automatically convert most of scientific

packages is a significant feature, and needs to be more robust - so

anyone is welcomed to try converting existing setup.py with toydist

(see toydist readme).

thanks,

David


This SF.Net email is sponsored by the Verizon Developer Community

Take advantage of Verizon’s best-in-class app development support

A streamlined, 14 day to market process makes app distribution fast and easy

Join now and get one step closer to millions of Verizon customers

http://p.sf.net/sfu/verizon-dev2dev


Matplotlib-devel mailing list

Matplotlib-devel@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/matplotlib-devel

Toydist (or distutils) does not break anything as is. It would be like
saying make breaks debian - it does not make much sense. As stated,
one of the goal of giving up distutils is to make packaging by os
vendors easier. In particular, by allowing to follow the FHS, and
making things more consistent. It should be possible to automatically
convert most packages to .deb (or .rpm) relatively easily. When you
look at the numpy .deb package, most of the issues are distutils
issues, and almost everything else can be done automatically.

Note that even ignoring the windows problem, there are systems to do
the kind of things I am talking about for linux-only systems (the
opensuse build service), because distributions are not always really
good at tracking fast changing softwares. IOW, traditional linux
packaging has some issues as well. And anyway, nothing prevents debian
or other OS vendors to package things as they want (as they do for R
packages).

David

···

On Mon, Dec 28, 2009 at 11:47 PM, Stefan Schwarzburg <stefan.schwarzburg@...564...> wrote:

Hi,
I would like to add a comment from the user perspective:

- the main reason why I'm not satisfied with pypi/distutils/etc. and why I
will not be satisfied with toydist (with the features you listed), is that
they break my installation (debian/ubuntu).

Stefan Schwarzburg wrote:

Hi,
I would like to add a comment from the user perspective:

- the main reason why I'm not satisfied with pypi/distutils/etc. and
why I will not be satisfied with toydist (with the features you
listed), is that they break my installation (debian/ubuntu). The main
advantage of these kinds of linux is their packaging system. So even
if there is a way to query if a python package is installed, I will
need three or more commands to check if it really is there (toydist,
distutils, aptitude and other packages installed by hand).

I am interested in adapting stdeb ( http://github.com/astraw/stdeb ) to
handle toydist distributions. Nevertheless, the goals of toydist seem
mostly unrelated to your issue with the exception that toydist will
hopefully make generating .deb packages easier and more robust.

See http://github.com/astraw/stdeb/issues#issue/16 for a wishlist item
that I think would solve your issue. In that ticket, I give the steps
required to solve the issue -- I don't think it looks too hard, and
patches will be reviewed with a goal of producing something that gets
merged.

-Andrew

David wrote:

Repository

The goal here is to have something like CRAN
(http://cran.r-project.org/web/views/), ideally with a build farm so
that whenever anyone submits a package to our repository, it would
automatically be checked, and built for windows/mac os x and maybe a
few major linux distributions. One could investigate the build service
from open suse to that end (http://en.opensuse.org/Build_Service),
which is based on xen VM to build installers in a reproducible way.

Do you here mean automatic generation of Ubuntu debs, Debian debs, Windows
MSI installer, Windows EXE installer, and so on? (If so then great!)

If this is the goal, I wonder if one looks outside of Python-land one
might find something that already does this -- there's a lot of different
package format, "Linux meta-distributions", "install everywhere packages"
and so on.

Of course, toydist could have such any such tool as a backend/in a pipeline.

What's next ?

At this point, I would like to ask for help and comments, in particular:
- Does all this make sense, or hopelessly intractable ?
- Besides the points I have mentioned, what else do you think is needed ?

Hmm. What I miss is the discussion of other native libraries which the
Python libraries need to bundle. Is it assumed that one want to continue
linking C and Fortran code directly into Python .so modules, like the
scipy library currently does?

Let me take CHOLMOD (sparse Cholesky) as an example.

- The Python package cvxopt use it, simply by linking about 20 C files
directly into the Python-loadable module (.so) which goes into the Python
site-packages (or wherever). This makes sure it just works. But, it
doesn't feel like the right way at all.

- scikits.sparse.cholmod OTOH simple specifies libraries=["cholmod"], and
leave it up to the end-user to make sure it is installed. Linux users
with root access can simply apt-get, but it is a pain for everybody else
(Windows, Mac, non-root Linux).

- Currently I'm making a Sage SPKG for CHOLMOD. This essentially gets the
job done by not bothering about the problem, not even using the
OS-installed Python.

Something that would spit out both Sage SPKGs, Ubuntu debs, Windows
installers, both with Python code and C/Fortran code or a mix (and put
both in the place preferred by the system in question), seems ideal. Of
course one would still need to make sure that the code builds properly
everywhere, but just solving the distribution part of this would be a huge
step ahead.

What I'm saying is that this is a software distribution problem in
general, and I'm afraid that Python-specific solutions are too narrow.

Dag Sverre

Do you here mean automatic generation of Ubuntu debs, Debian debs, Windows
MSI installer, Windows EXE installer, and so on? (If so then great!)

Yes (although this is not yet implemented). In particular on windows,
I want to implement a scheme so that you can convert from eggs to .exe
and vice et versa, so people can still install as exe (or msi), even
though the method would default to eggs.

If this is the goal, I wonder if one looks outside of Python-land one
might find something that already does this -- there's a lot of different
package format, "Linux meta-distributions", "install everywhere packages"
and so on.

Yes, there are things like 0install or autopackage. I think those are
deemed to fail, as long as it is not supported thoroughly by the
distribution. Instead, my goal here is much simpler: producing
rpm/deb. It does not solve every issue (install by non root, multiple
// versions), but one has to be realistic :slight_smile:

I think automatically built rpm/deb, easy integration with native
method can solve a lot of issues already.

- Currently I'm making a Sage SPKG for CHOLMOD. This essentially gets the
job done by not bothering about the problem, not even using the
OS-installed Python.

Something that would spit out both Sage SPKGs, Ubuntu debs, Windows
installers, both with Python code and C/Fortran code or a mix (and put
both in the place preferred by the system in question), seems ideal. Of
course one would still need to make sure that the code builds properly
everywhere, but just solving the distribution part of this would be a huge
step ahead.

On windows, this issue may be solved using eggs: enstaller has a
feature where dll put in a special location of an egg are installed in
python such as they are found by the OS loader. One could have
mechanisms based on $ORIGIN + rpath on linux to solve this issue for
local installs on Linux, etc...

But again, one has to be realistic on the goals. With toydist, I want
to remove all the pile of magic, hacks built on top of distutils so
that people can again hack their own solutions, as it should have been
from the start (that's a big plus of python in general). It won't
magically solve every issue out there, but it would hopefully help
people to make their own.

Bundling solutions like SAGE, EPD, etc... are still the most robust
ways to deal with those issues in general, and I do not intended to
replace those.

What I'm saying is that this is a software distribution problem in
general, and I'm afraid that Python-specific solutions are too narrow.

Distribution is a hard problem. Instead of pushing a very narrow (and
mostly ill-funded) view of how people should do things like
distutils/setuptools/pip/buildout do, I want people to be able to be
able to build their own solutions. No more "use this magic stick v
4.0.3.3.14svn1234, trust me it work you don't have to understand"
which is too prevalant with those tools, which has always felt deeply
unpythonic to me.

David

···

On Tue, Dec 29, 2009 at 3:49 AM, Dag Sverre Seljebotn <dagss@...806...> wrote:

Thanks, this cleared things up, and I like the direction this is heading.
Thanks a lot for doing this!

Dag Sverre

···

On Tue, Dec 29, 2009 at 3:49 AM, Dag Sverre Seljebotn > <dagss@...806...> wrote:

Do you here mean automatic generation of Ubuntu debs, Debian debs,
Windows
MSI installer, Windows EXE installer, and so on? (If so then great!)

Yes (although this is not yet implemented). In particular on windows,
I want to implement a scheme so that you can convert from eggs to .exe
and vice et versa, so people can still install as exe (or msi), even
though the method would default to eggs.

If this is the goal, I wonder if one looks outside of Python-land one
might find something that already does this -- there's a lot of
different
package format, "Linux meta-distributions", "install everywhere
packages"
and so on.

Yes, there are things like 0install or autopackage. I think those are
deemed to fail, as long as it is not supported thoroughly by the
distribution. Instead, my goal here is much simpler: producing
rpm/deb. It does not solve every issue (install by non root, multiple
// versions), but one has to be realistic :slight_smile:

I think automatically built rpm/deb, easy integration with native
method can solve a lot of issues already.

�- Currently I'm making a Sage SPKG for CHOLMOD. This essentially gets
the
job done by not bothering about the problem, not even using the
OS-installed Python.

Something that would spit out both Sage SPKGs, Ubuntu debs, Windows
installers, both with Python code and C/Fortran code or a mix (and put
both in the place preferred by the system in question), seems ideal. Of
course one would still need to make sure that the code builds properly
everywhere, but just solving the distribution part of this would be a
huge
step ahead.

On windows, this issue may be solved using eggs: enstaller has a
feature where dll put in a special location of an egg are installed in
python such as they are found by the OS loader. One could have
mechanisms based on $ORIGIN + rpath on linux to solve this issue for
local installs on Linux, etc...

But again, one has to be realistic on the goals. With toydist, I want
to remove all the pile of magic, hacks built on top of distutils so
that people can again hack their own solutions, as it should have been
from the start (that's a big plus of python in general). It won't
magically solve every issue out there, but it would hopefully help
people to make their own.

Bundling solutions like SAGE, EPD, etc... are still the most robust
ways to deal with those issues in general, and I do not intended to
replace those.

What I'm saying is that this is a software distribution problem in
general, and I'm afraid that Python-specific solutions are too narrow.

Distribution is a hard problem. Instead of pushing a very narrow (and
mostly ill-funded) view of how people should do things like
distutils/setuptools/pip/buildout do, I want people to be able to be
able to build their own solutions. No more "use this magic stick v
4.0.3.3.14svn1234, trust me it work you don't have to understand"
which is too prevalant with those tools, which has always felt deeply
unpythonic to me.

Hi David,

Executable: grin
module: grin
function: grin_main

Executable: grind
module: grin
function: grind_main

Have you thought at all about operations that are currently performed
by post-installation scripts? For example, it might be desirable for
the ipython or MayaVi windows installers to create a folder in the
Start menu that contains links the the executable and the
documentation. This is probably a secondary issue at this point in
toydist's development, but I think it is an important feature in the
long run.

Also, have you considered support for package extras (package variants
in Ports, allowing you to specify features that pull in additional
dependencies like traits[qt4])? Enthought makes good use of them in
ETS, and I think they would be worth keeping.

Darren

···

On Mon, Dec 28, 2009 at 9:03 AM, David Cournapeau <cournape@...149...> wrote:

Hi David,

Executable: grin
module: grin
function: grin_main

Executable: grind
module: grin
function: grind_main

Have you thought at all about operations that are currently performed
by post-installation scripts? For example, it might be desirable for
the ipython or MayaVi windows installers to create a folder in the
Start menu that contains links the the executable and the
documentation. This is probably a secondary issue at this point in
toydist's development, but I think it is an important feature in the
long run.

The main problem I see with post hooks is how to support them in
installers. For example, you would have a function which does the post
install, and declare it as a post install hook through decorator:

@hook.post_install
def myfunc():
    pass

The main issue is how to communicate data - that's a major issue in
every build system I know of (scons' solution is ugly: every function
takes an env argument, which is basically a giant global variable).

Also, have you considered support for package extras (package variants
in Ports, allowing you to specify features that pull in additional
dependencies like traits[qt4])? Enthought makes good use of them in
ETS, and I think they would be worth keeping.

The declarative format may declare flags as follows:

Flag: c_exts
    Description: Build (optional) C extensions
    Default: false

Library:
    if flag(c_exts):
        Extension: foo
             sources: foo.c

And this is automatically available at configure stage. It can be used
anywhere in Library, not just for Extension (you could use is within
the Requires section). I am considering adding more than Flag (flag
are boolean), if it does not make the format too complex. The use case
I have in mind is something like:

toydist configure --with-lapack-dir=/opt/mkl/lib

which I have wished to implement for numpy for ages.

David

···

On Wed, Dec 30, 2009 at 11:26 PM, Darren Dale <dsdale24@...149...> wrote:

On Mon, Dec 28, 2009 at 9:03 AM, David Cournapeau <cournape@...149...> wrote:

Does this example covers what you have in mind ? I am not so familiar
with this feature of setuptools:

Name: hello
Version: 1.0

Library:
    BuildRequires: paver, sphinx, numpy
    if os(windows)
        BuildRequires: pywin32
    Packages:
        hello
    Extension: hello._bar
        sources:
            src/hellomodule.c
    if os(linux)
        Extension: hello._linux_backend
            sources:
                src/linbackend.c

Note that instead of os(os_name), you can use flag(flag_name), where
flag are boolean variables which can be user defined:

http://github.com/cournape/toydist/blob/master/examples/simples/conditional/toysetup.info

http://github.com/cournape/toydist/blob/master/examples/var_example/toysetup.info

David

···

On Wed, Dec 30, 2009 at 11:26 PM, Darren Dale <dsdale24@...149...> wrote:

Hi David,

On Mon, Dec 28, 2009 at 9:03 AM, David Cournapeau <cournape@...149...> wrote:

Executable: grin
module: grin
function: grin_main

Executable: grind
module: grin
function: grind_main

Have you thought at all about operations that are currently performed
by post-installation scripts? For example, it might be desirable for
the ipython or MayaVi windows installers to create a folder in the
Start menu that contains links the the executable and the
documentation. This is probably a secondary issue at this point in
toydist's development, but I think it is an important feature in the
long run.

Also, have you considered support for package extras (package variants
in Ports, allowing you to specify features that pull in additional
dependencies like traits[qt4])? Enthought makes good use of them in
ETS, and I think they would be worth keeping.

I should defer to the description of extras in the setuptools
documentation. It is only a few paragraphs long:

http://peak.telecommunity.com/DevCenter/setuptools#declaring-extras-optional-features-with-their-own-dependencies

Darren

···

On Wed, Dec 30, 2009 at 11:16 AM, David Cournapeau <cournape@...149...> wrote:

On Wed, Dec 30, 2009 at 11:26 PM, Darren Dale <dsdale24@...149...> wrote:

Hi David,

On Mon, Dec 28, 2009 at 9:03 AM, David Cournapeau <cournape@...149...> wrote:

Executable: grin
module: grin
function: grin_main

Executable: grind
module: grin
function: grind_main

Have you thought at all about operations that are currently performed
by post-installation scripts? For example, it might be desirable for
the ipython or MayaVi windows installers to create a folder in the
Start menu that contains links the the executable and the
documentation. This is probably a secondary issue at this point in
toydist's development, but I think it is an important feature in the
long run.

Also, have you considered support for package extras (package variants
in Ports, allowing you to specify features that pull in additional
dependencies like traits[qt4])? Enthought makes good use of them in
ETS, and I think they would be worth keeping.

Does this example covers what you have in mind ? I am not so familiar
with this feature of setuptools:

Name: hello
Version: 1.0

Library:
BuildRequires: paver, sphinx, numpy
if os(windows)
BuildRequires: pywin32
Packages:
hello
Extension: hello._bar
sources:
src/hellomodule.c
if os(linux)
Extension: hello._linux_backend
sources:
src/linbackend.c

Note that instead of os(os_name), you can use flag(flag_name), where
flag are boolean variables which can be user defined:

http://github.com/cournape/toydist/blob/master/examples/simples/conditional/toysetup.info

http://github.com/cournape/toydist/blob/master/examples/var_example/toysetup.info

Ok, so there are two issues related to this feature:
  - supporting variant at the build stage
  - supporting different variants of the same package in the
dependency graph at install time

The first issue is definitely supported - I fixed a bug in toydist to
support this correctly, and this will be used when converting
setuptools-based setup.py which use the features argument.

The second issue is more challenging. It complicates the dependency
handling quite a bit, and may cause difficult situations to happen at
dependency resolution time. This becomes particularly messy if you mix
packages you build yourself with packages grabbed from a repository. I
wonder if there is a simpler solution which would give a similar
feature set.

cheers,

David

···

On Thu, Dec 31, 2009 at 6:06 AM, Darren Dale <dsdale24@...149...> wrote:

I should defer to the description of extras in the setuptools
documentation. It is only a few paragraphs long:

http://peak.telecommunity.com/DevCenter/setuptools#declaring-extras-optional-features-with-their-own-dependencies

AFAICT, in Debian, the same feature is given via virtual packages: you
would have:

python-matplotlib
python-matplotlib-basemap

for instance.

It is interesting to note that the same source package may be used to
generate both binary, end-user, packages.

And happy new year!

Gaël

···

On Sat, Jan 02, 2010 at 11:32:00AM +0900, David Cournapeau wrote:

[snip]
  - supporting different variants of the same package in the
dependency graph at install time

[snip]

The second issue is more challenging. It complicates the dependency
handling quite a bit, and may cause difficult situations to happen at
dependency resolution time. This becomes particularly messy if you mix
packages you build yourself with packages grabbed from a repository. I
wonder if there is a simpler solution which would give a similar
feature set.

I don't think virtual-packages entirely fix the issue. AFAIK, virtual
packages have two uses:
- handle dependencies where several packages may resolve one
particular dependency in an equivalent way (one good example is
LAPACK: both liblapack and libatlas provides the lapack feature)
- closer to this discussion, you can build several variants of the
same package, and each variant would resolve the dependency on a
virtual package handling the commonalities. For example, say we have
two numpy packages, one built with lapack (python-numpy-full), the
other without (python-numpy-core). What happens when a package foo
depends on numpy-full, but numpy-core is installed ? AFAICS, this can
only work as long as the set containing every variant can be ordered
(in the conventional set ordering sense), and the dependency can be
satisfied by the smallest one.

cheers,

David

···

On Sat, Jan 2, 2010 at 4:58 PM, Gael Varoquaux <gael.varoquaux@...427...> wrote:

On Sat, Jan 02, 2010 at 11:32:00AM +0900, David Cournapeau wrote:

[snip]
- supporting different variants of the same package in the
dependency graph at install time

[snip]

The second issue is more challenging. It complicates the dependency
handling quite a bit, and may cause difficult situations to happen at
dependency resolution time. This becomes particularly messy if you mix
packages you build yourself with packages grabbed from a repository. I
wonder if there is a simpler solution which would give a similar
feature set.

AFAICT, in Debian, the same feature is given via virtual packages: you
would have:

David Cournapeau wrote:

  

[snip]
  - supporting different variants of the same package in the
dependency graph at install time
      
[snip]
      
The second issue is more challenging. It complicates the dependency
handling quite a bit, and may cause difficult situations to happen at
dependency resolution time. This becomes particularly messy if you mix
packages you build yourself with packages grabbed from a repository. I
wonder if there is a simpler solution which would give a similar
feature set.
      

AFAICT, in Debian, the same feature is given via virtual packages: you
would have:
    
I don't think virtual-packages entirely fix the issue. AFAIK, virtual
packages have two uses:
- handle dependencies where several packages may resolve one
particular dependency in an equivalent way (one good example is
LAPACK: both liblapack and libatlas provides the lapack feature)
- closer to this discussion, you can build several variants of the
same package, and each variant would resolve the dependency on a
virtual package handling the commonalities. For example, say we have
two numpy packages, one built with lapack (python-numpy-full), the
other without (python-numpy-core). What happens when a package foo
depends on numpy-full, but numpy-core is installed ?

When you type "apt-get install my_new_package", the version resolution
system will tell you that apt-get will remove python-numpy-core and will
install python-numpy-full in the act of installing my_new_package.

AFAICS, this can
only work as long as the set containing every variant can be ordered
(in the conventional set ordering sense), and the dependency can be
satisfied by the smallest one.
  

Typically, the dependencies only depend on the smallest subset of what
they require (if they don't need lapack, they'd only depend on
python-numpy-core in your example), but yes, if there's an unsatisfiable
condition, then apt-get will raise an error and abort. In practice, this
system seems to work quite well, IMO.

Anyhow, here's the full Debian documentation:
http://www.debian.org/doc/debian-policy/ch-relationships.html

···

On Sat, Jan 2, 2010 at 4:58 PM, Gael Varoquaux > <gael.varoquaux@...427...> wrote:

On Sat, Jan 02, 2010 at 11:32:00AM +0900, David Cournapeau wrote:

Typically, the dependencies only depend on the smallest subset of what
they require (if they don't need lapack, they'd only depend on
python-numpy-core in your example), but yes, if there's an unsatisfiable
condition, then apt-get will raise an error and abort. In practice, this
system seems to work quite well, IMO.

Yes, but:
  - debian dependency resolution is complex. I think many people don't
realize how complex the problem really is (AFAIK, any correct scheme
to resolve dependencies in debian requires an algorithm which is
NP-complete )
  - introducing a lot of variants significantly slow down the whole thing.

I think it worths thinking whether our problems warrant such a complexity.

Anyhow, here's the full Debian documentation:
http://www.debian.org/doc/debian-policy/ch-relationships.html

This is not the part I am afraid of. This is:
http://people.debian.org/~dburrows/model.pdf

cheers,

David

···

On Sun, Jan 3, 2010 at 3:27 AM, Andrew Straw <strawman@...36...> wrote: