A talk about working with packages, zc.buildout and managing the application lifecycle.
Part 1: Working with packages
Distutils: builds and distributes a package, registers and uploads it to PyPI. The standard in the whole Python community. It’s not actively developed however and there are features missing.
on top of distutils. It corrects some of the problems with
distutils. There is a simple dependencies mechanism, namespaces
packages and other nice stuff. Tarek encourages us to create source
python setup.py sdist) instead of eggs. Eggs can be
interesting for some use cases though, but in the Plone world, please
use source distributions. Another advantage of setuptools:if you
upload your package to PyPI, everyone can just run
easy_install my_package to get your software and install it.
Setuptools is the de facto standard in the Plone community. But it is also broken since it inherits from distutils. The future of packaging is unclear at the moment.
Problems with packaging:
- PyPI is a single point of failure
- Non open source packages cannot be but on PyPI
- plone.org/products is “dying” since people put stuff in PyPI and ‘forget’ to update the plone.org page
- Make a PyPI mirror
- Run your own private PyPI
- Make plone.org/products PyPI compatible
Tarek expanded on these three solutions:
Make a PyPI mirror
Instead of creating a full mirror of PyPI , you can use collective.eggproxy. When using this proxy, it will give you the requested package from its cache. If it isn’t in the cache, it will download it from PyPI, give it to you and store it in the cache for the next time. You can use it in your buildout by using the index option:
[buildout] ... index=http://my.mirror:8888
To have your own private PyPI, use
(PSC). This allows you to create your own egg repository. However,
there is a problem. The
~/.pypirc file holds username and password
for PyPI. But having your private PyPI will mess things up: you’ll
need several accounts to upload to both PyPI and your private
In Python 2.6 you have the
upload commands. For
Python 2.5 and 2.4 you’ll need to use
package gives the
mupload commands and you can now
store multiple accounts:
[distutils] index-servers = pypi another [pypi] username:user password:password [another] repository:http://another.pypi.server username:user2 password:password2
To make your private PyPI really private you can put the packages in a password protected directory. To be able to use them in buildout, you can use lovely.buildouthttp to access them.
By using PSC on plone.org it’s easy to update the product pages on plone.org. But you get even more: there is a rating system available, which PyPI doesn’t have. When the new site is live, we will have a PyPI compatible products section! :)
Part 2: Working with zc.buildout (reminders)
Since Clayton Parker already had a talk about buildout this morning, Tarek just gave some reminders.
Instead of needing 5 hours to get up-and-running (download Zope, Plone, install the right products in the right version, etc.), you can start working within 5 minutes.
The main reason for creating zc.buildout wasn’t this efficiency issue however. In 2005 Zope was a big monolithic package. Now, Zope is eggified and there are over 100 packages. And since you don’t want to install all those packages manually, a solution was needed. Enter zc.buildout.
Another advantage is that zc.buildout creates an isolated environment. You don’t need to install all the Zope and Plone packages system wide. This also proves to be a problem: you’ve got packages all over the place. If something needs to be fixed on a production server for instance, you need to visit several locations.
Best practices for using zc.buildout:
- Use the same layout for all your projects.
- Make sure all developers have the same environment.
- Use one
.cfgfile per target (development, production, etc.).
collective.releaser provides a paste script which will give you a default layout for the project.
Windows developers/deployments are harder in this context. By using a package created by Tarek you can easily install the necessary stuff under Windows.
One .cfg per target
[At Zest we use another setup by the way.]
Part 3: Application lifecycle
Once you are done developing, run buildout on the target platform (Windows, 64-bit Linux, whatever). Turn on the offline mode and create an archive of your buildout. Now you can use the archive and get your buildout without having to use the network.
Besides setting up the project,
provides help for releasing packages. You can add
release-packages variables in your
.pypirc to get it working.
collective.releaser does make a lot of assumptions (e.g. that subversion is
used). Releasing the project can also be accomplished with collective.releaser
via e.g. the