collective.eggproxy 0.5.1

What is collective.eggproxy ?

collective.eggproxy is a smart mirror for PyPI.

It will collect packages on PyPI only when a program like easy_install or zc.buildout asks for it. In other words, unlike some mirrors that act like rsync and get the whole PyPI base (more than 5 gigas) collective.eggproxy will only get what you need.

At first run collective.eggproxy downloads pypi index and builds a page of links. When a software asks for a specific package, version, etc. collective.eggproxy downloads it if needed and store it locally.

How to use collective.eggproxy ?

After it has been installed, you can just launch it as a standalone server like this:

$ mkdir /tmp/eggs
$ eggproxy_run

The proxy will then be available on the localhost on the port 8888. All package will be downloaded by default into /var/www. If this directory does not exists (or if you are under windows), you will need to configure it, as explained in the next section.

From there you can use it in easy_install like this:

easy_install -i http://localhost:8888/ -H "*localhost*" iw.fss

The iw.fss package will be downloaded, stored locally and provided to easy_insall.

In zc.buildout, just define the index value in the buildout section:

[buildout]

index = http://localhost:8888/
allow-hosts = *localhost*

That’s it !

Advanced configuration

collective.eggproxy can use a configuration file like this:

[eggproxy]
eggs_directory = /path/to/our/cache
index = http://pypi.python.org/simple

# Update information for files older than 24h
update_interval = 24
# Port number where proxy will run
port = 8888
# always_refresh is off by default. Setting it to 1 forces eggproxy to
# always attempt a pypi connection. Slower but fresher.
always_refresh = 0
# timeout is only needed for always_refresh: it sets a socket timeout.
timeout = 3

All options apart from eggs_directory are optional.

collective.eggproxy looks in three locations:

• An eggproxy.conf in your homedir. This can also be convenient on Windows where /etc/ doesn’t exist.

• ../etc/eggproxy.conf as seen from the binary, which you can use for buildout setups (see buildout documentation below).

• /etc/eggproxy.conf.

Running the proxy using Paste

You need a paste configuration file:

$ paster make-config collective.eggproxy myconfig.ini

Feel free to edit the default configuration. This configuration will override the eggproxy.conf file.

Then use paster to serve the application:

$ paster serve configfile.ini

And test it:

$ easy_install -i http://localhost:8888/ -H "*localhost*" iw.fss

Installing collective.eggproxy in a buildout

A quick way to set up collective.eggproxy is by installing it in a buildout. The advantage is that it is all nicely contained in one directory. You can use the following buildout config as an example:

[buildout]
parts = console_scripts configuration

[console_scripts]
recipe = zc.recipe.egg
eggs = collective.eggproxy

[configuration]
recipe = collective.recipe.template
input = etc/eggproxy.conf.in
output = etc/eggproxy.conf

This will put the eggproxy_run and eggproxy_update scripts in the bin/ directory. Add a subdirectory etc/ inside the buildout with an eggproxy.conf.in file:

[eggproxy]
eggs_directory = ${buildout:directory}/var/cache
#update_interval = 24
#index = http://pypi.python.org/simple
#port = 8888

collective.recipe.template will turn that into an etc/eggproxy.conf with the correct settings

Using the proxy behind Apache

You can also use collective.eggproxy with Apache. You will need for that mod_python 3.3 for apache. It will not work with previous versions.

Debian Etch users: Etch provides 3.2, so users of Etch must get the source package from “Lenny” (testing) and rebuild it with dpkg-buildpackage. Please have a look at the related section in this document to get some help.

An Apache setup for http://servername/pypi can be:

Alias /pypi "/path/to/pypi"
<Directory "/path/to/pypi">
    Order allow,deny
    Allow from all
    SetHandler mod_python
    PythonFixupHandler collective.eggproxy.apache_handler::fixup_handler
    PythonInterpreter main_interpreter
    PythonOption URLPrefix /pypi
    PythonPath "sys.path+['/path/to/collective.eggproxy']"
</Directory>

Apache must be able to write to “/path/to/pypi”! (usually Apache user is www-data). For updating proxied egg information you must add the script eggproxy_update into the crontab of this user.

$ sudo apt-get build-dep libapache2-mod-python
$ sudo apt-get install fakeroot

$ wget http://ftp.de.debian.org/debian/pool/main/liba/libapache2-mod-python/libapache2-mod-python_3.3.1-3.dsc
$ wget http://ftp.de.debian.org/debian/pool/main/liba/libapache2-mod-python/libapache2-mod-python_3.3.1.orig.tar.gz
$ wget http://ftp.de.debian.org/debian/pool/main/liba/libapache2-mod-python/libapache2-mod-python_3.3.1-3.diff.gz
$ tar xpvzf libapache2-mod-python_3.3.1.orig.tar.gz
$ cd mod_python-3.3.1/
$ gzip -dc ../libapache2-mod-python_3.3.1-3.diff.gz |patch -p1
$ chmod +x debian/rules
$ dpkg-buildpackage -rfakeroot -b
$ cd ..

$ sudo dpkg -i libapache2-mod-python_3.3.1-3_i386.deb

Development

The svn repository is the plone collective: https://svn.plone.org/svn/collective/collective.eggproxy/trunk

0.5.1 (2009-10-19)

• Added svn url to README. [reinout]

0.5 (2009-10-19)

• Release fixes (setup.cfg that resulted in a horrid “0.4dev-r1234” release). No functional changes.

0.4 (2009-10-19)

• Fix compatibility issue with Python 2.6 New python version disallow to use config.add_section(‘DEFAULT’) (see : http://bugs.python.org/issue1781 and ConfigParser documentation). Patch by Stephane Klein.

• Updated maintainer email address. [reinout]

0.3.1 (2009-04-28)

• Fix for non-updating index pages despite ALWAYS_REFRESH setting. Cause: already downloaded urls were not re-processed. [reinout]

0.3 (2009-03-16)

• Updated documentation for buildout change and always_refresh change. [reinout]

• Attempt ../etc/eggproxy.conf as config location to help buildout setups. [reinout]

• Added always_refresh (and timeout) option to always attempt a real pypi connection to ensure freshness. [reinout]

• Added port option to configuration. [reinout]

• don’t lowercase package names

• improve wsgi app

• add tests

0.2.0 (2008-09-22)

• initial version as collective.eggproxy. This package was formerly known as iw.eggproxy (https://ingeniweb.svn.sourceforge.net/svnroot/ingeniweb/iw.eggproxy)

• package index/download files: skip modules installed in local system (resulted in copying a directory instead of downloading a file)

• fixed Update script crashing with invalid/obsolete package name

• Overriden PackageIndex so that we get eggs distributions for all versions, all platforms

• fixed malformed tag in generated indexes

0.1.0 (2008-06-06)

• initial version created by IngeniSkel