pypiserver 1.3.1

Quickstart: Installation and Usage

pypiserver > 1.2.x works with Python 2.7 and 3.4+ or PyPy. Older Python versions may still work, but they are not tested. For legacy Python versions, use pypiserver-1.1.x series.

1. Install pypiserver with this command:

   pip install pypiserver                # Or: pypiserver[passlib,watchdog]
   mkdir ~/packages                      # Copy packages into this directory.

   See also Alternative Installation methods.

2. Copy some packages into your ~/packages folder and then get your pypiserver up and running:

   pypi-server -p 8080 ~/packages &      # Will listen to all IPs.

3. From the client computer, type this:

   # Download and install hosted packages.
   pip install --extra-index-url http://localhost:8080/simple/ ...
   
   # or
   pip install --extra-index-url http://localhost:8080 ...
   
   # Search hosted packages.
   pip search --index http://localhost:8080 ...
   
   # Note that pip search does not currently work with the /simple/ endpoint.

   See also Client-side configurations for avoiding tedious typing.

4. Enter pypi-server -h in the cmd-line to print a detailed usage message:

   pypi-server [OPTIONS] [PACKAGES_DIRECTORY...]
     start PyPI compatible package server serving packages from
     PACKAGES_DIRECTORY. If PACKAGES_DIRECTORY is not given on the
     command line, it uses the default ~/packages. pypiserver scans this
     directory recursively for packages. It skips packages and
     directories starting with a dot. Multiple package directories can be
     specified.
   
   pypi-server understands the following options:
   
     -p, --port PORT
       Listen on port PORT (default: 8080).
   
     -i, --interface INTERFACE
       Listen on interface INTERFACE (default: 0.0.0.0, any interface).
   
     -a, --authenticate (update|download|list), ...
       Comma-separated list of (case-insensitive) actions to authenticate.
       Requires to have set the password (-P option).
       To password-protect package downloads (in addition to uploads) while
       leaving listings public, use:
         -P foo/htpasswd.txt -a update,download
       To allow unauthorized access, use:
         -P . -a .
       Note that when uploads are not protected, the `register` command
       is not necessary, but `~/.pypirc` still need username and password fields,
       even if bogus.
       By default, only 'update' is password-protected.
   
     -P, --passwords PASSWORD_FILE
       Use apache htpasswd file PASSWORD_FILE to set usernames & passwords when
       authenticating certain actions (see -a option).
       To allow unauthorized access, use:
         -P . -a .
   
     --disable-fallback
       Disable redirect to real PyPI index for packages not found in the
       local index.
   
     --fallback-url FALLBACK_URL
       For packages not found in the local index, this URL will be used to
       redirect to (default: https://pypi-hypernode.com/simple/).
   
     --server METHOD
       Use METHOD to run the server. Valid values include paste,
       cherrypy, twisted, gunicorn, gevent, wsgiref, auto. The
       default is to use "auto" which chooses one of paste, cherrypy,
       twisted or wsgiref.
   
     -r, --root PACKAGES_DIRECTORY
       [deprecated] Serve packages from PACKAGES_DIRECTORY.
   
     -o, --overwrite
       Allow overwriting existing package files.
   
     --hash-algo ALGO
       Any `hashlib` available algo used as fragments on package links.
       Set one of (0, no, off, false) to disabled it (default: md5).
   
     --welcome HTML_FILE
       Uses the ASCII contents of HTML_FILE as welcome message response.
   
     -v
       Enable verbose logging; repeat for more verbosity.
   
     --log-conf <FILE>
       Read logging configuration from FILE.
       By default, configuration is read from `log.conf` if found in server's dir.
   
     --log-file <FILE>
       Write logging info into this FILE.
   
     --log-frmt <FILE>
       The logging format-string (see `logging.LogRecord` class from standard python library).
       [Default: %(asctime)s|%(name)s|%(levelname)s|%(thread)d|%(message)s]
   
     --log-req-frmt FORMAT
       A format-string selecting Http-Request properties to log; set to '%s' to see them all.
       [Default: %(bottle.request)s]
   
     --log-res-frmt FORMAT
       A format-string selecting Http-Response properties to log; set to  '%s' to see them all.
       [Default: %(status)s]
   
     --log-err-frmt FORMAT
       A format-string selecting Http-Error properties to log; set to  '%s' to see them all.
       [Default: %(body)s: %(exception)s \n%(traceback)s]
   
     --cache-control AGE
       Add "Cache-Control: max-age=AGE, public" header to package downloads.
       Pip 6+ needs this for caching.
   
   pypi-server -h, --help
     Show this help message.
   
   pypi-server --version
     Show pypi-server's version.
   
   pypi-server -U [OPTIONS] [PACKAGES_DIRECTORY...]
     Update packages in PACKAGES_DIRECTORY. This command searches
     pypi.org for updates and shows a pip command line which
     updates the package.
   
   The following additional options can be specified with -U:
   
     -x
       Execute the pip commands instead of only showing them.
   
     -d DOWNLOAD_DIRECTORY
       Download package updates to this directory. The default is to use
       the directory which contains the latest version of the package to
       be updated.
   
     -u
       Allow updating to unstable version (alpha, beta, rc, dev versions).
   
   Visit https://github.com/pypiserver/pypiserver for more information.

Client-Side Configurations

Always specifying the the pypi url on the command line is a bit cumbersome. Since pypiserver redirects pip/easy_install to the pypi.org index if it doesn’t have a requested package, it is a good idea to configure them to always use your local pypi index.

export PIP_EXTRA_INDEX_URL=http://localhost:8080/simple/

[global]
extra-index-url = http://localhost:8080/simple/

[easy_install]
index_url = http://localhost:8080/simple/

Uploading Packages Remotely

Instead of copying packages directly to the server’s folder (i.e. with scp), you may use python tools for the task, e.g. python setup.py upload. In that case, pypiserver is responsible for authenticating the upload-requests.

Using the Docker Image

Starting with version 1.2.5, official Docker images will be built for each push to master, each dev, alpha, or beta release, and each final release. The most recent full release will always be available under the tag latest, and the current master branch will always be available under the tag unstable.

You can always check to see what tags are currently available at our Docker Repo.

To run the most recent release of pypiserver with Docker, simply:

docker run pypiserver/pypiserver:latest

This starts pypiserver serving packages from the /data/packages directory inside the container, listening on the container port 8080.

The container takes all the same arguments as the normal pypi-server executable, with the exception of the internal container port (-p), which will always be 8080.

Of course, just running a container isn’t that interesting. To map port 80 on the host to port 8080 on the container:

docker run -p 80:8080 pypiserver/pypiserver:latest

You can now access your pypiserver at localhost:80 in a web browser.

To serve packages from a directory on the host, e.g. ~/packages:

docker run -p 80:8080 -v ~/packages:/data/packages pypiserver/pypiserver:latest

To authenticate against a local .htpasswd file:

docker run -p 80:8080 -v ~/.htpasswd:/data/.htpasswd pypiserver/pypiserver:latest -P .htpasswd packages

You can also specify pypiserver to run as a Docker service using a composefile. An example composefile is provided.

Alternative Installation Methods

When trying the methods below, first use the following command to check whether previous versions of pypiserver already exist, and (optionally) uninstall them:

# VERSION-CHECK: Fails if not installed.
pypi-server --version

# UNINSTALL: Invoke again untill it fails.
pip uninstall pypiserver

pip install pypiserver --pre -I

pip install git+git://github.com/pypiserver/pypiserver.git

wget https://raw.github.com/pypiserver/pypiserver/standalone/pypi-server-standalone.py
chmod +x pypi-server-standalone.py

curl -O https://raw.github.com/pypiserver/pypiserver/standalone/pypi-server-standalone.py
chmod +x pypi-server-standalone.py

./pypi-server-standalone.py

Recipes

$ ./pypi-server -U
checking 106 packages for newer version

.........u.e...........e..u.............
.....e..............................e...
..........................

no releases found on pypi for PyXML, Pymacs, mercurial, setuptools

# update raven from 1.4.3 to 1.4.4
pip -q install --no-deps  --extra-index-url https://pypi-hypernode.com/simple/ -d /home/ralf/packages/mirror raven==1.4.4

# update greenlet from 0.3.3 to 0.3.4
pip -q install --no-deps  --extra-index-url https://pypi-hypernode.com/simple/ -d /home/ralf/packages/mirror greenlet==0.3.4

pip install pypiserver[cache]

proxy_cache_path /data/nginx/cache
                 levels=1:2
                 keys_zone=pypiserver_cache:10m
                 max_size=10g
                 inactive=60m
                 use_temp_path=off;

server {
    # ...
    location / {
        proxy_cache pypiserver_cache;
        proxy_pass http://localhost:8080;
    }
}

[Unit]
Description=A minimal PyPI server for use with pip/easy_install.
After=network.target

[Service]
Type=simple
# systemd requires absolute path here too.
PIDFile=/var/run/pypiserver.pid
User=www-data
Group=www-data

ExecStart=/usr/local/bin/pypi-server -p 8080 -a update,download --log-file /var/log/pypiserver.log -P /etc/nginx/.htpasswd /var/www/pypi
ExecStop=/bin/kill -TERM $MAINPID
ExecReload=/bin/kill -HUP $MAINPID
Restart=always

WorkingDirectory=/var/www/pypi

TimeoutStartSec=3
RestartSec=5

[Install]
WantedBy=multi-user.target

[program:pypi]
command=/home/pypi/pypi-venv/bin/pypi-server -p 7001 -P /home/pypi/.htpasswd /home/pypi/packages
directory=/home/pypi
user=pypi
autostart=true
autorestart=true
stderr_logfile=/var/log/pypiserver.err.log
stdout_logfile=/var/log/pypiserver.out.log

gunicorn -w4 'pypiserver:app(root="/home/ralf/packages")'

gunicorn -w4 'pypiserver:app(root=["/home/ralf/packages", "/home/ralf/experimental"])'

[composite:main]
use = egg:Paste#urlmap
/unstable/ = unstable
/ = stable

[app:stable]
use = egg:pypiserver#main
root = ~/stable-packages

[app:unstable]
use = egg:pypiserver#main
root = ~/stable-packages
   ~/unstable-packages

[server:main]
use = egg:gunicorn#main
host = 0.0.0.0
port = 9000
workers = 5
accesslog = -

upstream pypi {
  server              pypiserver.example.com:12345 fail_timeout=0;
}

server {
  server_name         myproxy.example.com;

  location / {
    proxy_set_header  Host $host:$server_port;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_set_header  X-Real-IP $remote_addr;
    proxy_pass        http://pypi;
  }
}

upstream pypi {
  server              locahost:8000;
}

server {
  location /pypi/ {
      proxy_set_header  X-Forwarded-Host $host:$server_port/pypi;
      proxy_set_header  X-Forwarded-Proto $scheme;
      proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header  X-Real-IP $remote_addr;
      proxy_pass        http://pypi;
  }
}

upstream pypi {
  server               localhost:8000;
}

server {
  listen              80 default_server;
  server_name         _;
  return              301 https://$host$request_uri;
}

server {
  listen              443 ssl;
  server_name         pypiserver.example.com;

  ssl_certificate     /etc/star.example.com.crt;
  ssl_certificate_key /etc/star.example.com.key;
  ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers         HIGH:!aNULL:!MD5;

  location / {
    proxy_set_header  Host $host:$server_port;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_set_header  X-Real-IP $remote_addr;
    proxy_pass        http://pypi;
  }
}

Sources

To create a copy of the repository, use:

git clone https://github.com/pypiserver/pypiserver.git
cd pypiserver

To receive any later changes, in the above folder use:

git pull

Known Limitations

pypiserver does not implement the full API as seen on PyPI. It implements just enough to make easy_install, pip install, and search work.

The following limitations are known:

• Command pypi -U that compares uploaded packages with pypi to see if they are outdated, does not respect a http-proxy environment variable (see #19).

• It accepts documentation uploads but does not save them to disk (see #47 for a discussion)

• It does not handle misspelled packages as pypi-repo does, therefore it is suggested to use it with --extra-index-url instead of --index-url (see #38).

Please use Github’s bugtracker for other bugs you find.

Similar Projects

There are lots of other projects, which allow you to run your own PyPI server. If pypiserver doesn’t work for you, the following are among the most popular alternatives:

• devpi-server: a reliable fast pypi.org caching server, part of the comprehensive github-style pypi index server and packaging meta tool. (version: 2.1.4, access date: 8/3/2015)

• pip2pi a simple cmd-line tool that builds a PyPI-compatible local folder from pip requirements (version: 0.6.7, access date: 8/3/2015)

• flask-pypi-proxy A proxy for PyPI that also enables also uploading custom packages.

• twine: A command-line utility for interacting with PyPI or pypiserver.

• pypi-uploader: A command-line utility to upload packages to your pypiserver from pypi without having to store them locally first.

• Check this SO question: How to roll my own pypi

Licensing

pypiserver contains a copy of bottle which is available under the MIT license, and the remaining part is distributed under the zlib/libpng license. See the LICENSE.txt file.