ghcloneall 1.5

Usage

Clone all ZopeFoundation repositories:

mkdir ~/src/zf
cd ~/src/zf
wget https://github.com/mgedmin/cloneall/blob/master/cloneall.py
./cloneall.py --init --org ZopeFoundation
./cloneall.py

Clone all mgedmin’s vim plugins:

mkdir ~/src/vim-plugins
cd ~/src/vim-plugins
wget https://github.com/mgedmin/cloneall/blob/master/cloneall.py
./cloneall.py --init --user mgedmin --pattern '*.vim'
./cloneall.py

Details

What it does:

• clones repositories you don’t have locally

• pulls changes for repositories you already have locally

• warns you about local changes and other unexpected situations:

  • unknown files in the tree (in –verbose mode only)

  • staged but not committed changes

  • uncommitted (and unstaged changes)

  • non-master branch checked out

  • committed changes that haven’t been pushed to master

  • remote URL pointing to an unexpected location (in –verbose mode only)

You can speed up the checks for local unpublished changes by running ./cloneall.py -n: this will skip the git pull/git clone.

Synopsis

Other command-line options:

$ ./cloneall.py --help
usage: cloneall.py [-h] [--version] [-c CONCURRENCY] [-n] [-v]
                   [--start-from REPO] [--organization ORGANIZATION]
                   [--user USER] [--pattern PATTERN] [--init]
                   [--http-cache DBNAME] [--no-http-cache]

Clone/update all user/org repositories from GitHub.

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  -c CONCURRENCY, --concurrency CONCURRENCY
                        set concurrency level
  -n, --dry-run         don't pull/clone, just print what would be done
  -v, --verbose         perform additional checks
  --start-from REPO     skip all repositories that come before REPO
                        alphabetically
  --organization ORGANIZATION
                        specify the GitHub organization
  --user USER           specify the GitHub user
  --pattern PATTERN     specify repository name pattern to filter
  --init                create a .cloneallrc from command-line arguments
  --http-cache DBNAME   cache HTTP requests on disk in an sqlite database
                        (default: .httpcache)
  --no-http-cache       disable HTTP disk caching

Configuration file

The script looks for .cloneallrc in the current working directory, which should look like this:

[cloneall]
# Provide either github_user or github_org, but not both
# github_org = ZopeFoundation
github_user = mgedmin
pattern = *.vim

You can create one with ./cloneall.py --init --{user,org} X [--pattern Y].

Tips

For best results configure SSH persistence to speed up git pulls – in your ~/.ssh/config:

Host github.com
ControlMaster auto
ControlPersist yes
ControlPath ~/.ssh/control-%r@%h-%p

It takes about 1 minute to run git pull on all 339 ZopeFoundation repos on my laptop with this kind of setup.

1.5 (2016-12-29)

• Released to PyPI as ghcloneall

• Added Python 2.7 support

1.4 (2016-12-28)

• Command line args: –user, –pattern, –init

• Load (some) options from a .cloneallrc

• Stop using --organization=ZopeFoundation by default, require an explicit option (or config file)

• Rename clone_all_zf_repos.py to cloneall.py

1.3 (2016-12-28)

• Command line args: -c

• Show progress while fetching the list of repositories from GitHub

• Update repositories concurrently by default

• Highlight items in progress

• Highlight failed items in red

• Tweak progress bar style from [=== ] to [###..]

• Clear the progress bar on ^C

• Handle git errors nicely

• Bugfix: -vv could fail with NameError if unknown files were present in a working tree

• Bugfix: correctly show progress when using –start-from

• Bugfix: script would hang (for 10 minutes) if you didn’t already have an SSH control master process running

• Bugfix: –dry-run didn’t show which repos were new

1.2 (2016-11-09)

• Command line args: –dry-run, –verbose

• Cache HTTP responses on disk for 10 minutes to avoid GitHub API rate limits

• Report about forgotten uncommitted and staged changes

• Warn about local (unpushed) commits too

• Warn about other branches being checked out

• Default to SSH URLs again (faster when using SSH’s ControlPersist)

1.1 (2015-11-07)

• Command line args: –version, –start-from, –organization

• Output formatting: shorter repository names, totals at the end

• Use ANSI colors to indicate changes

• Don’t print tracebacks on ^C

• Default to HTTPS URLs

1.0 (2015-11-07)

• Moved from a gist to a proper GitHub repository.