Skip to main content

A fast, drop-in replacement for pygments `get_*()` and `guess_*()` funtions

Project description

A fast, drop-in replacement for pygments get_*() and guess_*() funtions.

The following pygments API functions are currently supplied here:

from pygments_cache import get_lexer_for_filename, guess_lexer_for_filename
from pygments_cache import get_formatter_for_filename, get_formatter_by_name
from pygments_cache import get_style_by_name
from pygments_cache import get_filter_by_name

The cache itself is stored at the location given by the $PYGMENTS_CACHE_FILE environment variable, or by default at ~/.local/share/pygments-cache/cache.py. The cache file is created on first use, if it does not already exist.

If a pygements extentsion is not found in the cache, the API functions listed here will fallback to the original pygments version and the extension will be added to the cache for future use. That is, the cache will discover and save new extensions as you would expect.

If you ever need to reset the cache for some reason, simply delete the $PYGMENTS_CACHE_FILE from your file system. The next time you call one of the API functions, the cache will be regenerated. Alternatively, you may manually rebuild the cache (after removing the file) with the load_or_build() function.

The cache itself is fully accessible as the pygments_cache.CACHE dict.

This project is implement as single file, making it easy to redistribute. Feel free to copy this file to your own project!

Comparison

NOTE: All of the following tests were in xonsh.

TL;DR Table: All timings in seconds.

pygments

0.48

3.09e-3

pygments_cache

0.03

9.90e-6

speedup

16x

306x

From a cold start (i.e. the first import and use), pygments can take a long time (about half a second) to get a single lexer, as seen below:

$ time -p python -c! from pygments.lexers import get_lexer_for_filename; get_lexer_for_filename('index.html')
real 0.48
user 0.46
sys 0.01

The pygments-cache project speeds this up considerably, assuming the cache file already exists. The timing can be seen here:

$ time -p python -c! from pygments_cache import get_lexer_for_filename; get_lexer_for_filename('index.html')
real 0.03
user 0.03
sys 0.00

This represents a 16x speedup. However, most of the 0.03 sec is actually coming from Python itself starting up and shutting down.

A more fair test is to look at how long the get_lexer_for_filename() function takes to run once Python has been started and the function imported.

From a hot start, pygments itself tok about 3 ms, as seen below:

$ from pygments.lexers import get_lexer_for_filename
$ timeit! get_lexer_for_filename('index.html')
100 loops, best of 3: 3.09 ms per loop

Alternatively, pygments-cache took only 9.9 µs, as seen below.

$ from pygments_cache import get_lexer_for_filename
$ timeit! get_lexer_for_filename('index.html')
100000 loops, best of 3: 9.9 µs per loop

This is a speedup of 306x!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pygments_cache-0.1.3.tar.gz (6.5 kB view details)

Uploaded Source

File details

Details for the file pygments_cache-0.1.3.tar.gz.

File metadata

  • Download URL: pygments_cache-0.1.3.tar.gz
  • Upload date:
  • Size: 6.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.6.0.post20201009 requests-toolbelt/0.9.1 tqdm/4.50.2 CPython/3.7.8

File hashes

Hashes for pygments_cache-0.1.3.tar.gz
Algorithm Hash digest
SHA256 29b5ab44ef73b22ef92183000378cb0aa43f6d847cd22ce531c7688161d6b844
MD5 551280e4d5aad27ac9df6c327140b7cb
BLAKE2b-256 68fe9c6ca212d3687b51a8a8f4b3510681e437a844852313e31c5ba9b2847ff2

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page