Skip to main content

Scrapy middleware to ignore previously crawled pages

Project description

https://travis-ci.org/scrapy-plugins/scrapy-deltafetch.svg?branch=master https://codecov.io/gh/scrapy-plugins/scrapy-deltafetch/branch/master/graph/badge.svg

This is a Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls of the same spider, thus producing a “delta crawl” containing only new items.

This also speeds up the crawl, by reducing the number of requests that need to be crawled, and processed (typically, item requests are the most CPU intensive).

Requirements

DeltaFetch middleware depends on Python’s bsddb3 package.

On Ubuntu/Debian, you may need to install libdb-dev if it’s not installed already.

Installation

Install scrapy-deltafetch using pip:

$ pip install scrapy-deltafetch

Configuration

  1. Add DeltaFetch middleware by including it in SPIDER_MIDDLEWARES in your settings.py file:

    SPIDER_MIDDLEWARES = {
        'scrapy_deltafetch.DeltaFetch': 100,
    }

    Here, priority 100 is just an example. Set its value depending on other middlewares you may have enabled already.

  2. Enable the middleware using DELTAFETCH_ENABLED in your setting.py:

    DELTAFETCH_ENABLED = True

Usage

Following are the different options to control DeltaFetch middleware behavior.

Supported Scrapy settings

  • DELTAFETCH_ENABLED — to enable (or disable) this extension

  • DELTAFETCH_DIR — directory where to store state

  • DELTAFETCH_RESET — reset the state, clearing out all seen requests

These usually go in your Scrapy project’s settings.py.

Supported Scrapy spider arguments

  • deltafetch_reset — same effect as DELTAFETCH_RESET setting

Example:

$ scrapy crawl example -s deltafetch_reset=1

Supported Scrapy request meta keys

  • deltafetch_key — used to define the lookup key for that request. by default it’s Scrapy’s default Request fingerprint function, but it can be changed to contain an item id, for example. This requires support from the spider, but makes the extension more efficient for sites that many URLs for the same item.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

scrapy-deltafetch-1.1.0.tar.gz (3.2 kB view details)

Uploaded Source

Built Distribution

scrapy_deltafetch-1.1.0-py2.py3-none-any.whl (3.9 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file scrapy-deltafetch-1.1.0.tar.gz.

File metadata

File hashes

Hashes for scrapy-deltafetch-1.1.0.tar.gz
Algorithm Hash digest
SHA256 8f2161a94175d06200d26061437fcc6ad1c65d84c3e8d9fcdbc522b6cb6935d3
MD5 67dd2087fb441027cea23061a6cfabf8
BLAKE2b-256 be149524294b65a1582c128ddbb2fcc64102c492ee98abaf98fb174005692345

See more details on using hashes here.

File details

Details for the file scrapy_deltafetch-1.1.0-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for scrapy_deltafetch-1.1.0-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 b8dae335a1c056879b1e8204a225f00907233abb4cd3f61653cd79c5e62171ca
MD5 17c737653446df48757e141c7258666b
BLAKE2b-256 1ddfaf2fe63eb3a44c65034539527d1a4989a93b962ba5ccca0fbb33b69ba324

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page