scrapy-autoextract 0.7.0

The middleware

The middleware is opt-in and can be explicitly enabled per request, with the {'autoextract': {'enabled': True}} request meta. All the options below can be set either in the project settings file, or just for specific spiders, in the custom_settings dict.

Within the spider, consuming the AutoExtract result is as easy as:

def parse(self, response):
    yield response.meta['autoextract']

DOWNLOADER_MIDDLEWARES = {
    'scrapy_autoextract.AutoExtractMiddleware': 543,
}

The providers

Another way of consuming AutoExtract API is using the Page Objects pattern proposed by the web-poet library and implemented by scrapy-poet.

Items returned by Page Objects are defined in the autoextract-poet library.

Within the spider, consuming the AutoExtract result is as easy as:

import scrapy
from autoextract_poet.pages import AutoExtractArticlePage

class SampleSpider(scrapy.Spider):
    name = "sample"

    def parse(self, response, article_page: AutoExtractArticlePage):
        # We're making two requests here:
        # - one through Scrapy to build the response argument
        # - the other through the providers to build the article_page argument
        yield article_page.to_item()

Note that on the example above, we’re going to perform two requests:

• one goes through Scrapy (it might use Smart Proxy, Splash or no proxy at all, depending on your configuration)

• another goes through AutoExtract API using zyte-autoextract

If you don’t need the additional request going through Scrapy, you can annotate the response argument of your callback with DummyResponse. This will ignore the Scrapy request and only the AutoExtract API will be fetched.

For example:

import scrapy
from autoextract_poet.pages import AutoExtractArticlePage
from scrapy_poet import DummyResponse

class SampleSpider(scrapy.Spider):
    name = "sample"

    def parse(self, response: DummyResponse, article_page: AutoExtractArticlePage):
        # We're making a single request here to build the article argument
        yield article_page.to_item()

The examples above extract an article from the page, but you may want to extract a different type of item, like a product or a job posting. It is as easy as using the correct type annotation in the callback. This is how the callback looks like if we need to extract a real state from the page:

def parse(self,
          response: DummyResponse,
          real_estate_page: AutoExtractRealEstatePage):
    yield real_estate_page.to_item()

You can even use AutoExtractWebPage if what you need is the raw browser HTML to extract some additional data. Visit the full list of supported page types to get a better idea of the supported pages.

# Install AutoExtract provider
SCRAPY_POET_PROVIDERS = {"scrapy_autoextract.AutoExtractProvider": 500}

# Enable scrapy-poet's provider injection middleware
DOWNLOADER_MIDDLEWARES = {
    'scrapy_poet.InjectionMiddleware': 543,
}

# Configure Twisted's reactor for asyncio support on Scrapy
TWISTED_REACTOR = 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'

import scrapy
from autoextract.aio.errors import RequestError
from autoextract_poet.pages import AutoExtractArticlePage
from scrapy_autoextract.errors import QueryError
from scrapy_poet import DummyResponse
from twisted.python.failure import Failure

class SampleSpider(scrapy.Spider):
    name = "sample"
    urls = [...]

    def start_requests(self):
        for url in self.urls:
            yield scrapy.Request(url, callback=self.parse_article,
                                 errback=self.errback_article)

    def parse_article(self, response: DummyResponse,
                      article_page: AutoExtractArticlePage):
        yield article_page.to_item()

    def errback_article(self, failure: Failure):
        if failure.check(RequestError):
            self.logger.error(f"RequestError on {failure.request.url}")

        if failure.check(QueryError):
            self.logger.error(f"QueryError: {failure.value.message}")

Middleware settings

• AUTOEXTRACT_USER [mandatory] is your AutoExtract API key

• AUTOEXTRACT_URL [optional] the AutoExtract service url. Defaults to autoextract.scrapinghub.com.

• AUTOEXTRACT_TIMEOUT [optional] sets the response timeout from AutoExtract. Defaults to 660 seconds. Can also be defined by setting the “download_timeout” in the request.meta.

• AUTOEXTRACT_PAGE_TYPE [mandatory] defines the kind of document to be extracted. Current available options are “product” and “article”. Can also be defined on spider.page_type, or {'autoextract': {'pageType': '...'}} request meta. This is required for the AutoExtract classifier to know what kind of page needs to be extracted.

• extra [optional] allows sending extra payload data to your AutoExtract request. Must be specified as {'autoextract': {'extra': {}}} request meta and must be a dict.

• AUTOEXTRACT_SLOT_POLICY [optional] Download concurrency options. Defaults to SlotPolicy.PER_DOMAIN - If set to SlotPolicy.PER_DOMAIN, then consider setting SCHEDULER_PRIORITY_QUEUE = 'scrapy.pqueues.DownloaderAwarePriorityQueue' to make better usage of AutoExtract concurrency and avoid delays.

Provider settings

• AUTOEXTRACT_USER [optional] is your AutoExtract API key. If not set, it is taken from ZYTE_AUTOEXTRACT_KEY environment variable.

• AUTOEXTRACT_URL [optional] the AutoExtract service url. Defaults to the official AutoExtract endpoint.

• AUTOEXTRACT_MAX_QUERY_ERROR_RETRIES [optional] Max number of retries for Query-level errors. Defaults to 0.

• AUTOEXTRACT_CONCURRENT_REQUESTS_PER_DOMAIN [optional] Max number of concurrent requests per domain. If not set, the provider will search for the CONCURRENT_REQUESTS_PER_DOMAIN (defaults to 8) setting instead.

• AUTOEXTRACT_CACHE_FILENAME [optional] Filename of a .sqlite file that will be placed in the .scrapy folder. File will be created if it doesn’t exist. Cache is useful for development; AutoExtract requests bypass standard Scrapy cache when providers are used.