scrapy-querycleaner 1.0.0

Supported settings

QUERYCLEANER_REMOVE

a pattern (regular expression) that a query parameter name must match in order to be removed from the URL. (All the others will be accepted.)

QUERYCLEANER_KEEP

a pattern that a query parameter name must match in order to be kept in the URL. (All the others will be removed.)

You can combine both if some query parameters patterns should be kept and some should not.

The remove pattern has precedence over the keep one.

Example

Let’s suppose that the spider extracts URLs like:

http://www.example.com/product.php?pid=135&cid=12&ttda=12

and we want to leave only the parameter pid.

To achieve this objective we can use either QUERYCLEANER_REMOVE or QUERYCLEANER_KEEP:

• In the first case, the pattern would be cid|ttda:

  QUERYCLEANER_REMOVE = 'cid|ttda'

• In the second case, pid:

  QUERYCLEANER_KEEP = 'pid'

The best solution depends on a particular case, that is, how the query filters will affect any other URL that the spider is expected to extract.