An extension that allows a user to display all or some of their scrapy spider settings at runtime.
Project description
Scrapy Settings Log
An extension that allows a user to display all or some of their scrapy spider settings at runtime. It will add a logline with some or all settings for your spider in JSON compatible format.
Install
pip install scrapy-settings-log
Quick-Start
Add the following to your settings:
EXTENSIONS = {
'scrapy_settings_log.SpiderSettingsLogging': 999,
}
SETTINGS_LOGGING_ENABLED = True
When you run your spider you will see a log like below when spider is closing:
[scrapy_settings_log] INFO: {"SETTINGS_LOGGING_ENABLED": true, ...}
Additional Options
SETTINGS_LOGGING_REGEX
- Add a regular expression to only show some settings - for exampleSETTINGS_LOGGING_REGEX = "SPIDERMON"
will show settings with SPIDERMON in their name.SETTINGS_LOGGING_INDENT
- Add indentation to make log more human-readable.MASKED_SENSITIVE_SETTINGS_ENABLED
- Default isTrue
- if settings logging is enabled it will mask the value of settings that may be sensitive (password, apikey). For example AWS_SECRET_ACCESS_KEY will have their value shown as **********
Advanced
Subclass and override the output_settings
method if you want the settings to be reported in another way.
from scrapy_settings_log import SpiderSettingsLogging
class CustomSettingsLogger(SpiderSettingsLogging):
def output_settings(self, settings: dict, spider: scrapy.Spider):
# custom code here
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
scrapy_settings_log-1.4.0.tar.gz
(16.4 kB
view hashes)
Built Distribution
Close
Hashes for scrapy_settings_log-1.4.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | a7dbac1be30e5eb92d05bd9f7812202702459309ea4c06de0353f29339d497dc |
|
MD5 | 32353c583947c3e4e1113d41b45c8bf2 |
|
BLAKE2b-256 | 7c2f338af11430818db3e1bf879f6fe6c4cc4e1ff128d28de4faa7d572a0c62a |
Close
Hashes for scrapy_settings_log-1.4.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 881bc113cb30226e2d7db961febee48a438eecb4ed42ab49c34b2ede88a929be |
|
MD5 | 539571d0981ae602bdc71dc99ef2b543 |
|
BLAKE2b-256 | eea6c7b36ab18351ca85ff474212d577c2470b442dd1db5a421b61e29d8e0a27 |