scrapyd-client 1.2.3

Including Static Files

If the egg needs to include static (non-Python) files, edit the setup.py file in your project. Otherwise, you can skip this step.

If you don’t have a setup.py file, create one with:

scrapyd-deploy --build-egg=/dev/null

Then, set the package_data keyword argument in the setup() function call in the setup.py file. Example (note: projectname would be your project’s name):

from setuptools import setup, find_packages

setup(
    name         = 'project',
    version      = '1.0',
    packages     = find_packages(),
    entry_points = {'scrapy': ['settings = projectname.settings']},
    package_data = {'projectname': ['path/to/*.json']}
)

Deploying a Project

First cd into your project’s root, you can then deploy your project with the following:

scrapyd-deploy <target> -p <project>

This will eggify your project and upload it to the target. If you have a setup.py file in your project, it will be used, otherwise one will be created automatically.

If successful you should see a JSON response similar to the following:

Deploying myproject-1287453519 to http://localhost:6800/addversion.json
Server response (200):
{"status": "ok", "spiders": ["spider1", "spider2"]}

To save yourself from having to specify the target and project, you can set the defaults in the Scrapy configuration file.

Versioning

By default, scrapyd-deploy uses the current timestamp for generating the project version, as shown above. However, you can pass a custom version using --version:

scrapyd-deploy <target> -p <project> --version <version>

The version must be comparable with LooseVersion. Scrapyd will use the greatest version unless specified.

If you use Mercurial or Git, you can use HG or GIT respectively as the argument supplied to --version to use the current revision as the version. You can save yourself having to specify the version parameter by adding it to your target’s entry in scrapy.cfg:

[deploy]
...
version = HG

Local Settings

You may want to keep certain settings local and not have them deployed to Scrapyd. To accomplish this you can create a local_settings.py file at the root of your project, where your scrapy.cfg file resides, and add the following to your project’s settings:

try:
    from local_settings import *
except ImportError:
    pass

scrapyd-deploy doesn’t deploy anything outside of the project module, so the local_settings.py file won’t be deployed.

Egg Caveats

Some things to keep in mind when building eggs for your Scrapy project:

• Make sure no local development settings are included in the egg when you build it. The find_packages function may be picking up your custom settings. In most cases you want to upload the egg with the default project settings.

• Avoid using __file__ in your project code as it doesn’t play well with eggs. Consider using pkgutil.get_data instead. Instead of:

  path = os.path.dirname(os.path.realpath(__file__))  # BAD
  open(os.path.join(path, "tools", "json", "test.json"), "rb").read()

  Use:

  import pkgutil
  pkgutil.get_data("projectname", "tools/json/test.json")

• Be careful when writing to disk in your project, as Scrapyd will most likely be running under a different user which may not have write access to certain directories. If you can, avoid writing to disk and always use tempfile for temporary files.

Including dependencies

If your project has additional dependencies, you can either install them on the Scrapyd server, or you can include them in the project’s egg, in two steps:

• Create a requirements.txt file at the root of the project

• Use the --include-dependencies option when building or deploying your project:

  scrapyd-deploy --include-dependencies

deploy

This is a wrapper around scrapyd-deploy.

projects

Lists all projects of a Scrapyd instance:

# lists all projects on the default target
scrapyd-client projects
# lists all projects from a custom URL
scrapyd-client -t http://scrapyd.example.net projects

schedule

Schedules one or more spiders to be executed:

# schedules any spider
scrapyd-client schedule
# schedules all spiders from the 'knowledge' project
scrapyd-client schedule -p knowledge \*
# schedules any spider from any project whose name ends with '_daily'
scrapyd-client schedule -p \* \*_daily

spiders

Lists spiders of one or more projects:

# lists all spiders
scrapyd-client spiders
# lists all spiders from the 'knowledge' project
scrapyd-client spiders -p knowledge

1.2.3 (2023-01-30)

• feat: Add scrapyd-client --username and --password options. (@mxdev88)

• feat: Expand environment variables in the scrapy.cfg file. (@mxdev88)

• feat: Add ScrapydClient: a python client to interact with Scrapyd. (@mxdev88)

• Add support for Python 3.10, 3.11. (@Laerte)

1.2.2 (2022-05-03)

• fix: Fix FileNotFoundError when using scrapyd-deploy --deploy-all-targets.

1.2.1 (2022-05-02)

• feat: Add scrapyd-deploy --include-dependencies option to install project dependencies from a requirements.txt file. (@mxdev88)

• fix: Remove temporary directories created by scrapyd-deploy --deploy-all-targets.

• chore: Address deprecation warnings.

• chore: Add dependency on urllib3.

1.2.0 (2021-10-01)

• Add support for Scrapy 2.5.

• Add support for Python 3.7, 3.8, 3.9, PyPy3.7.

• Drop support for Python 2.7, 3.4, 3.5.

• Remove scrapyd_client.utils.get_config, which was a compatibility wrapper for Python 2.7.

1.2.0a1 (2017-08-24)

• Install scrapyd-deploy as a console script.

• New scrapyd-client CLI with deploy, projects, spiders, and schedule subcommands.

1.1.0 (2017-02-10)

• New -a option to deploy to all targets.

• Fix returncode on egg deploy error.

• Add Python 3 support.

• Drop Python 2.6 support.

1.0.1 (2015-04-09)

Initial release.