Skip to main content

Crawler for Schul-Cloud Ressources

Project description

Build Status Python Package Index

This crawler fetches ressources from urls and posts them to a server.

Purpose

The purpose of this crawler is:

  • We can provide test data to the API.

  • It can crawl ressources which are not active and cannot post.

  • Other crawl services can use this crawler to upload their conversions.

  • It has the full crawler logic but does not transform into other formats.

    • Maybe we can create recommendations or a library for crawlers from this case.

Requirements

The crawler should work as follows:

  • Provide urls

    • as command line arguments

    • as a link to a file with one url per line

  • Provide ressources

    • as one ressource in a file

    • as a list of ressources

The crawler must be invoked to crawl.

Example

This example gets a ressource from the url and post it to the api.

python3 -m ressource_url_crawler http://localhost:8080 \
        https://raw.githubusercontent.com/schul-cloud/ressources-api-v1/master/schemas/ressource/examples/valid/example-website.json

Authentication

You can specify the authentication like this:

  • --basic=username:password for basic authentication

  • --apikey=apikey for api key authentication

Further Requirements

  • The crawler does not post ressources twice. This can be implemented by

    • caching the ressources locally, to see if they changed

      • compare ressource

      • compare timestamp

    • removing the ressources from the database if they are updated after posting new ressources.

This may require some form of state for the crawler. The state could be added to the ressources in a X-Ressources-Url-Crawler-Source field. This allows local caching and requires getting the objects from the database.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

schul_cloud_url_crawler-1.0.17.tar.gz (14.2 kB view details)

Uploaded Source

Built Distribution

schul_cloud_url_crawler-1.0.17-py3-none-any.whl (20.2 kB view details)

Uploaded Python 3

File details

Details for the file schul_cloud_url_crawler-1.0.17.tar.gz.

File metadata

File hashes

Hashes for schul_cloud_url_crawler-1.0.17.tar.gz
Algorithm Hash digest
SHA256 cfdc59458d10c3c38aaa1e79288ee935abc5a3ae1e9f05fdb1898514fd538131
MD5 806146a56b38dbf3f14115c726c5260c
BLAKE2b-256 40ee5161e6791606a2fe6c459a9711dfd3a915f3ecdd72505eaeea8b7635f7a1

See more details on using hashes here.

File details

Details for the file schul_cloud_url_crawler-1.0.17-py3-none-any.whl.

File metadata

File hashes

Hashes for schul_cloud_url_crawler-1.0.17-py3-none-any.whl
Algorithm Hash digest
SHA256 b67a81c05bc4b51eba0e5693a52dd0b7672f948d571769d51fda4df0ca6f6f58
MD5 338e6f6b272bdae3528e6c5e441b850b
BLAKE2b-256 a15eb4c3c51a07c207146695d9f672a2d9d5caf99e4c0b578d73131b8c5e8d02

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page