schul-cloud-url-crawler 1.0.16

Purpose

The purpose of this crawler is:

• We can provide test data to the API.

• It can crawl ressources which are not active and cannot post.

• Other crawl services can use this crawler to upload their conversions.

• It has the full crawler logic but does not transform into other formats.

  • Maybe we can create recommendations or a library for crawlers from this case.

Requirements

The crawler should work as follows:

• Provide urls

  • as command line arguments

  • as a link to a file with one url per line

• Provide ressources

  • as one ressource in a file

  • as a list of ressources

The crawler must be invoked to crawl.

python3 -m ressource_url_crawler http://localhost:8080 \
        https://raw.githubusercontent.com/schul-cloud/ressources-api-v1/master/schemas/ressource/examples/valid/example-website.json

Further Requirements

• The crawler does not post ressources twice. This can be implemented by

  • caching the ressources locally, to see if they changed

    • compare ressource

    • compare timestamp

  • removing the ressources from the database if they are updated after posting new ressources.

This may require some form of state for the crawler. The state could be added to the ressources in a X-Ressources-Url-Crawler-Source field. This allows local caching and requires getting the objects from the database.