transmogrify.webcrawler

Crawling and feeding html content into a transmogrifier pipeline

These details have not been verified by PyPI

Project links

Homepage

Project description

Introduction

transmogrify.webcrawler: A source blueprint for crawling content from a site or local html files.
transmogrify.webcrawler.typerecognitor: A blueprint for assinging content type based on the mime-type as given by the webcrawler
transmogrify.webcrawler.cache: A blueprint that saves crawled content into a directory structure

transmogrify.webcrawler

A transmogrifier blueprint source which will crawl a url reading in all pages until all have been crawled.

Options

site_url: URL to start crawling. The URL will be treated as the base and any links outside this base will be ignored
ignore: Regular expressions for urls not to follow
alias_bases: Substitutions for url bases. This is useful where url to access is not the same as absolute urls of links in the pages
patterns: Regular expressions to substitute before html is parsed. New line seperated
subs: Text to replace
checkext: checkext
verbose: verbose
maxpage: maxpage
nonames: nonames
cache: cache

Keys inserted

The following set the keys items added to the pipeline

pathkey: default: _path. The path of the url not including the base
siteurlkey: default: _site_url. The base of the url
originkey: default: _origin. The original path in case retriving the url caused a redirection
contentkey: default: _content. The main content of the url
contentinfokey: default: _content_info. Headers returned by urlopen
sortorderkey: default: _sortoder. A count on when a link to this item was first encounted while crawling
backlinkskey: default: _backlinks. A list of tuples of which pages linked to this item. (url, path)

Tests

>>> testtransmogrifier(dontprint=['_content'], source="""
... [webcrawler]
... blueprint = transmogrify.webcrawler
... site_url  = file://%s/test_staticsite
... alias_bases = http://somerandomsite file:///
... """)
{'_backlinks': [],
 '_content_info': {'content-type': 'text/html'},
 '_origin': 'file://.../test_staticsite',
 '_path': '',
 '_site_url': 'file://.../test_staticsite/',
 '_sortorder': 0}
...

>>> testtransmogrifier(source=webcrawler, strip=['_content'])
{...
 '_path': '',
 ...}
{...
 '_path': 'file2.htm',
 ...}
{...
 '_path': 'subfolder',
 ...}
{...
 '_path': 'egenius-plone.gif',
 ...}
{...
 '_path': 'plone_schema.png',
 ...}
...

>>> source = """
... [webcrawler]
... blueprint = transmogrify.webcrawler
... site_url  = file://%s/test_staticsite
... alias_bases = http://somerandomsite file:///
... patterns =
...             (?s)<SCRIPT.*Abbreviation"\)
...             (?s)MakeLink\('(?P<u>[^']*)','(?P<a>[^']*)'\)
...     (?s)State=.*<body[^>]*>
... subs =
...     </head><body>
...             <a href="\g<u>">\g<a></a>
...     <br>
... """

External scripts used

http://svn.python.org/projects/python/trunk/Tools/webchecker/webchecker.py http://svn.python.org/projects/python/trunk/Tools/webchecker/websucker.py

TypeRecognitor

TypeRecognitor is a transmogrifier blue print which determines the plone type of the item from the mime_type in the header. It reads the mimetype from the headers in _content_info set by transmogrify.webrawler

>>> from os.path import dirname
>>> from os.path import abspath
>>> config = """
...
... [transmogrifier]
... pipeline =
...     webcrawler
...     typerecognitor
...     clean
...     printer
...
... [webcrawler]
... blueprint = transmogrify.webcrawler
... site_url  = file://%s/test_staticsite
...
... [typerecognitor]
... blueprint = transmogrify.webcrawler.typerecognitor
...
... [clean]
... blueprint = collective.transmogrifier.sections.manipulator
... delete =
...   file
...   text
...   image
...
... [printer]
... blueprint = collective.transmogrifier.sections.tests.pprinter
...
... """ % abspath(dirname(__file__)).replace('\\','/')

>>> from collective.transmogrifier.tests import registerConfig
>>> registerConfig(u'transmogrify.webcrawler.typerecognitor.test', config)

>>> from collective.transmogrifier.transmogrifier import Transmogrifier
>>> transmogrifier = Transmogrifier(plone)
>>> transmogrifier(u'transmogrify.webcrawler.typerecognitor.test')
{...
 '_mimetype': 'image/jpeg',
 ...
 '_path': 'cia-plone-view-source.jpg',
 ...
 '_type': 'Image',
 ...}
 ...