html-text 0.4.0

Install

Install with pip:

pip install html-text

The package depends on lxml, so you might need to install some additional packages: http://lxml.de/installation.html

Usage

Extract text from HTML:

>>> import html_text
>>> html_text.extract_text('<h1>Hello</h1> world!')
'Hello\n\nworld!'

>>> html_text.extract_text('<h1>Hello</h1> world!', guess_layout=False)
'Hello world!'

You can also pass already parsed lxml.html.HtmlElement:

>>> import html_text
>>> tree = html_text.parse_html('<h1>Hello</h1> world!')
>>> html_text.extract_text(tree)
'Hello\n\nworld!'

Or define a selector to extract text only from specific elements:

>>> import html_text
>>> sel = html_text.cleaned_selector('<h1>Hello</h1> world!')
>>> subsel = sel.xpath('//h1')
>>> html_text.selector_to_text(subsel)
'Hello'

Passed html will be first cleaned from invisible non-text content such as styles, and then text would be extracted. NB Selectors are not cleaned automatically you need to call html_text.cleaned_selector first.

Main functions:

• html_text.extract_text accepts html and returns extracted text.

• html_text.cleaned_selector accepts html as text or as lxml.html.HtmlElement, and returns cleaned parsel.Selector.

• html_text.selector_to_text accepts parsel.Selector and returns extracted text.

If guess_layout is True (default), a newline is added before and after newline_tags, and two newlines are added before and after double_newline_tags. This heuristic makes the extracted text more similar to how it is rendered in the browser. Default newline and double newline tags can be found in html_text.NEWLINE_TAGS and html_text.DOUBLE_NEWLINE_TAGS.

It is possible to customize how newlines are added, using newline_tags and double_newline_tags arguments (which are html_text.NEWLINE_TAGS and html_text.DOUBLE_NEWLINE_TAGS by default). For example, don’t add a newline after <div> tags:

>>> newline_tags = html_text.NEWLINE_TAGS - {'div'}
>>> html_text.extract_text('<div>Hello</div> world!',
...                        newline_tags=newline_tags)
'Hello world!'

Credits

The code is extracted from utilities used in several projects, written by Mikhail Korobov.

0.4.0 (2018-09-25)

This is a backwards-incompatible release: by default html_text functions now add newlines after elements, if appropriate, to make the extracted text to look more like how it is rendered in a browser.

To turn it off, pass guess_layout=False option to html_text functions.

• guess_layout option to to make extracted text look more like how it is rendered in browser.

• Add tests of layout extraction for real webpages.

0.3.0 (2017-10-12)

• Expose functions that operate on selectors, use .//text() to extract text from selector.

0.2.1 (2017-05-29)

• Packaging fix (include CHANGES.rst)

0.2.0 (2017-05-29)

• Fix unwanted joins of words with inline tags: spaces are added for inline tags too, but a heuristic is used to preserve punctuation without extra spaces.

• Accept parsed html trees.

0.1.1 (2017-01-16)

• Travis-CI and codecov.io integrations added

0.1.0 (2016-09-27)

• First release on PyPI.