inscriptis - HTML to text converter.
Project description
inscriptis
A python based HTML to text converter with minimal support for CSS.
Requirements
- Python 3.4+ (preferred) or Python 2.7+
- lxml
- requests
Usage
Command line
The command line client converts text files or text retrieved from Web pages to the corresponding text representation.
Installation
sudo python3 setup.py install
Command line parameters
usage: inscript.py [-h] [-o OUTPUT] [-e ENCODING] [-i] [-l] [-d] input
Converts HTML from file or url to a clean text version
positional arguments:
input Html input either from a file or an url (default:stdin)
optional arguments:
-h, --help show this help message and exit
-o OUTPUT, --output OUTPUT
Output file (default:stdout).
-e ENCODING, --encoding ENCODING
Content encoding for files (default:utf-8)
-i, --display-image-captions
Display image captions (default:false).
-l, --display-link-targets
Display link targets (default:false).
-d, --deduplicate-image-captions
Deduplicate image captions (default:false).
Examples
# convert the given page to text and output the result to the screen
inscript.py http://www.htwchur.ch
# convert the file to text and save the output to output.txt
inscript.py htwchur.html -o htwchur.txt
# convert the text provided via stdin and save the output to output.txt
echo '<body><p>Make it so!</p>></body>' | inscript.py -o htwchur.txt
Library
import urllib.request
from inscriptis import get_text
url = "http://www.informationscience.ch"
html = urllib.request.urlopen(url).read().decode('utf-8')
text = get_text(html)
print(text)
Unit tests
Test cases concerning the html to text conversion are located in the tests/html directory and consist of two files:
test-name.htmlandtest-name.txt
the latter one containing the reference text output for the given html file.
Text convertion output comparison and speed benchmarking
inscriptis offers a small benchmarking script that can compare different HTML to txt convertion approaches.
The script will run the different approaches on a list of URLs, url_list.txt, and save the text output into a time stamped folder in benchmarking/benchmarking_results for manual comparison.
Additionally the processing speed of every approach per URL is measured and saved in a text file called speed_comparisons.txt in the respective time stamped folder.
To run the benchmarking script execute run_benchmarking.py from within the folder benchmarking.
In def pipeline() set the which HTML -> Text algorithms to be executed by modifying
run_lynx = True
run_justext = True
run_html2text = True
run_beautifulsoup = True
run_inscriptis = True
In url_list.txt the URLs to be parsed can be specified by adding them to the file, one per line with no additional formatting. URLs need to be complete (including http:// or https://)
e.g.
http://www.informationscience.ch
https://en.wikipedia.org/wiki/Information_science
...
Flask Web Service
The Flask Web Service translates HTML pages to the corresponding plain text.
Requirements
- python3-flask
Startup
export FLASK_APP="web-service.py"
python3 -m flask run
Usage
The Web services receives the HTML file in the request body and returns the corresponding text. The file's encoding needs to be specified
in the Content-Type header (UTF-8 in the example below).
curl -X POST -H "Content-Type: text/html; encoding=UTF8" -d @test.html http://localhost:5000/get_text
Changelog
see Release notes.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file inscriptis-0.0.3.8.tar.gz.
File metadata
- Download URL: inscriptis-0.0.3.8.tar.gz
- Upload date:
- Size: 11.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.12.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.7.1 requests-toolbelt/0.9.1 tqdm/4.30.0 CPython/3.6.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 2088a1213e35d9994f02b11742633942cc59531a2fc134b21271d146ef87349c |
|
| MD5 | f859a8fa52a7d03626602d0e5a7ff9d2 |
|
| BLAKE2b-256 | b1527a5953b90acf71ba3d1e1e4dff9e3bf11bb4ce2b65f1f3cae8ad8656c291 |
File details
Details for the file inscriptis-0.0.3.8-py3-none-any.whl.
File metadata
- Download URL: inscriptis-0.0.3.8-py3-none-any.whl
- Upload date:
- Size: 17.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/1.12.1 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.7.1 requests-toolbelt/0.9.1 tqdm/4.30.0 CPython/3.6.7
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | c26ed6702910be261f4918b7fe5c657a295a2327d7216deaf4bb30b0a608fa98 |
|
| MD5 | 24a0b74c41f059a53ba6c4ef509db0a8 |
|
| BLAKE2b-256 | ed0808c8406897392afdefa23125274542263a2ea48e10f82c49a98e4eba0767 |
