mwparserfromhell

MWParserFromHell is a parser for MediaWiki wikicode.

These details have not been verified by PyPI

Project links

Development Status
- 4 - Beta
Environment
- Console
Intended Audience
- Developers
License
- OSI Approved :: MIT License
Operating System
- OS Independent
Programming Language
Topic
- Text Processing :: Markup

Project description

mwparserfromhell (the MediaWiki Parser from Hell) is a Python package that provides an easy-to-use and outrageously powerful parser for MediaWiki wikicode. It supports Python 3.5+.

Developed by Earwig with contributions from Σ, Legoktm, and others. Full documentation is available on ReadTheDocs. Development occurs on GitHub.

Installation

The easiest way to install the parser is through the Python Package Index; you can install the latest release with pip install mwparserfromhell (get pip). Make sure your pip is up-to-date first, especially on Windows.

Alternatively, get the latest development version:

git clone https://github.com/earwig/mwparserfromhell.git
cd mwparserfromhell
python setup.py install

The comprehensive unit testing suite requires pytest (pip install pytest) and can be run with python -m pytest.

Usage

Normal usage is rather straightforward (where text is page text):

>>> import mwparserfromhell
>>> wikicode = mwparserfromhell.parse(text)

wikicode is a mwparserfromhell.Wikicode object, which acts like an ordinary str object with some extra methods. For example:

>>> text = "I has a template! {{foo|bar|baz|eggs=spam}} See it?"
>>> wikicode = mwparserfromhell.parse(text)
>>> print(wikicode)
I has a template! {{foo|bar|baz|eggs=spam}} See it?
>>> templates = wikicode.filter_templates()
>>> print(templates)
['{{foo|bar|baz|eggs=spam}}']
>>> template = templates[0]
>>> print(template.name)
foo
>>> print(template.params)
['bar', 'baz', 'eggs=spam']
>>> print(template.get(1).value)
bar
>>> print(template.get("eggs").value)
spam

Since nodes can contain other nodes, getting nested templates is trivial:

>>> text = "{{foo|{{bar}}={{baz|{{spam}}}}}}"
>>> mwparserfromhell.parse(text).filter_templates()
['{{foo|{{bar}}={{baz|{{spam}}}}}}', '{{bar}}', '{{baz|{{spam}}}}', '{{spam}}']

You can also pass recursive=False to filter_templates() and explore templates manually. This is possible because nodes can contain additional Wikicode objects:

>>> code = mwparserfromhell.parse("{{foo|this {{includes a|template}}}}")
>>> print(code.filter_templates(recursive=False))
['{{foo|this {{includes a|template}}}}']
>>> foo = code.filter_templates(recursive=False)[0]
>>> print(foo.get(1).value)
this {{includes a|template}}
>>> print(foo.get(1).value.filter_templates()[0])
{{includes a|template}}
>>> print(foo.get(1).value.filter_templates()[0].get(1).value)
template

Templates can be easily modified to add, remove, or alter params. Wikicode objects can be treated like lists, with append(), insert(), remove(), replace(), and more. They also have a matches() method for comparing page or template names, which takes care of capitalization and whitespace:

>>> text = "{{cleanup}} '''Foo''' is a [[bar]]. {{uncategorized}}"
>>> code = mwparserfromhell.parse(text)
>>> for template in code.filter_templates():
...     if template.name.matches("Cleanup") and not template.has("date"):
...         template.add("date", "July 2012")
...
>>> print(code)
{{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{uncategorized}}
>>> code.replace("{{uncategorized}}", "{{bar-stub}}")
>>> print(code)
{{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{bar-stub}}
>>> print(code.filter_templates())
['{{cleanup|date=July 2012}}', '{{bar-stub}}']

You can then convert code back into a regular str object (for saving the page!) by calling str() on it:

>>> text = str(code)
>>> print(text)
{{cleanup|date=July 2012}} '''Foo''' is a [[bar]]. {{bar-stub}}
>>> text == code
True

Limitations

While the MediaWiki parser generates HTML and has access to the contents of templates, among other things, mwparserfromhell acts as a direct interface to the source code only. This has several implications:

Syntax elements produced by a template transclusion cannot be detected. For example, imagine a hypothetical page "Template:End-bold" that contained the text </b>. While MediaWiki would correctly understand that <b>foobar{{end-bold}} translates to <b>foobar</b>, mwparserfromhell has no way of examining the contents of {{end-bold}}. Instead, it would treat the bold tag as unfinished, possibly extending further down the page.
Templates adjacent to external links, as in http://example.com{{foo}}, are considered part of the link. In reality, this would depend on the contents of the template.
When different syntax elements cross over each other, as in {{echo|''Hello}}, world!'', the parser gets confused because this cannot be represented by an ordinary syntax tree. Instead, the parser will treat the first syntax construct as plain text. In this case, only the italic tag would be properly parsed.

Workaround: Since this commonly occurs with text formatting and text formatting is often not of interest to users, you may pass skip_style_tags=True to mwparserfromhell.parse(). This treats '' and ''' as plain text.

A future version of mwparserfromhell may include multiple parsing modes to get around this restriction more sensibly.

Additionally, the parser lacks awareness of certain wiki-specific settings:

Word-ending links are not supported, since the linktrail rules are language-specific.
Localized namespace names aren’t recognized, so file links (such as [[File:...]]) are treated as regular wikilinks.
Anything that looks like an XML tag is treated as a tag, even if it is not a recognized tag name, since the list of valid tags depends on loaded MediaWiki extensions.

Integration

mwparserfromhell is used by and originally developed for EarwigBot; Page objects have a parse method that essentially calls mwparserfromhell.parse() on page.get().

If you’re using Pywikibot, your code might look like this:

import mwparserfromhell
import pywikibot

def parse(title):
    site = pywikibot.Site()
    page = pywikibot.Page(site, title)
    text = page.get()
    return mwparserfromhell.parse(text)

If you’re not using a library, you can parse any page with the following Python 3 code (using the API and the requests library):

import mwparserfromhell
import requests

API_URL = "https://en.wikipedia.org/w/api.php"

def parse(title):
    params = {
        "action": "query",
        "prop": "revisions",
        "rvprop": "content",
        "rvslots": "main",
        "rvlimit": 1,
        "titles": title,
        "format": "json",
        "formatversion": "2",
    }
    headers = {"User-Agent": "My-Bot-Name/1.0"}
    req = requests.get(API_URL, headers=headers, params=params)
    res = req.json()
    revision = res["query"]["pages"][0]["revisions"][0]
    text = revision["slots"]["main"]["content"]
    return mwparserfromhell.parse(text)

Project details

These details have not been verified by PyPI

Project links

Development Status
- 4 - Beta
Environment
- Console
Intended Audience
- Developers
License
- OSI Approved :: MIT License
Operating System
- OS Independent
Programming Language
Topic
- Text Processing :: Markup

Release history Release notifications | RSS feed

0.6.6

Jan 4, 2024

0.6.5

Sep 5, 2023

0.6.4

Feb 14, 2022

0.6.3

Sep 2, 2021

This version

0.6.2

May 16, 2021

0.6

Dec 21, 2020

0.5.4

May 15, 2019

0.5.3

Mar 31, 2019

0.5.2

Nov 2, 2018

0.5.1

Mar 3, 2018

0.5

Jun 23, 2017

0.4.4

Dec 30, 2016

0.4.3

Oct 29, 2015

0.4.2

Jul 30, 2015

0.4.1

Jul 30, 2015

0.4

May 24, 2015

0.3.3

Apr 22, 2014

0.3.2

Sep 1, 2013

0.3.1

Aug 29, 2013

0.3

Aug 25, 2013

0.2

Jun 20, 2013

0.1.1

Sep 22, 2012

0.1

Aug 23, 2012

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mwparserfromhell-0.6.2.tar.gz (137.3 kB view details)

Uploaded May 16, 2021 Source

Built Distributions

mwparserfromhell-0.6.2-cp39-cp39-win_amd64.whl (103.3 kB view details)

Uploaded May 16, 2021 CPython 3.9 Windows x86-64

mwparserfromhell-0.6.2-cp39-cp39-win32.whl (100.7 kB view details)

Uploaded May 16, 2021 CPython 3.9 Windows x86

mwparserfromhell-0.6.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (179.4 kB view details)

Uploaded May 16, 2021 CPython 3.9 manylinux: glibc 2.5+ x86-64

mwparserfromhell-0.6.2-cp39-cp39-macosx_10_14_x86_64.whl (98.8 kB view details)

Uploaded May 16, 2021 CPython 3.9 macOS 10.14+ x86-64

mwparserfromhell-0.6.2-cp38-cp38-win_amd64.whl (103.4 kB view details)

Uploaded May 16, 2021 CPython 3.8 Windows x86-64

mwparserfromhell-0.6.2-cp38-cp38-win32.whl (100.7 kB view details)

Uploaded May 16, 2021 CPython 3.8 Windows x86

mwparserfromhell-0.6.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (184.6 kB view details)

Uploaded May 16, 2021 CPython 3.8 manylinux: glibc 2.5+ x86-64

mwparserfromhell-0.6.2-cp38-cp38-macosx_10_14_x86_64.whl (98.8 kB view details)

Uploaded May 16, 2021 CPython 3.8 macOS 10.14+ x86-64

mwparserfromhell-0.6.2-cp37-cp37m-win_amd64.whl (102.6 kB view details)

Uploaded May 16, 2021 CPython 3.7m Windows x86-64

mwparserfromhell-0.6.2-cp37-cp37m-win32.whl (99.8 kB view details)

Uploaded May 16, 2021 CPython 3.7m Windows x86

mwparserfromhell-0.6.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (177.8 kB view details)

Uploaded May 16, 2021 CPython 3.7m manylinux: glibc 2.5+ x86-64

mwparserfromhell-0.6.2-cp37-cp37m-macosx_10_14_x86_64.whl (98.7 kB view details)

Uploaded May 16, 2021 CPython 3.7m macOS 10.14+ x86-64

mwparserfromhell-0.6.2-cp36-cp36m-win_amd64.whl (102.6 kB view details)

Uploaded May 16, 2021 CPython 3.6m Windows x86-64

mwparserfromhell-0.6.2-cp36-cp36m-win32.whl (99.8 kB view details)

Uploaded May 16, 2021 CPython 3.6m Windows x86

mwparserfromhell-0.6.2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (177.8 kB view details)

Uploaded May 16, 2021 CPython 3.6m manylinux: glibc 2.5+ x86-64

mwparserfromhell-0.6.2-cp36-cp36m-macosx_10_14_x86_64.whl (98.7 kB view details)

Uploaded May 16, 2021 CPython 3.6m macOS 10.14+ x86-64

mwparserfromhell-0.6.2-cp35-cp35m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (177.8 kB view details)

Uploaded May 16, 2021 CPython 3.5m manylinux: glibc 2.5+ x86-64

mwparserfromhell-0.6.2-cp35-cp35m-macosx_10_14_x86_64.whl (98.7 kB view details)

Uploaded May 16, 2021 CPython 3.5m macOS 10.14+ x86-64

File details

Details for the file mwparserfromhell-0.6.2.tar.gz.

File metadata

Download URL: mwparserfromhell-0.6.2.tar.gz
Upload date: May 16, 2021
Size: 137.3 kB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/3.2.0 pkginfo/1.6.1 requests/2.25.1 setuptools/56.0.0 requests-toolbelt/0.9.1 tqdm/4.54.1 CPython/3.9.5

File hashes

Hashes for mwparserfromhell-0.6.2.tar.gz
Algorithm	Hash digest
SHA256	`d3f74c0101f81ff73c61985b67f2e7048a30dc5f6a578ea1544e69133988d874`
MD5	`8d22d8ca376ec206ea8cecc1daea99fe`
BLAKE2b-256	`357c2e0d48f42d7a5c48d788194d4853e49196460e125941649f409236ece717`

Algorithm	Hash digest
SHA256	`1e797c47fec1ccc855343b91ec26586300001119ade8f444aeb87b28fc9d6645`
MD5	`62e67d2e4c780d647ace5c0b9ce974f1`
BLAKE2b-256	`cb737ecd46395f54880cb315dda8410a16f4538aec1cc912143e17553245b394`

Algorithm	Hash digest
SHA256	`7a6e37b9cfc61d2f92d6f275b15cd9a8fdd1da4ce3382062537c1f5e22f5bac5`
MD5	`a7c108e44d99fc91fab47c986d34676a`
BLAKE2b-256	`5b0287a25ace9e5735aa7af63bfdd44f5b4f44b9f2b67f43696f53e229ed0d0a`

Algorithm	Hash digest
SHA256	`28e7782ab1518756e232e9a18d7f8ff68f389cf5c7812fc9fbb73cd059818a28`
MD5	`b1c354142490b5383b8fd6e22ba93bf4`
BLAKE2b-256	`bed3bce34538264ce2eeb4c13b1380d039058dfbc292cb06da91e3ef58c5b35a`

Algorithm	Hash digest
SHA256	`3008d62d9386487ddb63d45d50de62f2e3dd7b7047d46fb205f965f2f5b0650f`
MD5	`9dc33820965449322c61632dbe686629`
BLAKE2b-256	`adc50ded7d4005fa2ccd4aa0099471909da57e73c9d6200fa1a8a386d8944628`

Algorithm	Hash digest
SHA256	`1aae08cbb5764a43107b8d2dc4c95707b340324f41aa1ab82098a36cabcb50db`
MD5	`1144532ba1733dc5cbd465df2729dee6`
BLAKE2b-256	`9c886f42e2101d2b2f3eab7513f41a270d2daf2fb03288b92133601bfa1afc8d`

Algorithm	Hash digest
SHA256	`e4e6578e15ec48b89c70efc06276669377d3dbf49f414626b1882bf506bd4fdb`
MD5	`4c6efd314101669dc06136cffa3879d8`
BLAKE2b-256	`a77aa892b21c2ca39dc539961cc82e76bfb08a1ae257357eee87118e6477f6e6`

Algorithm	Hash digest
SHA256	`3167091e89cab14f8998dd26f0331ae9b7b03d46aaecde06058cb5e7bd88cca0`
MD5	`9f64325bd30dc1c1eb46f704c5af886a`
BLAKE2b-256	`8a49e70f29471405a655f4ff4cc9dd02a49b90de739476b937a9dc0e87ba3e2c`

Algorithm	Hash digest
SHA256	`ae6f8dc258d94e309a450936fdedd592c9d9d78f51db4606f87d6ca94822d6e4`
MD5	`9d67b6c95de76aaeff58158258b67246`
BLAKE2b-256	`bc8500c13a67c828a9267de51e0b04ecff880b1044aaebb9af6c21a80f9e2c25`

Algorithm	Hash digest
SHA256	`480991c919e7658b96800dfd8f0d95da413b37c90c0368d5e1aead83baa5e85c`
MD5	`3f45f953452c431867c0b31628ae21ef`
BLAKE2b-256	`453b1b4a83dc29189fb563a1b4e0ec1793e695a83cd7721b9d6722ed48ff52e2`

Algorithm	Hash digest
SHA256	`c5335d1e67017e66044e9b6334bfdff6dd70553f1fa2c16c23628c8dd8176be7`
MD5	`21d26200d3d29f87c77dd06b8d158c73`
BLAKE2b-256	`cab21ab6d2c1ecd72f6970d88193e87a2722718ef1cba2383d25fc03631c4afd`

Algorithm	Hash digest
SHA256	`97e199f19975d67801e0bff447495a526486987c1f216a14c928323544b5f594`
MD5	`fee98004603c37565ccd1ac822cb7865`
BLAKE2b-256	`e22d2ac4a127c9d89dad0f30336d1e8b1fb4bf2f68dab83f822cf4404ce0517b`

Algorithm	Hash digest
SHA256	`d0dc9b9cd48a1807e307d03ec758d63bb90d2a4d516366b1d2dddfa042fac8df`
MD5	`a4d32acd0dda1d65c70c0981ad439e12`
BLAKE2b-256	`ad56374c1c765009d43c1d613f65af4d0b4084a93a2a1c4c7d4677b3cda3779d`

Algorithm	Hash digest
SHA256	`a127ebb32f742d872acc2725070cd8059ce19418fe24e5fa3a85dc1fd02adb74`
MD5	`21f4d2b263d6a5f696f1781c61d13e08`
BLAKE2b-256	`5eb001611f76734422ef2e005aae5730bb603de74f65bd7e5b65eb201613b2a7`

Algorithm	Hash digest
SHA256	`6093fb4bc91161b8c49558d2a5fca4a7693f863194ca8541418bd9826af6d8dd`
MD5	`1631c7358da5ec7d69bd184e08552852`
BLAKE2b-256	`7640ca2cf20c31378c24712311d60657b1395aca6743b574a39c583aedae1a6c`

Algorithm	Hash digest
SHA256	`ca93e64aadea6785c940bb73ec50c9b3d37efe6c5bb5ad5d9846053797134f87`
MD5	`aa30f7343399d823b79a1e844a7c7be9`
BLAKE2b-256	`4937861f6a9542acc917158699c1c80a6dd351ddf2ee686c68ded3a9cc34de40`

Algorithm	Hash digest
SHA256	`0c33cb431dec434f61b66b49383bc57ee5d4bba2a16ffa72671c7e7e190a0458`
MD5	`558ee5478fd9e28980d388cbd8be780f`
BLAKE2b-256	`a3d429b3e65329159194398cd98dab9d79428eb68740896fc3891ad48747fa37`

Algorithm	Hash digest
SHA256	`465e086af36fc4c43bfe73d130f4a91e83ce8684f52ba492e233eb6f596e39e7`
MD5	`e9e5807464648e4dc0465017f92c44ac`
BLAKE2b-256	`8706d51c98aca9759a19dbf26b1e82200b3f71f108d6022769f56c0ed2beb5d1`

Algorithm	Hash digest
SHA256	`17b9fae14dce0995b30d7832748c54e9d36e92b618419bae929fceed7d39bb9d`
MD5	`7f5fef10b8b6632d11c08b544ffdb73f`
BLAKE2b-256	`435fc47b654bd09f73cadafb80b0e4a420a6aa923021605231a5b39a7e134d3a`

mwparserfromhell 0.6.2

Navigation

Verified details

Maintainers

Unverified details

Project links

Meta

Classifiers

Project description

Installation

Usage

Limitations

Integration

Project details

Verified details

Maintainers

Unverified details

Project links

Meta

Classifiers

Release history Release notifications | RSS feed

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata