Skip to main content

Edit diffs and type detection for Wikipedia

Project description

Build status

mwedittypes

Edit diffs and type detection for Wikipedia. The goal is to transform unstructured edits to Wikipedia articles into a structured summary of what actions were taken in the edit.

Installation

You can install mwedittypes with pip:

$ pip install mwedittypes

Example

If one revision of wikitext is as follows:

{{Short description|Austrian painter}}
'''Karl Josef Aigen''' (8 October 1684 – 22 October 1762) was a landscape painter, born at Olomouc.

and a second revision of wikitext is as follows:

{{Short description|Austrian landscape painter}}
'''Karl Josef Aigen''' (8 October 1684 – 22 October 1762) was a landscape painter, born at [[Olomouc]].

The changes that happened would be:

  • The addition of landscape to the short description template -- this would be registered as a Template change.
  • The changing of Olomouc to a Wikilink.
  • Notably, despite this change to the template and addition of a link, the "Text" of the article has not changed.

This repository would return this in the following structure: {'Template':{'change':1}, 'Wikilink':{'insert':1}.

Basic Usage

>>> from mwedittypes import EditTypes
>>> prev_wikitext = '{{Short description|Austrian painter}}'
>>> curr_wikitext = '{{Short description|Austrian [[landscape painter]]}}'
>>> et = EditTypes(prev_wikitext, curr_wikitext, lang='en', timeout=5)
>>> et.get_diff()
{'Wikilink': {'insert': 1}, 'Template': {'change': 1}}

Development

We are happy to receive contributions though will default to keeping the code here relatively general (not overly customized to individual use-cases). Please reach out or open an issue for the changes you would like to merge so that we can discuss beforehand.

Code Summary

The code for computing diffs and running edit-type detection can be found in two files:

  • mwedittypes/tree_differ.py: this is the first stage of the diffing pipeline that detects high-level changes.
  • mwedittypes/node_differ.py: this is the second stage of the diffing pipeline that takes the tree_differ output, further processes it, and counts up the edit types.

While the diffing/counting is not trivial, the trickiest part of the process is correctly parsing the wikitext into nodes (Templates, Wikilinks, etc.). This is almost all done via the amazing mwparserfromhell library with a few tweaks in the tree differ:

  • We use link namespace prefixes -- e.g., Category:, Image: -- to separate out categories and media from other wikilinks.
  • We identify some additional media files that are transcluded via templates -- e.g., infoboxes -- or gallery tags.
  • We also add some custom logic for parsing <gallery> tags to identify nested links, etc., which otherwise are treated as text by mwparserfromhell.
  • We use custom logic for converting wikitext into text to best match what words show up in the text of the article.

To accurately, but efficiently, describe the scale of textual changes in edits, we also use some regexes and heuristics to describe how much text was changed in an edit in the node differ. This is generally the toughest part of diffing text but because we do not need to visually describe the diff, just estimate the scale of how much changed, we can use relatively simple methods. To do this, we break down text changes into five categories and identify how much of each changed: paragraphs, sentences, words, punctuation, and whitespace.

Testing

The tests for node/tree differs are contained within the tests directory. They can be run via pytest. We are not even close to full coverage yet given the numerous node types (template, text, etc.) and four actions (insert/remove/change/move) and varying languages for e.g., Text or Category/Media nodes, but we are working on expanding coverage.

Releases

When a release is ready, there are a few simple steps to take:

  • Update the version number in setup.py and mwedittypes/__init__.py per semantic versioning and push to Github.
  • Create a tag with the same version number (easiest to do this as part of the next step on Github).
  • Make a Github release with the new version and summary of changes. This will trigger the publish to PyPi action.

Troubleshooting:

  • If something fails, you can always rollback the release (Github UI) and delete the associated tag (on your local repository: git tag -d [tag]; git push origin :[tag]) and retry.
  • This relies on a PyPi secret so you must have correct privileges on the Github repo.

Documentation

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

mwedittypes-1.0.2.tar.gz (29.2 kB view details)

Uploaded Source

Built Distribution

mwedittypes-1.0.2-py3-none-any.whl (27.8 kB view details)

Uploaded Python 3

File details

Details for the file mwedittypes-1.0.2.tar.gz.

File metadata

  • Download URL: mwedittypes-1.0.2.tar.gz
  • Upload date:
  • Size: 29.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.0 CPython/3.9.12

File hashes

Hashes for mwedittypes-1.0.2.tar.gz
Algorithm Hash digest
SHA256 dadd503df1efa6c9dc64d60384661cadcb3681c8d9d0e4c4fba1fc1cb88da222
MD5 2b06689be4af426c69d09abd3c191dfa
BLAKE2b-256 ec72f16c7bf1a2f3d594d8bdedb4130d01da0462da13b1722063f21efb54370f

See more details on using hashes here.

Provenance

File details

Details for the file mwedittypes-1.0.2-py3-none-any.whl.

File metadata

  • Download URL: mwedittypes-1.0.2-py3-none-any.whl
  • Upload date:
  • Size: 27.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.0 CPython/3.9.12

File hashes

Hashes for mwedittypes-1.0.2-py3-none-any.whl
Algorithm Hash digest
SHA256 ffc4bac9c0fd91db480692c33fceaad3fab3c2b307a087f41ea4f63bcb4349a5
MD5 e5f6d257087bafee3c7f28aa4117c669
BLAKE2b-256 9669d91d0b2f7f4f24669f7e3d59e0d60b020e81f42f21eafa45b1d0b30091b5

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page