mwsql 0.1.3

Encoding errors

Wikimedia SQL dumps use utf-8 encoding. Unfortunately, some fields can contain non-recognized characters, raising an encoding error when attempting to parse the dump file. If this happens while reading in the file, it’s recommended to try again using a different encoding. latin-1 will sometimes solve the problem; if not, you’re encouraged to try with other encodings. If iterating over the rows throws an encoding error, you can try changing the encoding. In this case, you don’t need to recreate the dump – just pass in a new encoding via the dump.encoding attribute.

Parsing errors

Some Wikimedia SQL dumps contain string-type fields that are sometimes not correctly parsed, resulting in fields being split up into several parts. This is more likely to happen when parsing dumps containing file names from Wikimedia Commons or containing external links with many query parameters. If you’re parsing any of the other dumps, you’re unlikely to run into this issue.

In most cases, this issue affects a relatively very small proportion of the total rows parsed. For instance, Wikimedia Commons page dump contains approximately 99 million entries, out of which ~13.000 are incorrectly parsed. Wikimedia Commons page links on the other hand, contains ~760M records, and only 20 are wrongly parsed.

This issue is most commonly caused by the parser mistaking a single quote (or apostrophe, as they’re identical) within a string for the single quote that marks the end of said string. There’s currently no known workaround other than manually removing the rows that contain more fields than expected, or if they are relatively few, manually merging the split fields.

Future versions of mwsql will improve the parser to correctly identify when single quotes should be treated as string delimiters and when they should be escaped. For now, it’s essential to be aware that this problem exists.