Character Tables and Tokenizer for Mathics and the Wolfram Language.
Project description
Mathics Character Tables and Scanner
This repository really contains two things:
extensive tables describing WL symbols and operators their properties
a tokenizer or scanner portion for the Wolfram Language.
With respect to the first item, there is a commented YAML that contains a full set of translation between:
Wolfram Language named characters,
their Unicode/ASCII equivalents and Unicode and WL code-points,
Operator name (if symbol is an operator),
Operator precedence (if an operator)
Keyboard escape sequences for the symbol
Uses
The scanner and character tables are used inside Mathics. However information can also be used by other programs for tokenizing and formatting Wolfram Language code.
For example, tables are used in mathics-pygments, a Pygments-based lexer and highlighter for Mathematica/Wolfram Language source code.
This library may be useful if you need to work with Wolfram Language named character and convert them to various formats.
Usage
For tokenizing and scanning Wolfram Language code, use the mathics_scanner.tokenizer.Tokenizer class.
To convert between Wolfram Language named characters and Unicode/ASCII, use the mathics_scanner.characters.replace_wl_with_plain_text and mathics_scanner.characters.replace_unicode_with_wl functions.
To convert between qualified names of named characters (such FormalA for \[FormalA]) and Wolfram’s internal representation use the m̀athics_scanner.characters.named_characters dictionary.
To regenerate JSON-format tables run:
$ mathics-generate-json-table
Without options mathics-generate-json-table produces the maximum set of correspondences.
In most applications though you may need just a few of these. The --field option can be used to narrow the list of entries to output in JSON. Run mathics-generate-json-table --help for a full list of fields.
Implementation
For notes on the implementation of the packages or details on the conversion scheme please read Implementation.
Contributing
Please feel encouraged to contribute to this package or Mathics! Create your own fork, make the desired changes, commit, and make a pull request.
License
Mathics is released under the GNU General Public License Version 3 (GPL3).
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
File details
Details for the file Mathics_Scanner-1.2.2.tar.gz.
File metadata
- Download URL: Mathics_Scanner-1.2.2.tar.gz
- Upload date:
- Size: 84.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.9.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 2164caca9c16e4a1c9a9c729f14f2a34f35b2e5e387aa5f5c1825cf780f5e047 |
|
| MD5 | 7e014dbe60d7c2dbbd014319acad4400 |
|
| BLAKE2b-256 | 64b6c9187029e9e7fa1a5a3e3ff08308d541d047484c8463d97cb5da10fe35ce |
Provenance
File details
Details for the file Mathics_Scanner-1.2.2-py3.9.egg.
File metadata
- Download URL: Mathics_Scanner-1.2.2-py3.9.egg
- Upload date:
- Size: 76.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.9.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 3e3db7641041672a526175b95acff02b19a8baf66de07692345c3c3bd99ca736 |
|
| MD5 | c9d6fe243926c2717fcadf9582c03461 |
|
| BLAKE2b-256 | 9d99955685b9f11fdeb1a58a827bf8f874d1542ad6a9a98598980f22cfa1c14c |
Provenance
File details
Details for the file Mathics_Scanner-1.2.2-py3.8.egg.
File metadata
- Download URL: Mathics_Scanner-1.2.2-py3.8.egg
- Upload date:
- Size: 76.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.9.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 4149fc94c1c34cbf3ae8ebc52e95c9735b9f9a269bfa44bd6c8d9afcbdca8bad |
|
| MD5 | d1ef95f1a2185c237747fa201879e273 |
|
| BLAKE2b-256 | 7078815277e2a12e3d0e74cde28d7db465c8d052a923c5579e5fceb3c1f341b9 |
Provenance
File details
Details for the file Mathics_Scanner-1.2.2-py3.7.egg.
File metadata
- Download URL: Mathics_Scanner-1.2.2-py3.7.egg
- Upload date:
- Size: 76.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.9.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 08804956b8463479fb3367ec300d65f5586f5ba88aea25d5a388d07ea75ebedd |
|
| MD5 | 6166f0fa49fc80fc19a48879d17394c8 |
|
| BLAKE2b-256 | 511010b1adbd2b30573b65d6b49f6dea9af16e1531c6fc7858b542dd9ee67ad1 |
Provenance
File details
Details for the file Mathics_Scanner-1.2.2-py3.6.egg.
File metadata
- Download URL: Mathics_Scanner-1.2.2-py3.6.egg
- Upload date:
- Size: 77.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.9.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 7ba03ce0ddab095789e8345e82075d6652d262a5431c1122f2860ca5f58050e4 |
|
| MD5 | 3c31286734f51a1a318dedbda227ab09 |
|
| BLAKE2b-256 | 0b822d73df2324f5001fcdff45f0683e98727ba12fd51b8f2f2f5e230b2ab64c |
Provenance
File details
Details for the file Mathics_Scanner-1.2.2-py3-none-any.whl.
File metadata
- Download URL: Mathics_Scanner-1.2.2-py3-none-any.whl
- Upload date:
- Size: 71.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.9.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 44430a3799f8a1b1c1c73c220d95d8708f493f59ffb5fe03f825f3f8dc80ee47 |
|
| MD5 | cc606af26d5e5f8acb55ee03a1b911d3 |
|
| BLAKE2b-256 | 114daa7429f5e73014430581b2a1037880f452614db07d861c27c4e7a3df97d8 |
Provenance
File details
Details for the file Mathics_Scanner-1.2.2-py2.py3-none-any.whl.
File metadata
- Download URL: Mathics_Scanner-1.2.2-py2.py3-none-any.whl
- Upload date:
- Size: 71.6 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.1 importlib_metadata/4.0.1 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.9.5
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 00026b562e1e2c6f7368eb6e6c6588a5aa0b757194ca177fada548df103a4179 |
|
| MD5 | 0e67cf004e63ce26256e1499589062e8 |
|
| BLAKE2b-256 | b037ed78240af39d4abb0320a2d156745093f457592f7705aa15efa6395dafd5 |
