MLCommons datasets format.
Project description
mlcroissant 🥐
Discover mlcroissant 🥐
with this
introduction tutorial in Google Colab.
Python requirements
Python version >= 3.10.
If you do not have a Python environment:
python3 -m venv ~/py3
source ~/py3/bin/activate
Install
python -m pip install ".[dev]"
The command can fail, for example, due to missing dependencies, e.g.:
Failed to build pygraphviz
ERROR: Could not build wheels for pygraphviz, which is required to install pyproject.toml-based projects
This can be fixed by running
sudo apt-get install python3-dev graphviz libgraphviz-dev pkg-config
Conda installation
Conda can help create a consistent environment. It can also be useful to install packages without root access. To use Conda, run:
conda create --name croissant python=3.10 -y
conda activate croissant
conda install graphviz
python3 -m pip install ".[dev]"
Verify/load a Croissant dataset
mlcroissant validate --jsonld ../../datasets/titanic/metadata.json
The command:
- Exits with 0, prints
Done
and displays encountered warnings, when no error was found in the file. - Exits with 1 and displays all encountered errors/warnings, otherwise.
Similarly, you can generate a dataset by launching:
mlcroissant load \
--jsonld ../../datasets/titanic/metadata.json \
--record_set passengers \
--num_records 10
Loading a distribution
via git+https
If the encodingFormat
of a distribution
is git+https
, please provide the username and password by setting the CROISSANT_GIT_USERNAME
and CROISSANT_GIT_PASSWORD
environment variables. These will be used to construct the authentication necessary to load the distribution.
Note that, for datasets hosted on HuggingFace, CROISSANT_GIT_USERNAME
and CROISSANT_GIT_PASSWORD
should correspond respectively to your HuggingFace's username and User Access Token. User Access Tokens can be generated following this guide.
Loading a distribution
via HTTP with Basic Auth
If the contentUrl
of a distribution
requires authentication via Basic Auth, please provide the username and password by setting the CROISSANT_BASIC_AUTH_USERNAME
and CROISSANT_BASIC_AUTH_PASSWORD
environment variables. These will be used to construct the authentication necessary to load the distribution.
Programmatically build JSON-LD files
You can programmatically build Croissant JSON-LD files using the Python API.
import mlcroissant as mlc
metadata=mlc.nodes.Metadata(
name="...",
)
metadata.to_json() # this returns the JSON-LD file.
Add new properties to the standard
Nodes (Metadata, RecordSets, etc) implement PEP 681. So you can declare RDF triplets using the dataclass syntax.
Example 1: implement CreativeWork:
@mlc_dataclasses.dataclass
class CreativeWork(Node):
JSONLD_TYPE = SDO.CreativeWork # https://schema.org/CreativeWork
name: str | None = mlc_dataclasses.jsonld_field(
cardinality="ONE", # Cardinality can be ONE or MANY
default=None, # Specify the default value in Python
description="The name of the item.", # The full description
input_types=[SDO.Text], # The schema.org type
url=SDO.name, # The URL of the property
)
Example 2: implement RecordSet:
@mlc_dataclasses.dataclass
class RecordSet(Node):
JSONLD_TYPE = constants.ML_COMMONS_RECORD_SET_TYPE
fields: list[Field] = mlc_dataclasses.jsonld_field(
cardinality="MANY", # Example with cardinality=="MANY"
default_factory=list,
description=(
"A data element that appears in the records of the RecordSet (e.g., one"
" column of a table)."
),
input_types=[Field], # Types can also be other nodes (here `Field`)
url=constants.ML_COMMONS_FIELD,
)
Example 3: specify a version (by default all versions):
@mlc_dataclasses.dataclass
class Field(Node):
is_enumeration: bool | None = mlc_dataclasses.jsonld_field(
default=None,
input_types=[SDO.Boolean],
url=constants.ML_COMMONS_IS_ENUMERATION,
versions=[CroissantVersion.V_0_8], # `is_enumeration` is only valid for v0.8, not v1.0
)
Run tests
All tests can be run from the Makefile:
make tests
Note that git lfs
should be installed to successfully pass all tests:
git lfs install
Design
The most important modules in the library are:
mlcroissant/_src/structure_graph
is responsible for the static analysis of the Croissant files. We convert Croissant files to a Python representation called "structure graph" (using NetworkX). In the process, we catch any static analysis issues (e.g., a missing mandatory field or a logic problem in the file).mlcroissant/_src/operation_graph
is responsible for the dynamic analysis of the Croissant files (i.e., actually loading the dataset by yielding examples). We convert the structure graph into an "operation graph". Operations are the unit transformations that allow to build the dataset (likeDownload
,Extract
, etc).
Other important modules are:
mlcroissant/_src/core
defines all needed core internals. For instance,Issues
are a way to track errors and warning during the analysis of Croissant files.mlcroissant/__init__.py
declares the public API withmlcroissant.Dataset
.
For the full design, refer to the design doc for an overview of the implementation.
Caching. By default, all downloaded/extracted files are cached in ~/.cache/croissant
, but you can overwrite this by setting the environment variable $CROISSANT_CACHE
.
Contribute
All contributions are welcome! We even have good first issues to start in the project. Refer to the GitHub project for more detailed user stories and read above how the repo is designed.
An easy way to contribute to mlcroissant
is using Croissant's configured codespaces.
To start a codespace:
- On Croissant's main repo page, click on the
<Code>
button and select theCodespaces
tab. You can start a new codespace by clicking on the+
sign on the left side of the tab. By default, the codespace will start on Croissant'smain
branch, unless you select otherwise from the branches drop-down menu on the left side. - While building the environment, your codespaces will install all
mlcroissant
's required dependencies - so that you can start coding right away! Of course, you can further personalize your codespace. - To start contributing to Croissant:
- Create a new branch from the
Terminal
tab in the bottom panel of your codespace withgit checkout -b feature/my-awesome-new-feature
- You can create new commits, and run most git commands from the
Source Control
tab in the left panel of your codespace. Alternatively, use theTerminal
in the bottom panel of your codespace. - Iterate on your code until all tests are green (you can run tests with
make pytest
or form theTests
tab in the left panel of your codespace). - Open a pull request (PR) with the main branch of https://github.com/mlcommons/croissant, and ask for feedback!
- Create a new branch from the
Alternatively, you can contribute to mlcroissant
using the "classic" GitHub workflow:
Debug
You can debug the validation of the file using the --debug
flag:
mlcroissant validate --jsonld ../../datasets/titanic/metadata.json --debug
This will:
- print extra information, like the generated nodes;
- save the generated structure graph to a folder indicated in the logs.
Publishing packages
To publish a package,
- Bump the version in
croissant/python/mlcroissant/pyproject.toml
, and merge your PR. - Publish a new release in GitHub, and add a tag to it with the newest version in
pyproject.toml
. Ensure that the new release is marked aslatest
. The workflow scriptpython-publish.yml
will trigger and publish the package to PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file mlcroissant-1.0.10.tar.gz
.
File metadata
- Download URL: mlcroissant-1.0.10.tar.gz
- Upload date:
- Size: 96.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.1 CPython/3.12.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 419c162888e78486df50aafe45c75f91866dd84d57aff9da02c4d158d74d5b74 |
|
MD5 | 9831ec47009650221131c096b3a03bae |
|
BLAKE2b-256 | 7a087cd3d34382dcb7c847c70c6e6ebfffcd5f7fadb749dfea9ff07dffd5b438 |
File details
Details for the file mlcroissant-1.0.10-py2.py3-none-any.whl
.
File metadata
- Download URL: mlcroissant-1.0.10-py2.py3-none-any.whl
- Upload date:
- Size: 136.5 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/5.1.1 CPython/3.12.7
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 49ac4b57acb6c5eec82667c6e53dade0b0d9310625b1be0385bc66360ddb1fba |
|
MD5 | 7caf9e5bff459cf727b9e4e31878c663 |
|
BLAKE2b-256 | 74c9c5f6d52fc9d77461cd9bc84e73d8c57e8387a6a9d9ab6bfa7bb22738591b |