Skip to main content

A JIT implementation for Marshmallow to speed up dumping and loading objects.

Project description

*************************************************************
:fire:toastedmarshmallow:fire:: Makes Marshmallow Toasty Fast
*************************************************************

Toasted Marshmallow implements a JIT for marshmallow that speeds up dumping
objects 10-25X (depending on your schema). Toasted Marshmallow allows you to
have the great API that
`Marshmallow <https://github.com/marshmallow-code/marshmallow>`_ provides
without having to sacrifice performance!

::

Benchmark Result:
Original Time: 2682.61 usec/dump
Optimized Time: 176.38 usec/dump
Optimized (Cython) Time: 125.77 usec/dump
Speed up: 15.21x
Cython Speed up: 21.33x

Even ``PyPy`` benefits from ``toastedmarshmallow``!

::

Benchmark Result:
Original Time: 189.78 usec/dump
Optimized Time: 20.03 usec/dump
Speed up: 9.48x

Installing toastedmarshmallow
-----------------------------

.. code-block:: bash

pip install toastedmarshmallow

This will *also* install a slightly-forked ``marshmallow`` that includes some
hooks Toastedmarshmallow needs enable the JIT to run before falling back
to the original marshmallow code. These changes are minimal making it easier
to track upstream. You can find the changes
`Here <https://github.com/marshmallow-code/marshmallow/pull/629/files>`_.

This means you should **remove** ``marshmallow`` from your requirements and
replace it with ``toastedmarshmallow``. By default there is no
difference unless you explicitly enable Toasted Marshmallow.

Enabling Toasted Marshmallow
----------------------------

Enabling Toasted Marshmallow on an existing Schema is just one line of code,
set the ``jit`` property on any ``Schema`` instance to
``toastedmarshmallow.Jit``. For example:

.. code-block:: python

from datetime import date
import toastedmarshmallow
from marshmallow import Schema, fields, pprint

class ArtistSchema(Schema):
name = fields.Str()

class AlbumSchema(Schema):
title = fields.Str()
release_date = fields.Date()
artist = fields.Nested(ArtistSchema())

schema = AlbumSchema()
# Specify the jit method as toastedmarshmallow's jit
schema.jit = toastedmarshmallow.Jit
# And that's it! Your dump methods are 15x faster!

It's also possible to use the ``Meta`` class on the ``Marshmallow`` schema
to specify all instances of a given ``Schema`` should be optimized:

.. code-block:: python

import toastedmarshmallow
from marshmallow import Schema, fields, pprint

class ArtistSchema(Schema):
class Meta:
jit = toastedMarshmallow.Jit
name = fields.Str()

You can also enable Toasted Marshmallow globally by setting the environment
variable ``MARSHMALLOW_SCHEMA_DEFAULT_JIT`` to ``toastedmarshmallow.Jit`` .
Future versions of Toasted Marshmallow may make this the default.

How it works
------------

Toasted Marshmallow works by generating code at runtime to optimize dumping
objects without going through layers and layers of reflection. The generated
code optimistically assumes the objects being passed in are schematically valid,
falling back to the original marshmallow code on failure.

For example, taking ``AlbumSchema`` from above, Toastedmarshmallow will
generate the following 3 methods:

.. code-block:: python

def InstanceSerializer(obj):
res = {}
value = obj.release_date; value = value() if callable(value) else value; res["release_date"] = _field_release_date__serialize(value, "release_date", obj)
value = obj.artist; value = value() if callable(value) else value; res["artist"] = _field_artist__serialize(value, "artist", obj)
value = obj.title; value = value() if callable(value) else value; value = str(value) if value is not None else None; res["title"] = value
return res

def DictSerializer(obj):
res = {}
if "release_date" in obj:
value = obj["release_date"]; value = value() if callable(value) else value; res["release_date"] = _field_release_date__serialize(value, "release_date", obj)
if "artist" in obj:
value = obj["artist"]; value = value() if callable(value) else value; res["artist"] = _field_artist__serialize(value, "artist", obj)
if "title" in obj:
value = obj["title"]; value = value() if callable(value) else value; value = str(value) if value is not None else None; res["title"] = value
return res

def HybridSerializer(obj):
res = {}
try:
value = obj["release_date"]
except (KeyError, AttributeError, IndexError, TypeError):
value = obj.release_date
value = value; value = value() if callable(value) else value; res["release_date"] = _field_release_date__serialize(value, "release_date", obj)
try:
value = obj["artist"]
except (KeyError, AttributeError, IndexError, TypeError):
value = obj.artist
value = value; value = value() if callable(value) else value; res["artist"] = _field_artist__serialize(value, "artist", obj)
try:
value = obj["title"]
except (KeyError, AttributeError, IndexError, TypeError):
value = obj.title
value = value; value = value() if callable(value) else value; value = str(value) if value is not None else None; res["title"] = value
return res

Toastedmarshmallow will invoke the proper serializer based upon the input.

Since Toastedmarshmallow is generating code at runtime, it's critical you
re-use Schema objects. If you're creating a new Schema object every time you
serialize/deserialize an object you'll likely have much worse performance.

:zap::microscope: Experimental :microscope::zap:
--------------------------------------------------

Toastedmarshmallow also has an experimental Cython based jit. It takes the
generated code above and runs it through Cython first, getting another 1.5x
win. Generally the generated Python code is fast enough, but this is a useful
option when you've got to squeeze out every last bit of performance.

To use the Cython jit, replace `Jit` with `CythonJit`:

.. code-block:: python

schema.jit = toastedmarshmallow.CythonJit


Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

toastedmarshmallow-0.2.1.tar.gz (52.4 kB view details)

Uploaded Source

Built Distributions

toastedmarshmallow-0.2.1-py3.6.egg (116.5 kB view details)

Uploaded Source

toastedmarshmallow-0.2.1-py2.py3-none-any.whl (54.9 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file toastedmarshmallow-0.2.1.tar.gz.

File metadata

File hashes

Hashes for toastedmarshmallow-0.2.1.tar.gz
Algorithm Hash digest
SHA256 fa7e9cf5d4df045d0d0c7230de0f8abdf02a911500c0878c4b75a5dd877031a5
MD5 f79608f40c8f48714ddddd21bd466c31
BLAKE2b-256 89e5c469b430766ab7025f25002b16ac7062c3f5d090d8ea53566d366148f897

See more details on using hashes here.

File details

Details for the file toastedmarshmallow-0.2.1-py3.6.egg.

File metadata

File hashes

Hashes for toastedmarshmallow-0.2.1-py3.6.egg
Algorithm Hash digest
SHA256 81ac63873f4533080c9cf0e6277d3ab131103c6efe8ef03835253c851967a19f
MD5 179018fae422bc992d0d1929bdfb1120
BLAKE2b-256 780e4d1e0d28226aabd7bcb650ddce625c7a17d096f99a1a4b5fff68392fd97f

See more details on using hashes here.

File details

Details for the file toastedmarshmallow-0.2.1-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for toastedmarshmallow-0.2.1-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 f887b7a7a0dfba07632dbb82bd292738c18a6426547c61f90b4cc4d55ddd1c4d
MD5 40f7700bbb3c99acac62b8b4ecd177ac
BLAKE2b-256 5f80fef2c40e9186aa80829a415478dae86fa9c5cfbc678d99203746e7391d59

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page