itemadapter

The ItemAdapter class is a wrapper for data container objects, providing a common interface to handle objects of different types in an uniform manner, regardless of their underlying implementation.

Currently supported types are:

• dict
• scrapy.item.Item
• dataclass-based classes
• attrs-based classes

Requirements

• Python 3.6+
• scrapy: optional, needed to interact with scrapy items
• dataclasses (stdlib in Python 3.7+, or its backport in Python 3.6): optional, needed to interact with dataclass-based items
• attrs: optional, needed to interact with attrs-based items

Installation

itemadapter is available on PyPI, it can be installed with pip:

pip install itemadapter

License

itemadapter is distributed under a BSD-3 license.

Basic usage

The following is a simple example using a dataclass object. Consider the following type definition:

>>> from dataclasses import dataclass
>>> from itemadapter import ItemAdapter, is_item
>>> @dataclass
... class InventoryItem:
...     name: str
...     price: float
...     stock: int
>>>

The ItemAdapter object can be treated much like a dictionary:

>>> obj = InventoryItem(name='foo', price=20.5, stock=10)
>>> is_item(obj)
True
>>> adapter = ItemAdapter(obj)
>>> len(adapter)
3
>>> adapter["name"]
'foo'
>>> adapter.get("price")
20.5
>>>

The wrapped object is modified in-place:

>>> adapter["name"] = "bar"
>>> adapter.update({"price": 12.7, "stock": 9})
>>> adapter.item
InventoryItem(name='bar', price=12.7, stock=9)
>>> adapter.item is obj
True
>>>

Converting to dict

The ItemAdapter class provides the asdict method, which converts nested items recursively. Consider the following example:

>>> from dataclasses import dataclass
>>> from itemadapter import ItemAdapter
>>> @dataclass
... class Price:
...     value: int
...     currency: str
>>> @dataclass
... class Product:
...     name: str
...     price: Price
>>>

>>> item = Product("Stuff", Price(42, "UYU"))
>>> adapter = ItemAdapter(item)
>>> adapter.asdict()
{'name': 'Stuff', 'price': {'value': 42, 'currency': 'UYU'}}
>>>

Note that just passing an adapter object to the dict built-in also works, but it doesn't traverse the object recursively converting nested items:

>>> dict(adapter)
{'name': 'Stuff', 'price': Price(value=42, currency='UYU')}
>>>

Public API

ItemAdapter class

class itemadapter.adapter.ItemAdapter(item: Any)

ItemAdapter implements the MutableMapping interface, providing a dict-like API to manipulate data for the object it wraps (which is modified in-place).

Some additional methods are available:

get_field_meta(field_name: str) -> MappingProxyType

Return a MappingProxyType object, which is a read-only mapping with metadata about the given field. If the item class does not support field metadata, or there is no metadata for the given field, an empty object is returned.

The returned value is taken from the following sources, depending on the item type:

• scrapy.item.Field for scrapy.item.Items
• dataclasses.field.metadata for dataclass-based items
• attr.Attribute.metadata for attrs-based items

field_names() -> KeysView

Return a keys view with the names of all the defined fields for the item.

asdict() -> dict

Return a dict object with the contents of the adapter. This works slightly different than calling dict(adapter), because it's applied recursively to nested items (if there are any).

is_item function

itemadapter.utils.is_item(obj: Any) -> bool

Return True if the given object belongs to one of the supported types, False otherwise.

get_field_meta_from_class function

itemadapter.utils.get_field_meta_from_class(item_class: type, field_name: str) -> MappingProxyType

Given an item class and a field name, return a MappingProxyType object, which is a read-only mapping with metadata about the given field. If the item class does not support field metadata, or there is no metadata for the given field, an empty object is returned.

Metadata support

scrapy.item.Item, dataclass and attrs objects allow the inclusion of arbitrary field metadata. This can be retrieved from an item instance with the itemadapter.adapter.ItemAdapter.get_field_meta method, or from an item class with the itemadapter.utils.get_field_meta_from_class function. The definition procedure depends on the underlying type.

scrapy.item.Item objects

>>> from scrapy.item import Item, Field
>>> from itemadapter import ItemAdapter
>>> class InventoryItem(Item):
...     name = Field(serializer=str)
...     value = Field(serializer=int, limit=100)
...
>>> adapter = ItemAdapter(InventoryItem(name="foo", value=10))
>>> adapter.get_field_meta("name")
mappingproxy({'serializer': <class 'str'>})
>>> adapter.get_field_meta("value")
mappingproxy({'serializer': <class 'int'>, 'limit': 100})
>>>

dataclass objects

>>> from dataclasses import dataclass, field
>>> @dataclass
... class InventoryItem:
...     name: str = field(metadata={"serializer": str})
...     value: int = field(metadata={"serializer": int, "limit": 100})
...
>>> adapter = ItemAdapter(InventoryItem(name="foo", value=10))
>>> adapter.get_field_meta("name")
mappingproxy({'serializer': <class 'str'>})
>>> adapter.get_field_meta("value")
mappingproxy({'serializer': <class 'int'>, 'limit': 100})
>>>

attrs objects

>>> import attr
>>> @attr.s
... class InventoryItem:
...     name = attr.ib(metadata={"serializer": str})
...     value = attr.ib(metadata={"serializer": int, "limit": 100})
...
>>> adapter = ItemAdapter(InventoryItem(name="foo", value=10))
>>> adapter.get_field_meta("name")
mappingproxy({'serializer': <class 'str'>})
>>> adapter.get_field_meta("value")
mappingproxy({'serializer': <class 'int'>, 'limit': 100})
>>>

More examples

scrapy.item.Item objects

>>> from scrapy.item import Item, Field
>>> from itemadapter import ItemAdapter
>>> class InventoryItem(Item):
...     name = Field()
...     price = Field()
...
>>> item = InventoryItem(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
{'name': 'bar', 'price': 5}
>>>

dict

>>> from itemadapter import ItemAdapter
>>> item = dict(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
{'name': 'bar', 'price': 5}
>>>

dataclass objects

>>> from dataclasses import dataclass
>>> from itemadapter import ItemAdapter
>>> @dataclass
... class InventoryItem:
...     name: str
...     price: int
...
>>> item = InventoryItem(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
InventoryItem(name='bar', price=5)
>>>

attrs objects

>>> import attr
>>> from itemadapter import ItemAdapter
>>> @attr.s
... class InventoryItem:
...     name = attr.ib()
...     price = attr.ib()
...
>>> item = InventoryItem(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
InventoryItem(name='bar', price=5)
>>>

Changelog

See the full changelog