itemadapter

The ItemAdapter class is a wrapper for data container objects, providing a common interface to handle objects of different types in an uniform manner, regardless of their underlying implementation.

Currently supported types are:

• dict
• scrapy.item.Item
• dataclass-based classes
• attrs-based classes

Requirements

• Python 3.5+
• scrapy: optional, needed to interact with scrapy items
• dataclasses (stdlib in Python 3.7+, or its backport in Python 3.6): optional, needed to interact with dataclass-based items
• attrs: optional, needed to interact with attrs-based items

Installation

itemadapter is available on PyPI, it can be installed with pip:

pip install itemadapter

License

itemadapter is distributed under a BSD-3 license.

Basic usage

The following is a simple example using a dataclass object. Consider the following type definition:

from dataclasses import dataclass
from itemadapter import ItemAdapter, is_item

@dataclass
class InventoryItem:
    name: str
    price: float
    stock: int

The ItemAdapter object can be treated much like a dictionary:

>>> obj = InventoryItem(name='foo', price=20.5, stock=10)
>>> is_item(obj)
True
>>> adapter = ItemAdapter(obj)
>>> len(adapter)
3
>>> adapter["name"]
'foo'
>>> adapter.get("price")
20.5

The wrapped object is modified in-place:

>>> adapter["name"] = "bar"
>>> adapter.update({"price": 12.7, "stock": 9})
>>> adapter.item
InventoryItem(name='bar', price=12.7, stock=9)
>>> adapter.item is obj
True

Converting to dictionary

Passing an ItemAdapter to the dict built-in will create a new dictionary:

>>> dict(adapter)
{'name': 'bar', 'price': 12.7, 'stock': 9}

For more examples using different types, refer to the examples section below.

Public API

ItemAdapter class

class itemadapter.adapter.ItemAdapter(item: Any)

ItemAdapter implements the MutableMapping interface, providing a dict-like API to manipulate data for the object it wraps (which is modified in-place).

Two additional methods are available:

get_field_meta(field_name: str) -> MappingProxyType

Return a MappingProxyType object, which is a read-only mapping with metadata about the given field. If the item class does not support field metadata, or there is no metadata for the given field, an empty object is returned.

The returned value is taken from the following sources, depending on the item type:

• scrapy.item.Field for scrapy.item.Items
• dataclasses.field.metadata for dataclass-based items
• attr.Attribute.metadata for attrs-based items

field_names() -> List[str]

Return a list with the names of all the defined fields for the item.

is_item function

itemadapter.utils.is_item(obj: Any) -> bool

Return True if the given object belongs to one of the supported types, False otherwise.

Metadata support

scrapy.item.Item, dataclass and attrs objects allow the inclusion of arbitrary field metadata, which can be retrieved with the ItemAdapter.get_field_meta method. The definition procedure depends on the underlying type.

scrapy.item.Item objects

>>> from scrapy.item import Item, Field
>>> from itemadapter import ItemAdapter
>>> class InventoryItem(Item):
...     name = Field(serializer=str)
...     value = Field(serializer=int, limit=100)
...
>>> adapter = ItemAdapter(InventoryItem(name="foo", value=10))
>>> adapter.get_field_meta("name")
mappingproxy({'serializer': <class 'str'>})
>>> adapter.get_field_meta("value")
mappingproxy({'serializer': <class 'int'>, 'limit': 100})

dataclass objects

>>> from dataclasses import dataclass, field
>>> @dataclass
... class InventoryItem:
...     name: str = field(metadata={"serializer": str})
...     value: int = field(metadata={"serializer": int, "limit": 100})
...
>>> adapter = ItemAdapter(InventoryItem(name="foo", value=10))
>>> adapter.get_field_meta("name")
mappingproxy({'serializer': <class 'str'>})
>>> adapter.get_field_meta("value")
mappingproxy({'serializer': <class 'int'>, 'limit': 100})

attrs objects

>>> import attr
>>> @attr.s
... class InventoryItem:
...     name = attr.ib(metadata={"serializer": str})
...     value = attr.ib(metadata={"serializer": int, "limit": 100})
...
>>> adapter = ItemAdapter(InventoryItem(name="foo", value=10))
>>> adapter.get_field_meta("name")
mappingproxy({'serializer': <class 'str'>})
>>> adapter.get_field_meta("value")
mappingproxy({'serializer': <class 'int'>})

More examples

scrapy.item.Item objects

>>> from scrapy.item import Item, Field
>>> from itemadapter import ItemAdapter
>>> class InventoryItem(Item):
...     name = Field()
...     price = Field()
...
>>> item = InventoryItem(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
{'name': 'bar', 'price': 5}

dict

>>> from itemadapter import ItemAdapter
>>> item = dict(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
{'name': 'bar', 'price': 5}

dataclass objects

>>> from dataclasses import dataclass
>>> from itemadapter import ItemAdapter
>>> @dataclass
... class InventoryItem:
...     name: str
...     price: int
...
>>> item = InventoryItem(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
InventoryItem(name='bar', price=5)

attrs objects

>>> import attr
>>> from itemadapter import ItemAdapter
>>> @attr.s
... class InventoryItem:
...     name = attr.ib()
...     price = attr.ib()
...
>>> item = InventoryItem(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
InventoryItem(name='bar', price=5)