Common interface for data container classes
Project description
itemadapter
The ItemAdapter
class is a wrapper for data container objects, providing a
common interface to handle objects of different types in an uniform manner,
regardless of their underlying implementation.
Currently supported types are:
scrapy.item.Item
dict
dataclass
-based classesattrs
-based classespydantic
-based classes (pydantic>=2
not yet supported)
Additionally, interaction with arbitrary types is supported, by implementing
a pre-defined interface (see extending itemadapter
).
Requirements
- Python 3.8+
scrapy
: optional, needed to interact withscrapy
itemsattrs
: optional, needed to interact withattrs
-based itemspydantic
: optional, needed to interact withpydantic
-based items (pydantic>=2
not yet supported)
Installation
itemadapter
is available on PyPI
, it can be installed with pip
:
pip install itemadapter
License
itemadapter
is distributed under a BSD-3 license.
Basic usage
The following is a simple example using a dataclass
object.
Consider the following type definition:
>>> from dataclasses import dataclass
>>> from itemadapter import ItemAdapter
>>> @dataclass
... class InventoryItem:
... name: str
... price: float
... stock: int
>>>
An ItemAdapter
object can be treated much like a dictionary:
>>> obj = InventoryItem(name='foo', price=20.5, stock=10)
>>> ItemAdapter.is_item(obj)
True
>>> adapter = ItemAdapter(obj)
>>> len(adapter)
3
>>> adapter["name"]
'foo'
>>> adapter.get("price")
20.5
>>>
The wrapped object is modified in-place:
>>> adapter["name"] = "bar"
>>> adapter.update({"price": 12.7, "stock": 9})
>>> adapter.item
InventoryItem(name='bar', price=12.7, stock=9)
>>> adapter.item is obj
True
>>>
Converting to dict
The ItemAdapter
class provides the asdict
method, which converts
nested items recursively. Consider the following example:
>>> from dataclasses import dataclass
>>> from itemadapter import ItemAdapter
>>> @dataclass
... class Price:
... value: int
... currency: str
>>> @dataclass
... class Product:
... name: str
... price: Price
>>>
>>> item = Product("Stuff", Price(42, "UYU"))
>>> adapter = ItemAdapter(item)
>>> adapter.asdict()
{'name': 'Stuff', 'price': {'value': 42, 'currency': 'UYU'}}
>>>
Note that just passing an adapter object to the dict
built-in also works,
but it doesn't traverse the object recursively converting nested items:
>>> dict(adapter)
{'name': 'Stuff', 'price': Price(value=42, currency='UYU')}
>>>
API reference
Built-in adapters
The following adapters are included by default:
itemadapter.adapter.ScrapyItemAdapter
: handlesScrapy
itemsitemadapter.adapter.DictAdapter
: handlesPython
dictionariesitemadapter.adapter.DataclassAdapter
: handlesdataclass
objectsitemadapter.adapter.AttrsAdapter
: handlesattrs
objectsitemadapter.adapter.PydanticAdapter
: handlespydantic
objects
class itemadapter.adapter.ItemAdapter(item: Any)
This is the main entrypoint for the package. Tipically, user code
wraps an item using this class, and proceeds to handle it with the provided interface.
ItemAdapter
implements the
MutableMapping
interface, providing a dict
-like API to manipulate data for the object it wraps
(which is modified in-place).
Attributes
class attribute ADAPTER_CLASSES: Iterable
Stores the currently registered adapter classes.
The order in which the adapters are registered is important. When an ItemAdapter
object is
created for a specific item, the registered adapters are traversed in order and the first
adapter class to return True
for the is_item
class method is used for all subsequent
operations. The default order is the one defined in the
built-in adapters section.
The default implementation uses a
collections.deque
to support efficient addition/deletion of adapters classes to both ends, but if you are
deriving a subclass (see the section on extending itemadapter
for additional information), any other iterable (e.g. list
, tuple
) will work.
Methods
class method is_item(item: Any) -> bool
Return True
if any of the registed adapters can handle the item
(i.e. if any of them returns True
for its is_item
method with
item
as argument), False
otherwise.
class method is_item_class(item_class: type) -> bool
Return True
if any of the registered adapters can handle the item class
(i.e. if any of them returns True
for its is_item_class
method with
item_class
as argument), False
otherwise.
class method get_field_meta_from_class(item_class: type, field_name: str) -> MappingProxyType
Return a types.MappingProxyType
object, which is a read-only mapping with metadata about the given field. If the item class does not
support field metadata, or there is no metadata for the given field, an empty object is returned.
The returned value is taken from the following sources, depending on the item type:
scrapy.item.Field
forscrapy.item.Item
sdataclasses.field.metadata
fordataclass
-based itemsattr.Attribute.metadata
forattrs
-based itemspydantic.fields.FieldInfo
forpydantic
-based items
class method get_field_names_from_class(item_class: type) -> Optional[list[str]]
Return a list with the names of all the fields defined for the item class. If an item class doesn't support defining fields upfront, None is returned.
get_field_meta(field_name: str) -> MappingProxyType
Return metadata for the given field, if available. Unless overriden in a custom adapter class, by default
this method calls the adapter's get_field_meta_from_class
method, passing the wrapped item's class.
field_names() -> collections.abc.KeysView
Return a keys view with the names of all the defined fields for the item.
asdict() -> dict
Return a dict
object with the contents of the adapter. This works slightly different than
calling dict(adapter)
, because it's applied recursively to nested items (if there are any).
function itemadapter.utils.is_item(obj: Any) -> bool
Return True
if the given object belongs to (at least) one of the supported types,
False
otherwise. This is an alias, using the itemadapter.adapter.ItemAdapter.is_item
class method is encouraged for better performance.
function itemadapter.utils.get_field_meta_from_class(item_class: type, field_name: str) -> types.MappingProxyType
Alias for itemadapter.adapter.ItemAdapter.get_field_meta_from_class
Metadata support
scrapy.item.Item
, dataclass
, attrs
, and pydantic
objects allow the definition of
arbitrary field metadata. This can be accessed through a
MappingProxyType
object, which can be retrieved from an item instance with
itemadapter.adapter.ItemAdapter.get_field_meta
, or from an item class
with the itemadapter.adapter.ItemAdapter.get_field_meta_from_class
method (or its alias itemadapter.utils.get_field_meta_from_class
).
The source of the data depends on the underlying type (see the docs for
ItemAdapter.get_field_meta_from_class
).
scrapy.item.Item
objects
>>> from scrapy.item import Item, Field
>>> from itemadapter import ItemAdapter
>>> class InventoryItem(Item):
... name = Field(serializer=str)
... value = Field(serializer=int, limit=100)
...
>>> adapter = ItemAdapter(InventoryItem(name="foo", value=10))
>>> adapter.get_field_meta("name")
mappingproxy({'serializer': <class 'str'>})
>>> adapter.get_field_meta("value")
mappingproxy({'serializer': <class 'int'>, 'limit': 100})
>>>
dataclass
objects
>>> from dataclasses import dataclass, field
>>> @dataclass
... class InventoryItem:
... name: str = field(metadata={"serializer": str})
... value: int = field(metadata={"serializer": int, "limit": 100})
...
>>> adapter = ItemAdapter(InventoryItem(name="foo", value=10))
>>> adapter.get_field_meta("name")
mappingproxy({'serializer': <class 'str'>})
>>> adapter.get_field_meta("value")
mappingproxy({'serializer': <class 'int'>, 'limit': 100})
>>>
attrs
objects
>>> import attr
>>> @attr.s
... class InventoryItem:
... name = attr.ib(metadata={"serializer": str})
... value = attr.ib(metadata={"serializer": int, "limit": 100})
...
>>> adapter = ItemAdapter(InventoryItem(name="foo", value=10))
>>> adapter.get_field_meta("name")
mappingproxy({'serializer': <class 'str'>})
>>> adapter.get_field_meta("value")
mappingproxy({'serializer': <class 'int'>, 'limit': 100})
>>>
pydantic
objects
>>> from pydantic import BaseModel, Field
>>> class InventoryItem(BaseModel):
... name: str = Field(serializer=str)
... value: int = Field(serializer=int, limit=100)
...
>>> adapter = ItemAdapter(InventoryItem(name="foo", value=10))
>>> adapter.get_field_meta("name")
mappingproxy({'serializer': <class 'str'>})
>>> adapter.get_field_meta("value")
mappingproxy({'serializer': <class 'int'>, 'limit': 100})
>>>
Extending itemadapter
This package allows to handle arbitrary item classes, by implementing an adapter interface:
class itemadapter.adapter.AdapterInterface(item: Any)
Abstract Base Class for adapters. An adapter that handles a specific type of item must
inherit from this class and implement the abstract methods defined on it. AdapterInterface
inherits from collections.abc.MutableMapping
,
so all methods from the MutableMapping
interface must be implemented as well.
-
class method
is_item_class(cls, item_class: type) -> bool
Return
True
if the adapter can handle the given item class,False
otherwise. Abstract (mandatory). -
class method
is_item(cls, item: Any) -> bool
Return
True
if the adapter can handle the given item,False
otherwise. The default implementation callscls.is_item_class(item.__class__)
. -
class method
get_field_meta_from_class(cls, item_class: type) -> bool
Return metadata for the given item class and field name, if available. By default, this method returns an empty
MappingProxyType
object. Please supply your own method definition if you want to handle field metadata based on custom logic. See the section on metadata support for additional information. -
method
get_field_meta(self, field_name: str) -> types.MappingProxyType
Return metadata for the given field name, if available. It's usually not necessary to override this method, since the
itemadapter.adapter.AdapterInterface
base class provides a default implementation that callsItemAdapter.get_field_meta_from_class
with the wrapped item's class as argument. See the section on metadata support for additional information. -
method
field_names(self) -> collections.abc.KeysView
:Return a dynamic view of the item's field names. By default, this method returns the result of calling
keys()
on the current adapter, i.e., its return value depends on the implementation of the methods from theMutableMapping
interface (more specifically, it depends on the return value of__iter__
).You might want to override this method if you want a way to get all fields for an item, whether or not they are populated. For instance, Scrapy uses this method to define column names when exporting items to CSV.
Registering an adapter
Add your custom adapter class to the itemadapter.adapter.ItemAdapter.ADAPTER_CLASSES
class attribute in order to handle custom item classes:
Example
>>> from itemadapter.adapter import ItemAdapter
>>> from tests.test_interface import BaseFakeItemAdapter, FakeItemClass
>>>
>>> ItemAdapter.ADAPTER_CLASSES.appendleft(BaseFakeItemAdapter)
>>> item = FakeItemClass()
>>> adapter = ItemAdapter(item)
>>> adapter
<ItemAdapter for FakeItemClass()>
>>>
Multiple adapter classes
If you need to have different handlers and/or priorities for different cases
you can subclass the ItemAdapter
class and set the ADAPTER_CLASSES
attribute as needed:
Example
>>> from itemadapter.adapter import (
... ItemAdapter,
... AttrsAdapter,
... DataclassAdapter,
... DictAdapter,
... PydanticAdapter,
... ScrapyItemAdapter,
... )
>>> from scrapy.item import Item, Field
>>>
>>> class BuiltinTypesItemAdapter(ItemAdapter):
... ADAPTER_CLASSES = [DictAdapter, DataclassAdapter]
...
>>> class ThirdPartyTypesItemAdapter(ItemAdapter):
... ADAPTER_CLASSES = [AttrsAdapter, PydanticAdapter, ScrapyItemAdapter]
...
>>> class ScrapyItem(Item):
... foo = Field()
...
>>> BuiltinTypesItemAdapter.is_item(dict())
True
>>> ThirdPartyTypesItemAdapter.is_item(dict())
False
>>> BuiltinTypesItemAdapter.is_item(ScrapyItem(foo="bar"))
False
>>> ThirdPartyTypesItemAdapter.is_item(ScrapyItem(foo="bar"))
True
>>>
More examples
scrapy.item.Item
objects
>>> from scrapy.item import Item, Field
>>> from itemadapter import ItemAdapter
>>> class InventoryItem(Item):
... name = Field()
... price = Field()
...
>>> item = InventoryItem(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
{'name': 'bar', 'price': 5}
>>>
dict
>>> from itemadapter import ItemAdapter
>>> item = dict(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
{'name': 'bar', 'price': 5}
>>>
dataclass
objects
>>> from dataclasses import dataclass
>>> from itemadapter import ItemAdapter
>>> @dataclass
... class InventoryItem:
... name: str
... price: int
...
>>> item = InventoryItem(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
InventoryItem(name='bar', price=5)
>>>
attrs
objects
>>> import attr
>>> from itemadapter import ItemAdapter
>>> @attr.s
... class InventoryItem:
... name = attr.ib()
... price = attr.ib()
...
>>> item = InventoryItem(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
InventoryItem(name='bar', price=5)
>>>
pydantic
objects
>>> from pydantic import BaseModel
>>> from itemadapter import ItemAdapter
>>> class InventoryItem(BaseModel):
... name: str
... price: int
...
>>> item = InventoryItem(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
InventoryItem(name='bar', price=5)
>>>
Changelog
See the full changelog
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for itemadapter-0.9.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cfd108c9d5205d056fcac402ec8f8e9d799ce9066911eec1cd521ea442f87af1 |
|
MD5 | 0d7cc20dfb1256ded3b0d205932b8c3e |
|
BLAKE2b-256 | 399371cd02a96887da62e248647da70af60bf1538bbe445b4a0735f84126fc46 |