Skip to main content

Vocabulary library Plone. Central, Pluggable, TTW, with IMS VDEX Support

Project description

Overview

  • ATVocabularyManager offers central through the Plone management of dynamic vocabularies.

  • This product is based on Archetypes and made to work with Archetypes as well as with other Products. It is intended use is within Archetypes Fields. Using it as a vocabulary provider for CMFMetadata worked out fine too. Integration with different other products will work as well.

  • to use a managed vocabulary simply add the term vocabulary = NamedVocabulary("myvocabulary") to the fields of your Archetypes Schema, import NamedVocabulary from this Product and create your vocabulary with id myvocabulary in ‘portal_vocabularies’ tool (available through Plone Site-Setup).

  • ATVocabularyManager supports:

    simple flat key

    value vocabularies,

    tree like

    hierachical vocabularies (see Limitations),

    IMS Vocabulary Definition Exchange Format (VDEX) aware vocabularies

    with XML Import and Export. VDEX is i18n-aware by its nature and does not need LinguaPlone!

    Vocabularies are pluggable types.

    ATVocabularyManager is prepared for extension with your special vocabulary type. ArchGenXML will help you here. Each vocabulary term needs to be an CMF aware content type. Reuse normal rich content as a vocabularies.

ArchGenXML 1.4+ code-generator does full integration of ATVocabularyManager: via tagged value defined named vocabularies are registered transparently, VDEX-XML files are imported at install-time, stub vocabularies are created at install time and custom types are registered by just providing appropriate stereotypes.

ATVM is Linguaplone compatible (only tested with SimpleVocabulary, and TreeVocabulary). Add a simple vocabulary with some items, install and configure Linguaplone, translate the vocabulary to the language(s) of your choice, translate every item to the language(s) of your choice. NamedVocabulary() will return the vocabulary as usual, the keys will stay the same disregarding language settings, the values will show in the currently selected language. VDEX vocabularies are not using LinguaPlone, but are i18n-aware (imo much better than everything else).

You can do hierachy-aware searches on treevocabularies (for more information on this see doc/search_treevocabulary.txt) attention: curently certain changes in the term hierachy require a catalog rebuild (see Limitations).

Installation

See docs/INSTALL.txt.

To speed up ATVocabularyManager you might want to associate it with a Cache-Manager.

Dependencies

  • Archetypes 1.5.x and above.

  • Targets with Plone 3.1+

  • imsvdex (usally it get fetched automatically as dependency, otherwise easy_install imsvdex - )

  • Linguaplone (optional: for tests and multilingual vocabularies)

Tests

If you’re using a buildout run the tests using

bin/instance test -s Products.ATVocabularyManager

To run a certain testmodule run

bin/instance test -s Products.ATVocabularyManager -m testSimpleVocabulary

Limitations

TreeVocabulary catalog updates

If you are using the hierachy-aware catalog search support of tree-vocabularies you need to rebuild the uid_catalog and portal_catalog if you move a vocabularyterm that has other terms below it.

Creation/Edit of VDEX TTW is difficult.

Take an editor of your choice, create the vocab and upload it or help writing an UI.

Known Bugs

Support

Usally the authors are offering professional support. The classical well-working community support is found at the mailing-lists and IRC-channels announced at plone.org:http://plone.org

Credits

Several parts code was created for the ZUCCARO project. ZUCCARO (Zope-based Universally Configurable Classes for Academic Research Online) is a database framework for the Humanities developed by the Bibliotheca Hertziana, Max Planck Institute for Art History For further information: “zuccaro.biblhertz.it”:http://zuccaro.biblhertz.it/

Author

  • Jens Klein <jens@bluedynamics.com>

  • Major contributions by Harald Friessnegger ‘frisi’: i18n, cleanup, more cleanup of hierachical vocabularies.

Contributors

Several people committed smaller fixes and translations, thank you all!

TODO

Tasks

  • create an issue: simplevocabs do not get alphabetically ordered, nore do they use the order defined by getObjectPositionInParent (see issue #4)

  • drop the try from Products.Archetypes.interfaces.vocabulary import IVocabularyTerm

  • we should add the parameter ‘lang’ to IVocabularyTerm.getTermValue [jens] agrees, but we should make it optional (normally translations should happen transparently)

  • move getTermPath to separate hierarchicalvocabulary interface? [hf] this would allow us to differentiate hierarchical vocabularies by their interface we’d should to make namedvocbaulary.getKeyPathForTerms aware of this too then.

  • (missing in alias) mark methods getVocabularyValue and getVocabularyKey as deprecated and use the methods getTermValue and getTermKey instead [jens] XXX

  • use zope3 interfaces for terms useful for example in NamedVocabulary.getKeyPathForTerms (see comment there)

  • resolve TODOs (XXX, FIXME) marked in the code ;-)

Ideas for features

selection-depth

basic idea:

fields can specify selection depth (0 unlimited, 1 first level, 2 seconde level, etc) that gets displayed in the Displaylist.

why do we need that:

if a certain field does not need the fine-grained selection possibilities a vocabulary offers, no new vocabulary has to be defined.

variable displaylists

basic idea:

field can specify which attribute of the vocabularyterms is used for the displaylist.

why could we need that:

in certain cases the same vocabulary is needed, but another field or field combination should be used for display in the selection.

risks/drawbacks:

more complicated handling of cached displaylists (different dictionaries have to be handeled, not just one)

caching displaylists

basic idea:

caching of displaylists and vocabularydicts using ZCacheable (the only thing we’d have to care about is invalidating the cache when a term gets moved or changed)

why do we need that:

fetching vocabulary displaylists costs a lot. at the moment the displaylist for the whole vocab gets fetched everytime a content type that uses the vocabulary is edited and even VIEWED!

[jens] will implement this for the vdex vocabulary in the near future: DONE
open tasks: Invalidate cache after upload of a vdex.

(Auto?)-Assign to a ram-cache-manager.

generic setup support

basic idea

vdex already can be ex and imported to xml. we shoud realize it for simple and term too [jens] decision needed: use vdex, some own xml or better atxml from marshall or just simple Excel importable CSV?) Jens decision is: vdex (implemented import) or csv (todo).

mixins

common hierarchic functionality of vdex and tree and possibly other types should be coded in a mixin. code and test once, use everywhere

catalog updates

current limitation:

every change on the structure of a vocabulary (eg: moving a term up the hierarchy) has to trigger not only the invalidation of the cache but also the reindexing of all terms down the hierarchy (uid_catalog search on getTermUidPath, catalog.manage_reindexIndex(ids=(‘getTermUidPath’,)))

currently this is a limitation of treevocabulary. vdex dont need to get cataloged

need to reindex objects in uid_catalog and portal_catalog:

Which method do we need to hook?

in portal_catalog for indexes used by content types

the worse problem is to reindex the objects that define custom indexes on their path-methods. we don’t know which catalog they use (could be portal_catalog, but also any other)

brainstorming:

  • store catalog and index in vocab or the atvmtool

  • provide a hook somewhere that can be implemented

  • maybe use references instead of text-linefields to be able to find out which content uses a certain term.

Simple Vocabulary

Simple vocabularies are flat vocabularies.

Creating a simple vocabulary

SimpleVocabularies can be created in the vocabulary library:

>>> atvm = self.portal.portal_vocabularies
>>> self.setRoles(['Manager'])
>>> _ = atvm.invokeFactory('SimpleVocabulary', 'testvocab', title="test vocabulary")

You can fetch a vocabulary calling getVocabularyByName on the vocabulary library

>>> simple = atvm.getVocabularyByName('testvocab')
>>> simple
<SimpleVocabulary at /plone/portal_vocabularies/testvocab>
>>> simple.Title()
'test vocabulary'

Creating simple vocabulary terms

Simple vocabularies can only contain SimpleVocabularyTerms.

>>> simple.allowedContentTypes()
[<FactoryTypeInformation at /plone/portal_types/SimpleVocabularyTerm>]

You can add terms using invokeFactory or the method addTerm:

>>> _ = simple.invokeFactory('SimpleVocabularyTerm', 'term1')
>>> simple.addTerm('term2', 'first time')
True

addTerm can ignore duplicate keys, and returns whether adding has been successfull.

>>> simple.addTerm('term2', 'second time', silentignore=True)
False
>>> simple.getDisplayList(self)
<DisplayList [('term1', ''), ('term2', 'first time')] at ...>

Batch creation

For creating simple vocabularies in python code you can use a convenience method atvm provides for you:

>>> from Products.ATVocabularyManager.utils.vocabs import createSimpleVocabs

This needs to be fed with a dictionary in the following format:

>>> testvocabs = {}
>>> testvocabs['sorting'] = (
...     ('c', u'Alpha'),
...     ('a', u'Zeppelin'),
...     ('y', u'Charly'))
>>> createSimpleVocabs(atvm, testvocabs)
>>> sorting = atvm.getVocabularyByName('sorting')
>>> sorting.contentIds()
['c', 'a', 'y']

Sorting

You can define the sort order of the vocabularyterms within a simplevocabulary by choosing one of the values in Sort method.

The default sort order is alphabetically by Vales (Title):

>>> sorting.getField('sortMethod').vocabulary.keys()
['getObjPositionInParent', 'lexicographic_values', 'lexicographic_keys']
>>> sorting.getSortMethod()
'lexicographic_values'

Sorting by Values

>>> sorting.getDisplayList(self)
<DisplayList [('c', 'Alpha'), ('y', 'Charly'), ('a', 'Zeppelin')] at ...>

Sorting by keys

>>> from Products.ATVocabularyManager.config import SORT_METHOD_LEXICO_KEYS
>>> sorting.setSortMethod(SORT_METHOD_LEXICO_KEYS)
>>> sorting.getDisplayList(self)
<DisplayList [('a', 'Zeppelin'), ('c', 'Alpha'), ('y', 'Charly')] at ...>

Sorting by folder position

>>> sorting.listFolderContents()
[<SimpleVocabularyTerm .../sorting/c>, <SimpleVocabularyTerm .../sorting/a>, <SimpleVocabularyTerm .../sorting/y>]
>>> from Products.ATVocabularyManager.config import SORT_METHOD_FOLDER_ORDER
>>> sorting.setSortMethod(SORT_METHOD_FOLDER_ORDER)
>>> sorting.getDisplayList(self)
<DisplayList [('c', 'Alpha'), ('a', 'Zeppelin'), ('y', 'Charly')] ...>

Linguaplone Support

Simple vocabularies have full Linguaplone Support, but also work w/o having Linguaplone installed:

>>> self.portal.portal_quickinstaller.isProductInstalled('LinguaPlone')
False

If Linguaplone is not installed, our vocabulary works without causing troubles

>>> self._createTestVocabulary()
>>> vocab = self.atvm.teststates
>>> vocab.getVocabularyDict(vocab)
{'ger': 'Germany', 'fin': 'Finland', 'aut': 'Austria', 'nor': 'Norway'}

If Linguaplone is installed, the simplevocabulary is fully supporting translated vocabularyterms:

XXX go on porting the tests/testSimpleVocabulary testTranslations here

Sorting

Also sorting on translated vocabularies works as excepted: XXX write tests for the following

  • keys are the same for every language, so sorting by key should result in the same order in every language

  • sorting on title might result in completely differnt order

  • sorting on position in folder might be differnt to, since translations have their separate order

Searches on hierarchical vocabularies

This tests demonstrates how ATVM enables you to make your contenttype searchable using a treevocabulary field.

imagine the following vocabulary:

engine-type

  • electrical

  • mechanical

    • otto engine

    • diesel engine

a search for engine==mechanical should return all entries for otto and diesel engines

You can skip this tutorial if you are no developer and simply want to learn how to do hierarchy-aware searches on your content type. This is demonstrated in the Product ATVocabularyManagerExample

Setup Vocabulary

let’s create our vocabulary using the utility methods provided by ATVM:

>>> from Products.CMFCore.utils import getToolByName
>>> atvm = getToolByName(self.portal, 'portal_vocabularies')
>>> from Products.ATVocabularyManager.utils.vocabs import createHierarchicalVocabs

>>> dictionary = {
...     ('electrical', 'electrical engines'): {},
...     ('mechanical', 'mechanial engines'): {
...        ('otto', 'ottomotor'): {},
...        ('diesel', 'diesel engine'): {}
...     }
... }
>>> hierarchicalVocabs = {}
>>> hierarchicalVocabs[('enginetypes', 'Diffetent types of engines')] = dictionary
>>> createHierarchicalVocabs(atvm, hierarchicalVocabs)

now we’ve got our vocabulary available:

>>> engines = atvm.getVocabularyByName('enginetypes')
>>> engines
<TreeVocabulary at /plone/portal_vocabularies/enginetypes>

Background Information

This small section tells you how ATMV achieves that an ottomotor is recognized as a mechanical engine too.

The key of a vocabulary term is the UID of the canonical object (in case the vocabulary is translated into different languages) and can be obtained using the method getTermKey

>>> ottomotor = engines.mechanical.otto
>>> ottomotor.getTermKey() == ottomotor.UID()
True

A TreeVocabularyTerm implements the method getTermKeyPath that returns a list of vocabularyterm-keys of the term itself and all the terms in the hierarchy above it.

>>> ottoPath = ottomotor.getTermKeyPath()
>>> engines.mechanical.getTermKey() in ottoPath
True

Each VocabularyTerm is indexed using a KeywordIndex on getTermKeyPath in uid_catalog.

The keypath is is available as a metadata-column in uid_catalog, so it can be fetched for via a catalog query without invoking real objects.

our ottomotor term can be found searching for the ottomotor uid:

>>> uid_catalog = getToolByName(self.portal, 'uid_catalog')
>>> uid_catalog(getTermKeyPath=ottomotor.UID())[0].getObject()
<TreeVocabularyTerm at .../enginetypes/mechanical/otto>

a search for a machanical engine returns 3 vocabularies: the ottomotor, the dieselengine and the term for mechanicalengine too:

>>> result = uid_catalog(getTermKeyPath=engines.mechanical.UID())
>>> [brain.id for brain in result]
['mechanical', 'diesel', 'otto']

a change in the hierarchy of our vocabulary will reindex all vocabularies below the changed term.

XXX make new term below engines: ‘fossil’’ and move mechanical there. after that the otto and diesel should be available under fossil too

Make your content type searchable

To be able to search for content types assiciated to terms not only directly but also down the hierarchy we need to index our object under the TermKeyPath(s) of the associated vocabularyterm(s).

ATVM provides a fast implemented utility method for this purpose that is working only on the catalog.

You can utilize getKeyPathForTerms in NamedVocabulary:

>>> from Products.ATVocabularyManager import NamedVocabulary
>>> nv = NamedVocabulary('engines')
>>> keyPath = nv.getKeyPathForTerms(self.portal, ottomotor.getTermKey())
>>> engines.mechanical.getTermKey() in keyPath
True
>>> ottomotor.getTermKey() in keyPath
True

All you need to do is to implement a method that uses the getKeyPathForTerms method and define it as index method:

# .. somewhere in the schemadefinition

ReferenceField(
  name='myfield',
  widget=...
  vocabulary=NamedVocabulary("""myvocab"""),
  index="KeywordIndex",
  index_method="someIndexMethod"
),

# .. somewhere in your content type definition

define someIndexMethod(self):
  """used to index myfield
  """
  # all we have to know is the field name of the field this
  # index method belongs to
  fieldName = 'myfield'

  # the following code need not be touched
  termUID = self.getField(fieldName).get(self)
  return self.getField(fieldName).vocabulary.getKeyPathForTerms(self, termUID)

See ATVocabularyManagerExample for a more detailed explanation how to use ATVocabularyManager in your content type.

_ATVocabularyManagerExample: https://svn.plone.org/svn/archetypes/ATVocabularyManagerExample/

Changelog

1.6.1 (2011-05-02)

  • Avoid deprecation warnings [WouterVH]

  • Update metadata-files and folder-structure to current conventions. [WouterVH]

  • Add z3c.autoinclude entry point for automatic ZCML loading in Plone 3.3+. [WouterVH]

  • Add MANIFEST.in to include top-level *.txt-files. [WouterVH]

1.6 - 2011-02-22

  • Don’t fetch version and shortdesc from metadata.xml - this is wrong. [jensens]

1.6.0a2 - 2011-02-18

  • Changed method of dermining if LinguaPlone is installed. Asking QI does not work in Plone4. [naro]

  • Added some missing dependencies, and a new extra section to setup.py. This helps to get a simple buildout cfg for testing against plone4. [do3cc]

  • Added some sleeps between installs in the tests, else genericsetup creates ids twice, and fails. [do3cc]

  • Temporary deactivated two checks in the tests. With Products.CMFQuickInstaller>=3.0.3, the portal_vocabularies don’t get deleted during uninstall. [do3cc]

  • Restructured test setup, since call semantics changed. [do3cc]

1.6.0a1 - 2010-07-16

  • Fixed deprecated API call to registerType without Package Name. Works now for Plone4. [thet]

  • Added Products.LinguaPlone as test dependency to extras_require and begun fixing tests. [thet]

  • Removed old slow VDEX implementation; added stub for import/export of vocabs with generixsetup; added support for vdex import with genericsetup. [jensens]

1.4.2 - 2008-06-04

before 1.4

  • Workaround for #46 Upload from Windows/IE creates wrong ID. [jensens]

  • Made new VdexFileVocabulary as default for upload form. [jensens]

  • Add alternative vdex vocabulary, much faster. [jensens]

  • Added convinience method to vocabulary library (tool) which returns a dictionary with key=vocbaulary id and value=title of vocabulary (language aware. [jensens]

  • Added convinience function to make it easier to fetch a value from a vocabulary dictionary (utils.vocabs.fetchValueByKeyFromVocabularyDict). [jensens]

  • Adding css handles to various elements of portal_vocabulary user interface, so that if desired, one can easily dumb down the user interface for less technical clients with css, but not override the template [andrewburkhalter]

  • Language negotation to make sure the cache is language aware. [jensens]

  • In types/simple/vocabulary.py, applied patch of issue #4 (sorting of simple vocabularies) [fRiSi]

  • In doc/simplevocabulary.txt, add doctest testing the sorting of simplevocabularies sorting might not yet work with translated vocabularies. I added a section to test this, but did not yet write the tests there. [fRiSi]

  • event.py, configure.zcml, interfaces.py, types/simple/term.py. [rocky]

  • “renamed events” are now fired when the title of any term is updated. [rocky]

  • “deleted events” are now fired when a term is deleted from a vocabulary; this particular feature will only work when running on Zope 2.9 or higher (note: this doesn’t make ATVM dependent on Zope 2.9) [rocky]

  • types/tree/vocabulary.py, types/simple/vocabulary.py replaced if not instance is None: with a try-except. if you are on a type that is not linguaplone enabled, and using a vocabulary you got an error when creating an object of this type because getLanguage raised an AttributeError. [fRiSi]

  • types/tree/term.py to fix http://plone.org/products/atvocabularymanager/issues/26 in the meantime i solved the problem with the “hack solution” to take away the IVocabulary interface from the term that is extending the vocabulary. [fRiSi] (see the issue above for more information.)

  • Makefile added a new target clean that removes stale pyc files and -~ backup files [fRiSi]

  • namedvocabulary.py used to not add the keypath of a term if the search for a term uid in uid-catalog returned more than one value. proceeding and just taking the first result is better than silently ignoring the path for this item. [fRiSi]

  • In Extensions/Install.py, did a cleaner check for BadRequest Error in case the tool already existed at install time (also added basic tests for installing/uninstalling/and reinstalling the product. [fRiSi]

  • In Extensions/Install.py, self.portal.portal_properties.navtree_properties.metaTypesNotToList is a tuple by default (on a freshly created plonesite) ATVM creates a list at installation time and removes the tool from this list at uninstallation. If other products get installed between installing and uninstalling, they possibly change metaTypesNotToList back to a tuple again. changed the installer and uninstaller to operate on and return a tuple here. (should fix http://plone.org/products/atvocabularymanager/issues/10) [fRiSi]

  • tests/testATVocabularyManager.py tests for the installation and uninstallation of the product. agreed with jens that we could do a zexp of the vocabularies at uninstallation in case this was done by accident. (http://plone.org/products/atvocabularymanager/issues/22/) [fRiSi]

  • types/simple/term.py: removed bug, which stops me from adding new terms. [jensens]

  • In types/simple/term.py, SimpleVocabularyTerms now return the canoncial’s id as key. This differs from the current behaviour but is essential for translated vocabularies. [fRiSi]

  • In types/simple/term.py,getTermKeyPath now returns a list containing the key of the term, not it’s uid. This is consistent with the interface. [fRiSi]

  • types/simple/vocabulary.py, types/tree/vocabulary.py GetVocabularyDict now uses instance.getLanguage in favour of portal_lanuages.getPreferredLanguage (otherwhise passing instance would be useless in most cases)

  • tests/testSimpleVocabulary.py: Added a test that checks if translations return the same key and if dictionaries are translated.

  • backports.py

    • Improved the docstrings to be more explicit.

    • getTermKey now states that all translations of a term have the same key.

    • getTermValue: now has an optional lang parameter making --kwargs needless.

  • Add sorted simple vocabulary [gotcha]

  • Run i18ndude and update fr and nl [gotcha]

  • Merged branch supporting catalog searches on hierarchical vocabularies to trunk (https://svn.plone.org/svn/archetypes/ATVocabularyManager/branches/treevocab-frisi) [fRiSi]

  • Merged current trunk into this branch. [fRiSi]

  • Utils/vocabs.py utility methods for batch creating vocabularies.[fRiSi]

  • types/simple/term.py types/tree/term.py Marked methods getVocabularyValue and getVocabularyKey as deprecated and used those defined in IVocabularyTerm to make terms implement their interface. [fRiSi]

  • types/tree/vocabulary.py types/simple/vocabulary.py Improved linguaplone support and also added a testcase. [fRiSi]

  • tests/testTreeVocabulary.py Testing if treevocabulary is linguaplone aware. [fRiSi]

  • doc/search_treevocabulary.txt: A doctest explaining how ATVM performs searches on hierarchical vocabularies, and how to enable them for your content type. [fRiSi]

  • Added dutch translations, thanks to Atopia. [jladage]

  • AllowedTypesByIface is broken, it seems - adding manual override to fix this in ATVM. [optilude]

  • Removed workflows from ATVM types on installation. [ferri]

  • Added brazilian portuguese translation and portal_vocabularies title i18n support. [ferri]

  • Added i18n and Czech translation. [naro]

  • Uncommented code in Install.py to make portal_vocabularylib not show in the navigation tree. [panjunyong]

  • Use StringWidget instead of IDWidget for Simple vocaublary term. [panjunyong]

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

Products.ATVocabularyManager-1.6.1.tar.gz (78.4 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page