Skip to main content

Invenio module for record classification.

Project description

https://img.shields.io/travis/inveniosoftware/invenio-classifier.svg https://img.shields.io/coveralls/inveniosoftware/invenio-classifier.svg https://img.shields.io/github/tag/inveniosoftware/invenio-classifier.svg https://img.shields.io/pypi/dm/invenio-classifier.svg https://img.shields.io/github/license/inveniosoftware/invenio-classifier.svg

Invenio module for record classification.

This is an experimental development preview release.

Features

Classifier automatically extracts keywords from fulltext documents. The automatic assignment of keywords to textual documents has clear benefits in the digital library environment as it aids catalogization, classification and retrieval of documents.

Keyword extraction is simple

 Dependencies. Classifier requires Python RDFLib in order to process the RDF/SKOS taxonomy.

In order to extract relevant keywords from a document fulltext.pdf based on a controlled vocabulary thesaurus.rdf, you would run Classifier as follows:

$ inveniomanage classifier extract -k thesaurus.rdf -f fulltext.pdf

Launching inveniomanage classifier --help shows the options available.

As an example, running classifier on document nucl-th/0204033 using the high-energy physics RDF/SKOS taxonomy (HEP.rdf) would yield the following results (based on the HEP taxonomy from October 10th 2008):

Input file: 0204033.pdf

Author keywords:
Dense matter
Saturation
Unstable nuclei

Composite keywords:
10  nucleus: stability [36, 14]
6  saturation: density [25, 31]
6  energy: symmetry [35, 11]
4  nucleon: density [13, 31]
3  energy: Coulomb [35, 3]
2  energy: density [35, 31]
2  nuclear matter: asymmetry [21, 2]
1  n: matter [54, 36]
1  n: density [54, 31]
1  n: mass [54, 16]

Single keywords:
61  K0
23  equation of state
12  slope
4  mass number
4  nuclide
3  nuclear model
3  mass formula
2  charge distribution
2  elastic scattering
2  binding energy

Thesaurus

Classifier performs an extraction of keywords based on the recurrence of specific terms, taken from a controlled vocabulary. A controlled vocabulary is a thesaurus of all the terms that are relevant in a specific context. When a context is defined by a discipline or branch of knowledge then the vocabulary is said to be a subject thesaurus. Various existing subject thesauri can be found here.

A subject thesaurus can be expressed in several different formats. Different institutions/disciplines have developed different ways of representing their vocabulary systems. The taxonomy used by classifier is expressed in RDF/SKOS. It allows not only to list keywords but to specify relations between the keywords and alternative ways to represent the same keyword.

<Concept rdf:about="http://cern.ch/thesauri/HEP.rdf#scalar">
 <composite rdf:resource="http://cern.ch/thesauri/HEP.rdf#Composite.fieldtheoryscalar"/>
 <prefLabel xml:lang="en">scalar</prefLabel>
 <note xml:lang="en">nostandalone</note>
</Concept>

<Concept rdf:about="http://cern.ch/thesauri/HEP.rdf#fieldtheory">
 <composite rdf:resource="http://cern.ch/thesauri/HEP.rdf#Composite.fieldtheoryscalar"/>
 <prefLabel xml:lang="en">field theory</prefLabel>
 <altLabel xml:lang="en">QFT</altLabel>
 <hiddenLabel xml:lang="en">/field theor\w*/</hiddenLabel>
 <note xml:lang="en">nostandalone</note>
</Concept>

<Concept rdf:about="http://cern.ch/thesauri/HEP.rdf#Composite.fieldtheoryscalar">
 <compositeOf rdf:resource="http://cern.ch/thesauri/HEP.rdf#scalar"/>
 <compositeOf rdf:resource="http://cern.ch/thesauri/HEP.rdf#fieldtheory"/>
 <prefLabel xml:lang="en">field theory: scalar</prefLabel>
 <altLabel xml:lang="en">scalar field</altLabel>
</Concept>

In RDF/SKOS, every keyword is wrapped around a concept which encapsulates the full semantics and hierarchical status of a term - including synonyms, alternative forms, broader concepts, notes and so on - rather than just a plain keyword.

The specification of the SKOS language and various manuals that aid the building of a semantic thesaurus can be found at the SKOS W3C website. Furthermore, Classifier can function on top of an extended version of SKOS, which includes special elements such as key chains, composite keywords and special annotations.

Keyword extraction

Classifier computes the keywords of a fulltext document based on the frequency of thesaurus terms in it. In other words, it calculates how many times a thesaurus keyword (and its alternative and hidden labels, defined in the taxonomy) appears in a text and it ranks the results. Unlike other similar systems, Classifier does not use any machine learning or AI methodologies - a just plain phrase matching using regular expressions: it exploits the conformation and richness of the thesaurus to produce accurate results. It is then clear that Classifier performs best on top of rich, well-structured, subject thesauri expressed in the RDF/SKOS language.

Happy hacking and thanks for flying Invenio-Classifier.

Invenio Development Team

Changes

Version 0.1.0 (release 2015-08-19)

  • Initial public release.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

invenio-classifier-0.1.0.tar.gz (86.2 kB view details)

Uploaded Source

Built Distribution

invenio_classifier-0.1.0-py2.py3-none-any.whl (71.1 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file invenio-classifier-0.1.0.tar.gz.

File metadata

File hashes

Hashes for invenio-classifier-0.1.0.tar.gz
Algorithm Hash digest
SHA256 a5b0fd9b3a4e30a23a5f01d86da934fe95c3d4809b7152e35ae6b3a2ec568753
MD5 b879ead8f38effa92cc93c0e70eed443
BLAKE2b-256 f73afdc0ebdc41a472ad1eac9a4619ec82ec5d51c3cb70d2fb84220e1ddc9bc4

See more details on using hashes here.

Provenance

File details

Details for the file invenio_classifier-0.1.0-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for invenio_classifier-0.1.0-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 1e7c460ea93d906b98923a72cdd0890ad72459b6ac7ae5d83b672d8f3387bebc
MD5 8ac83baf61c1a8febf6a750db39ee604
BLAKE2b-256 7051c5a93a8b3c823988706e8a841e39314628ec2f8e2786ddbfbb960f81b0d2

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page