Content Term Extraction using POS Tagging
Project description
This package determines important terms within a given piece of content. It uses linguistic tools such as Parts-Of-Speech (POS) and some simple statistical analysis to determine the terms and their strength.
Detailed Documentation
Term Extraction
This package implements text term extraction by making use of a simple Parts-Of-Speech (POS) tagging algorithm.
http://bioie.ldc.upenn.edu/wiki/index.php/Part-of-Speech
The POS Tagger
POS Taggers use a lexicon to mark words with a tag. A list of available tags can be found at:
http://bioie.ldc.upenn.edu/wiki/index.php/POS_tags
Since words can have multiple tags, the determination of the correct tag is not always simple. This implementation, however, does not try to infer linguistic use and simply chooses the first tag in the lexicon.
>>> from topia.termextract import tag >>> tagger = tag.Tagger() >>> tagger <Tagger for english>
To get the tagger ready for its work, we need to initialize it. In this implementation the lexicon is loaded.
>>> tagger.initialize()
Now we are ready to rock and roll.
Tokenizing
The first step of tagging is to tokenize the text into terms.
>>> tagger.tokenize('This is a simple example.')
['This', 'is', 'a', 'simple', 'example', '.']
While most tokenizers ignore punctuation, it is important for us to keep it, since we need it later for the term extraction. Let’s now look at some more complex cases:
Quoted Text
>>> tagger.tokenize('This is a "simple" example.') ['This', 'is', 'a', '"', 'simple', '"', 'example', '.']>>> tagger.tokenize('"This is a simple example."') ['"', 'This', 'is', 'a', 'simple', 'example', '."']Non-letters within words.
>>> tagger.tokenize('Parts-Of-Speech') ['Parts-Of-Speech']>>> tagger.tokenize('amazon.com') ['amazon.com']>>> tagger.tokenize('Go to amazon.com.') ['Go', 'to', 'amazon.com', '.']Various punctuation.
>>> tagger.tokenize('Quick, go to amazon.com.') ['Quick', ',', 'go', 'to', 'amazon.com', '.']>>> tagger.tokenize('Live free; or die?') ['Live', 'free', ';', 'or', 'die', '?']Tolerance to incorrect punctuation.
>>> tagger.tokenize('Hi , I am here.') ['Hi', ',', 'I', 'am', 'here', '.']Possessive structures.
>>> tagger.tokenize("my parents' car") ['my', 'parents', "'", 'car'] >>> tagger.tokenize("my father's car") ['my', 'father', "'s", 'car']Numbers.
>>> tagger.tokenize("12.4") ['12.4'] >>> tagger.tokenize("-12.4") ['-12.4'] >>> tagger.tokenize("$12.40") ['$12.40']Dates.
>>> tagger.tokenize("10/3/2009") ['10/3/2009'] >>> tagger.tokenize("3.10.2009") ['3.10.2009']
Okay, that’s it.
Tagging
The next step is tagging. Tagging is done in two phases. During the first phase terms are assigned a tag by looking at the lexicon and the normalized form is set to the term itself. In the second phase, a set of rules is applied to each tagged term and the tagging and normalization is tweaked.
>>> tagger('This is a simple example.')
[['This', 'DT', 'This'],
['is', 'VBZ', 'is'],
['a', 'DT', 'a'],
['simple', 'JJ', 'simple'],
['example', 'NN', 'example'],
['.', '.', '.']]
So wow, this determination was dead on. Let’s try a plural form noun and see what happens:
>>> tagger('These are simple examples.')
[['These', 'DT', 'These'],
['are', 'VBP', 'are'],
['simple', 'JJ', 'simple'],
['examples', 'NNS', 'example'],
['.', '.', '.']]
So far so good. Let’s test a few more cases:
>>> tagger("The fox's tail is red.") [['The', 'DT', 'The'], ['fox', 'NN', 'fox'], ["'s", 'POS', "'s"], ['tail', 'NN', 'tail'], ['is', 'VBZ', 'is'], ['red', 'JJ', 'red'], ['.', '.', '.']]>>> tagger("The fox can't really jump over the fox's tail.") [['The', 'DT', 'The'], ['fox', 'NN', 'fox'], ['can', 'MD', 'can'], ["'t", 'RB', "'t"], ['really', 'RB', 'really'], ['jump', 'VB', 'jump'], ['over', 'IN', 'over'], ['the', 'DT', 'the'], ['fox', 'NN', 'fox'], ["'s", 'POS', "'s"], ['tail', 'NN', 'tail'], ['.', '.', '.']]
Rules
Correct Default Noun Tag
>>> tagger('Ikea') [['Ikea', 'NN', 'Ikea']] >>> tagger('Ikeas') [['Ikeas', 'NNS', 'Ikea']]Verify proper nouns at beginning of sentence.
>>> tagger('. Police') [['.', '.', '.'], ['police', 'NN', 'police']] >>> tagger('Police') [['police', 'NN', 'police']] >>> tagger('. Stephan') [['.', '.', '.'], ['Stephan', 'NNP', 'Stephan']]Determine Verb after Modal Verb
>>> tagger('The fox can jump') [['The', 'DT', 'The'], ['fox', 'NN', 'fox'], ['can', 'MD', 'can'], ['jump', 'VB', 'jump']] >>> tagger("The fox can't jump") [['The', 'DT', 'The'], ['fox', 'NN', 'fox'], ['can', 'MD', 'can'], ["'t", 'RB', "'t"], ['jump', 'VB', 'jump']] >>> tagger('The fox can really jump') [['The', 'DT', 'The'], ['fox', 'NN', 'fox'], ['can', 'MD', 'can'], ['really', 'RB', 'really'], ['jump', 'VB', 'jump']]Normalize Plural Forms
>>> tagger('examples') [['examples', 'NNS', 'example']] >>> tagger('stresses') [['stresses', 'NNS', 'stress']] >>> tagger('cherries') [['cherries', 'NNS', 'cherry']]Some cases that do not work:
>>> tagger('men') [['men', 'NNS', 'men']] >>> tagger('feet') [['feet', 'NNS', 'feet']]
Term Extraction
Now that we can tag a text, let’s have a look at the term extractions.
>>> from topia.termextract import extract >>> extractor = extract.TermExtractor() >>> extractor <TermExtractor using <Tagger for english>>
As you can see, the extractor maintains a tagger:
>>> extractor.tagger <Tagger for english>
When creating an extractor, you can also pass in a tagger to avoid frequent tagger initialization:
>>> extractor = extract.TermExtractor(tagger) >>> extractor.tagger is tagger True
Let’s get the terms for a simple text.
>>> extractor("The fox can't jump over the fox's tail.")
[]
We got no terms. That’s because by default at least 3 occurences of a term must be detected, if the term consists of a single word.
The extractor maintains a filter component. Let’s register the trivial permissive filter, which simply return everything that the extractor suggests:
>>> extractor.filter = extract.permissiveFilter
>>> extractor("The fox can't jump over the fox's tail.")
[('tail', 1, 1), ('fox', 2, 1)]
But let’s look at the default filter again, since it allows tweaking its parameters:
>>> extractor.filter = extract.DefaultFilter(singleStrengthMinOccur=2)
>>> extractor("The fox can't jump over the fox's tail.")
[('fox', 2, 1)]
Let’s now have a look at multi-word terms. Oftentimes multi-word nouns and proper names occur only once or twice in a text. But they are often great terms! To handle this scenario, the concept of “strength” was introduced. Currently the strength is simply the amount of words in the term. By default, all terms with a strength larger than 1 are selected regardless of the number of occurances.
>>> extractor('The German consul of Boston resides in Newton.')
[('German consul', 1, 2)]
An Exmaple - A News Article
This document provides a simple example of extracting the terms of a BBC article from May 29, 2009. We will use several term extraction tools to compare the outcome.
>>> text =''' ... Police shut Palestinian theatre in Jerusalem. ... ... Israeli police have shut down a Palestinian theatre in East Jerusalem. ... ... The action, on Thursday, prevented the closing event of an international ... literature festival from taking place. ... ... Police said they were acting on a court order, issued after intelligence ... indicated that the Palestinian Authority was involved in the event. ... ... Israel has occupied East Jerusalem since 1967 and has annexed the ... area. This is not recognised by the international community. ... ... The British consul-general in Jerusalem , Richard Makepeace, was ... attending the event. ... ... "I think all lovers of literature would regard this as a very ... regrettable moment and regrettable decision," he added. ... ... Mr Makepeace said the festival's closing event would be reorganised to ... take place at the British Council in Jerusalem. ... ... The Israeli authorities often take action against events in East ... Jerusalem they see as connected to the Palestinian Authority. ... ... Saturday's opening event at the same theatre was also shut down. ... ... A police notice said the closure was on the orders of Israel's internal ... security minister on the grounds of a breach of interim peace accords ... from the 1990s. ... ... These laid the framework for talks on establishing a Palestinian state ... alongside Israel, but left the status of Jerusalem to be determined by ... further negotiation. ... ... Israel has annexed East Jerusalem and declares it part of its eternal ... capital. ... ... Palestinians hope to establish their capital in the area. ... '''
Yahoo Keyword Extractor
Yahoo provides a service that extracts terms from a piece of content using its immense search database.
http://developer.yahoo.com/search/content/V1/termExtraction.html
As you can see, the result is excellent:
<ResultSet> <Result>british consul general</Result> <Result>east jerusalem</Result> <Result>literature festival</Result> <Result>richard makepeace</Result> <Result>international literature</Result> <Result>israeli authorities</Result> <Result>eternal capital</Result> <Result>peace accords</Result> <Result>security minister</Result> <Result>israeli police</Result> <Result>internal security</Result> <Result>palestinian state</Result> <Result>palestinian authority</Result> <Result>british council</Result> <Result>palestinians</Result> <Result>negotiation</Result> <Result>breach</Result> <Result>1990s</Result> <Result>closure</Result> <Result>israel</Result> </ResultSet>
Unfortunately, the service allows only 5000 requests per 24 hours. Also, there is no strength indicator on the terms.
TreeTagger
A POS tagger that uses some linguistics to tag a text. Here is its output:
Police NNS Police shut VVD shut Palestinian JJ Palestinian theatre NN theatre in IN in Jerusalem NP Jerusalem . SENT . Israeli JJ Israeli police NNS police have VHP have shut VVN shut down RP down a DT a Palestinian JJ Palestinian theatre NN theatre in IN in East NP East Jerusalem NP Jerusalem . SENT . The DT the action NN action , , , on IN on Thursday NP Thursday , , , prevented VVD prevent the DT the closing NN closing event NN event of IN of an DT an international JJ international literature NN literature festival NN festival from IN from taking VVG take place NN place . SENT . Police NNS Police said VVD say they PP they were VBD be acting VVG act on IN on a DT a court NN court order NN order , , , issued VVN issue after IN after intelligence NN intelligence indicated VVN indicate that IN that the DT the Palestinian NP Palestinian Authority NP Authority was VBD be involved VVN involve in IN in the DT the event NN event . SENT . Israel NP Israel has VHZ have occupied VVN occupy East NP East Jerusalem NP Jerusalem since IN since 1967 CD @card@ and CC and has VHZ have annexed VVN annex the DT the area NN area . SENT . This DT this is VBZ be not RB not recognised VVN recognise by IN by the DT the international JJ international community NN community . SENT . The DT the British JJ British consul-general NN <unknown> in IN in Jerusalem NP Jerusalem , , , Richard NP Richard Makepeace NP Makepeace , , , was VBD be attending VVG attend the DT the event NN event . SENT . " `` " I PP I think VVP think all DT all lovers NNS lover of IN of literature NN literature would MD would regard VV regard this DT this as IN as a DT a very RB very regrettable JJ regrettable moment NN moment and CC and regrettable JJ regrettable decision NN decision , , , " '' " he PP he added VVD add . SENT . Mr NP Mr Makepeace NP Makepeace said VVD say the DT the festival NN festival 's POS 's closing NN closing event NN event would MD would be VB be reorganised VVN <unknown> to TO to take VV take place NN place at IN at the DT the British NP British Council NP Council in IN in Jerusalem NP Jerusalem . SENT . The DT the Israeli JJ Israeli authorities NNS authority often RB often take VVP take action NN action against IN against events NNS event in IN in East NP East Jerusalem NP Jerusalem they PP they see VVP see as RB as connected VVN connect to TO to the DT the Palestinian JJ Palestinian Authority NP Authority . SENT . Saturday NP Saturday 's POS 's opening NN opening event NN event at IN at the DT the same JJ same theatre NN theatre was VBD be also RB also shut VVN shut down RP down . SENT . A DT a police NN police notice NN notice said VVD say the DT the closure NN closure was VBD be on IN on the DT the orders NNS order of IN of Israel NP Israel 's POS 's internal JJ internal security NN security minister NN minister on IN on the DT the grounds NNS ground of IN of a DT a breach NN breach of IN of interim JJ interim peace NN peace accords NNS accord from IN from the DT the 1990s NNS 1990s . SENT . These DT these laid VVD lay the DT the framework NN framework for IN for talks NNS talk on IN on establishing VVG establish a DT a Palestinian JJ Palestinian state NN state alongside IN alongside Israel NP Israel , , , but CC but left VVD leave the DT the status NN status of IN of Jerusalem NP Jerusalem to TO to be VB be determined VVN determine by IN by further JJR further negotiation NN negotiation . SENT . Israel NP Israel has VHZ have annexed VVN annex East NP East Jerusalem NP Jerusalem and CC and declares VVZ declare it PP it part NN part of IN of its PP$ its eternal JJ eternal capital NN capital . SENT . Palestinians NPS Palestinians hope VVP hope to TO to establish VV establish their PP$ their capital NN capital in IN in the DT the area NN area . SENT .
As you can see, the identification of TreeTagger is pretty good, but the output would need some analysis to produce a useful set of terms. Furthermore, TreeTagger is not free for commercial use.
Topia’s Term Extractor
Topia’s Term Extractor tries to produce results somewhere between a POS tagger like TreeTagger and Yahoo Keyword Extraction.
Since we are only interested in nouns, a very simple POS tagging algorithm can be deployed, which will provide good results most of the time. We then use some simple statistics and linguistics to produce a narrow but strong list of terms for the content.
>>> from topia.termextract import extract >>> extractor = extract.TermExtractor()
Let’s look at the result of the tagger first:
>>> printTaggedTerms(extractor.tagger(text)) #doctest: +REPORT_NDIFF police NN police shut VBN shut Palestinian JJ Palestinian theatre NN theatre in IN in Jerusalem NNP Jerusalem . . . Israeli JJ Israeli police NN police have VBP have shut VBN shut down RB down a DT a Palestinian JJ Palestinian theatre NN theatre in IN in East NNP East Jerusalem NNP Jerusalem . . . The DT The action NN action , , , on IN on Thursday NNP Thursday , , , prevented VBN prevented the DT the closing VBG closing event NN event of IN of an DT an international JJ international literature NN literature festival NN festival from IN from taking VBG taking place NN place . . . police NN police said VBD said they PRP they were VBD were acting VBG acting on IN on a DT a court NN court order NN order , , , issued VBN issued after IN after intelligence NN intelligence indicated VBD indicated that IN that the DT the Palestinian JJ Palestinian Authority NNP Authority was VBD was involved VBN involved in IN in the DT the event NN event . . . Israel NNP Israel has VBZ has occupied VBN occupied East NNP East Jerusalem NNP Jerusalem since IN since 1967 NN 1967 and CC and has VBZ has annexed VBD annexed the DT the area NN area . . . This DT This is VBZ is not RB not recognised VBD recognised by IN by the DT the international JJ international community NN community . . . The DT The British JJ British consul-general NN consul-general in IN in Jerusalem NNP Jerusalem , , , Richard NNP Richard Makepeace NNP Makepeace , , , was VBD was attending VBG attending the DT the event NN event . . . " " " I PRP I think VBP think all DT all lovers NNS lover of IN of literature NN literature would MD would regard VB regard this DT this as IN as a DT a very RB very regrettable JJ regrettable moment NN moment and CC and regrettable JJ regrettable decision NN decision ," , ," he PRP he added VBD added . . . Mr NNP Mr Makepeace NNP Makepeace said VBD said the DT the festival NN festival 's POS 's closing VBG closing event NN event would MD would be VB be reorganised NN reorganised to TO to take VB take place NN place at IN at the DT the British JJ British Council NNP Council in IN in Jerusalem NNP Jerusalem . . . The DT The Israeli JJ Israeli authorities NNS authority often RB often take VB take action NN action against IN against events NNS event in IN in East NNP East Jerusalem NNP Jerusalem they PRP they see VB see as IN as connected VBN connected to TO to the DT the Palestinian JJ Palestinian Authority NNP Authority . . . Saturday NNP Saturday 's POS 's opening NN opening event NN event at IN at the DT the same JJ same theatre NN theatre was VBD was also RB also shut VBN shut down RB down . . . A DT A police NN police notice NN notice said VBD said the DT the closure NN closure was VBD was on IN on the DT the orders NNS order of IN of Israel NNP Israel 's POS 's internal JJ internal security NN security minister NN minister on IN on the DT the grounds NNS ground of IN of a DT a breach NN breach of IN of interim JJ interim peace NN peace accords NNS accord from IN from the DT the 1990 NN 1990 s PRP s . . . These DT These laid VBN laid the DT the framework NN framework for IN for talks NNS talk on IN on establishing VBG establishing a DT a Palestinian JJ Palestinian state NN state alongside IN alongside Israel NNP Israel , , , but CC but left VBN left the DT the status NN status of IN of Jerusalem NNP Jerusalem to TO to be VB be determined VBN determined by IN by further JJ further negotiation NN negotiation . . . Israel NNP Israel has VBZ has annexed VBD annexed East NNP East Jerusalem NNP Jerusalem and CC and declares VBZ declares it PRP it part NN part of IN of its PRP$ its eternal JJ eternal capital NN capital . . . Palestinians NNPS Palestinian hope NN hope to TO to establish VB establish their PRP$ their capital NN capital in IN in the DT the area NN area . . .
Let’s now apply the extractor.
>>> sorted(extractor(text))
[('British Council', 1, 2),
('British consul-general', 1, 2),
('East', 4, 1),
('East Jerusalem', 4, 2),
('Israel', 4, 1),
('Israeli authorities', 1, 2),
('Israeli police', 1, 2),
('Jerusalem', 8, 1),
('Mr Makepeace', 1, 2),
('Palestinian', 6, 1),
('Palestinian Authority', 2, 2),
('Palestinian state', 1, 2),
('Palestinian theatre', 2, 2),
('Palestinians hope', 1, 2),
('Richard Makepeace', 1, 2),
('court order', 1, 2),
('event', 6, 1),
('literature festival', 1, 2),
('opening event', 1, 2),
('peace accords', 1, 2),
('police', 4, 1),
('police notice', 1, 2),
('security minister', 1, 2),
('theatre', 3, 1)]
CHANGES
1.1.0 (2009-06-29)
Improved the dictionary a little bit to improve real scenarios.
1.0.0 (2009-05-30)
Initial Release
Part-Of-Speech Text Tagging using existing lexicon ans very simplisitc linguistic rules.
Term Extraction based on occurances and term strength.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
