Python NLTK pos_tag不返回正确的词性标记

拥有这个:

text = word_tokenize("The quick brown fox jumps over the lazy dog") 

并运行:

 nltk.pos_tag(text) 

我得到:

 [('The', 'DT'), ('quick', 'NN'), ('brown', 'NN'), ('fox', 'NN'), ('jumps', 'NNS'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'NN'), ('dog', 'NN')] 

这是不正确的。 句子中quick brown lazy的标签应该是:

 ('quick', 'JJ'), ('brown', 'JJ') , ('lazy', 'JJ') 

通过他们的在线工具进行testing,结果相同。 quickbrownfox应该是形容词不是名词。

总之

NLTK并不完美。 事实上,没有一个模型是完美的。

注意:

从NLTK版本3.1开始,默认的pos_tag函数不再是旧的MaxEnt英文pickle 。

现在是@ Honnibal实现的感知器标记器 ,请参阅nltk.tag.pos_tag

 >>> import inspect >>> print inspect.getsource(pos_tag) def pos_tag(tokens, tagset=None): tagger = PerceptronTagger() return _pos_tag(tokens, tagset, tagger) 

还是更好,但不完美:

 >>> from nltk import pos_tag >>> pos_tag("The quick brown fox jumps over the lazy dog".split()) [('The', 'DT'), ('quick', 'JJ'), ('brown', 'NN'), ('fox', 'NN'), ('jumps', 'VBZ'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'JJ'), ('dog', 'NN')] 

在某些时候,如果有人想要TL;DR解决scheme,请参阅https://github.com/alvations/nltk_cli


长久以来

尝试使用其他标记(请参阅https://github.com/nltk/nltk/tree/develop/nltk/tag ),例如

  • HunPos
  • 斯坦福大学POS
  • 塞纳

使用来自NLTK的默认MaxEnt POS标记器,即nltk.pos_tag

 >>> from nltk import word_tokenize, pos_tag >>> text = "The quick brown fox jumps over the lazy dog" >>> pos_tag(word_tokenize(text)) [('The', 'DT'), ('quick', 'NN'), ('brown', 'NN'), ('fox', 'NN'), ('jumps', 'NNS'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'NN'), ('dog', 'NN')] 

使用斯坦福POS机

 $ cd ~ $ wget http://nlp.stanford.edu/software/stanford-postagger-2015-04-20.zip $ unzip stanford-postagger-2015-04-20.zip $ mv stanford-postagger-2015-04-20 stanford-postagger $ python >>> from os.path import expanduser >>> home = expanduser("~") >>> from nltk.tag.stanford import POSTagger >>> _path_to_model = home + '/stanford-postagger/models/english-bidirectional-distsim.tagger' >>> _path_to_jar = home + '/stanford-postagger/stanford-postagger.jar' >>> st = POSTagger(path_to_model=_path_to_model, path_to_jar=_path_to_jar) >>> text = "The quick brown fox jumps over the lazy dog" >>> st.tag(text.split()) [(u'The', u'DT'), (u'quick', u'JJ'), (u'brown', u'JJ'), (u'fox', u'NN'), (u'jumps', u'VBZ'), (u'over', u'IN'), (u'the', u'DT'), (u'lazy', u'JJ'), (u'dog', u'NN')] 

使用HunPOS (注意:默认编码是ISO-8859-1而不是UTF8):

 $ cd ~ $ wget https://hunpos.googlecode.com/files/hunpos-1.0-linux.tgz $ tar zxvf hunpos-1.0-linux.tgz $ wget https://hunpos.googlecode.com/files/en_wsj.model.gz $ gzip -d en_wsj.model.gz $ mv en_wsj.model hunpos-1.0-linux/ $ python >>> from os.path import expanduser >>> home = expanduser("~") >>> from nltk.tag.hunpos import HunposTagger >>> _path_to_bin = home + '/hunpos-1.0-linux/hunpos-tag' >>> _path_to_model = home + '/hunpos-1.0-linux/en_wsj.model' >>> ht = HunposTagger(path_to_model=_path_to_model, path_to_bin=_path_to_bin) >>> text = "The quick brown fox jumps over the lazy dog" >>> ht.tag(text.split()) [('The', 'DT'), ('quick', 'JJ'), ('brown', 'JJ'), ('fox', 'NN'), ('jumps', 'NNS'), ('over', 'IN'), ('the', 'DT'), ('lazy', 'JJ'), ('dog', 'NN')] 

使用Senna (确保你有最新版本的NLTK,API有一些变化):

 $ cd ~ $ wget http://ronan.collobert.com/senna/senna-v3.0.tgz $ tar zxvf senna-v3.0.tgz $ python >>> from os.path import expanduser >>> home = expanduser("~") >>> from nltk.tag.senna import SennaTagger >>> st = SennaTagger(home+'/senna') >>> text = "The quick brown fox jumps over the lazy dog" >>> st.tag(text.split()) [('The', u'DT'), ('quick', u'JJ'), ('brown', u'JJ'), ('fox', u'NN'), ('jumps', u'VBZ'), ('over', u'IN'), ('the', u'DT'), ('lazy', u'JJ'), ('dog', u'NN')] 

或者尝试build立一个更好的POS标签

  • Ngram Tagger: http ://streamhacker.com/2008/11/03/part-of-speech-tagging-with-nltk-part-1/
  • 词组/正则expression式
  • build立你自己的布里尔(阅读代码这是一个非常有趣的标记, http: //www.nltk.org/_modules/nltk/tag/brill.html),请参阅http://streamhacker.com/2008/12/03/部分的语音标记与- NLTK部分-3 /
  • Perceptron Tagger: https : //honnibal.wordpress.com/2013/09/11/a-good-part-of-speechpos-tagger-in-about-200-lines-of-python/
  • LDA Tagger: http : //scm.io/blog/hack/2015/02/lda-intentions/

在stackoverflow上抱怨pos_tag准确性包括

  • POS标签 – NLTK认为名词是形容词
  • python NLTK POS tagger没有按预期运行
  • 如何使用NLTK pos标签获得更好的结果
  • NLTK中的pos_tag不会正确标记语句

有关NLTK HunPos的问题包括

  • 如何在nltk中用hunpos标记文本文件?
  • 有谁知道如何在nltk上configurationhunpos包装类?

NLTK和斯坦福POS机的问题包括

  • 将stanford pos tagger导入nltk时遇到麻烦
  • Java命令在NLTK Stanford POS Tagger中失败
  • 在NLTK Python中使用Stanford POS Tagger时出错
  • 如何提高斯坦福大学的NLP Tagger和NLTK的速度
  • Nltk stanford pos tagger错误:Java命令失败
  • 在NLTK中实例化和使用StanfordTagger
  • 在NLTK中运行Stanford POS标记导致在Windows上“不是有效的Win32应用程序”