python – 这是正确的tfidf吗?
内容导读
互联网集市收集整理的这篇技术教程文章主要介绍了python – 这是正确的tfidf吗?,小编现在分享给大家,供广大互联网技能从业者学习和参考。文章包含3247字,纯文字阅读大概需要5分钟。
内容图文
![python – 这是正确的tfidf吗?](/upload/InfoBanner/zyjiaocheng/789/7ad4999208584610a026e7b563cdbebb.jpg)
我试图从文件中获取tfidf.但我不认为它给了我正确的价值观,或者我可能做错了什么.请建议.代码和输出如下:
from sklearn.feature_extraction.text import TfidfVectorizer
books = ["Hello there this is first book to be read by wordcount script.", "This is second book to be read by wordcount script. It has some additionl information.", "just third book."]
vectorizer = TfidfVectorizer()
response = vectorizer.fit_transform(books)
feature_names = vectorizer.get_feature_names()
for col in response.nonzero()[1]:
print feature_names[col], '-', response[0, col]
更新1 :(由juanpa.arrivillaga建议)
vectorizer = TfidfVectorizer(smooth_idf=False)
输出:
script - 0.269290317245
wordcount - 0.269290317245
by - 0.269290317245
read - 0.269290317245
be - 0.269290317245
to - 0.269290317245
book - 0.209127954024
first - 0.354084405732
is - 0.269290317245
this - 0.269290317245
there - 0.354084405732
hello - 0.354084405732
information - 0.0
...
更新1后的输出:
script - 0.256536760895
wordcount - 0.256536760895
by - 0.256536760895
read - 0.256536760895
be - 0.256536760895
to - 0.256536760895
book - 0.182528018244
first - 0.383055542114
is - 0.256536760895
this - 0.256536760895
there - 0.383055542114
hello - 0.383055542114
information - 0.0
...
根据我的理解,tfidf是= tf * idf.我手动计算它的方式如下:
文档1:“你好,这是第一本由wordcount脚本读取的书.”
文档2:“这是wordcount脚本读取的第二本书.它有一些附加信息.”
文件3:“只是第三本书.”
Tfidf for Hello:
tf= 1/12(total terms in document 1)= 0.08333333333
idf= log(3(total documents)/1(no. of document with term in it))= 0.47712125472
0.08333333333*0.47712125472= 0.03976008865
这与下面不同(你好 – 0.354084405732).
更新1后手动计算:
tf = 1
idf= log(nd/df) +1 = log (3/1) +1= 0.47712125472 + 1= 1.47712
tfidf = tf*idf = 1* 1.47712= 1.47712
(与idf平滑后的代码输出“hello – 0.383055542114”不同)
任何有助于了解最新情况的人都非常感谢..
解决方法:
这是没有平滑或标准化的输出:
In [2]: from sklearn.feature_extraction.text import TfidfVectorizer
...: books = ["Hello there this is first book to be read by wordcount script.", "This is second book to be read by wordcount sc
...: ript. It has some additionl information.", "just third book."]
...: vectorizer = TfidfVectorizer(smooth_idf=False, norm=None)
...: response = vectorizer.fit_transform(books)
...: feature_names = vectorizer.get_feature_names()
...: for col in response.nonzero()[1]:
...: print(feature_names[col], '-', response[0, col])
...:
hello - 2.09861228867
there - 2.09861228867
this - 1.40546510811
is - 1.40546510811
first - 2.09861228867
book - 1.0
to - 1.40546510811
be - 1.40546510811
read - 1.40546510811
by - 1.40546510811
wordcount - 1.40546510811
script - 1.40546510811
this - 1.40546510811
is - 1.40546510811
book - 1.0
to - 1.40546510811
be - 1.40546510811
read - 1.40546510811
by - 1.40546510811
wordcount - 1.40546510811
script - 1.40546510811
second - 0.0
it - 0.0
has - 0.0
some - 0.0
additionl - 0.0
information - 0.0
book - 1.0
just - 0.0
third - 0.0
所以考虑“你好”的结果:
hello - 2.09861228867
现在,手动:
In [3]: import math
In [4]: tf = 1
In [5]: idf = math.log(3/1) + 1
In [6]: tf*idf
Out[6]: 2.09861228866811
手动计算的问题是您使用的是log base 10,但是您需要使用自然对数.
如果您仍然感觉到要经历平滑和标准化步骤的强烈愿望,那么这应该让您正确地做到这一点.
内容总结
以上是互联网集市为您收集整理的python – 这是正确的tfidf吗?全部内容,希望文章能够帮你解决python – 这是正确的tfidf吗?所遇到的程序开发问题。 如果觉得互联网集市技术教程内容还不错,欢迎将互联网集市网站推荐给程序员好友。
内容备注
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 gblab@vip.qq.com 举报,一经查实,本站将立刻删除。
内容手机端
扫描二维码推送至手机访问。