Python 对Twitter tweet的元素 (Word, Screen Name, Hash Tag)的词汇多样性分析

CODE:

#!/usr/bin/python 
# -*- coding: utf-8 -*-

'''
Created on 2014-7-3
@author: guaguastd
@name: tweet_lexical_diversity.py
'''

# Compute lexical diversity
def lexical_diversity(tokens):
    return 1.0*len(set(tokens))/len(tokens)

# Compute the average number of words per tweet
def average_words(statuses):
    total_words = sum([len(s.split()) for s in statuses])
    return 1.0*total_words/len(statuses)
    
if __name__ == '__main__':

    # import login, see http://blog.csdn.net/guaguastd/article/details/31706155 
    from login import oauth_login

    # get the twitter access api
    twitter_api = oauth_login()
    
    # import tweet, see http://blog.csdn.net/guaguastd/article/details/36163301
    from tweets import tweet

    while 1:
        query = raw_input('\nInput the query (eg. #MentionSomeoneImportantForYou, exit to quit): ')
        
        if query == 'exit':
            print 'Successfully exit!'
            break
        
        status_texts,screen_names,hashtags,words = tweet(twitter_api, query)  
        
        for token in (words, screen_names, hashtags):
            print '\rLexical diversity of %s: ' % token
            print lexical_diversity(token)

        for status in (status_texts,):
            print '\rAverage words of %s: ' % status
            print average_words(status)

Input the query (eg. #MentionSomeoneImportantForYou, exit to quit): #MentionSomeoneImportantForYou
Length of statuses 1

Lexical diversity of [u'#MentionSomeoneImportantForYou', u'@RihannaDaily']: 
1.0

Lexical diversity of [u'RihannaDaily']: 
1.0

Lexical diversity of [u'MentionSomeoneImportantForYou']: 
1.0

Average words of [u'#MentionSomeoneImportantForYou @RihannaDaily']: 
2.0

Input the query (eg. #MentionSomeoneImportantForYou, exit to quit): 

原文：http://blog.csdn.net/guaguastd/article/details/36614833