What’s the most efficient way in Java to get the 50 most frequent words with their frequency out of a text?
I want to search around ~1,000,000 texts with each have around ~10,000 words and hope that it works in a reasonable time frame.
Share
Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
What’s the most efficient way in Java to get the 50 most frequent words with their frequency out of a text?
I want to search around ~1,000,000 texts with each have around ~10,000 words and hope that it works in a reasonable time frame.
Most efficient would probably be using a Patricia trie that links to a max-heap. Every time you read a word,
findit on the trie, go to the heap andincrease-key. If it’s not in the trie,addit and set its key in the heap appropriately.With a Fibonacci heap,
increase-keyisO(1).A not so unreasonable solution is to use a
Map<String, Integer>, adding the count every time a word is encountered, and then custom-sorting itsentrySet()based on the count to get the top 50.If the
O(N log N)sort is unacceptable, use selection algorithm to find the top 50 inO(N).Which technique is better really depends on what you’re asking for (i.e. the comment whether this is more of an
[algorithm]question than a[java]question is very telling).The
Map<String, Integer>followed by selection algorithm is most practical, but the Patricia trie solution clearly beats it in space efficiency alone (since common prefixes are not stored redundantly).