Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
This is a college project:
I have a database ( mysql or postgresql doesn’t matter ) with 100 billion of posts and I need to search ( as fast as possible ) a generic keyword.
Every post is 500-1000 keywords.
This isn’t only a database issue but also a software ( for indexing or other ) issue.
How can I do that ?
I could use some advanced search engine technology but I don’t know which.
You might want to check out Sphinx. It’s a full text search engine that handles distributed indexes. You can have parts of the data spread over many computers. And querying a single server can send the query to the other servers, and gather the results from each. It has pretty good speed, but you probably can’t do 100 billion posts on a single machine.
You probably won’t be able to do something like this in MySQL or Postgresql. While you could store all the data, MySQL and Postgres lack the full text indexing and search speed that a real full text index will bring you.
MySQL has a way to compile in support for the Sphinx storage engine, which, although the data will still be stored in Sphinx separate from MySQL, you can still query the Sphinx search engine using anything that connects to MySQL, and also perform joins to other tables that are in your MySQL database. However, if you just want to perform simple searches on the documents, and don’t require joining to other data, you can just use the native PHP interface.