Question

I need to build a website where we display data that is refreshed every 5minutes
in a text file with a | separator.

I currently use Java to do this.

What I do now:
I grab the textfile for every request through the website and process it and then display
the data to the end user, this works fine, since Java can go through like 5000 lines of data fast, and when I filter it it is still extremely fast.

However now the management wants the following:
They added 3 textfiles with the | separator to it, and now want me to also read
those files and match the information on certain fields, and if there is a match
also display that information to the end user.

I think soon enough, although Java is fast, I will run into trouble when 10 people
want that information and I have to run through 4 total files matching the information.

What can I do to make this process super fast?

My Creative solutions so far:

-Leave it this way, since Java is fast and end users can wait (probably less then 1second)

-Have a background process that dumps new data into a MySQL database every 5minutes, since databases are extremely good at getting the same data from multiple tables.

Thank you!

Answer 1

Both solutions are good. A few things to keep in mind:

• If you’re going to keep the user waiting for more than 1 second, consider making a landing page and showing a “loading info – please wait” message while your page loads the information (using AJAX, for example).
• Chances are management will add more files later. Coding a database solution now might save you from hacking it now and having to do it the right way later anyway.
• Other reasons to use the database are the DBMS extra advantages in caching, locking, joining…