I have a task and would like to develop in my mind how i should go around programming this.
I will probably be given a csv format database which would have minimum 36 million lines of data. In the future, users would need to search this “database” through CGI/perl interface based on some conditions depending on multiple column values and display the matching rows.
How should I using perl read the csv format (probably using CSV parser from CPAN) and store into what type of database?
Key priority would be speed of searching of the database.
Any sample coding would appreciated
You probably want to go with a proper database solution. The easiest to set up (depending on your familiarity with RDBMSes) is probably MySQL. Once you have that set up you want to look into Perl modules for interfacing with the database.
DBIx::Classis the “in thing” these days, and as such, there are many people using it who can answer questions.Oh, and for your CSV parsing, look at
Text::CSV, if you don’t want to load it directly into the database (and if your RDBMS of choice doesn’t support ingesting of CSV files directly).