Question

I have an old project at our company that uses shell scripting a lot. Most popular commands are: grep, sed, sort. And it was OK for now. But one thing really bothers me. Sometimes we have an input data from other companies that is in csv-like formats. These input file are connected by several IDs like they are just dumps of some DBMS.

Instead of importing the data into some DBMS there are pages of shell scripts that do ‘cut’-ing, ‘paste’-ing and ‘join’-ing in combination of sort to emulate real db.

The best solution seems to be use SQLite, but my worries are about the size of the data it can handle and the speed of work. And it seems like SQLite will make it more difficult to use such features of shell as ‘sed’.

Can someone advise or describe your success story?

Answer 1

• size: SQLite can hanlde huge datasets, easily go into the Terabytes, according to docs
• speed: propery indexed queries and joins are on the O(logn) or O(nlogn) instead of the O(n) or O(n^2) you get from sed, cut, paste and join.
• sed:
  • if you use it to simply transform a field, it’s just as easy to do a one- or two-liner on Python to handle it.
  • if you use it to transform the table structure, it’s a totally different task, either trivially solved by ALTER TABLE, or maybe a huge processing project.