Question

I had an pre-interview task, which I have completed and the solution works, however I was marked down and did not get an interview due to having used a TADODataset. I basically imported a CSV file which populated the dataset, the data had to be processed in a specific way, so I used Filtering and Sorting of the dataset to make sure that the data was ordered in the way I wanted it and then I did the logic processing in a while loop. The feedback that was received said that this was bad as it would be very slow for large files.

My main question here is if using an in memory dataset is slow for processing large files, what would have been better way to access the information from the csv file. Should I have used String Lists or something like that?

Answer 1

It really depends on how “big” and the available resources(in this case RAM) for the task.

“The feedback that was received said that this was bad as it would be very slow for large files.”

CSV files are usually used for moving data around(in most cases that I’ve encountered files are ~1MB+ up to ~10MB, but that’s not to say that others would not dump more data in CSV format) without worrying too much(if at all) about import/export since it is extremely simplistic.

Suppose you have a 80MB CSV file, now that’s a file you want to process in chunks, otherwise(depending on your processing) you can eat hundreds of MB of RAM, in this case what I would do is:

while dataToProcess do begin
  // step1
  read <X> lines from file, where <X> is the max number of lines 
  you read in one go, if there are less lines(i.e. you're down to 50 lines and X is 100)
  to process, then you read those
  // step2
  process information
  // step3
  generate output, database inserts, etc.
end;

In the above case, you’re not loading 80MB of data into RAM, but only a few hundred KB, and the rest you use for processing, i.e. linked lists, dynamic insert queries(batch insert), etc.

“…however I was marked down and did not get an interview due to having used a TADODataset.”

I’m not surprised, they were probably looking to see if you’re capable of creating algorithm(s) and provide simple solutions on the spot, but without using “ready-made” solutions.

They were probably thinking of seeing you use dynamic arrays and creating one(or more) sorting algorithm(s).

“Should I have used String Lists or something like that?”

The response might have been the same, again, I think they wanted to see how you “work”.