Question

i have about 1000 .csv files with daily data stretching back 30 years.
i am building a RESTful app for myself where ideally the user will input the stock symbol of interest
and the time frame of interest which could go more than 10 years of daily data.

Design 1

I was thinking of the following flow:

1) user input symbol='GOOG'
2) user input startDate='01-01-1997', endDate=getTodayDate()
3) client requests location of GOOG.csv file in dropbox server or EC2
4) server returns GOOG.csv to client
5) client parses .csv file until all data is collected and plotted

OR

Design 2

1) user input symbol='GOOG'
2) user input startDate='01-01-1997', endDate=getSystemDate()
3) client requests location of GOOG.csv file in dropbox server or EC2
4) server-side converts GOOG.csv file to GOOG.json file 
5) server returns GOOG.json object to client side for plotting

OR

Design 3

0) During maintenance time, convert all 1000 .csv files to .json objects
1) user input symbol='GOOG'
2) user input startDate='01-01-1997', endDate=getSystemDate()
3) client requests location of GOOG.json file in dropbox server or EC2 
4) server returns GOOG.json object to client side for plotting

My end goal after this would be to plot up to 6 different graphs in the same 1 chart. As well as plotting historical portfolio returns.

Which design is appropriate, does not eat too much memory and delivers good performance?

Answer 1

from your short description I would go with Design 2 and enforce some caching on the server or storing the converted json items on the fly so that you don’t have to convert them again later that day. If it turns out that you have too much load or memory consumption on the server side you can always move to Design 3 at a later point.