Question

Objective:

To create an Index that accommodates a pre-existing set of price data from a csv file. I can build an index using list comprehensions. If it’s done in that way, the construction would give me a filtered list of length 86,772–when run over 1/3/2007-8/30/2012 for 42 times (i.e. 10 minute intervals). However, my data of prices coming from the csv is length: 62,034. Observe that the difference in length is due to data cleaning issues.

That said, I am not sure how to overcome the apparent mismatch between the real data and this pre-built (list comp) dataframe.

Attempt:

Am I using the first two lines incorrectly?

data=pd.read_csv('___.csv', parse_dates={'datetime':[0,1]}).set_index('datetime')

dt_index = pd.DatetimeIndex([datetime.combine(i.date,i.time) for i in data.index])

ts = pd.Series(data.prices.values, dt_index)

Questions:

1. As I understand it, I should use ‘combine’ since I want the index construction to be completely informed by my csv file. And, ‘combine’ returns a new datetime object whose date components are equal to the given date object’s, and whose time components are equal to the given time object’s.

2. When I parse_dates, is it lumping the time and date together and considering it to be a ‘date’?

3. Is there a better way to achieve the stated objective?

Traceback Error:

AttributeError: ‘unicode’ object has no attribute ‘date’

Answer 1

You can write this neatly as follows:

ts = df1.prices

Here’s an example:

In [1]: df = pd.read_csv('prices.csv',
                  parse_dates={'datetime': [0,1]}).set_index('datetime')

In [2]: df # dataframe
Out[2]: 
                    prices  duty
datetime                 
2012-11-12 10:00:00      1     0
2012-12-12 10:00:00      2     0
2012-12-12 10:00:00      3     1


In [3]: df.prices # timeseries
Out[3]: 
datetime
2012-11-12 10:00:00    1
2012-12-12 10:00:00    2
2012-12-12 11:00:00    3
Name: prices

In [4]: ts = df.prices

You can groupby date like so (similar to this example from the docs):

In [5]: key = lambda x: x.date()

In [6]: df.groupby(key).sum()
Out[6]: 
            prices  duty
2012-11-12       1     0
2012-12-12       5     1

In [7]: ts.groupby(key).sum()
Out[7]: 
2012-11-12    1
2012-12-12    5

Where prices.csv contains:

date,time,prices,duty
11/12/2012,10:00,1,0
12/12/2012,10:00,2,0
12/12/2012,11:00,3,1