Question

In SQLite I would like to find the standard deviation of the first differences of a (logged) series that I define with GROUP BY. My data provider gives me a daily price series, but I would like to find annualized daily volatility (the standard deviation of daily returns — first difference of the natural log of the series — over each year). I can bring the data to R, then use ddply(), but I would like to do this entirely in SQLite. I tried the difference() function from the RSQLite.extfunctions package, but my usage is wrong. I expected it to work like diff() in R, but I can’t find much documentation.

This generates some data.

stocks <- 5
years <- 5
list.n <- as.list(rep(252, stocks * years))
list.mean <- as.list(rep(0, stocks * years))
list.sd <- as.list(abs(runif(stocks * years, min = 0, max = 0.1)))
list.po <- as.list(runif(n = stocks, min = 25, max = 100)) 
list.ret <- mapply(rnorm, n = list.n, mean = list.mean, sd = list.sd, SIMPLIFY = F)
my.price <- function(po, ret) po * exp(cumsum(ret))
list.price <- mapply(my.price, po = list.po, ret = list.ret, SIMPLIFY = F)
gvkey <- rep(seq(stocks), each = 252 * years)
day <- rep(seq(252), n = stocks * years)
fyr <- rep(seq(years), n = stocks, each = 252)
data.dly <- data.frame(gvkey, fyr, day, p = unlist(list.price))

Here is how I would do it with ddply() and the result.

# I could do this easily with ddply and subset
library(plyr)
data.dly <- ddply(data.dly, .(gvkey, fyr), transform, vol = sd(diff(log(p))))
data.ann <- subset(data.dly, day == 252)
head(data.ann)
     gvkey fyr day         p         vol
252      1   1 252  86.08568 0.077287182
504      1   2 252  43.32113 0.066741862
756      1   3 252  68.69734 0.084419564
1008     1   4 252  75.37267 0.006003969
1260     1   5 252  17.53583 0.083688727
1512     2   1 252 168.44656 0.035959492

And here is my (failed) SQLite attempt and error.

# but I can't figure it out in SQLite
library(RSQLite)
library(RSQLite.extfuns)
db <- dbConnect(SQLite())
init_extensions(db)
[1] TRUE
dbWriteTable(db, name = "data_dly", value = data.dly)
[1] TRUE
temp <- dbGetQuery(db, "SELECT stdev(difference(log(p))) FROM data_dly GROUP BY gvkey, fyr ORDER BY gvkey, fyr, day")
Error in sqliteExecStatement(con, statement, bind.data) : 
  RS-DBI driver: (error in statement: wrong number of arguments to function difference())

Does difference() need a comma separated list of numbers? Can I do this entirely in SQLite? Or do I need to perform in R? Thanks!

Answer 1

Try this where data.dly is the data frame in the post:

library(sqldf)
out <- sqldf("select A.gvkey, A.fyr, stdev(log(A.p) - log(B.p)) vol
    from `data.dly` A join `data.dly` B 
    where A.day = B.day + 1 
        and A.gvkey = B.gvkey 
        and A.fyr = B.fyr 
    group by A.gvkey, A.fyr")

This gives:

> head(out)
  gvkey fyr        vol
1     1   1 0.09312510
2     1   2 0.01905447
3     1   3 0.01651095
4     1   4 0.06962667
5     1   5 0.05243940
6     2   1 0.03039751