Question

I have data that more or less looks like this (don’t know how to paste plots):

library(reshape2)
library(ggplot2)

df <- cbind(runif(2000,0,1000), rep(0,n=2000))
for (i in 1:nrow(df)) {
  df[i,2] <- runif(1, df[i,1], (10000-2*df[i,1])) 
}
colnames(df) <- c("x","y")
df.1 <- melt(data.frame(df), id="x")

p <- ggplot(df.1, aes(x=x, y=value))
p <- p + geom_point()
p <- p + geom_smooth()
p

Instead of the smooth line shown, I need one straight line at the bottom 5% and one straight line at the top 95%.
An issue is that I have millions of points, so I suppose data.table is a good way forward:

library(data.table)
dt <- data.table(df)
dt[,xbin:=0]
for (i in 0:100) {
  x1 <- i*100
  x2 <- (i+1)*100
  dt[x>=x1 & x<x2, xbin:=x2]
}
setkey(dt,xbin)
result1.dt <- dt[,list(ymin=min(y), ymax=max(y)), by=key(dt)]
result1.df <- data.frame(result1.dt)

p <- p + geom_line(data=result1.df, aes(x=xbin, y=ymin))
p <- p + geom_line(data=result1.df, aes(x=xbin, y=ymax))
p

The lines are not yet straight, but from here it is trivial to fix that.

Instead of min and max, how can I with data.table get 5th and 95th percentiles? Am I reinventing the wheel, i.e. is there already a statistical method for this (and a function)?

Answer 1

You can use stat_quantile to add these lines to your plot.

p + stat_quantile(quantiles = c(0.05,0.95))

This uses quantile regression, specifically the rq function from the quantreg package.