Question

I’m trying to implement a simple gradient boosting algorithm for regression in R. This is what I came up with so far but the error is not plateauing like I had expected. Any suggestions?

data("OrchardSprays")
niter  <- 10
learn  <- 0.05
y      <- OrchardSprays$decrease
yhat   <- rep(0,nrow(OrchardSprays))
weight <- rep(1,nrow(OrchardSprays))
loss   <- function(y,yhat) (y - yhat)^2

for (i in seq(niter))
{
    model  <- lm(decrease~.,weights=weight,data=OrchardSprays)
    yhat   <- yhat + weight * (predict(model) - yhat) / i
    error  <- mean(loss(y,yhat))
    weight <- weight + learn * (loss(y,yhat) - error) / error
    cat(i,"error:",error,"\n")
}

output:

1 error: 319.5881 
2 error: 318.6175 
3 error: 317.9368 
4 error: 317.6112 
5 error: 317.6369 
6 error: 317.9772 
7 error: 318.5833 
8 error: 319.4047 
9 error: 320.3939 
10 error: 321.5086 

Answer 1

I’ll admit to not having written a weight-optimizer in ages, so I may be off base. I’d start by recording the yhat vector on every iteration. See if the values are either oscillating or disappearing towards zero (as I’m not sure whether you’re helping or hurting by dividing by i ) .
Similarly, take a look at the R^2 values from each iteration of lm(). If they’re very close to 1 you may simply have run into the currently prescribed sensitivity limit of lm().

It would be helpful if you could provide the source of your algorithm so we could check the code against the equations you’re implementing.

Update: A quick look at wikipedia yields the following: “Several open-source R packages are available: gbm,[6] mboost, gbev.” I strongly recommend you study those packages, including their source code, to see if they’ll meet your needs.