Question

Question: I am testing functions in a package that I am developing and would like to know if you can suggest some general guidelines for how to do this. The functions include a large range of statistical modeling, transformations, subsetting, and plotting. Is there a ‘standard’ or some sufficient test?

An Example: the test that prompted me ask this question,

The function dtheta:

dtheta <- function(x) {
  ## find the quantile of the mean
  q.mean <- mean(mean(x) >= x)
  ## find the quantiles of ucl and lcl (q.mean +/- 0.15)
  q.ucl  <- q.mean + 0.15
  q.lcl  <- q.mean - 0.15
  qs <- c(q.lcl, q.mean, q.ucl)
  ## find the lcl, mean, and ucl of the vector
  c(quantile(x,qs), var(x), sqrt(var(x))/mean(x))
}

Step 1: make test data:

set.seed(100) # per Dirk's recommendation
test <- rnorm(100000,10,1)

Step 2: compare the expected output from the function with the actual output from the function:

 expected <- quantile(test, c(0.35, 0.65, 0.5))
 actual   <- dtheta(test)[1:3]
 signif(expected,2) %in% signif(actual,2)

Step 3: maybe do another test

test2 <- runif(100000, 0, 100)
expected <- c(35, 50, 65)
actual   <- dtheta(test2)
expected %in% signif(actual,2)

Step 4: if true, consider function ‘functional’

Answer 1

It depends on what exactly you want to test. Next to Dirks recommendations, svUnit or the RUnit package VitoshKa mentioned, I’d like to add a few things :

• Indeed, set the seed, but make sure you try the function with different seeds as well. Some functions fail only once every ten times you try. Especially when optimization is involved, this becomes crucial. replicate() is a nice function to use in this context.
• Think very well about the input you want to test. You should test a number of “odd” cases that don’t really resemble the “perfect” dataset. I always test at least 10 (simulated) datasets of different sizes.
• Fool-proof the function: I also throw in some data types that are not the ones the function is meant for. Wrong type input is likely going to happen at one point, and the last thing you want is a function returning a bogus result without a warning. If you use that function later on in some other code, debugging that code can and will! be hell. Been there, done that, bought the t-shirt…

An example on extended testing of datasets: what would you like to see as output in these cases? Is this the result you’d expect? Not according to the test you did.

> test3 <- rep(12,100000) # data with only 1 value
> expected <- c(12, 12, 12)
> actual   <- dtheta(test3) 
Error in quantile.default(x, qs) : 'probs' outside [0,1]

>  test4 <- rbinom(100000,30,0.5) # large dataset with a limited amount of values
>  expected <- quantile(test4,c(0.35, 0.50, 0.65))
>  actual   <- dtheta(test4)
>  expected %in% signif(actual,2)
[1] FALSE  TRUE  TRUE

> test5 <- runif(100,0,100) # small dataset. 
> expected <- c(35, 50, 65)
> actual   <- dtheta(test5)
> expected %in% signif(actual,2)
[1] FALSE FALSE FALSE

edit : corrected code so tests are a bit more senseful.