Question

I’m writing a small benchmarking library in C which is used to benchmark single functions. The way it works is that you supply the benchmarking function with a pointer to a void function without parameters and the number iterations. The function then returns a struct with information about

However when looking at the individual results I see that the first call takes up a lot of time and then the calls after takes up very little time.

So I’m wondering:

1. Is this due to instruction caching?
2. If yes to first question, how does benchmarking tools generally conform to this? Is the first call excluded?
3. If yes to first question, is there a case when caching is not applied to a function?
4. If yes to first question, does caching occur on whole functions or segments of a function?
5. If yes to first question, is there anything else I should consider that I should read up on and understand better?

---

Data and code

The function which retrieves the time:

double currentTime()
{
    struct timeval time;
    struct timezone timezone;
    gettimeofday(&time, &timezone);
    return time.tv_sec + time.tv_usec * 1e-6;
}

Test result:

0.000319
0.000000
0.000000
0.000000
0.000001
0.000000
0.000000
0.000000
0.000000
0.000000
Total time: 0.000320
Average time: 0.000032
Worst time: 0.000319
Best time: 0.000000

Result without optimizing flag:

13.425430
13.349757
13.482863
13.129472
13.020705
13.672982
13.027595
13.139602
13.028962
13.107892
Total time: 132.385260
Average time: 13.238526
Worst time: 13.672982
Best time: 13.020705

The function being tested which produces this result:

void test()
{
    unsigned int i = 0;
    while(i++ < UINT_MAX){}
}

Make file:

CC = gcc
MAIN = main.c
SOURCES = lib/tb_time_handling.c lib/tb_rendering.c tb_benching.c
OUTPUT = main
FLAGS = -Wall -pedantic -O2

all: main

main: 
    $(CC) $(MAIN) $(SOURCES) -o $(OUTPUT) $(FLAGS)

Github repo with all code:

https://github.com/Ancide/TinyBench

Edit: Forgot to mention compiler and compiler flags

Edit 2: Added git repo with all code in case someone wants to see everything

Edit 3: Added results withouth O2 flag

Answer 1

This is normal in virtual memory operating systems. The first call to a function tends to generate a page fault which loads the code from the executable file into RAM. If you are interested in actual code perf then you ignore the first call. If you are interested in realistic perf measurements then you don’t ignore it.