Question

I am attempting to do what I originally thought would be pretty simple. To wit:

For every file in a list of input files:

1. open the file with png.Decode()
2. scan every pixel in the file and test to see if it is “grey”.
3. Return the percentage of “grey” pixels in the image.

This is the function I am calling:

func greyLevel(fname string) (float64, string) {
    f, err := os.Open(fname)
    if err != nil {
            return -1.0, "can't open file"
    }
    defer f.Close()

    i, err := png.Decode(f)
    if err != nil {
            return -1.0, "unable to decode"
    }

    bounds := i.Bounds()

    var lo uint32 = 122 // Low grey RGB value.
    var hi uint32 = 134 // High grey RGB value.
    var gpix float64    // Grey pixel count.
    var opix float64    // Other (non-grey) pixel count.
    var tpix float64    // Total pixels.

    for x := bounds.Min.X; x < bounds.Max.X; x++ {
            for y := bounds.Min.Y; y < bounds.Max.Y; y++ {
                    r, g, b, _ := i.At(x, y).RGBA()
                    if ((r/255)-1 > lo && (r/255)-1 < hi) &&
                            ((g/255)-1 > lo && (g/255)-1 < hi) &&
                            ((b/255)-1 > lo && (b/255)-1 < hi) {
                            gpix++
                    } else {
                            opix++
                    }
                    tpix++
            }
    }
    return (gpix / tpix) * 100, ""
} 

func main() {
    srcDir := flag.String("s", "", "Directory containing image files.")
    threshold := flag.Float64("t", 65.0, "Threshold (in percent) of grey pixels.")
    flag.Parse()

    dirlist, direrr := ioutil.ReadDir(*srcDir)
    if direrr != nil {
            log.Fatalf("Error reading %s: %s\n", *srcDir, direrr)
    }

    for f := range dirlist {
            src := path.Join(*srcDir, dirlist[f].Name())

            level, msg := greyLevel(src)

            if msg != "" {
                    log.Printf("error processing %s: %s\n", src, msg)
                    continue
            }

            if level >= *threshold {
                    log.Printf("%s is grey (%2.2f%%)\n", src, level)
            } else {
                    log.Printf("%s is not grey (%2.2f%%)\n", src, level)
            }
    }
}

The files are relatively small (960×720, 8-bit RGB)

I am calling ioutil.ReadDir() to generate a list of files, looping over the slice and calling greyLevel().

After about 155 files (out of a list of >4000) the script panics with:

runtime: memory allocated by OS not in usable range
runtime: out of memory: cannot allocate 2818048-byte block (534708224 in use)
throw: out of memory

I figure there is something simple I am missing. I thought that Go would de-allocate the memory allocated in greyLevels() but I guess not?

Follow up:

After inserting runtime.GC() after every call to greyLevels, the memory usage evens out. Last night I was teting to about 800 images then stopped. Today I let it run over the entire input set, approximately 6800 images.

After 1500 images, top looks like this:

top - 10:30:11 up 41 days, 11:47,  2 users,  load average: 1.46, 1.25, 0.88
Tasks: 135 total,   2 running, 131 sleeping,   1 stopped,   1 zombie
Cpu(s): 49.8%us,  5.1%sy,  0.2%ni, 29.6%id, 15.0%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:   3090304k total,  2921108k used,   169196k free,     2840k buffers
Swap:  3135484k total,    31500k used,  3103984k free,   640676k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
28474 mtw       20   0 2311m 1.8g  412 R   99 60.5  16:48.52 8.out

And remained steady after processing another 5000 images.

Answer 1

It appears that you are using a 32-bit machine. It is likely that the program runs out of memory because Go’s garbage collector is conservative. A conservative garbage collector may fail to detect that some region of memory is no longer in use. There is currently no workaround for this in Go programs other than avoiding data structures that the garbage collector cannot handle (such as: struct {...; binaryData [256]byte})

Try to call runtime.GC() in each iteration of the loop in which you are calling function greyLevel. Maybe it will help the program to process more images.

If calling runtime.GC() fails to improve the situation you may want to change your strategy so that the program processes a smaller number of PNG files per run.