Question

UPD: Turns out, it is a question about error handling in Go

I have written a simple web crawler, that generates the addresses of web-pages on the “main thread”, fetches the actual pages in one go-routine and gives the content back via a chan, and writes it to a file in another go-routine. However, after about 300,000 iterations of this process, i get the following error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x38 pc=0x22e9]</pre>

The error “stacktrace” is quite long, and unfotunately, I don’t have it now (I will post it later, after repeating the experiment).

Is there a need to somehow manage the memory, close the channels or something like that?

Below is the code with some omissions.

package main

import (
    "fmt"
    "io/ioutil"
    "net/http"
    "strconv"
)

func main() {
    channel := make(chan []byte)

    for i:=0; i < 20; i++ {
        go fetcher(generateLink(), channel)
    }

    for a:=0; ; a++ {
        go writeToFile(strconv.Itoa(a), <-channel)
        go fetcher(generateLink(), channel)
        fmt.Println(strconv.Itoa(a))
    }
}

func fetcher(url string, channel chan []byte) {

    resp, err := http.Get(url)
    if err != nil {
        channel <- []byte("")
    }
    defer resp.Body.Close()
    body, err := ioutil.ReadAll(resp.Body)
    channel <- body
}

func writeToFile(filename string, bytes []byte) {
    ioutil.WriteFile(filename+".html", bytes, 0644)
}

func generateLink() string {
    ...
}

Answer 1

panic: runtime error: invalid memory address or nil pointer deference

This means you tried to use a nil pointer. This is a programmer error, not an out of memory error. Since I don’t have access to the actual trace or code, there is no way I can help you further.

Edit: After taking another look at your code I think I found a possible source of the error.

resp, err := http.Get(url)
if err != nil {
    channel <- []byte("")
}
defer resp.Body.Close()

You end up referencing resp in order to get the field Body even when err != nil. If err !=nil, it is quite likely resp is nil.