Question

I am doing an assignment, in which I have a large text file (1gb). I am supposed to parse this text file and store it in a tree for some operations. The problem I am facing is the time it takes to completely parse the whole file. It takes about 40 min to completely parse the file. Can anyone please show me how to do it efficiently in a few minutes?
My code is

int main()
{
FILE * file=fopen("data.txt","r");
char line[1000];
char *token;

while(fgets(line,1000,file)!=NULL) 
{
    token=strtok(line,"     ");
    while(token!=NULL)
    {
        cout<<token<<endl;
        token=strtok(NULL,"     ");
    }
}

fclose(file);
return 0;

}

Answer 1

Personally, I would guess that printing the tokens is the biggest time sink. Try this instead and see if it runs faster:

#include <iostream>
#include <fstream>

int main() {
    std::ios_base::sync_with_stdio(false);
    std::ifstream in("data.txt", std::ios_base::binary);
    for (std::string token; in >> token; ) {
        if (++count / 100000 == 0) {
            std::cout << "read " << count << " tokens\n";
        }
    }
    std::cout << "read " << count << " tokens\n";
}