Question

I am trying to read 200,000 records from a file and then use tokenizer to parse the string and remove the quotes which are around each part. But the running time is very high compared to normally reading a string. It took 25 seconds just to read these records (0.0001 second per record????). Is there any problem with my programming or if not is there a faster way to do this?

int main()
{
    int counter = 0;
    std::string getcontent;
    std::vector<std::string> line;
    std::vector< std::vector<std::string> > lines;

    boost::escaped_list_separator<char> sep( '\\', '*', '"' ) ;
    boost::tokenizer<> tok(getcontent);

    std::ifstream openfile ("test.txt");

    if(openfile.is_open())
    {
        while(!openfile.eof())
        {
            getline(openfile,getcontent);

            // THIS LINE TAKES A LOT OF TIME
            boost::tokenizer<> tok(getcontent); 

            for (boost::tokenizer<>::iterator beg=tok.begin(); beg!=tok.end(); ++beg){
                line.push_back(*beg);
            }

            lines.push_back(line);
            line.clear();
            counter++;
        }
        openfile.close();
    }
    else std::cout << "No such file" << std::endl;

    return 0;
}

Answer 1

At least if I’m reading this correctly, I think I’d take a rather more C-like approach. Instead of reading a line, then breaking it up into tokens, and stripping out the characters you don’t want, I’d read a character at a time, and based on the character I read, decide whether to add it to the current token, end the token and add it to the current line, or end the line and add it to the vector of lines:

#include <vector>
#include <string>
#include <stdio.h>
#include <time.h>

std::vector<std::vector<std::string> > read_tokens(char const *filename) {
    std::vector<std::vector<std::string> > lines;
    FILE *infile= fopen(filename, "r");

    int ch;

    std::vector<std::string> line;
    std::string token;

    while (EOF != (ch = getc(infile))) {
        switch(ch) {
            case '\n':
                lines.push_back(line);
                line.clear();
                token.clear();
                break;
            case '"':
                break;
            case '*':
                line.push_back(token);
                token.clear();
                break;
            default:
                token.push_back(ch);
        }
    }
    return lines;
}

int main() {
    clock_t start = clock();
    std::vector<std::vector<std::string> > lines = read_tokens("sample_tokens.txt");
    clock_t finish = clock();
    printf("%f seconds\n", double(finish-start)/CLOCKS_PER_SEC);
    return 0;
}

Doing a quick test with this on a file with a little over 200K copies of the sample you gave in the comment, it’s reading and (apparently) tokenizing the data in ~3.5 second with gcc or ~4.5 seconds with VC++. I’d be a little surprised to see anything get a whole lot faster (at least without faster hardware).

As an aside, this is handling memory about as you originally did, which (at least in my opinion) is pretty strong evidence that managing memory in the vector probably isn’t a major bottleneck.