Question

I am Trying to read data from text files and save tokens in SQLite databae running on windows
The Files size is about 300MB , the thing is that I am not using threads yet , the application collapses and stuck with “not responding” message
here is my code

QDir dir(ui->lineEdit->text());
if(dir.exists() && ui->lineEdit->text()!=""){
    CreateTables();

    dir.setFilter(QDir::Files | QDir::Hidden | QDir::NoSymLinks);
    dir.setSorting(QDir::Size | QDir::Reversed);

    QFileInfoList list = dir.entryInfoList();
    for (int i = 0; i < list.size(); ++i)  {
        QFileInfo fileInfo = list.at(i);
        QFile file(ui->lineEdit->text()+"/"+ fileInfo.fileName());

        if (file.open(QIODevice::ReadOnly | QIODevice::Text)) {
            QSqlQuery qrry;
            qrry.prepare( "INSERT INTO documents (path) VALUES ('"+ui->lineEdit->text()+"/"+ fileInfo.fileName()+"')" );
            qrry.exec();

            QString line;
            QTextStream in(&file);
            int lineCount=0;
            while (!in.atEnd()) {

                line = in.readLine();
                lineCount++;
                QRegExp rx("(\\ |\\,|\\.|\\:|\\t)"); 
                QStringList line_tokens = line.split(rx);

                for(int i=0;i<line_tokens.length();i++){
                    if(line_tokens[i].length()>3){
                        QSqlQuery qry;
                        qry.prepare( "INSERT INTO tokens (token,path,line) VALUES ('"+line_tokens[i]+"', '"+ui->lineEdit->text()+"/"+ fileInfo.fileName()+"','"+QString::number(lineCount)+"')" );
                        qry.exec();
                    }
                }
            }
        }
        else{
            QMessageBox::information(this,"File Read Error","Couldn't Open File, Please Make Sure That This File Is Accessable And Readable !");
        }
        std::cout << std::endl;
    }
}
else{
    QMessageBox::critical(this,"Directory choosen is Not Valid","Please Make Sure That You Have Choosen A Valid Directory!");
}

it works for a small file sizes but when it comes to huge amount of data it becomes gives the “not responding” message

will threads save the application from being collapsed ??
and how may I implements threads to do this job for more thant 20 files of 15 MB each

Answer 1

First, you can simply… wait until it process all the data. If you should run this program only once, it is ok.

Then, you should not generate such stream of requests to database. It is much better to run one single big insertion, than a lot of small. You can use batch execution.

Then, it’s probably no reason of trying to parallel writing to Db. I think, it’s smart enough to manage writing efficiently.

But it worth paralleling parsing. The best solution here is to use QtConcurrent framework. It is made exactly for such tasks.