Question

I have written a program using Free Pascal and the Lazarus IDE. In brief, it recursively scans directories and ‘does stuff’ (hashes) with each file and then it outputs the hash values and the file name into a StringGrid, which is refreshed with each successive file.

It works great for up to a few thousand files, but when you get to the tens of thousands, it really slows down, processing maybe one file every half a second, even if it’s only a small file of a few Kb.

The main part of the code responsible is below. Can anyone see why my program slows down when the numbers of files in the grid exceeds tens of thousands?

procedure TForm1.HashFile(FileIterator: TFileIterator);
var
  SizeOfFile : int64;
  NameOfFileToHash, fileHashValue, PercentageProgress : string; 
  FI : TFileIterator;                      //File Iterator class
  SG : TStringGrid;

begin
  FI := TFileIterator.Create;
  SG := TStringGrid.Create(self);
  SizeOfFile := 0;
  fileHashValue := '';

  if StopScan = FALSE then                     // If Stop button clicked, cancel scan
    begin
    NameOfFileToHash := (FileIterator.FileName);
    SizeOfFile := FileSize(NameofFileToHash);
    StatusBar1.SimpleText := 'Currently Hashing: ' + NameOfFileToHash;
    fileHashValue := CalcTheHashFile(NameOfFileToHash); // Custom function, see below

    // Now lets update the stringgrid and text fields

    // StringGrid Elements:
    // Col 0 is FileCounter. Col 1 is File Name. Col 2 is Hash
    StringGrid1.rowcount:= FileCounter+1;
    StringGrid1.Cells[0,FileCounter] := IntToStr(FileCounter);
    Stringgrid1.Cells[1,FileCounter] := NameOfFileToHash;
    Stringgrid1.Cells[2,FileCounter] := UpperCase(fileHashValue);

    // Dynamically scroll the list so the user always has the most recently hashed
    // file insight and expand the columns in lie with their content width

    StringGrid1.row := FileCounter;
    StringGrid1.col := 1;
    StringGrid1.AutoSizeColumns;

    // Progress Status Elements: Most of these vars are global vars

    NoOfFilesExamined.Caption := IntToStr(FileCounter);
    PercentageProgress := IntToStr((FileCounter * 100) DIV NoOfFilesInDir2);
    Edit1.Caption := PercentageProgress + '%';
    TotalBytesRead := TotalBytesRead + SizeOfFile;
    edtTotalBytesExamined.Caption := FormatByteSize(TotalBytesRead);

    Application.ProcessMessages;
    FileCounter := FileCounter+1;
    end;
  SG.Free;
  FI.Free;
end;

The full source code is available from my SourceForge page, https://sourceforge.net/projects/quickhash/ under ‘Files’ –> ‘Source Code’ if you need that.

Any help appreciated

Ted

Answer 1

Well as a Delphi bloke two things popped out at me.
AutoSizeColumns. If you are lucky it’s doing nothing. If it’s marching down all columns in 10,000 rows, every update and doing a GetTextLength to see if it fits, then repainting the grid….

So job 1 would be to preset some column sizes and comment that out.
Do it once at the end of the scan at most.

Who is going to want to see all 10,000 + rows at once?

I think I’d stream them out to file, and show the last 1 – page full to indicate progress. Then I’d drive my ui from the file, using a simple get a page full scenario.
Depends on what you are doing with the data, but you could reload file for further analyis, do a compare for changes.

Even if you stick with in memory Have a TList? of THashRecord. Then drive your display from that, you’ll have a chance then.