Home/ Questions/Q 6677425
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-26T04:07:06+00:00

I tested writing to a text file simply via shell command or php. Using

  • 0

I tested writing to a text file simply via shell command or php. Using a while loop for 1 million cycles, appending cycle number into the file (as $i\n). I thought this must be fastest way to store data on the disk. Where in shell or php, it took about 40s for 1 million records.

Alternatively, I used fast key/value databases such as QDBM, GDBM, etc. The overall time was even lesser (15-30s). What is the theoretical explanation for faster storage of data via simple database systems? I thought the rate-limiting should be hardware (e.g the speed of hard disk or CPU).

Another Question: When increasing the cycle from 100k to 1M, the overall time was increased 10 times; but when increasing from 1M to 10M, it took 100 times longer. What threshold makes the system slow this way?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    The bottleneck for writing data to disk is usually the disk I/O. The fastest continuous output can be achieved when writing contiguous blocks to the disk(s).

    I assume that the file was opened and closed 1 million times in your shell script. Each close issues a synchronization of the file to disk, which probably causes a wait (depending on file system and storage solution).

    • 0