Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I need to write around 103 sparse double arrays to disk (one at a time) and read them individually later in the program.
EDIT: Apologies for not framing the question clearly earlier. To be specific I am looking to store as much as possible in memory and save the currently unused variables on the disk. I am working on linux.
The fastest way would be to buffer the I/O. Instead of writing each array individually, you’d first copy as many as you can to a buffer. Once that buffer is full you would write the entire buffer to disk, clear the buffer, and repeat. This minimizes the amount of writes that occur to the disk and will increase I/O efficiency.
If you plan on reading the arrays later in sequential order, I recommend you also buffer the reads so it reads more that it needs and you can work out of the buffer.
You could take it a step further and use asynchronous read/write operations so that your program can process other tasks while waiting on the disk.
If you are concerned about the size on disk it will consume, you can add another layer that will compress/uncompress the data stream as you write/read to and from the disk.