I want to write a program that just like grep command to deal with my sql file. I hope it can use all of my cores when it working. Using MPI or just Pthreads, it doesn’t matter. But, How can I divide the file in to 4 files? I mean, if I have a txt file with 700mb size, how to make every thread deal with 1/4 of that file.
In my opinion, I need know how many lines of this file, then, set different file offset per thread. Is it a good way, and how to get the number of total lines?
maybe get how many lines the file have will take a long time:
madper@myhost ~ % time wc -l lineitem.tbl
6001215 lineitem.tbl
wc -l lineitem.tbl 0.20s user 0.40s system 4% cpu 12.996 total
What you want to do isn’t very easy. Frameworks like Google’s MapReduce and Apache Hadoop have been created to handle problems like this. In the general case, splitting up an arbitrary large file for arbitrary parallel processing is a difficult problem, and it’s best to use a dedicated framework for this task. But in your case, since you seem to have very specific requirements, you might be able to do something like:
Split the file into N chunks (where N = number of cores). Obviously, this won’t respect line boundaries, so you’ll need a “splitter” routine which searches for the next new line character in each chunk. (This may be tricky to get right and avoid overlapping, but it’s basically what Hadoop or MapReduce does.) You can then process each chunk in parallel.
Of course, this might not be effective as you think. If the entire file doesn’t fit in memory, you’re going to run into I/O problems. Just because you are parallelizing CPU usage doesn’t necessarily mean you’re parallelizing I/O. This is why distributed processing frameworks usually also split data across multiple hard drives or cluster nodes.