Question

I’m dealing with large data, so finding a good way for reading data is really important.
I’m just a little bit confused about different reading methods.

1.f=gzip.open(file,'r')
      for line in f:
          process line
     #how can I process nth line? can I?
2.f=gzip.open(file,'r').readlines()
  #f is a list
  f[10000]
  #we can process nth line

3.f=gzip.open(file,'r')
  while True:
       linelist=list(islice(f,4))

4.for line in fileinput.input():
  process line

What’s the difference between 2 and 3 ? I just find their memory usage is the same. islice() also needs to first load the whole file into memory (but just later take bit by bit).
And I hear the 4th method is the least memory-consuming, it’s really processing bit by bit, right?
For 10GB-scale file, which file-reading method would you recommend? Any thought/information is welcomed.
thx

edit: I think one of my problem is I need to pick out specific lines randomly sometimes.
say:

f1=open(inputfile1, 'r')
while True:
    line_group1 = list(islice(f1, 3))
    if not line_group1:
        break
    #then process specific lines say, the second line.
    processed 2nd line
    if ( ....):
           LIST1.append(line_group1[0])
           LIST1.append(processed 2nd line)
           LIST1.append(line_group1[2])

And then sth. like

with open(file,'r') as f,
    for line in f:
       # process line

may not work, am I correct?

Answer 1

You forgot –

with open(...) as f:
    for line in f:
        <do something with line>

The with statement handles opening and closing the file, including if an exception is raised in the inner block. The for line in f treats the file object f as an iterable, which automatically uses buffered IO and memory management so you don’t have to worry about large files.

Both 2,3 are not advised for large files as they read & load the entire file contents in memory before processing starts. To read large files you need to find ways to not read the entire file in one single go.

There should be one — and preferably only one — obvious way to do it.