Question

I’m using scrapy to scrape some data.
i got 2 spiders = spider a and b

spider a fetches some data and write it to a file.
spider b reads that data.

the problem is that spider b gets an empty file.

I can see that the file is filled after spider a finishes hes job.

I spent few hour to figure this out.

code snippet:

spider a

f = open('file.txt', 'a+')
f.write("str")
f.write("\n")
f.close()

spider b

f = open('file.txt')
for line in f:
  print line
f.close()

Nothing comes out, what is wrong with this?

EDIT:I got it to work.
I open a file with ‘a’
write a line and then close the file.
this goes for all lines.

the code snippet was revised.

Answer 1

(1) You want f.close() instead of f.close in “spider b”. The way it is now, you are not properly closing the file, so its buffers aren’t getting flushed.

Also, regarding your use of f.flush() in “spider a” be sure you are aware of this note from the documentation:

Note flush() does not necessarily write the file’s data to disk. Use
flush() followed by os.fsync() to ensure this behavior.

In fact, unless you have a good reason, it might be better to just close() the file. If you do have/want to flush it, be sure you also use os.sfsync().

(2) Also, instead of

for line in f:
   print f

try this:

for line in f:
   print line

As a final note, take a look at using with to manage open files. For instance

with open('file.txt') as f:
   # put file related actions here
   for line in f:
      print line

will automatically close the file at the end of the block for you or if an exception is encountered.