Home/ Questions/Q 160627
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-11T11:04:06+00:00

I have large data files stored in S3 that I need to analyze. Each

  • 0

I have large data files stored in S3 that I need to analyze. Each batch consists of ~50 files, each of which can be analyzed independently.

I’d like to setup parallel downloads of the S3 data into the EC2 instance, and setup triggers that start the analysis process on each file that downloads.

Are there any libraries that handle an async download, trigger on complete model?

If not, I’m thinking of setting up multiple download processes with pyprocessing, each of which will download and analyze a single piece of the file. Does that sound reasonable or are there better alternatives?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Answering my own question, I ended up writing a simple modification to the Amazon S3 python library that lets you download the file in chunks or read it line by line. Available here.

    • 0