Question

In my GAE code I upload several files to 3rd party (not mine) server:

for argument in files_arguments:
    files_counter +=1
    file_size = len(self.request.get(argument))/(1024.0*1024.0)
    if file_size <= 1:
        files.append(MultipartParam('file1', self.request.get(argument), 'file'+str(files_counter)+'.jpg', 'application/x-www-form-urlencoded'))
        # sending file
        data, headers = multipart_encode(files)
        # try 3 times to send the file
        for i in range(3):
            try:
                result = urlfetch.fetch(url=self.request.get('upload_url'), 
                                        payload=''.join(data),
                                        method=urlfetch.POST,
                                        headers=headers,
                                        deadline=10
                                        )              
                break
            except DownloadError:
                logging.error('(POST) ... DownloadError during file upload, attempt ' + str(i))
                pass
            except DeadlineExceededError:
                logging.error('(POST) ... DeadlineExceededError during file upload, attempt ' + str(i))
                pass
        if result.status_code == 200:

and on ~10th file I am getting that DeadlineExceededError.
What can I do to avoid that?

Answer 1

• If you have not done so already; re-factor your code so that you can defer the execution to a task. The deadline for Tasks is 10 minutes vs the 30 seconds for a normal request.
• Further split down your problem so that each file upload is in it’s own task. This also has the added benefit of making your ‘retries’ code redundant as tasks can automatically retry when they fail.
• If each upload takes longer than 10mins, then you will have to look at using Backends to handle the long-running processes.