Question

self.agent = Agent(reactor, pool=pool)
self.deferred = self.agent.request(
            'GET',
            self.url,
            Headers({'User-Agent': ['Mozilla/5.0']})
        )

self.deferred.addCallback(self.gotResponse)

But gotResponse called for every part of received data, not for all of it. I could collect it, but how to know that i got all the data?

EDIT:

I found this (from the words “If the response body has been completely received“) but still have no idea how to implement this. I mean, what does “the failure will wrap…” mean?

Answer 1

The twisted documentation provides an example of how to do this.

From http://twistedmatrix.com/documents/current/web/howto/client.html

from pprint import pformat

from twisted.internet import reactor
from twisted.internet.defer import Deferred
from twisted.internet.protocol import Protocol
from twisted.web.client import Agent
from twisted.web.http_headers import Headers

class BeginningPrinter(Protocol):
    def __init__(self, finished):
        self.finished = finished
        self.remaining = 1024 * 10

    def dataReceived(self, bytes):
        if self.remaining:
            display = bytes[:self.remaining]
            print 'Some data received:'
            print display
            self.remaining -= len(display)

    def connectionLost(self, reason):
        print 'Finished receiving body:', reason.getErrorMessage()
        self.finished.callback(None)

agent = Agent(reactor)
d = agent.request(
    'GET',
    'http://example.com/',
    Headers({'User-Agent': ['Twisted Web Client Example']}),
    None)

def cbRequest(response):
    print 'Response version:', response.version
    print 'Response code:', response.code
    print 'Response phrase:', response.phrase
    print 'Response headers:'
    print pformat(list(response.headers.getAllRawHeaders()))
    finished = Deferred()
    response.deliverBody(BeginningPrinter(finished))
    return finished
d.addCallback(cbRequest)

def cbShutdown(ignored):
    reactor.stop()
d.addBoth(cbShutdown)

reactor.run()

The connectionLost() method of BeginningPrinter will be called when the request has completed.

Response version: ('HTTP', 1, 0)
Response code: 302
Response phrase: Found
Response headers:
[('Location', ['http://www.iana.org/domains/example/']), ('Server', ['BigIP'])]
Finished receiving body: Response body fully received

It looks like checking if reason.check(twisted.web.client.ResponseDone) will tell you whether or not it was successful.