Introduction
Deferred
objects are
signals that a function you have called does not yet have the data you want
available. When a function returns a Deferred object, your calling function
attaches callbacks to it to handle the data when available.
This document addresses the other half of the question: writing functions that return Deferreds, that is, constructing Deferred objects, arranging for them to be returned immediately without blocking until data is available, and firing their callbacks when the data is available.
Assumed knowledge
This document assumes that you are familiar with the asynchronous model used by Twisted, and with using deferreds returned by functions.
Class overview
This is an overview API reference for Deferred from the point of creating a Deferred and firing its callbacks and errbacks. It is not meant to be a substitute for the docstrings in the Deferred class, but can provide guidelines for its use.
There is a parallel overview of functions used by calling function which the Deferred is returned to at Using Deferreds.
Basic Callback Functions
callback(result)
Run success callbacks with the given result. This can only be run once. Later calls to this or
errback
will raisetwisted.internet.defer.AlreadyCalledError
. If further callbacks or errbacks are added after this point, addCallbacks will run the callbacks immediately.errback(failure)
Run error callbacks with the given failure. This can only be run once. Later calls to this or
callback
will raisetwisted.internet.defer.AlreadyCalledError
. If further callbacks or errbacks are added after this point, addCallbacks will run the callbacks immediately.
What Deferreds don't do: make your code asynchronous
Deferreds do not make the code magically not block.
Let's take this function as an example:
from twisted.internet import defer TARGET = 10000 def largeFibonnaciNumber(): # create a Deferred object to return: d = defer.Deferred() # calculate the ten thousandth Fibonnaci number first = 0 second = 1 for i in xrange(TARGET - 1): new = first + second first = second second = new if i % 100 == 0: print "Progress: calculating the %dth Fibonnaci number" % i # give the Deferred the answer to pass to the callbacks: d.callback(second) # return the Deferred with the answer: return d import time timeBefore = time.time() # call the function and get our Deferred d = largeFibonnaciNumber() timeAfter = time.time() print "Total time taken for largeFibonnaciNumber call: %0.3f seconds" % (timeAfter - timeBefore) # add a callback to it to print the number def printNumber(number): print "The %dth Fibonacci number is %d" % (TARGET, number) print "Adding the callback now." d.addCallback(printNumber)
You will notice that despite creating a Deferred in the
largeFibonnaciNumber
function, these things happened:
- the "Total time taken for largeFibonnaciNumber call" output shows that the function did not return immediately as asynchronous functions are expected to do; and
- rather than the callback being added before the result was available and called after the result is available, it isn't even added until after the calculation has been completed.
The function completed its calculation before returning, blocking the process until it had finished, which is exactly what asynchronous functions are not meant to do. Deferreds are not a non-blocking talisman: they are a signal for asynchronous functions to use to pass results onto callbacks, but using them does not guarantee that you have an asynchronous function.
Technical Details
Deferreds greatly simplify the process of writing asynchronous code by providing a standard for registering callbacks, but there are some subtle and sometimes confusing rules that you need to follow if you are going to use them. This mostly applies to people who are writing new systems that use Deferreds internally, and not writers of applications that just add callbacks to Deferreds produced and processed by other systems. Nevertheless, it is good to know.
Deferreds are one-shot. A generalization of the Deferred API to generic
event-sources is in progress -- watch this space for updates! -- but Deferred
itself is only for events that occur once. You can only call
Deferred.callback
or Deferred.errback
once. The
processing chain continues each time you add new callbacks to an
already-called-back-to Deferred.
The important consequence of this is that sometimes, addCallbacks
will call its argument synchronously, and sometimes it will not. In
situations where callbacks modify state, it is highly desirable for the chain
of processing to halt until all callbacks are added. For this, it is possible
to pause
and unpause
a Deferred's processing chain while you are adding lots of callbacks.
Be careful when you use these methods! If you pause
a Deferred,
it is your responsibility to make sure that you unpause it; code that
calls callback
or errback
should
never call unpause
, as this would negate its
usefulness!
Advanced Processing Chain Control
pause()
Cease calling any methods as they are added, and do not respond to
callback
, untilself.unpause()
is called.unpause()
If
callback
has been called on this Deferred already, call all the callbacks that have been added to this Deferred sincepause
was called.Whether it was called or not, this will put this Deferred in a state where further calls to
addCallbacks
orcallback
will work as normal.
Returning Deferreds from synchronous functions
Sometimes you might wish to return a Deferred from a synchronous function. There are several reasons why, the major two are maintaining API compatibility with another version of your function which returns a Deferred, or allowing for the possiblity that in the future your function might need to be asynchronous.
In the Using Deferreds reference, we gave the following example of a synchronous function:
def synchronousIsValidUser(user): ''' Return true if user is a valid user, false otherwise ''' return user in ["Alice", "Angus", "Agnes"]
While we can require that callers of our function wrap our synchronous
result in a Deferred using maybeDeferred
, for the sake of API
compatibility it is better to return a Deferred ourself using defer.succeed
:
from twisted.internet import defer def immediateIsValidUser(user): ''' Returns a Deferred resulting in true if user is a valid user, false otherwise ''' result = user in ["Alice", "Angus", "Agnes"] # return a Deferred object already called back with the value of result return defer.succeed(result)
There is an equivalent defer.fail
method to return a Deferred with the
errback chain already fired.
See also
- twisted.flow, a mechanism for interpolating computationally intensive tasks without threading.