This document describes Scrapy . For development docs, go here.
Scrapy uses signals extensively to notify when certain events occur. You can catch some of those signals in your Scrapy project (using an extension, for example) to perform additional tasks or extend Scrapy to add functionality not provided out of the box.
Even though signals provide several arguments, the handlers that catch them don’t need to accept all of them - the signal dispatching mechanism will only deliver the arguments that the handler receives.
Finally, for more detailed information about signals internals see the documentation of pydispatcher (the which the signal dispatching mechanism is based on).
Some signals support returning Twisted deferreds from their handlers, see the Built-in signals reference below to know which ones.
Here’s the list of Scrapy built-in signals and their meaning.
Sent when the Scrapy engine is started (for example, when a crawling process has started).
This signal supports returning deferreds from their handlers.
Sent when the Scrapy engine is stopped (for example, when a crawling process has finished).
This signal supports returning deferreds from their handlers.
Sent when the engine receives a new scraped item from the spider, and right before the item is sent to the Item Pipeline.
This signal supports returning deferreds from their handlers.
Parameters: |
|
---|
Sent after an item has passed all the Item Pipeline stages without being dropped. Same as item_scraped() if there are no pipelines enabled.
This signal supports returning deferreds from their handlers.
Parameters: |
|
---|
Sent after an item has been dropped from the Item Pipeline when some stage raised a DropItem exception.
This signal supports returning deferreds from their handlers.
Parameters: |
|
---|
Sent after a spider has been closed. This can be used to release per-spider resources reserved on spider_opened.
This signal supports returning deferreds from their handlers.
Parameters: |
|
---|
Sent after a spider has been opened for crawling. This is typically used to reserve per-spider resources, but can be used for any task that needs to be performed when a spider is opened.
This signal supports returning deferreds from their handlers.
Parameters: | spider (BaseSpider object) – the spider which has been opened |
---|
Sent when a spider has gone idle, which means the spider has no further:
- requests waiting to be downloaded
- requests scheduled
- items being processed in the item pipeline
If the idle state persists after all handlers of this signal have finished, the engine starts closing the spider. After the spider has finished closing, the spider_closed signal is sent.
You can, for example, schedule some requests in your spider_idle handler to prevent the spider from being closed.
This signal does not support returning deferreds from their handlers.
Parameters: | spider (BaseSpider object) – the spider which has gone idle |
---|
Sent when the engine receives a Request from a spider.
This signal does not support returning deferreds from their handlers.
Parameters: |
|
---|
Sent when the engine receives a new Response from the downloader.
This signal does not support returning deferreds from their handlers.
Parameters: |
|
---|
Sent by the downloader right after a HTTPResponse is downloaded.
This signal does not support returning deferreds from their handlers.
Parameters: |
|
---|