[Top]
Web
Web.Crawler
Web.Crawler.Policy
|
Class Web.Crawler.Policy
- Description
-
The crawler policy object.
- Variable
max_concurrent_fetchers
-
int max_concurrent_fetchers
- Description
-
Maximum number of fetchers. Defaults to 100.
- Variable
max_bits_per_second_total
-
int max_bits_per_second_total
- Description
-
Maximum number of bits per second. Defaults to off (0).
- Variable
max_bits_per_second_per_host
-
int max_bits_per_second_per_host
- Description
-
Maximum number of bits per second, per host. Defaults to off (0).
- Variable
bandwidth_throttling_floating_window_width
-
int bandwidth_throttling_floating_window_width
- Description
-
Bandwidth throttling floating window width. Defaults to 30.
- Variable
max_concurrent_fetchers_per_host
-
int max_concurrent_fetchers_per_host
- Description
-
Maximum concurrent fetchers per host. Defaults to 1.
- Variable
min_delay_per_host
-
int min_delay_per_host
- Description
-
Minimum delay per host. Defaults to 0.
|