[Top]
Web
Web.Crawler
Web.Crawler.Crawler
|
Method Web.Crawler.Crawler()->create()
- Method
create
-
void Web.Crawler.Crawler(Queue _queue, function _page_cb, function _error_cb, function _done_cb, function _prepare_cb, string|array(string)|Standards.URI|array(Standards.URI) start_uri, mixed ... _args)
- Parameter _page_cb
-
function called when a page is retreived. Arguments are:
Standards.URI uri, mixed data, mapping headers, mixed ... args.
should return an array containing additional links found within data
that will be analyzed for insertion into the crawler queue (assuming
they are allowed by the allow/deny rulesets.
- Parameter _error_cb
-
function called when an error is received from a server. Arguments are:
Standards.URI real_uri, int status_code, mapping headers,
mixed ... args. Returns void.
- Parameter _done_cb
-
function called when crawl is complete. Accepts mixed ... args and
returns void.
- Parameter _prepare_cb
-
argument called before a uri is retrieved. may be used to alter
the request. Argument is Standards.URI uri. Returns array with
element 0 of Standards.URI uri, element 1 is a header mapping for the
outgoing request.
- Parameter start_uri
-
location to start the crawl from.
- Parameter _args
-
optional arguments sent as the last argument to the callback
functions.
|