Lines Matching refs:Crawler
74 multi object of the crawler object so that the corresponding pages gets
313 class Crawler(object):
328 """Init crawler URL, links lists, logger, and creates a cookie temp file.
371 """Deletes cookie file when Crawler instances are destroyed."""
511 Links are copied to the links list of the crawler object, which holds all
512 the links found from all retrievers that the crawler object created. The
513 Crawler object exists as far as a specific site is examined and the
538 """Runs the Crawler.
548 If no registration page is found, the Crawler object will give up its try
604 url: will be used as an argument to create a Crawler object later.
611 """Execution of thread creates a Crawler object and runs it.
617 self.page_found = Crawler(self._url).Run()
621 """Calls the Run function of WorkerThread which creates & runs a Crawler obj.
623 The crawler object runs concurrently, examining one site each.
628 """Creates threaded Crawler objects.
661 """Runs Crawler objects using python threads.
678 self.logger.info('URL fed to a crawler thread: %s', url)
752 CrawlerClass = Crawler
