refactor(crawl): parallel plugins via JobExecutor; per-plugin throttle
- Remove global crawl_slot gate; all CrawlJobs share only executor semaphore - max_concurrent_jobs = max(24, n_plugins+8) for crawl-all + aggregator headroom - BaseHTTPPlugin max_concurrency 3->2; fpw multi-URL plugins 4->2 - fetch_all: short random delay before each request to ease single-host pressure Made-with: Cursor
This commit is contained in:
@@ -13,7 +13,7 @@ class FpwProxyListDownloadPlugin(BaseHTTPPlugin):
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.max_concurrency = 4
|
||||
self.max_concurrency = 2
|
||||
self.api_pairs = [
|
||||
("http", "https://www.proxy-list.download/api/v1/get?type=http"),
|
||||
("https", "https://www.proxy-list.download/api/v1/get?type=https"),
|
||||
|
||||
Reference in New Issue
Block a user