si-meng-spec/ProxyPool
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(crawl): throttle concurrent CrawlJobs and relax fpw/proxyscrape HTTP

Browse Source 
- CrawlJob waits on crawl_slot before JobExecutor semaphore so crawl-all does not fill slots while queued
- BaseHTTPPlugin: longer connect budget for slow international links
- proxyscrape: jsDelivr mirror + longer GitHub/API phases
- fpw_*: higher timeouts/retries; lower internal concurrency on heavy multi-URL plugins

Made-with: Cursor
This commit is contained in:
祀梦
2026-04-05 13:48:41 +08:00
parent 2c98abaf91
commit 957cee3100
13 changed files with 116 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
app/plugins/fpw_spys_one.py
View File

@@ -130,7 +130,7 @@ class FpwSpysOnePlugin(BaseHTTPPlugin):
async def _one(proto: str, url: str, xf5: str) -> Tuple[str, str]:
data = {**form_base, "xf5": xf5}
html = await self.fetch_post(url, data=data, timeout=14, retries=1)
html = await self.fetch_post(url, data=data, timeout=25, retries=2)
return proto, html or ""
pairs = await asyncio.gather(