si-meng-spec/ProxyPool
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: 全面修复代码问题并优化架构

Browse Source 
修复问题：
- 添加缺失的 httpx 依赖到 requirements.txt
- 修复前端批量删除参数格式与后端不匹配（数组->对象数组）
- 移除 app/api/main.py 中重复创建 app 的冗余代码
- 修复 Plugins.vue v-model 直接修改 store 状态的 Vue 警告
- 修复 README 端口/启动命令文档与实际配置不一致
- 修正 pytest.ini 过时配置 (asyncio_default_fixture_loop_scope)
- 修复 WebUI index.html 语言设置为 zh-CN
- 修复 .gitignore 错误忽略 tests/ 目录

后端优化：
- 修复调度器默认间隔从 5 秒改为 30 分钟，避免无节制验证
- 修复 validate_all_now 在调度器停止时无法执行的 bug
- 设置保存后热更新运行中调度器的验证间隔
- 将 update_score 优化为原子单事务 SQL，消除并发竞态
- 导出功能改为真正的流式分批读取（iter_batches），降低大导出内存占用
- ProxyResponse Schema 补齐 response_time_ms 字段
- 日志级别改为从配置动态读取，不再硬编码 INFO
- 清理 validator_service 中的冗余 try/finally 代码

插件健壮性：
- 修复 ip3366/ip89/kuaidaili/proxylist_download/speedx/yundaili/proxyscrape
  的端口范围检查和 IPv6 地址解析问题（改用 rsplit + 1-65535 校验）
- 修复 PluginService.list_plugins 并发竞争条件
- 修复 run_all_plugins 去重逻辑与数据库 UNIQUE 约束保持一致
- 修复 proxyscrape 异常时错误跳过 fallback 的 bug

测试：
- 新增 7 个插件解析单元测试
- 新增 update_score 自动删除和 iter_batches 流式读取测试
- 全部 74 个测试通过
This commit is contained in:
祀梦
2026-04-04 21:03:43 +08:00
parent 875e61f17e
commit 4ef7931941
27 changed files with 212 additions and 113 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
app/plugins/ip3366.py
View File

@@ -53,8 +53,11 @@ class Ip3366Plugin(BaseHTTPPlugin):
protocol = tds[4].get_text(strip=True).lower() if len(tds) > 4 else "http"
if protocol not in VALID_PROTOCOLS:
protocol = "http"
if re.match(r"^\d+\.\d+\.\d+\.\d+$", ip) and port.isdigit():
results.append(ProxyRaw(ip, int(port), protocol))
if re.match(r"^\d+\.\d+\.\d+\.\d+$", ip) and port.isdigit() and 1 <= int(port) <= 65535:
try:
results.append(ProxyRaw(ip, int(port), protocol))
except ValueError:
continue
if results:
logger.info(f"{self.display_name} 解析完成，获得 {len(results)} 个潜在代理")

7
app/plugins/ip89.py
View File

@@ -34,8 +34,11 @@ class Ip89Plugin(BaseHTTPPlugin):
if len(tds) >= 2:
ip = tds[0].get_text(strip=True)
port = tds[1].get_text(strip=True)
if re.match(r"^\d+\.\d+\.\d+\.\d+$", ip) and port.isdigit():
results.append(ProxyRaw(ip, int(port), "http"))
if re.match(r"^\d+\.\d+\.\d+\.\d+$", ip) and port.isdigit() and 1 <= int(port) <= 65535:
try:
results.append(ProxyRaw(ip, int(port), "http"))
except ValueError:
continue
await asyncio.sleep(random.uniform(1, 2))

7
app/plugins/kuaidaili.py
View File

@@ -61,8 +61,11 @@ class KuaiDaiLiPlugin(BaseHTTPPlugin):
protocol = tds[4].get_text(strip=True).lower() if len(tds) > 4 else "http"
if protocol not in VALID_PROTOCOLS:
protocol = "http"
if re.match(r"^\d+\.\d+\.\d+\.\d+$", ip) and port.isdigit():
results.append(ProxyRaw(ip, int(port), protocol))
if re.match(r"^\d+\.\d+\.\d+\.\d+$", ip) and port.isdigit() and 1 <= int(port) <= 65535:
try:
results.append(ProxyRaw(ip, int(port), protocol))
except ValueError:
continue
await asyncio.sleep(random.uniform(5, 8))
if results:

12
app/plugins/proxylist_download.py
View File

@@ -59,12 +59,14 @@ class ProxyListDownloadPlugin(BaseHTTPPlugin):
line = line.strip()
if not line or ":" not in line:
continue
parts = line.split(":")
if len(parts) >= 2:
ip = parts[0].strip()
port = parts[1].strip()
if ip and port.isdigit():
ip, _, port = line.rpartition(":")
ip = ip.strip()
port = port.strip()
if ip and port.isdigit() and 1 <= int(port) <= 65535:
try:
results.append(ProxyRaw(ip, int(port), protocol))
except ValueError:
continue
return results
async def crawl(self) -> List[ProxyRaw]:

14
app/plugins/proxyscrape.py
View File

@@ -42,12 +42,14 @@ class ProxyScrapePlugin(BaseHTTPPlugin):
line = line.strip()
if not line or ":" not in line:
continue
parts = line.split(":")
if len(parts) >= 2:
ip = parts[0].strip()
port_str = parts[1].strip()
if port_str.isdigit():
ip, _, port_str = line.rpartition(":")
ip = ip.strip()
port_str = port_str.strip()
if port_str.isdigit() and 1 <= int(port_str) <= 65535:
try:
proxies.append(ProxyRaw(ip, int(port_str), protocol))
except ValueError:
continue
return proxies
async def crawl(self) -> List[ProxyRaw]:
@@ -71,7 +73,7 @@ class ProxyScrapePlugin(BaseHTTPPlugin):
htmls.append("")
except Exception:
htmls.append("")
done_protocols.add(protocols[i])
# 异常时不加入 done_protocols，以便触发 API fallback
fallback_protocols = []
for protocol, html in zip(protocols, htmls):

18
app/plugins/speedx.py
View File

@@ -42,15 +42,17 @@ class SpeedXPlugin(BaseHTTPPlugin):
line = line.strip()
if not line or ":" not in line:
continue
parts = line.split(":")
if len(parts) >= 2:
ip = parts[0].strip()
port = parts[1].strip()
if not re.match(r"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$", ip):
continue
if not port.isdigit() or not (1 <= int(port) <= 65535):
continue
ip, _, port = line.rpartition(":")
ip = ip.strip()
port = port.strip()
if not re.match(r"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$", ip):
continue
if not port.isdigit() or not (1 <= int(port) <= 65535):
continue
try:
results.append(ProxyRaw(ip, int(port), protocol))
except ValueError:
continue
return results
async def crawl(self) -> List[ProxyRaw]:

13
app/plugins/yundaili.py
View File

@@ -40,17 +40,18 @@ class YunDaiLiPlugin(BaseHTTPPlugin):
line = line.strip()
if not line or ":" not in line:
continue
parts = line.split(":")
if len(parts) < 2:
continue
ip = parts[0].strip()
port_str = parts[1].strip()
ip, _, port_str = line.rpartition(":")
ip = ip.strip()
port_str = port_str.strip()
if not re.match(r"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$", ip):
continue
if not port_str.isdigit() or not (1 <= int(port_str) <= 65535):
continue
final_protocol = protocol if protocol in VALID_PROTOCOLS else "http"
results.append(ProxyRaw(ip, int(port_str), final_protocol))
try:
results.append(ProxyRaw(ip, int(port_str), final_protocol))
except ValueError:
continue
count += 1
if count: