test: skip network/live crawl by default; fix settings e2e key
- pytest_collection_modifyitems: skip @pytest.mark.network unless PROXYPOOL_RUN_NETWORK_TESTS=1 - Document opt-in in tests/README.md - e2e: replace removed crawl_timeout with validation_timeout Made-with: Cursor
This commit is contained in:
@@ -24,22 +24,24 @@ tests/
|
|||||||
|
|
||||||
## 网络与真实调用
|
## 网络与真实调用
|
||||||
|
|
||||||
集成测试与 E2E **不再 mock** `PluginRunner` / `ValidatorService`:会发起真实 HTTP 爬取与代理验证(视设置而定)。运行全量 `pytest` 需要 **可用的出站网络**,且含 `network` / `slow` 标记的用例可能耗时数分钟。
|
带 `@pytest.mark.network` 的用例会发起真实 HTTP 爬取,**默认不运行**(避免全量 `pytest` 卡住数十分种)。
|
||||||
|
|
||||||
跳过需外网的用例(例如离线快速检查):
|
需要跑外网验收时(PowerShell 示例):
|
||||||
|
|
||||||
|
```powershell
|
||||||
|
$env:PROXYPOOL_RUN_NETWORK_TESTS="1"; pytest tests/integration/test_plugins_live_crawl.py -v --tb=short
|
||||||
|
```
|
||||||
|
|
||||||
|
离线快速检查直接:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
pytest -m "not network"
|
pytest
|
||||||
```
|
```
|
||||||
|
|
||||||
**插件爬取验收**(`test_plugins_live_crawl.py`):
|
**插件爬取验收**(`test_plugins_live_crawl.py`):
|
||||||
|
|
||||||
- 核心 8 插件:必须至少 1 条代理且无 Runner 失败。
|
- 核心 8 插件:必须至少 1 条代理且无 Runner 失败(依赖稳定外网)。
|
||||||
- `fpw_*`:对照 [Free_Proxy_Website](https://github.com/cyubuchen/Free_Proxy_Website) 的公开源,允许 0 条(国际网络差异),使用更长超时。
|
- `fpw_*`:允许部分源 0 条;单插件可能接近各插件 `crawl_timeout_seconds` 上限。
|
||||||
|
|
||||||
```bash
|
|
||||||
pytest tests/integration/test_plugins_live_crawl.py -v
|
|
||||||
```
|
|
||||||
|
|
||||||
## 运行测试
|
## 运行测试
|
||||||
|
|
||||||
|
|||||||
@@ -11,6 +11,26 @@ if sys.platform == "win32":
|
|||||||
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
|
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
|
|
||||||
|
def _network_tests_enabled() -> bool:
|
||||||
|
v = os.environ.get("PROXYPOOL_RUN_NETWORK_TESTS", "").strip().lower()
|
||||||
|
return v in ("1", "true", "yes", "on")
|
||||||
|
|
||||||
|
|
||||||
|
def pytest_collection_modifyitems(config, items) -> None:
|
||||||
|
"""默认跳过 @pytest.mark.network,避免全量 pytest 触发真实爬取卡住数十分种。"""
|
||||||
|
if _network_tests_enabled():
|
||||||
|
return
|
||||||
|
skip = pytest.mark.skip(
|
||||||
|
reason=(
|
||||||
|
"外网/真实爬取用例默认跳过。需要验收时设置环境变量 "
|
||||||
|
"PROXYPOOL_RUN_NETWORK_TESTS=1 后再运行对应文件或 -m network。"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
for item in items:
|
||||||
|
if "network" in item.keywords:
|
||||||
|
item.add_marker(skip)
|
||||||
import pytest_asyncio
|
import pytest_asyncio
|
||||||
from typing import AsyncGenerator
|
from typing import AsyncGenerator
|
||||||
from httpx import AsyncClient, ASGITransport
|
from httpx import AsyncClient, ASGITransport
|
||||||
|
|||||||
@@ -155,9 +155,9 @@ class TestFullWorkflow:
|
|||||||
assert response.status_code == 200
|
assert response.status_code == 200
|
||||||
original_settings = response.json()["data"]
|
original_settings = response.json()["data"]
|
||||||
|
|
||||||
# 2. 修改设置
|
# 2. 修改设置(爬取限时已改为各插件 crawl_timeout_seconds,不再使用全局 crawl_timeout)
|
||||||
new_settings = original_settings.copy()
|
new_settings = original_settings.copy()
|
||||||
new_settings["crawl_timeout"] = 45
|
new_settings["validation_timeout"] = 8
|
||||||
new_settings["auto_validate"] = not original_settings["auto_validate"]
|
new_settings["auto_validate"] = not original_settings["auto_validate"]
|
||||||
|
|
||||||
response = await client.post("/api/settings", json=new_settings)
|
response = await client.post("/api/settings", json=new_settings)
|
||||||
@@ -166,7 +166,7 @@ class TestFullWorkflow:
|
|||||||
# 3. 验证设置已保存
|
# 3. 验证设置已保存
|
||||||
response = await client.get("/api/settings")
|
response = await client.get("/api/settings")
|
||||||
saved_settings = response.json()["data"]
|
saved_settings = response.json()["data"]
|
||||||
assert saved_settings["crawl_timeout"] == 45
|
assert saved_settings["validation_timeout"] == 8
|
||||||
|
|
||||||
# 4. 恢复原始设置
|
# 4. 恢复原始设置
|
||||||
response = await client.post("/api/settings", json=original_settings)
|
response = await client.post("/api/settings", json=original_settings)
|
||||||
|
|||||||
Reference in New Issue
Block a user