simon/hottub
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

proxy pool: spread rule34video enrichment over checked public proxies

Browse Source 
Rework the unused ALL_PROXIES list into util/proxy_pool.rs:
- hourly sweep of the public lists, TLS-verified cloudflare trace checks,
  >=2 successes with a median under 1.5s, one proxy per exit IP; proxies
  that show our own IP are dropped, two failures in a row evict
- per-site sets checked against a site URL (rule34video: robots.txt),
  round-robin, 10-min per-site cooldown on 429/challenge pages, live
  request outcomes fed back
- started from main (PROXY_POOL=0 disables; PROXY_POOL_LISTS,
  PROXY_POOL_CONCURRENCY), /api/proxies shows its status without credentials
- wreq "socks" feature enabled: SOCKS proxies silently failed before

rule34video enrichment is back, through the pool: full detail pages
(1080p..360p formats, tags, artist), 8 concurrent, one 4s deadline per
listing; late pages keep running and land in the 2h detail cache, which
cached listings are re-merged with on every serve. No ready proxies means
no enrichment (no direct fallback). Requester::get_via sends a pooled
request with certificate checks, sharing the session's cookie jar.

Also adds an ignored live probe showing /embed/ shares the 429 limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AEp724zDpCWNaSeCrMSvjx
This commit is contained in:
Simon
2026-11-05 21:59:28 +00:00
parent 4f83dae3fa
commit 29d526b5fc
11 changed files with 1204 additions and 211 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
SKILL.md
View File

@@ -64,6 +64,9 @@ Runtime env vars:
- `PROXY` optional. Any value other than `"0"` enables proxy mode in the shared requester.
- `BURP_URL` optional. Outbound HTTP proxy used when `PROXY` is enabled.
- `FLARE_URL` optional but strongly recommended for provider work. Used for FlareSolverr fallback and some providers that call it directly.
- `PROXY_POOL` optional. `0` disables the public-proxy pool used by rule34video enrichment (default on).
- `PROXY_POOL_LISTS` optional. Comma-separated proxy list URLs replacing the default TheSpeedX lists.
- `PROXY_POOL_CONCURRENCY` optional. Cap on concurrent proxy checks (default 500).
- `DOMAIN` optional. Used for the `/` redirect target.
- `DISCORD_WEBHOOK` optional. Enables `/api/test` and provider error reporting to Discord.