Simon 29d526b5fc proxy pool: spread rule34video enrichment over checked public proxies ...

Rework the unused ALL_PROXIES list into util/proxy_pool.rs:
- hourly sweep of the public lists, TLS-verified cloudflare trace checks,
  >=2 successes with a median under 1.5s, one proxy per exit IP; proxies
  that show our own IP are dropped, two failures in a row evict
- per-site sets checked against a site URL (rule34video: robots.txt),
  round-robin, 10-min per-site cooldown on 429/challenge pages, live
  request outcomes fed back
- started from main (PROXY_POOL=0 disables; PROXY_POOL_LISTS,
  PROXY_POOL_CONCURRENCY), /api/proxies shows its status without credentials
- wreq "socks" feature enabled: SOCKS proxies silently failed before

rule34video enrichment is back, through the pool: full detail pages
(1080p..360p formats, tags, artist), 8 concurrent, one 4s deadline per
listing; late pages keep running and land in the 2h detail cache, which
cached listings are re-merged with on every serve. No ready proxies means
no enrichment (no direct fallback). Requester::get_via sends a pooled
request with certificate checks, sharing the session's cookie jar.

Also adds an ignored live probe showing /embed/ shares the 429 limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AEp724zDpCWNaSeCrMSvjx

2026-11-05 21:59:28 +00:00

1.8 KiB

Raw Blame History

View Raw