proxy pool: spread rule34video enrichment over checked public proxies
Rework the unused ALL_PROXIES list into util/proxy_pool.rs: - hourly sweep of the public lists, TLS-verified cloudflare trace checks, >=2 successes with a median under 1.5s, one proxy per exit IP; proxies that show our own IP are dropped, two failures in a row evict - per-site sets checked against a site URL (rule34video: robots.txt), round-robin, 10-min per-site cooldown on 429/challenge pages, live request outcomes fed back - started from main (PROXY_POOL=0 disables; PROXY_POOL_LISTS, PROXY_POOL_CONCURRENCY), /api/proxies shows its status without credentials - wreq "socks" feature enabled: SOCKS proxies silently failed before rule34video enrichment is back, through the pool: full detail pages (1080p..360p formats, tags, artist), 8 concurrent, one 4s deadline per listing; late pages keep running and land in the 2h detail cache, which cached listings are re-merged with on every serve. No ready proxies means no enrichment (no direct fallback). Requester::get_via sends a pooled request with certificate checks, sharing the session's cookie jar. Also adds an ignored live probe showing /embed/ shares the 429 limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AEp724zDpCWNaSeCrMSvjx
This commit is contained in:
@@ -20,7 +20,7 @@ ntex-files = "3"
|
||||
serde = "1.0.228"
|
||||
serde_json = "1.0.145"
|
||||
tokio = { version = "1.49", features = ["full"] }
|
||||
wreq = { version = "6.0.0-rc.31", features = ["cookies", "multipart", "json", "gzip", "brotli", "deflate", "zstd"] }
|
||||
wreq = { version = "6.0.0-rc.31", features = ["cookies", "multipart", "json", "gzip", "brotli", "deflate", "zstd", "socks"] }
|
||||
wreq-util = "3.0.0-rc.14"
|
||||
percent-encoding = "2.3.2"
|
||||
capitalize = "0.3.4"
|
||||
|
||||
Reference in New Issue
Block a user