Compare commits

...

28 Commits

Author SHA1 Message Date
Simon
cf0ae15f93 hanime: play through a redirect proxy; the signed stream links expire
Items now carry /proxy/hanime/{slug}/{height}.m3u8, which does a fresh
handshake per play and 302s to the current playlist. The handshake moved
to util/hanime_resolve.rs so single-provider builds keep the route.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-11-01 07:48:35 +00:00
Simon
aa00abdd5b rule34video: probe the space-less artist slug (e.g. /models/biggreen/) for multi-word queries
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-27 21:45:09 +00:00
Simon
e5a2afcc70 eporner: stream through the server; the signed CDN links are IP-locked
The redirect proxy from 6ba65f1 still played the one-minute "watch on
eporner.com" clip away from home: checked from foreign networks
(check-host.net nodes in BR/DE/HU), a freshly signed link answers 302 ->
static.eporner.com/na.mp4, while the IP that resolved it gets the file. A
redirect therefore only works for a player behind the server's public IP.

/proxy/eporner/{alnum_id}/{height}.mp4 now streams the mp4: resolves the
link (cached 90 min, re-resolved when the CDN answers with a redirect),
forwards Range, and returns a sized body so 206 responses keep
Content-Length. It is no longer a redirect proxy, so app clients keep the
URLs in formats. Verified locally: HEAD, 0-1, mid-file and open-ended ranges
return 206 with Content-Length/Content-Range and valid MP4 data.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-22 20:46:14 +00:00
Simon
6ba65f1457 eporner: play through a redirect proxy; the signed CDN links expire after 2h
Expired eporner mp4 links are answered with a 302 to static.eporner.com/
na.mp4, a one-minute "watch it on eporner.com" clip. The links expire
exactly two hours after the /xhr/video call, so anything played from a
favourite, history entry or a list left open past that hit the decoy.

New /proxy/eporner/{alnum_id}/{height} re-resolves the hash/xhr chain at
play time and 302s to the matching (else best) quality. The eporner
provider and melonstube's eporner destinations now emit these proxy URLs;
verified app clients get the best quality as video.url via the existing
redirect-proxy rule. Verified live: proxy -> fresh CDN link -> 206 video/mp4
for app and non-app clients.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-22 18:09:23 +00:00
Simon
b131910ea2 rule34video: make queries robust (single-page results, safe search paths, 429-proof enrichment)
- Parser no longer requires a pagination block: single-page results
  (small tags, short searches) used to parse to zero items.
- Search path is slugged like the site's own form: "2b/9s" 404'd, "%20"
  is refused with 403 and "&" truncated the query.
- Probed archives need >= 4 videos to beat keyword search ("elf girl" was a
  one-video tag); conclusive query -> target resolutions are remembered for 6h
  so later pages and the prefetch skip the probes.
- Enriched details are cached per video URL for 2h (tokens last ~12h);
  detail fetches go through get_probe so a lasting 429 stops the batch instead
  of escalating each page to Jina/FlareSolverr. Archive failures serve stale
  cache.
- requester: get_probe waits before retrying 429/5xx and compares redirect
  paths percent-decoded (non-ASCII archives like /tags/pokémon-species/ were
  treated as redirected away); get_with_headers backs off 1/2/4s on 429.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-21 20:17:58 +00:00
Simon
921ad93149 requester: treat DDoS-Guard interstitials as challenges (FlareSolverr fallback); mock-origin test
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-19 20:38:17 +00:00
Simon
ad2abe0057 rule34video: artist, category and tag archives; resolve bare queries by probing; credit artists as uploader
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-19 17:06:00 +00:00
Simon
0aa8929f29 Docs: the redirect-proxy rule applies to verified Hot Tub app clients only
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-19 07:53:08 +00:00
Simon
6c69b46395 Move redirecting-proxy formats to video.url for verified app clients; document the rule
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-19 07:48:34 +00:00
Simon
f900987434 archivebate: verified app clients get the proxy URL as item url and no formats
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-19 07:39:48 +00:00
Simon
30271b9b7c archivebate: app clients get resolved media URLs with headers, resolved concurrently
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-18 19:30:55 +00:00
Simon
a4364e5be3 archivebate: keep the proxied format when enrichment times out; skip dead search candidates
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-17 19:54:49 +00:00
Simon
edf6732f8e Query resolution todo, slug helper tests, fix txxx decoder test fixture
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-17 06:19:18 +00:00
Simon
697836b914 coomer: creator feeds are a bare array; single-word queries that are creator ids go to the creator 2026-10-17 06:16:24 +00:00
Simon
5598216381 sextb: probe actress/studio for names outside the loaded catalogs 2026-10-17 06:07:34 +00:00
Simon
4c29d27b3e redgifs: single-word queries that are creator names go to the creator 2026-10-17 06:05:54 +00:00
Simon
a486bc5502 erome: single-word queries that are uploader names go to the uploader 2026-10-17 06:03:58 +00:00
Simon
270b327f73 fikfap: single-word queries that are profile names go to the creator 2026-10-17 06:01:34 +00:00
Simon
813e2ec2ca clapdat: resolve tag/user names without prefix 2026-10-17 05:58:53 +00:00
Simon
26ef830f6e pornhub: probe pornstar/model/channel for unknown free-form queries 2026-10-17 05:56:31 +00:00
Simon
e34801f6f8 thepornbunny: resolve pornstar/studio/tag names; get_probe treats redirect-away as a miss 2026-10-17 05:53:43 +00:00
Simon
2646ca5659 xgroovy: resolve pornstar/channel names without prefix 2026-10-17 05:51:40 +00:00
Simon
981ec29ad8 xhamster: resolve pornstar/channel names without prefix 2026-10-17 05:50:41 +00:00
Simon
76b9d4e058 xvideos: resolve channel/pornstar names without prefix; channel archives via the JSON video list 2026-10-17 05:48:58 +00:00
Simon
59ea0495a6 porn4fans: resolve model/tag names without prefix 2026-10-17 05:40:28 +00:00
Simon
1f95549167 youporn: resolve pornstar/channel/amateur names without prefix 2026-10-17 05:38:47 +00:00
Simon
1e56fe2da5 tube8: resolve pornstar/channel names without prefix 2026-10-17 05:37:07 +00:00
Simon
bdde02fa8c Add get_probe/probe_first_archive; fullporner and fullpornerorg resolve pornstar names without prefix 2026-10-17 05:34:15 +00:00
37 changed files with 2226 additions and 366 deletions

View File

@@ -59,7 +59,7 @@ This is the current implementation inventory as of this snapshot of the repo. Us
| `redtube` | `mainstream-tube` | no | no | Mainstream archive. |
| `ripnsfw` | `onlyfans` | no | yes | Leaked-creator aggregator for ripnsfw.com — the entire catalogue (~2.5k rows) is a single client-side-rendered page whose data source is a published Google Sheet exported as CSV (`GET /data.csv`, no auth/CF gate); the provider fetches and parses that CSV directly instead of scraping any HTML, replicating the site's own JS `parseCSV`/`csvSplit` column layout (`THUMB,NEW,NOME,DOODSTREAM,LULUSTREAM,<link3>,DOOD,<guests>,SITE(bunkr links),TITLE,ID,#(tags),POST,,DATE,SIZE`). Feed/search/pagination/sort are all done in-memory over the fetched rows (no native site pagination exists — `perPage`-sized pages are sliced locally), with a 180s in-process cache to avoid re-fetching the CSV on every request. `tag:`/`category:`/`cat:` and `model:`/`uploader:` query prefixes filter by the parsed tag list / model name; bare keywords substring-match model name, guests, and tags. `video.url` is the site's client-rendered `https://ripnsfw.com/model/{slug}/{id}` route — the host has no server-side SPA fallback, so this 404s to any plain HTTP client (its custom 404 page stashes the path and JS-redirects to `/`, which restores the route — works in a real browser/webview only); `check.py`'s `_CF_PROTECTED_HOSTS` allowlist includes `ripnsfw.com` for this reason. Media is never hosted on ripnsfw.com itself: each row carries a Doodstream and/or Lulustream embed link, resolved into `formats[]` (Lulustream first, matching the site's own default-player order) via the **existing, previously-unused** `src/proxies/doodstream.rs` / `src/proxies/lulustream.rs` redirect proxies (`build_proxy_url(&options, "doodstream"/"lulustream", &strip_url_scheme(link))`) — no new proxy code was written. Both proxies resolve correctly to real signed CDN URLs (verified against live data), but neither CDN's final edge is reachable from this repo's dev sandbox: Lulustream's `*.tnmr.org` edge 403s datacenter/proxy IPs regardless of UA or TLS impersonation, and Doodstream's final `*.dood.video` edge resolves to a loopback/refused address from non-residential egress — both are IP-reputation-style CDN blocks, not code defects (same class of sandbox limitation already documented for `camsoda`/`hentaihaven`/`animeidhentai`), so `check.py` allowlists both by suffix (`_CF_PROTECTED_SUFFIXES`). No thumbnail proxy needed (`cdn.ripnsfw.com/thumbs/...` is directly hotlinkable, no CF/referer gate). No `/api/uploaders` (model pages exist but have no dedicated profile/stats schema confirmed). |
| `rule34gen` | `ai` | no | no | AI group example. |
| `rule34video` | `hentai-animation` | no | no | Hentai group example. |
| `rule34video` | `hentai-animation` | no | no | Hentai group example. **Archives:** artists are `/models/{slug}/`, franchises/genres `/categories/{slug}/`, characters and everything else `/tags/{slug}/`; each is paged and sorted through `?mode=async&function=get_block&block_id=custom_list_videos_common_videos&sort_by={sort}&from={page}` (unknown slugs 404). Prefix queries `artist:`/`model:`/`creator:`, `category:`/`franchise:`/`series:`, `tag:`/`character:`; a bare query is probed as artist, then category, then tag, then resolved through `/search_ajax.php?tag=` (characters are tagged `name (franchise)`; an unambiguous exact base-name match wins), else keyword search. Cards are enriched from the detail page, which also yields the credited artist as `uploader` (`rule34video:{slug}`) and artist names as tags. |
| `sextb` | `jav` | no | no | JAV family provider. sextb.net is Cloudflare-gated, so listing/pagination markdown is fetched through a hand-built Jina mirror URL (`mirror_url()`); the target is already an absolute `https://sextb.net/...` URL, so the scheme must be stripped before prepending the `https://r.jina.ai/http://` prefix — concatenating them directly (the pre-fix bug) produces a malformed `.../http://https://...` URL that Jina rejects outright. |
| `shooshtime` | `onlyfans` | no | yes | Redirect proxy plus dedicated media route. Listing cards are `div.card` (card→`a.title` for the link/title, `a.image img` for thumb/preview, `div.duration`, `div.list div.box` for views+date, `div.name a.text` for uploader) — the site redesigned from an older `div.item.thumb`/`span.description` layout, so re-verify these selectors against `{BASE_URL}/videos/` (the real default-sort URL, not the bare homepage) if the feed ever goes empty again. |
| `spankbang` | `mainstream-tube` | no | yes | Best template for redirect proxy plus anti-bot fetches. |
@@ -83,7 +83,7 @@ This is the current implementation inventory as of this snapshot of the repo. Us
| `fullporner` | `mainstream-tube` | no | no | HTML scraper for fullporner.com; thumbnail IDs derived from `/thumb/{id}.jpg` URLs and used to build direct `xiaoshenke.net/vid/{id}/720` media redirect URLs (Referer + User-Agent headers required); supports cat:/category:/pornstar:/star: shortcut queries; no proxy needed. |
| `cumlouder` | `mainstream-tube` | no | no | HTML scraper for cumlouder.com (Spanish/amateur tube, same platform family as superporn). Cards are `a.muestra-escena[href^="/porn-video/"]` with lazy thumb in `img[data-src]` (protocol-relative `//th0cdn.cumlouder.com/...`, direct, no Referer/proxy), `h2` title, `.vistas` views, `.minutos` duration (`06:14 m`); relative date only ("2 years ago"), so `uploadedAt` is not set. Default feed `/porn/` (page N≥2 is the bare `/N/` path, not `/porn/N/`); category `/porn-videos/{slug}/{N}/`; tag `/tag/{slug}/{N}/`; series `/series/{slug}/{N}/`. **The site's own `/search/` (POST `form[search]` or `?q=`) is behind a Cloudflare managed challenge — 403 even in a real browser — so keyword queries are routed instead: a query matching a hardcoded category (72, scraped from `/categories/`) goes to the category archive, anything else to `/tag/{slug}/` (unknown tags 404 → empty list).** Pornstar pages live at `/girl/{slug}/{N}/` (same cards/pagination; unknown slug = fast 404). A free-form keyword that is not a category is tried as a pornstar page first, then as a tag archive; `girl:`/`pornstar:`/`star:`/`model:` force the pornstar page. Supports `cat:`/`category:`, `tag:`, `series:` prefixes and the `categories` option. A Cloudflare challenge on the raw listing fetch is handed to the requester's FlareSolverr chain. `video.url` is the detail page, which yt-dlp resolves natively via the generic html5 extractor (the page embeds a signed `m5cdnst.cumlouder.com/*.mp4?secure=…` `<source>`, no Referer needed), so no `formats`/enrichment. The origin intermittently answers a first, uncached request with a Cloudflare 522 (a retry succeeds) — the listing fetch retries up to 3×, and `check.py`/yt-dlp may show a one-off `HTTP=-1`/522 on a cold detail page that is not a provider fault. No proxy needed. |
| `thepornbunny` | `mainstream-tube` | no | yes | KVS-style HTML scraper for thepornbunny.com; 24 items per site page; thumbnails at `https://www.thepornbunny.com/images/thumb/{id}.webp` from `data-original` attribute (no proxy needed); studio exposed as uploader; pornstar names in tags; `/proxy/thepornbunny/{slug}` fetches the video page, extracts `generate_mp4(enc_data, key, rnd, video_id)` args, decrypts `enc_data` via PBKDF2-HMAC-SHA512+AES-256-CBC to get an OK.ru session key, calls `api.ok.ru/fb.do?method=video.get&session_key=KEY&vids=RND` to get signed CDN URLs, and returns 302 to the best-quality okcdn.ru/vkuser.net MP4 URL (no special client headers needed); supports sort: new/popular/rated, 20 hardcoded categories via `categories` option, and tag:/category:/studio:/pornstar: query shortcuts. |
| `eporner` | `mainstream-tube` | no | no | HTML scraper for eporner.com (5M+ videos); card selector `div.mb[data-id]` with inline duration/rating/views/uploader; thumbnails at `static-eu-cdn.eporner.com` (no proxy needed); pagination uses `/{N}/` suffix (page 1 = no suffix, page 2 = `/2/`); supports sort: new/popular/rated/best; 65 hardcoded categories, plus `cat:`/`category:`, `tag:`, `pornstar:`/`star:`, `uploader:`/`profile:` query-prefix routing to the matching archive, a bare-keyword category-name match, and a background-loaded pornstar name→URL map from `/pornstar-list/` (falls back to `/tag/{slug}` keyword search otherwise). `video.url` is the detail page — **not** yt-dlp-resolvable and not directly playable on its own: opening it bare serves a small restricted/preview clip. Real playback needs per-card enrichment (bounded `buffer_unordered(6)`): fetch the detail page for its 32-hex `hash`, derive the site's own hashed token (`calc_hash` — split into four 8-hex chunks, each parsed as u32 and base36-encoded, concatenated; reverse-engineered from the player JS), then call the site's own `GET /xhr/video/{alnum_id}?hash={calc}&device=generic&domain=www.eporner.com&fallback=false` JSON API for direct per-resolution mp4 URLs into `formats[]`. Separately from the hash dance, the resolved CDN mp4 URLs enforce their own Referer check: a request with no Referer or one pointing at eporner.com gets the real file, but any other non-empty Referer gets a 302 to a static `.../na.mp4` ("not available") decoy clip — the exact "this video is only available at eporner.com"-style short clip a client sees if its default (or app-injected) Referer doesn't match, so every format explicitly declares `http_headers: {Referer: "https://www.eporner.com/"}` rather than relying on the client's default. No proxy needed (formats are direct CDN URLs, just header-gated). **2026-09-21:** the `calc_hash`/hash-regex/`alnum_id_from_href`/`resolve_formats` logic moved to `src/util/eporner_resolve.rs` so `melonstube` (a meta-search aggregator that redirects out to eporner.com for ~34% of its own cards) can reuse the exact same resolution without a cross-provider `use`, which would break either provider's `HOT_TUB_PROVIDER=<id>` single-provider build (only the one selected provider module gets compiled; `src/util/*` always does). `eporner.rs` itself now just calls `crate::util::eporner_resolve::resolve_formats(...)`. |
| `eporner` | `mainstream-tube` | no | yes | HTML scraper for eporner.com (5M+ videos); card selector `div.mb[data-id]` with inline duration/rating/views/uploader; thumbnails at `static-eu-cdn.eporner.com` (no proxy needed); pagination uses `/{N}/` suffix (page 1 = no suffix, page 2 = `/2/`); supports sort: new/popular/rated/best; 65 hardcoded categories, plus `cat:`/`category:`, `tag:`, `pornstar:`/`star:`, `uploader:`/`profile:` query-prefix routing to the matching archive, a bare-keyword category-name match, and a background-loaded pornstar name→URL map from `/pornstar-list/` (falls back to `/tag/{slug}` keyword search otherwise). `video.url` is the detail page — **not** yt-dlp-resolvable and not directly playable on its own: opening it bare serves a small restricted/preview clip. Real playback needs per-card enrichment (bounded `buffer_unordered(6)`): fetch the detail page for its 32-hex `hash`, derive the site's own hashed token (`calc_hash` — split into four 8-hex chunks, each parsed as u32 and base36-encoded, concatenated; reverse-engineered from the player JS), then call the site's own `GET /xhr/video/{alnum_id}?hash={calc}&device=generic&domain=www.eporner.com&fallback=false` JSON API for direct per-resolution mp4 URLs into `formats[]`. Separately from the hash dance, the resolved CDN mp4 URLs enforce their own Referer check: a request with no Referer or one pointing at eporner.com gets the real file, but any other non-empty Referer gets a 302 to a static `.../na.mp4` ("not available") decoy clip — the exact "this video is only available at eporner.com"-style short clip a client sees if its default (or app-injected) Referer doesn't match, so every format explicitly declares `http_headers: {Referer: "https://www.eporner.com/"}` rather than relying on the client's default. No proxy needed (formats are direct CDN URLs, just header-gated). **2026-09-21:** the `calc_hash`/hash-regex/`alnum_id_from_href`/`resolve_formats` logic moved to `src/util/eporner_resolve.rs` so `melonstube` (a meta-search aggregator that redirects out to eporner.com for ~34% of its own cards) can reuse the exact same resolution without a cross-provider `use`, which would break either provider's `HOT_TUB_PROVIDER=<id>` single-provider build (only the one selected provider module gets compiled; `src/util/*` always does). `eporner.rs` itself now just calls `crate::util::eporner_resolve::resolve_formats(...)`. **2026-10-22:** the signed CDN URLs expire exactly two hours after the `/xhr/video` call (the leading number in `/{expiry}_{ip}_{n}/`) and an expired link gets the same `na.mp4` decoy, so favourites, history and long-open lists played the one-minute clip. The links are also **IP-locked**: checked from foreign networks (check-host.net nodes) a fresh link answers 302 → `na.mp4`, while the resolving IP gets the file, so a redirect only works for a player behind the server's own public IP (a Jina fetch had misleadingly suggested otherwise). Formats therefore point at the streaming proxy `/proxy/eporner/{alnum_id}/{height}.mp4`: it resolves (cached 90 min per id/height, re-resolved when the CDN answers with a redirect), forwards `Range`, and returns a sized body so `206` responses keep `Content-Length` for AVPlayer. Not a redirect proxy, so app clients keep these in `formats`. All eporner playback bandwidth now flows through the server. melonstube's eporner destinations use the same `resolve_proxied_formats`. |
| `xnxx` | `mainstream-tube` | no | no | HTML scraper for xnxx.com (10M+ videos); unified card parser handles two formats: `div.thumb-block[data-eid]` (search) and `div.thumb-block.video[data-video='{"id":...}']` (hits); eid extracted from `/video-{eid}/{slug}` URL path; thumbnails at `thumb-cdn77.xnxx-cdn.com` and `thumbs-gcore.xnxx-cdn.com` (no proxy, no Referer needed); 0-indexed pagination (page 1 = `/hits`, page N = `/hits/{N-1}`); default feed is `/hits` (most-viewed — xnxx has no chronological listing); search via `/search/{slug}` (works for keywords and tags); supports `tag:`, `cat:`, `category:` query shortcuts; yt-dlp resolves `video.url` natively (XNXX extractor, returns 4-7 HLS formats); no proxy needed. |
| `xgroovy` | `mainstream-tube` | no | no | KVS (Kernel Video Sharing)-style HTML scraper for xgroovy.com; card selector `div.item[data-video-id]` with `data-views`/`data-rating` as clean numeric attributes directly on the card (no text-parsing needed, unlike xhamster); title/url from `a.popito[href] > strong.title`; thumbnails at `i.xgroovy.com` (no proxy, no Referer needed) from `img.thumb[data-jpg|src]`, preview clip from `img[data-preview]`; duration rendered as free text ("N min"/"N sec", not colon-separated) via a bespoke `parse_duration` helper; path-segment pagination (`/new/` = page 1, `/new/2/` = page 2, unlike xhamster's `?page=N` query string); feeds: `/new/` (default), categories via `/categories/{slug}/` (73 hardcoded categories as `categories` option), pornstars via `/pornstars/{slug}/`, channels via `/channels/{slug}/`; search via `/search/{dashed-query}/` (multi-word queries are lowercased and dash-joined); uploader kind inferred from URL path (`/pornstars/` vs `/channels/`) via `div.author-link > a`; supports `cat:`/`category:` query shortcuts plus static category name matching; yt-dlp resolves `video.url` via the generic html5-embed extractor (no dedicated xgroovy extractor, but resolves cleanly to a 1080p format); no proxy needed. |
| `xhamster` | `mainstream-tube` | no | no | HTML scraper for xhamster.com; card selector `div[data-video-type="video"]` with `data-video-id`; thumbnails via `img[data-role="thumb-preview-img"]` at `ic-vt-nss.xhcdn.com` (no proxy, no Referer needed); pagination via `?page=N` query param (browse feeds use infinite-scroll so only search reliably returns different content per page); feeds: `/newest` (default), `/most-viewed`, `/best`; categories via `/categories/{slug}`; channels via `/channels/{slug}`; 43 hardcoded categories as `categories` option; uploader type inferred from URL path (`/channels/` → channel, `/creators/` → creator, `/pornstars/` → pornstar); supports `cat:`/`category:` and `channel:` query shortcuts, plus static category name matching; preview mp4 clips from `data-previewvideo` attribute; yt-dlp resolves `video.url` natively (xHamster extractor, 28 formats); no proxy needed. |
@@ -126,6 +126,7 @@ These resolve a provider-specific input into a `302 Location`.
These return binary media or images, sometimes rewriting manifests or forwarding cookies/referers.
- `/proxy/shooshtime-media/{endpoint}*`
- `/proxy/eporner/{alnum_id}/{height}.mp4` (streams the IP-locked signed mp4 with Range passthrough; link cached 90 min)
- `/proxy/noodlemagazine/{endpoint}*`
- `/proxy/noodlemagazine-thumb/{endpoint}*`
- `/proxy/hanime-cdn/{endpoint}*`

View File

@@ -26,6 +26,7 @@ These rules cut across every step below. They are intentionally grouped at the t
### Output Shape
- **Verified Hot Tub app clients cannot follow redirects in `formats[].url`.** Redirect proxies (`REDIRECT_PROXIES` in `src/proxy.rs`) are for `video.url` in the app response; `api.rs` moves such formats to `url` for verified app clients as a safety net. Other clients keep the redirect-proxy URLs in `formats`.
- Use `formats` for real media URLs (HLS, multi-quality, signed token), and keep `video.url` on a stable page URL. Never set `video.url` to a tokenized stream that needs a `Referer` to play — the `url` field has no header contract.
- If thumbnails need a `Referer` and the player has no per-thumb header mechanism, proxy them via `/proxy/<id>-thumb/...`. The same applies to manifests with relative URIs.
- Keep filter `id` values stable and machine-targeted. Display `title`s change; ids route.

View File

@@ -0,0 +1,42 @@
# Seamless query resolution todo
Goal: a free-form query ("gal ritchie", "family strokes") should reach the matching
pornstar / studio / channel / tag archive **without** a `pornstar:` / `studio:` / `tag:` prefix,
by asking the site (probe) and falling back to keyword search. Prefixes stay as explicit overrides.
Pattern (see `hooporn`, `cumlouder`): after known-category matching, probe page 1 of each entity
archive for the slug with `Requester::get_probe` (404 = instant `None`, Cloudflare challenge goes to
FlareSolverr, 5xx/429 retried once), use the first archive that yields cards (page N stays in that
archive), else keyword search. A probe is only valid after checking what the site answers for an
unknown slug (404 vs soft-404 that lists unrelated videos).
Status: `[ ]` open, `[x]` done and verified live (feed of the archive + unknown name falls back),
`[-]` not applicable (reason).
| | Provider | Prefix kinds today | Plan / evidence |
|---|---|---|---|
| [x] | `hooporn` | cat, pornstar, studio | reference implementation |
| [x] | `cumlouder` | cat, tag, series, girl | reference implementation (girl page, then tag) |
| [x] | `fullporner` | cat, pornstar | probes `/pornstar/{slug}` (unknown = 404 with a suggestion grid, so status decides); `pornstar:` slug now dashed |
| [x] | `fullpornerorg` | pornstar | probes `/videos/pornstar/{slug}/` (404 for unknown) |
| [x] | `tube8` | tag, cat, channel, pornstar | probes pornstar then channel |
| [x] | `youporn` | tag, channel, pornstar, amateur | probes pornstar, channel, amateur (plus existing background shortcut map) |
| [x] | `porn4fans` | tag, cat, model | probes `/models/{slug}/` then `/tags/{slug}/` (both 404 when unknown) |
| [x] | `xvideos` | tag, cat, uploader/channel/profile | probes `/channels/{slug}/videos/new/0` (JSON; pornstar profiles redirect there). **Also fixed**: the old `channel:`/`uploader:` prefix returned nothing (landing page cards are `/prof-video-click/` links); channels are now read from the JSON video list |
| [x] | `xhamster` | cat, channel | added `Pornstar` target; probes pornstar then channel |
| [x] | `xgroovy` | cat, pornstar, channel | probes pornstar then channel |
| [x] | `thepornbunny` | tag, cat, studio, pornstar | probes pornstar, studio, tag (unknown slugs redirect home: `get_probe` now treats a redirect to another path as a miss) |
| [x] | `pornhub` | channel, pornstar, model, user | background catalogs already matched listed names; unknown names are probed (`pornstar`, `model`, `channels`) and cached in `tag_map` (misses remembered) |
| [x] | `clapdat` | tag, user | probes tag then user |
| [x] | `fikfap` | tag, user | single-word query that is an existing profile (profile endpoint) goes to its posts, empty profile falls back to search |
| [x] | `erome` | uploader, user | single-word query that is an existing `/{name}` with albums goes to the uploader |
| [x] | `redgifs` | tag, user | single-word query that is an existing creator (users endpoint) goes to the creator, empty falls back |
| [x] | `sextb` | (catalog matched) | catalog (actress/studio/genre) match first, then probes `/actress/{slug}` and `/studio/{slug}` for names outside the catalog |
| [x] | `coomer` | u:, @service:user | single-word query that is an existing creator id (`/api/v1/{service}/user/{id}/profile`) goes to the creator. **Also fixed**: creator feeds are a bare JSON array, which the parser rejected (every creator feed was empty) |
| [-] | `ripnsfw`, `hsex`, `omgxxx`, `eporner`, `borntobefuck`, `archivebate` | various | already seamless: bare names match background-loaded creator / studio / tag catalogs (or the whole catalog is filtered client-side) |
| [-] | `perfectgirls`, `okxxx`, `pornhat`, `perverzija`, `porntrex` | sites/models/stars | bare names already resolve through the background-loaded `tag_map`; these clones build their own `wreq` client with no `ServerOptions`, so a live probe needs a separate refactor onto the shared requester |
| [-] | `homoxxx` | models, pornstars, channels, categories, tags | legacy provider (own client, unpaginated archives, no `/models/` index on the site); not worth a probe until it is rewritten |
| [-] | `kwiky`, `animeidhentai`, `hentaitv`, `xnxx`, `blowjobspro`, `txxx` | tag/cat | search already is the entity lookup, or the site has no pornstar/studio archives to probe (txxx: the API rejects `models`/`channels` sections) |
| [-] | `fyptt` | cat | by design: bare category words are common search terms and must not be hijacked |
| [-] | `notfans` | tag, category | tag URLs are hashed ids, so an unknown name cannot be probed; search covers it |
| [-] | `allpornstream`, `jable`, `wowxxx`, `hornyleak`, `hotbunny`, `hentaimama`, `tikporn`, `camsoda` | actor/producer, model, studio, ... | Cloudflare-gated or down from this host (403/522, FlareSolverr wedged), so a probe cannot be verified here; each would add one or two FlareSolverr solves per search. Candidates for the same pattern once checked from the production network |

View File

@@ -714,6 +714,15 @@ async fn videos_post(
.collect();
}
// The Hot Tub app cannot follow a redirect in `formats[].url`: for verified
// app clients, formats that point at a redirecting `/proxy/...` become the
// item's `url` instead.
if clientversion.is_verified_hottub() {
for video in video_items.iter_mut() {
move_redirect_formats_to_url(video);
}
}
if let Some(literal_query) = literal_query.as_deref() {
#[cfg(feature = "debug")]
let before = video_items.len();
@@ -794,6 +803,30 @@ async fn videos_post(
Ok(web::HttpResponse::Ok().json(&videos))
}
/// Moves formats that point at a redirecting proxy to `video.url`.
///
/// If direct (non-redirecting) formats remain they are kept and `url` is left
/// alone; otherwise `url` becomes the first (best-listed) redirecting proxy URL
/// and `formats` is dropped entirely.
pub(crate) fn move_redirect_formats_to_url(video: &mut VideoItem) {
let Some(formats) = video.formats.take() else {
return;
};
let (redirecting, direct): (Vec<_>, Vec<_>) = formats
.into_iter()
.partition(|format| crate::proxy::is_redirect_proxy_url(&format.url));
if redirecting.is_empty() {
video.formats = Some(direct);
return;
}
if direct.is_empty() {
video.url = redirecting[0].url.clone();
video.formats = None;
} else {
video.formats = Some(direct);
}
}
async fn uploaders_post(
uploader_request: web::types::Json<UploadersRequest>,
cache: web::types::State<VideoCache>,
@@ -1082,4 +1115,53 @@ mod tests {
assert_eq!(item.uploader.as_deref(), Some("FreeusePorn"));
assert_eq!(item.formats.as_ref().map(|formats| formats.len()), Some(2));
}
fn item_with_formats(urls: &[&str]) -> VideoItem {
let mut item = VideoItem::new(
"id".to_string(),
"t".to_string(),
"https://site.example/watch/1".to_string(),
"c".to_string(),
"https://thumb".to_string(),
10,
);
item.formats = Some(
urls.iter()
.map(|u| crate::videos::VideoFormat::new(u.to_string(), "q".to_string(), "mp4".to_string()))
.collect(),
);
item
}
#[test]
fn redirect_only_formats_move_to_url() {
let mut item = item_with_formats(&[
"http://h/proxy/hqporner/1080",
"http://h/proxy/hqporner/720",
]);
move_redirect_formats_to_url(&mut item);
assert_eq!(item.url, "http://h/proxy/hqporner/1080");
assert!(item.formats.is_none());
}
#[test]
fn direct_formats_are_kept_and_redirect_ones_dropped() {
let mut item = item_with_formats(&[
"http://h/proxy/hqporner/1080",
"https://cdn.example/v.mp4",
]);
move_redirect_formats_to_url(&mut item);
assert_eq!(item.url, "https://site.example/watch/1");
let formats = item.formats.expect("direct format kept");
assert_eq!(formats.len(), 1);
assert_eq!(formats[0].url, "https://cdn.example/v.mp4");
}
#[test]
fn media_proxy_formats_are_untouched() {
let mut item = item_with_formats(&["http://h/proxy/noodlemagazine/x.mp4"]);
move_redirect_formats_to_url(&mut item);
assert_eq!(item.url, "https://site.example/watch/1");
assert_eq!(item.formats.map(|f| f.len()), Some(1));
}
}

View File

@@ -799,6 +799,39 @@ impl ArchivebateProvider {
.map(|value| value.to_string())
}
/// The proxied format needs no network, so an item always plays through
/// it even when the best-effort header enrichment below times out.
fn with_default_format(item: VideoItem, options: &ServerOptions) -> VideoItem {
let format_url = Self::proxied_video(options, &item.url);
if format_url.is_empty() || item.formats.is_some() {
return item;
}
let mut item = item;
item.formats = Some(vec![VideoFormat::new(
format_url,
"source".to_string(),
"mp4".to_string(),
)]);
item
}
/// For verified Hot Tub app clients: no `formats` at all. The item's `url`
/// is the lazy `/proxy/archivebate` redirect, which the app plays directly
/// (it resolves the media at playback time).
fn proxy_as_item_url(videos: Vec<VideoItem>, options: &ServerOptions) -> Vec<VideoItem> {
videos
.into_iter()
.map(|mut video| {
let proxy_url = Self::proxied_video(options, &video.url);
if !proxy_url.is_empty() {
video.url = proxy_url;
}
video.formats = None;
video
})
.collect()
}
async fn enrich_video(&self, item: VideoItem, options: &ServerOptions) -> VideoItem {
let page_url = item.url.clone();
let format_url = Self::proxied_video(options, &page_url);
@@ -1148,11 +1181,15 @@ impl ArchivebateProvider {
platform: candidate.platform.clone(),
gender: candidate.gender.clone(),
};
let items = self
// A candidate whose profile page is gone (the search index lists
// deleted profiles) must not abort the whole search: try the next.
match self
.fetch_items_for_target(cache.clone(), &target, 1, per_page, &options)
.await?;
if !items.is_empty() {
return Ok(items);
.await
{
Ok(items) if !items.is_empty() => return Ok(items),
Ok(_) => {}
Err(_) => {}
}
}
@@ -1188,6 +1225,14 @@ impl Provider for ArchivebateProvider {
if videos.is_empty() {
return videos;
}
let is_app_client = options
.client_version
.as_ref()
.map(ClientVersion::is_verified_hottub)
.unwrap_or(false);
if is_app_client {
return Self::proxy_as_item_url(videos, &options);
}
stream::iter(videos.into_iter().map(|video| {
let provider = self.clone();
let options = options.clone();
@@ -1199,11 +1244,11 @@ impl Provider for ArchivebateProvider {
.await;
match timeout_result {
Ok(enriched) => enriched,
Err(_) => video,
Err(_) => Self::with_default_format(video, &options),
}
}
}))
.buffer_unordered(4)
.buffered(4)
.collect::<Vec<_>>()
.await
}
@@ -1218,3 +1263,52 @@ impl Provider for ArchivebateProvider {
Some(self.build_channel(clientversion))
}
}
#[cfg(test)]
mod tests {
use super::*;
fn options() -> ServerOptions {
ServerOptions {
client_version: None,
featured: None,
category: None,
sites: None,
filter: None,
language: None,
public_url_base: Some("http://hottub.example".to_string()),
requester: None,
network: None,
stars: None,
categories: None,
duration: None,
sort: None,
sexuality: None,
}
}
#[test]
fn app_clients_get_the_proxy_url_and_no_formats() {
let mut item = VideoItem::new(
"archivebate:1".to_string(),
"t".to_string(),
"https://archivebate.com/watch/1".to_string(),
CHANNEL_ID.to_string(),
"https://cdn.example/t.jpg".to_string(),
10,
);
item.formats = Some(vec![VideoFormat::new(
"x".to_string(),
"source".to_string(),
"mp4".to_string(),
)]);
let out = ArchivebateProvider::proxy_as_item_url(vec![item], &options());
assert_eq!(out.len(), 1);
assert_eq!(
out[0].url,
"http://hottub.example/proxy/archivebate/archivebate.com/watch/1"
);
assert!(out[0].formats.is_none());
}
}

View File

@@ -1,7 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{
Provider, build_proxy_url, report_provider_error, requester_or_default, strip_url_scheme,
Provider, build_proxy_url, probe_first_archive, report_provider_error, requester_or_default,
slug_dashed, strip_url_scheme,
};
use crate::status::*;
use crate::util::time::parse_time_to_seconds;
@@ -149,6 +150,29 @@ impl ClapdatProvider {
}
}
/// A free-form query that names a tag or a user goes to that archive
/// (probed on the site; unknown names 404), else keyword search.
async fn resolve_keyword(&self, query: &str, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let candidates = [
Target::Tag { slug: slug.clone() },
Target::User { username: slug },
];
let urls: Vec<String> = candidates
.iter()
.filter_map(|t| self.listing_url(t, 1))
.collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
body.contains("/video/")
})
.await?;
candidates.into_iter().nth(index)
}
fn listing_url(&self, target: &Target, page: u16) -> Option<String> {
let page = page.max(1);
match target {
@@ -491,7 +515,12 @@ impl Provider for ClapdatProvider {
sort
};
let query_value = query.unwrap_or_default();
let target = self.resolve_target(&query_value, &sort_value);
let mut target = self.resolve_target(&query_value, &sort_value);
if let Target::Search { query } = &target {
if let Some(found) = self.resolve_keyword(query, &options).await {
target = found;
}
}
let stubs = match &target {
Target::Search { query } => match self

View File

@@ -8,7 +8,7 @@ use wreq::Version;
use crate::{
DbPool,
api::ClientVersion,
providers::{Provider, requester_or_default},
providers::{Provider, probe_first_archive_with, requester_or_default},
status::{Channel, ChannelOption, FilterOption},
util::cache::VideoCache,
videos::{ServerOptions, VideoFormat, VideoItem},
@@ -147,6 +147,43 @@ impl CoomerProvider {
}
}
/// A single-word query that is an existing creator id (the profile
/// endpoint answers 404 otherwise) goes to that creator's posts, trying the
/// filtered service first, else keyword search.
async fn resolve_keyword(
query: &str,
service_filter: Option<&str>,
options: &ServerOptions,
) -> Option<Target> {
let name = query.trim().trim_start_matches('@');
if name.is_empty()
|| !name
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-' || c == '.')
{
return None;
}
let mut services = vec![Service::Onlyfans, Service::Fansly, Service::Candfans];
if let Some(preferred) = Self::resolve_service(service_filter) {
services.retain(|s| s.as_str() != preferred.as_str());
services.insert(0, preferred);
}
let urls: Vec<String> = services
.iter()
.map(|s| format!("{BASE_URL}/api/v1/{}/user/{name}/profile", s.as_str()))
.collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) =
probe_first_archive_with(&mut requester, &urls, Self::json_headers(), |body| {
body.contains("\"id\"")
})
.await?;
Some(Target::Uploader {
service: services[index].clone(),
username: name.to_string(),
})
}
fn json_headers() -> Vec<(String, String)> {
vec![
("Referer".to_string(), REFERER.to_string()),
@@ -290,13 +327,16 @@ impl CoomerProvider {
)));
}
}
let posts: ApiPostsResponse = serde_json::from_str(&body).map_err(|err| {
let posts: ApiPostsPayload = serde_json::from_str(&body).map_err(|err| {
Error::from(format!(
"json parse failed for {url}: {err}; body[:200]={}",
&body.chars().take(200).collect::<String>()
))
})?;
Ok(posts.posts)
Ok(match posts {
ApiPostsPayload::List(posts) => posts,
ApiPostsPayload::Wrapped(wrapped) => wrapped.posts,
})
}
async fn fetch_items(
@@ -543,6 +583,14 @@ struct ApiError {
error: String,
}
/// The global feed wraps posts in `{posts: [...]}`; a creator's feed is a bare array.
#[derive(Debug, Deserialize)]
#[serde(untagged)]
enum ApiPostsPayload {
List(Vec<ApiPost>),
Wrapped(ApiPostsResponse),
}
#[derive(Debug, Deserialize)]
struct ApiPostsResponse {
#[serde(default)]
@@ -601,8 +649,13 @@ impl Provider for CoomerProvider {
.clamp(1, 64);
let service_filter = options.categories.as_deref().or(options.sites.as_deref());
let target =
let mut target =
Self::resolve_target(&sort_value, query.as_deref(), service_filter);
if let Target::Search { query } = &target {
if let Some(found) = Self::resolve_keyword(query, service_filter, &options).await {
target = found;
}
}
// Bump the ignored `Sort` variant warning off by referencing it.
let _ = Sort::New;
@@ -849,4 +902,18 @@ mod tests {
let post: ApiPost = serde_json::from_str(json).expect("parse");
assert!(CoomerProvider::build_video_item(post).is_none());
}
#[test]
fn parses_bare_array_and_wrapped_post_payloads() {
let bare = r#"[{"id":"1","user":"u","service":"onlyfans","title":"t","file":{},"attachments":[]}]"#;
match serde_json::from_str::<ApiPostsPayload>(bare).expect("bare array") {
ApiPostsPayload::List(posts) => assert_eq!(posts.len(), 1),
other => panic!("expected list, got {other:?}"),
}
let wrapped = r#"{"posts":[{"id":"1"}],"count":1}"#;
match serde_json::from_str::<ApiPostsPayload>(wrapped).expect("wrapped") {
ApiPostsPayload::Wrapped(w) => assert_eq!(w.posts.len(), 1),
other => panic!("expected wrapped, got {other:?}"),
}
}
}

View File

@@ -440,15 +440,26 @@ impl EpornerProvider {
Ok(cards)
}
async fn build_item(requester: &mut Requester, card: EpornerCard) -> VideoItem {
async fn build_item(
requester: &mut Requester,
options: &ServerOptions,
card: EpornerCard,
) -> VideoItem {
// The hash/xhr resolution + Referer handling live in
// `crate::util::eporner_resolve` (shared with other providers that
// redirect out to eporner.com, e.g. `melonstube`) rather than here,
// since single-provider builds can't depend on another provider's
// module — see that module's doc comment.
let formats =
crate::util::eporner_resolve::resolve_formats(requester, &card.alnum_id, &card.page_url)
.await;
// Formats point at the eporner streaming proxy: the CDN links expire
// after two hours and only serve the server's IP; anything else plays
// a one-minute "watch on eporner.com" clip.
let formats = crate::util::eporner_resolve::resolve_proxied_formats(
requester,
options,
&card.alnum_id,
&card.page_url,
)
.await;
let mut item = VideoItem::new(
card.id,
@@ -648,7 +659,8 @@ impl EpornerProvider {
let cards = Self::parse_list_cards(&html)?;
let items = stream::iter(cards.into_iter().take(per_page.max(1)).map(|card| {
let mut req = requester.clone();
async move { Self::build_item(&mut req, card).await }
let options = &options;
async move { Self::build_item(&mut req, options, card).await }
}))
.buffer_unordered(ENRICH_CONCURRENCY)
.collect::<Vec<_>>()

View File

@@ -1,6 +1,6 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::providers::{Provider, probe_first_archive, report_provider_error, requester_or_default};
use crate::status::*;
use crate::util::parse_abbreviated_number;
use crate::videos::{ServerOptions, VideoItem};
@@ -164,6 +164,26 @@ impl EromeProvider {
(!slug.is_empty()).then_some(slug)
}
/// A single-word query that is an existing uploader name (`/{name}` answers
/// 404 otherwise) goes to that uploader's albums, else keyword search.
async fn resolve_keyword(&self, query: &str, options: &ServerOptions) -> Option<Target> {
let slug = query.trim().trim_start_matches('@');
if slug.is_empty()
|| !slug
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-')
{
return None;
}
let url = format!("{}/{slug}", self.url);
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
probe_first_archive(&mut requester, &[url], &format!("{}/", self.url), |body| {
body.contains("album-link")
})
.await
.map(|_| Target::UploaderPosts { slug: slug.to_string() })
}
fn resolve_target(&self, query: &str, options: &ServerOptions, sort: &str) -> Target {
if let Some(site) = options.sites.as_deref() {
let normalized = site.trim();
@@ -367,7 +387,12 @@ impl Provider for EromeProvider {
};
let query_value = query.unwrap_or_default();
let page_value = page.parse::<u16>().unwrap_or(1);
let target = self.resolve_target(&query_value, &options, &sort_value);
let mut target = self.resolve_target(&query_value, &options, &sort_value);
if let Target::Search { query, .. } = &target {
if let Some(found) = self.resolve_keyword(query, &options).await {
target = found;
}
}
let url = self.build_url_for_target(&target, page_value);
match self.fetch_page(&options, &url).await {

View File

@@ -639,6 +639,24 @@ impl Provider for FikfapProvider {
.map(str::trim)
.filter(|value| !value.is_empty());
let target = Self::pick_target(normalized_query, &sort, &options);
// A single-word query that is an existing profile name goes to that
// creator's posts (checked against the profile endpoint); a profile
// without posts falls back to the keyword search.
if let Target::Search(query) = &target {
if !query.contains(char::is_whitespace) {
if let Ok(Some(profile)) = self.fetch_user_profile(query, &options).await {
let user = Target::User(profile.username);
if let Ok(items) = self
.fetch_target_items(user, page, per_page, &options)
.await
{
if !items.is_empty() {
return items;
}
}
}
}
}
match self.fetch_target_items(target, page, per_page, &options).await {
Ok(items) => items,

View File

@@ -1,6 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::providers::{
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::time::parse_time_to_seconds;
@@ -134,13 +136,13 @@ impl FullpornerProvider {
}
if let Some(slug) = q.strip_prefix("pornstar:") {
return Target::Pornstar {
slug: slug.trim().to_string(),
slug: slug_dashed(slug),
page,
};
}
if let Some(slug) = q.strip_prefix("star:") {
return Target::Pornstar {
slug: slug.trim().to_string(),
slug: slug_dashed(slug),
page,
};
}
@@ -153,6 +155,25 @@ impl FullpornerProvider {
Target::Latest { page }
}
/// A free-form query that names a pornstar goes to her archive (probed on
/// the site; unknown names 404), otherwise it stays a keyword search.
async fn resolve_keyword(query: &str, page: u32, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let probe = Target::Pornstar { slug: slug.clone(), page: 1 };
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
probe_first_archive(
&mut requester,
&[Self::build_listing_url(&probe)],
&format!("{BASE_URL}/"),
|body| body.contains("class=\"video-card\""),
)
.await
.map(|_| Target::Pornstar { slug, page })
}
fn selector(value: &str) -> Result<Selector> {
Selector::parse(value)
.map_err(|e| Error::from(format!("selector `{value}` failed: {e}")))
@@ -394,7 +415,10 @@ impl Provider for FullpornerProvider {
.filter(|q| !q.is_empty())
.map(ToOwned::to_owned);
let target = Self::pick_target(normalized_query.as_deref(), page);
let mut target = Self::pick_target(normalized_query.as_deref(), page);
if let Target::Search { query, .. } = &target {
target = Self::resolve_keyword(query, page, &options).await.unwrap_or(target);
}
match Self::fetch_page(&target, per_page, &options).await {
Ok(items) => items,

View File

@@ -1,6 +1,6 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{
use crate::providers::{probe_first_archive, slug_dashed,
Provider, report_provider_error, report_provider_error_background, requester_or_default,
};
use crate::status::*;
@@ -109,12 +109,12 @@ impl FullpornerorgProvider {
if let Some(q) = query.map(str::trim).filter(|q| !q.is_empty()) {
if let Some(slug) = q.strip_prefix("pornstar:") {
return Target::Pornstar {
slug: slug.trim().to_string(),
slug: slug_dashed(slug),
};
}
if let Some(slug) = q.strip_prefix("star:") {
return Target::Pornstar {
slug: slug.trim().to_string(),
slug: slug_dashed(slug),
};
}
return Target::Search {
@@ -153,6 +153,25 @@ impl FullpornerorgProvider {
}
}
/// A free-form query that names a pornstar goes to her archive (probed on
/// the site; unknown names 404), otherwise it stays a keyword search.
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let target = Target::Pornstar { slug };
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
probe_first_archive(
&mut requester,
&[Self::build_listing_url(&target, 1)],
&format!("{BASE_URL}/"),
|body| body.contains("loop-video"),
)
.await
.map(|_| target)
}
fn html_headers(referer: &str) -> Vec<(String, String)> {
vec![
("User-Agent".to_string(), BROWSER_UA.to_string()),
@@ -446,7 +465,10 @@ impl Provider for FullpornerorgProvider {
.filter(|q| !q.is_empty())
.map(ToOwned::to_owned);
let target = Self::pick_target(normalized_query.as_deref());
let mut target = Self::pick_target(normalized_query.as_deref());
if let Target::Search { query } = &target {
target = Self::resolve_keyword(query, &options).await.unwrap_or(target);
}
match Self::fetch_page(target, page, per_page, &options).await {
Ok(items) => items,

View File

@@ -1,13 +1,8 @@
use aes_gcm::aead::{Aead, KeyInit, Payload};
use aes_gcm::{Aes256Gcm, Key, Nonce};
use async_trait::async_trait;
use base64::Engine;
use error_chain::error_chain;
use futures::future::join_all;
use serde_json::json;
use sha2::{Digest, Sha256};
use std::sync::{Arc, Mutex, OnceLock};
use std::time::{SystemTime, UNIX_EPOCH};
use std::time::SystemTime;
use std::vec;
use crate::DbPool;
@@ -16,6 +11,7 @@ use crate::db;
use crate::providers::{Provider, report_provider_error, report_provider_error_background};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::hanime_resolve::{self as resolve, HanimeSource, SITE_ORIGIN};
use crate::videos::{self, ServerOptions, VideoItem};
pub const CHANNEL_METADATA: crate::providers::ProviderChannelMetadata =
@@ -34,14 +30,6 @@ error_chain! {
/// Full catalogue dump. Ignores every query parameter, so filtering, sorting and
/// pagination all happen client side. ~860 KB compressed, CDN cached for an hour.
const INDEX_URL: &str = "https://guest.freeanimehentai.net/api/v11/search_hvs";
const HANDSHAKE_URL: &str = "https://auth.hanime.tv/api/v11/handshake";
const SITE_ORIGIN: &str = "https://hanime.tv";
/// Envelope encryption used for the handshake request/response bodies.
const HANDSHAKE_KEY_SEED: &[u8] = b"htv-insecure-handshake-v1";
const HANDSHAKE_AAD: &[u8] = b"htv-insecure-v1";
/// Pieces of the `x-signature` pre-image, as baked into the site's wasm module.
const SIGNATURE_SECRET: &str = "Xkdi29";
const SIGNATURE_SALT: &str = "mn2";
const INDEX_TTL_SECS: u64 = 3600;
type IndexCache = OnceLock<Mutex<Option<(SystemTime, Arc<Vec<HanimeSearchResult>>)>>>;
@@ -76,99 +64,6 @@ struct HanimeSearchResult {
released_at_unix: u64,
}
#[derive(serde::Deserialize, Debug)]
struct HanimeHandshakePayload {
#[serde(default)]
sources: Vec<HanimeSource>,
}
#[derive(serde::Deserialize, Debug, Clone)]
struct HanimeSource {
#[serde(default)]
src: String,
#[serde(default)]
height: u32,
#[serde(default)]
label: String,
/// `"normal"` for the free streams, `"promotion"` for the premium-only teaser
/// entries which always carry an empty `src`.
#[serde(default)]
kind: String,
}
fn b64() -> base64::engine::general_purpose::GeneralPurpose {
base64::engine::general_purpose::URL_SAFE_NO_PAD
}
fn b64_decode(value: &str) -> Result<Vec<u8>> {
b64()
.decode(value.trim_end_matches('='))
.map_err(|e| Error::from(format!("base64url decode failed: {e}")))
}
fn handshake_cipher() -> Aes256Gcm {
let digest = Sha256::digest(HANDSHAKE_KEY_SEED);
let key = Key::<Aes256Gcm>::from_slice(digest.as_slice());
Aes256Gcm::new(key)
}
/// `{"v":1,"alg":"AES-256-GCM","iv":..,"tag":..,"data":..}`, base64url encoded.
fn seal_envelope(plaintext: &[u8]) -> Result<String> {
let mut iv = [0u8; 12];
rand::fill(&mut iv);
let sealed = handshake_cipher()
.encrypt(
Nonce::from_slice(&iv),
Payload {
msg: plaintext,
aad: HANDSHAKE_AAD,
},
)
.map_err(|e| Error::from(format!("handshake encrypt failed: {e}")))?;
if sealed.len() < 16 {
return Err(Error::from("handshake ciphertext too short"));
}
let (data, tag) = sealed.split_at(sealed.len() - 16);
let envelope = json!({
"v": 1,
"alg": "AES-256-GCM",
"iv": b64().encode(iv),
"tag": b64().encode(tag),
"data": b64().encode(data),
});
Ok(b64().encode(envelope.to_string().as_bytes()))
}
fn open_envelope(token: &str) -> Result<Vec<u8>> {
let envelope: serde_json::Value = serde_json::from_slice(&b64_decode(token)?)
.map_err(|e| Error::from(format!("handshake envelope is not JSON: {e}")))?;
let field = |name: &str| -> Result<Vec<u8>> {
let raw = envelope
.get(name)
.and_then(|v| v.as_str())
.ok_or_else(|| Error::from(format!("handshake envelope missing {name}")))?;
b64_decode(raw)
};
let iv = field("iv")?;
let mut ciphertext = field("data")?;
ciphertext.extend_from_slice(&field("tag")?);
handshake_cipher()
.decrypt(
Nonce::from_slice(&iv),
Payload {
msg: &ciphertext,
aad: HANDSHAKE_AAD,
},
)
.map_err(|e| Error::from(format!("handshake decrypt failed: {e}")))
}
fn signature_for(timestamp: u64) -> String {
let pre_image =
format!("{timestamp},{SIGNATURE_SECRET},{SITE_ORIGIN},{SIGNATURE_SALT},{timestamp}");
hex::encode(Sha256::digest(pre_image.as_bytes()))
}
#[derive(Debug, Clone)]
pub struct HanimeProvider;
@@ -242,6 +137,25 @@ impl HanimeProvider {
}
}
/// Fresh free sources for a slug, best quality first.
async fn fetch_sources(&self, slug: &str, options: &ServerOptions) -> Result<Vec<HanimeSource>> {
let mut requester =
crate::providers::requester_or_default(options, module_path!(), "missing_requester");
resolve::fetch_sources(&mut requester, slug)
.await
.map_err(Error::from)
}
/// The stream URLs expire, so players get a stable redirect that does a fresh
/// handshake on every play instead of the signed link itself.
fn proxied_url(options: &ServerOptions, slug: &str, source: &HanimeSource) -> String {
crate::providers::build_proxy_url(
options,
"hanime",
&format!("{slug}/{}.m3u8", source.height),
)
}
fn db_key(slug: &str) -> String {
format!("https://hanime.tv/videos/hentai/{slug}")
}
@@ -319,86 +233,6 @@ impl HanimeProvider {
Ok(items)
}
/// Returns the free `sources` for a slug, best quality first. The auth host
/// occasionally refuses one of a burst of parallel connections, so retry once.
async fn fetch_sources(&self, slug: &str, options: &ServerOptions) -> Result<Vec<HanimeSource>> {
match self.handshake(slug, options).await {
Ok(sources) => Ok(sources),
Err(_) => {
ntex::time::sleep(ntex::time::Millis(500)).await;
self.handshake(slug, options).await
}
}
}
async fn handshake(&self, slug: &str, options: &ServerOptions) -> Result<Vec<HanimeSource>> {
let timestamp = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs();
let token = seal_envelope(
json!({
"timestamp_unix": timestamp,
"directive": "htv_player_handshake",
"slug": slug,
})
.to_string()
.as_bytes(),
)?;
let mut requester =
crate::providers::requester_or_default(options, module_path!(), "missing_requester");
let response = requester
.post_json(
HANDSHAKE_URL,
&json!({ "token": token }),
vec![
("Origin".to_string(), SITE_ORIGIN.to_string()),
("Referer".to_string(), format!("{SITE_ORIGIN}/")),
("Accept".to_string(), "application/json".to_string()),
("x-signature-version".to_string(), "web2".to_string()),
("x-signature".to_string(), signature_for(timestamp)),
("x-time".to_string(), timestamp.to_string()),
],
)
.await
.map_err(|e| Error::from(format!("handshake request failed: {e}")))?;
let status = response.status();
if !status.is_success() {
return Err(Error::from(format!("handshake returned HTTP {status}")));
}
// The interesting part of the answer travels in a response header, not the body.
let encrypted = response
.headers()
.get("x-token")
.and_then(|v| v.to_str().ok())
.map(|v| v.to_string())
.ok_or_else(|| Error::from("handshake response is missing x-token"))?;
let payload: HanimeHandshakePayload = serde_json::from_slice(&open_envelope(&encrypted)?)
.map_err(|e| Error::from(format!("Failed to parse handshake payload: {e}")))?;
let mut sources: Vec<HanimeSource> = payload
.sources
.into_iter()
.filter(|s| s.kind == "normal" && !s.src.is_empty())
.collect();
sources.sort_by(|a, b| b.height.cmp(&a.height));
match sources.is_empty() {
true => Err(Error::from("handshake returned no playable sources")),
false => Ok(sources),
}
}
fn source_url(source: &HanimeSource) -> String {
match source.src.starts_with("http") {
true => source.src.clone(),
false => format!("{SITE_ORIGIN}{}", source.src),
}
}
fn quality_label(source: &HanimeSource) -> String {
match source.label.trim_end_matches('p') {
"" => source.height.to_string(),
@@ -452,15 +286,15 @@ impl HanimeProvider {
.iter()
.map(|source| {
videos::VideoFormat::new(
Self::source_url(source),
Self::proxied_url(&options, &hit.slug, source),
Self::quality_label(source),
"m3u8".to_string(),
)
})
.collect();
let stream_url = Self::source_url(&sources[0]);
let stream_url = Self::proxied_url(&options, &hit.slug, &sources[0]);
let duration = self
.fetch_duration(&stream_url, &options)
.fetch_duration(&resolve::source_url(&sources[0]), &options)
.await
.unwrap_or_default();
if let Ok(mut conn) = pool.get() {
@@ -490,11 +324,8 @@ impl HanimeProvider {
.and_then(|mut conn| db::get_video(&mut conn, db_key.clone()).ok().flatten());
match db_result {
Some(video_url) if video_url.contains("m3u8") || video_url.contains("/hls/") => {
let duration = self
.fetch_duration(&video_url, &options)
.await
.unwrap_or_default();
Some(video_url) if video_url.contains("/proxy/hanime/") => {
let duration = 0;
let formats = vec![videos::VideoFormat::new(
video_url.clone(),
"720".to_string(),

View File

@@ -430,40 +430,13 @@ impl HoopornProvider {
.await
}
/// Fetches a listing page. A real 404 (unknown archive, page past the end)
/// is `None` and answers immediately; a Cloudflare challenge is handed to
/// the requester's full chain (FlareSolverr).
/// Fetches a listing page; `None` for a real 404 (see `Requester::get_probe`).
async fn fetch_html(requester: &mut Requester, url: &str) -> Result<Option<String>> {
let headers = vec![("Referer".to_string(), format!("{BASE_URL}/"))];
let mut last_err = String::new();
for _ in 0..2 {
match requester.get_raw_with_headers(url, headers.clone()).await {
Ok(response) => {
let status = response.status().as_u16();
let body = response.text().await.unwrap_or_default();
if status == 404 {
return Ok(None);
}
if status == 403
|| status == 503
|| Requester::looks_like_cloudflare_challenge(&body)
{
return requester
.get_with_headers(url, headers, None)
.await
.map(Some)
.map_err(|e| Error::from(format!("fetch failed for {url}: {e}")));
}
if status >= 500 || status == 429 {
last_err = format!("HTTP {status}");
continue;
}
return Ok(Some(body));
}
Err(e) => last_err = e.to_string(),
}
}
Err(Error::from(format!("fetch failed for {url}: {last_err}")))
requester
.get_probe(url, headers)
.await
.map_err(|e| Error::from(format!("fetch failed for {url}: {e}")))
}
async fn fetch(target: &Target, page: u32, options: &ServerOptions) -> Result<Vec<VideoItem>> {

View File

@@ -494,11 +494,18 @@ impl MelonstubeProvider {
Self::parse_kvs_formats(&html, &origin)
}
/// Through the eporner streaming proxy: its CDN links expire after two
/// hours and only serve the server's IP (else a "watch on eporner.com" clip).
async fn resolve_eporner(
requester: &mut Requester,
options: &ServerOptions,
detail_url: &str,
) -> Option<Vec<VideoFormat>> {
crate::util::eporner_resolve::resolve_public_formats(requester, detail_url).await
let alnum_id = crate::util::eporner_resolve::alnum_id_from_href(detail_url)?;
crate::util::eporner_resolve::resolve_proxied_formats(
requester, options, &alnum_id, detail_url,
)
.await
}
fn xhamster_mirror_mp4_regex() -> &'static Regex {
@@ -632,10 +639,11 @@ impl MelonstubeProvider {
async fn resolve_source_formats(
kind: SourceKind,
requester: &mut Requester,
options: &ServerOptions,
detail_url: &str,
) -> Option<Vec<VideoFormat>> {
match kind {
SourceKind::Eporner => Self::resolve_eporner(requester, detail_url).await,
SourceKind::Eporner => Self::resolve_eporner(requester, options, detail_url).await,
SourceKind::Kvs => Self::resolve_kvs(requester, detail_url, false).await,
SourceKind::KvsCfGated => Self::resolve_kvs(requester, detail_url, true).await,
SourceKind::XhamsterMirror => Self::resolve_xhamster_mirror(requester, detail_url).await,
@@ -904,6 +912,7 @@ impl MelonstubeProvider {
if !pending.is_empty() {
let resolved = stream::iter(pending.into_iter().map(|(idx, kind, dest_url)| {
let mut req = requester.clone();
let options = &options;
async move {
// Bounded so one slow/hung third-party destination (e.g.
// an unreachable FlareSolverr fallback) can't stall the
@@ -911,7 +920,7 @@ impl MelonstubeProvider {
// its plain destination URL with no formats.
let formats = tokio::time::timeout(
std::time::Duration::from_secs(ENRICH_TIMEOUT_SECS),
Self::resolve_source_formats(kind, &mut req, &dest_url),
Self::resolve_source_formats(kind, &mut req, options, &dest_url),
)
.await
.ok()

View File

@@ -612,6 +612,66 @@ pub fn report_provider_error_background(provider_name: &str, context: &str, msg:
});
}
/// `Some(slug)` joined with `-` ("Gal Ritchie" -> "gal-ritchie") for archive URLs.
pub fn slug_dashed(value: &str) -> String {
let mut out = String::new();
let mut last_dash = true;
for c in value.trim().chars() {
if c.is_alphanumeric() {
out.extend(c.to_lowercase());
last_dash = false;
} else if !last_dash {
out.push('-');
last_dash = true;
}
}
out.trim_matches('-').to_string()
}
/// Same as [`slug_dashed`] without separators ("Family Strokes" -> "familystrokes").
pub fn slug_joined(value: &str) -> String {
slug_dashed(value).replace('-', "")
}
/// Probes candidate archive URLs (pornstar / studio / channel / tag pages) in
/// order and returns the index and body of the first that exists and
/// `has_items` accepts. A miss (404, empty page, transport error) moves on to
/// the next candidate; the caller falls back to keyword search on `None`.
/// Uses `Requester::get_probe`, so an unknown archive answers instantly and a
/// Cloudflare challenge is solved through FlareSolverr.
pub async fn probe_first_archive(
requester: &mut Requester,
urls: &[String],
referer: &str,
has_items: impl Fn(&str) -> bool,
) -> Option<(usize, String)> {
probe_first_archive_with(
requester,
urls,
vec![("Referer".to_string(), referer.to_string())],
has_items,
)
.await
}
/// [`probe_first_archive`] with caller-supplied request headers (JSON APIs
/// that demand a special `Accept`, extra tokens, ...).
pub async fn probe_first_archive_with(
requester: &mut Requester,
urls: &[String],
headers: Vec<(String, String)>,
has_items: impl Fn(&str) -> bool,
) -> Option<(usize, String)> {
for (index, url) in urls.iter().enumerate() {
if let Ok(Some(body)) = requester.get_probe(url, headers.clone()).await {
if has_items(&body) {
return Some((index, body));
}
}
}
None
}
pub fn requester_or_default(
options: &ServerOptions,
provider_name: &str,
@@ -1777,3 +1837,16 @@ mod tests {
);
}
}
#[cfg(test)]
mod slug_tests {
use super::{slug_dashed, slug_joined};
#[test]
fn slugs_names_for_archive_urls() {
assert_eq!(slug_dashed(" Gal Ritchie "), "gal-ritchie");
assert_eq!(slug_dashed("Mia Khalifa!!"), "mia-khalifa");
assert_eq!(slug_joined("Family Strokes"), "familystrokes");
assert_eq!(slug_dashed("???"), "");
}
}

View File

@@ -2,6 +2,7 @@ use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{
Provider, report_provider_error, report_provider_error_background, requester_or_default,
probe_first_archive,
};
use crate::status::*;
use crate::util::cache::VideoCache;
@@ -260,6 +261,34 @@ impl Porn4fansProvider {
Target::Latest
}
/// A free-form query that names a model or a tag goes to that archive
/// (probed on the site; unknown names 404), else keyword search.
async fn resolve_keyword(&self, query: &str, options: &ServerOptions) -> Option<Target> {
let slug = Self::slugify(query);
if slug.is_empty() {
return None;
}
let candidates = [
Target::Model { slug: slug.clone() },
Target::Tag { slug },
];
let urls: Vec<String> = candidates
.iter()
.map(|t| match t {
Target::Model { slug } => format!("{}/models/{slug}/", self.url),
Target::Tag { slug } => format!("{}/tags/{slug}/", self.url),
_ => String::new(),
})
.collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) =
probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
body.contains("custom_list_videos")
})
.await?;
candidates.into_iter().nth(index)
}
/// Build the async `get_block` URL plus the Referer to send with it.
fn build_block_url(&self, target: &Target, sort: &str, page: u32) -> (String, String) {
let sort_by = Self::sort_by(sort);
@@ -788,7 +817,12 @@ impl Provider for Porn4fansProvider {
options: ServerOptions,
) -> Vec<VideoItem> {
let page = page.parse::<u32>().unwrap_or(1).max(1);
let target = self.target_from_request(query.as_deref(), options.categories.as_deref());
let mut target = self.target_from_request(query.as_deref(), options.categories.as_deref());
if let Target::Search { query } = &target {
if let Some(found) = self.resolve_keyword(query, &options).await {
target = found;
}
}
match self
.get_videos_for_target(cache, &target, &sort, page, options)

View File

@@ -1,6 +1,9 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, proxy_aylo_thumb, report_provider_error, requester_or_default};
use crate::providers::{
Provider, probe_first_archive, proxy_aylo_thumb, report_provider_error, requester_or_default,
slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::parse_abbreviated_number;
@@ -45,6 +48,8 @@ error_chain! {
pub struct PornhubProvider {
url: String,
tag_map: Arc<RwLock<HashMap<String, TagInfo>>>,
/// Queries already probed on the site that are neither a pornstar, model nor channel.
probe_misses: Arc<RwLock<std::collections::HashSet<String>>>,
}
#[derive(Debug, Clone, Copy)]
@@ -91,6 +96,7 @@ impl PornhubProvider {
let provider = Self {
url: BASE_URL.to_string(),
tag_map: Arc::new(RwLock::new(HashMap::new())),
probe_misses: Arc::new(RwLock::new(std::collections::HashSet::new())),
};
provider.spawn_initial_load();
provider
@@ -588,6 +594,43 @@ impl PornhubProvider {
values.push(normalized.to_string());
}
/// A free-form query that is not in the loaded catalogs may still name a
/// pornstar, model or channel. Probe the site (unknown names 404 or
/// redirect away) and remember the hit in `tag_map` so later pages and
/// queries resolve without another request.
async fn learn_query_target(&self, query: &str, options: &ServerOptions) {
let key = query.trim().to_ascii_lowercase();
let slug = slug_dashed(query);
if slug.is_empty() || self.probe_misses.read().unwrap().contains(&key) {
return;
}
let kinds = [
QueryTargetKind::Pornstar,
QueryTargetKind::Model,
QueryTargetKind::Channel,
];
let urls: Vec<String> = kinds
.iter()
.map(|kind| format!("{}/{}/{slug}/videos", self.url, kind.path_segment()))
.collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "learn_query_target");
let hit = probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
body.contains("pcVideoListItem")
})
.await;
match hit {
Some((index, _)) => {
self.tag_map.write().unwrap().insert(
key,
TagInfo { kind: kinds[index], slug, title: query.trim().to_string() },
);
}
None => {
self.probe_misses.write().unwrap().insert(key);
}
}
}
async fn fetch_listing(
&self,
cache: VideoCache,
@@ -596,6 +639,11 @@ impl PornhubProvider {
query: Option<&str>,
options: ServerOptions,
) -> Result<Vec<VideoItem>> {
if let Some(q) = query.map(str::trim).filter(|q| !q.is_empty()) {
if self.parse_query_target(q).is_none() {
self.learn_query_target(q, &options).await;
}
}
if query.is_some() && self.tag_map.read().unwrap().is_empty() {
let _ = Self::load_tags(&self.url, Arc::clone(&self.tag_map)).await;
}

View File

@@ -560,6 +560,24 @@ impl Provider for RedgifsProvider {
let sort = Sort::from_sort_id(&sort);
let target = Self::pick_target(normalized_query, sort, &options);
// A single-word query that is an existing creator name goes to that
// creator's gifs (checked against the users endpoint); a creator
// without gifs falls back to the keyword search.
if let Target::Query(text, _) = &target {
if !text.contains(char::is_whitespace) {
if let Ok(Some(profile)) = self.fetch_user_profile(text, &options).await {
let creator = Target::Creator(profile.username);
if let Ok(items) = self
.fetch_target_items(creator, page, per_page, &options)
.await
{
if !items.is_empty() {
return items;
}
}
}
}
}
match self.fetch_target_items(target, page, per_page, &options).await {
Ok(items) => items,

View File

@@ -1,6 +1,6 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::Provider;
use crate::providers::{Provider, requester_or_default, slug_dashed};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::discord::send_discord_error_report;
@@ -12,10 +12,23 @@ use async_trait::async_trait;
use error_chain::error_chain;
use futures::stream::{self, StreamExt};
use htmlentity::entity::{ICodedDataTrait, decode};
use std::time::{SystemTime, UNIX_EPOCH};
use scraper::{Html, Selector};
use std::collections::HashMap;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
use percent_encoding::{NON_ALPHANUMERIC, utf8_percent_encode};
use std::vec;
const ENRICH_CONCURRENCY: usize = 6;
const ENRICH_CONCURRENCY: usize = 4;
/// The signed `v-acctoken` format URLs last ~12h; keep details well inside that.
const DETAIL_TTL: Duration = Duration::from_secs(2 * 60 * 60);
const DETAIL_CACHE_SIZE: usize = 5000;
const TARGET_TTL: Duration = Duration::from_secs(6 * 60 * 60);
const TARGET_CACHE_SIZE: usize = 2000;
/// A probed archive smaller than this loses to keyword search: "elf girl" is a
/// one-video tag, while the search finds hundreds.
const MIN_PROBED_ARCHIVE_ITEMS: usize = 4;
pub const CHANNEL_METADATA: crate::providers::ProviderChannelMetadata =
crate::providers::ProviderChannelMetadata {
@@ -36,15 +49,66 @@ error_chain! {
}
}
#[derive(Debug, Clone)]
/// What a request resolves to. Artists live under `/models/`, franchises and
/// genres under `/categories/`, characters and everything else under `/tags/`.
#[derive(Debug, Clone, PartialEq, Eq)]
enum Target {
Latest,
Search(String),
Artist(String),
Category(String),
Tag(String),
}
impl Target {
/// Archive path (`models/{slug}`, ...) for the archive targets.
fn archive_path(&self) -> Option<String> {
match self {
Target::Artist(slug) => Some(format!("models/{slug}")),
Target::Category(slug) => Some(format!("categories/{slug}")),
Target::Tag(slug) => Some(format!("tags/{slug}")),
_ => None,
}
}
}
#[derive(Clone)]
pub struct Rule34videoProvider {
url: String,
/// Enriched items (formats, tags, artist) by video URL. Every listing costs
/// ~23 detail pages, again for the prefetched next page; refetching them
/// on each request is what runs into the site's 429 rate limit.
details: VideoCache,
/// Bare query -> resolved target, so later pages skip the archive probes.
targets: Arc<Mutex<HashMap<String, (Instant, Target)>>>,
}
impl Rule34videoProvider {
pub fn new() -> Self {
let mut details = VideoCache::new();
details.max_size(DETAIL_CACHE_SIZE);
Rule34videoProvider {
url: "https://rule34video.com".to_string(),
details,
targets: Arc::new(Mutex::new(HashMap::new())),
}
}
fn remembered_target(&self, query: &str) -> Option<Target> {
let targets = self.targets.lock().ok()?;
let (time, target) = targets.get(&slug_dashed(query))?;
(time.elapsed() < TARGET_TTL).then(|| target.clone())
}
fn remember_target(&self, query: &str, target: &Target) {
if let Ok(mut targets) = self.targets.lock() {
if targets.len() >= TARGET_CACHE_SIZE {
targets.retain(|_, (time, _)| time.elapsed() < TARGET_TTL);
if targets.len() >= TARGET_CACHE_SIZE {
targets.clear();
}
}
targets.insert(slug_dashed(query), (Instant::now(), target.clone()));
}
}
@@ -93,6 +157,192 @@ impl Rule34videoProvider {
}
}
/// Explicit `artist:` / `category:` / `tag:` style prefixes (the server has
/// already stripped a leading `#`).
fn parse_prefixed(query: &str) -> Option<Target> {
let (kind, value) = query.trim().split_once(':')?;
let slug = slug_dashed(value);
if slug.is_empty() {
return None;
}
match kind.trim().to_ascii_lowercase().as_str() {
"artist" | "artists" | "model" | "models" | "author" | "creator" | "uploader" => {
Some(Target::Artist(slug))
}
"cat" | "category" | "categories" | "franchise" | "series" | "game" => {
Some(Target::Category(slug))
}
"tag" | "tags" | "character" | "char" => Some(Target::Tag(slug)),
_ => None,
}
}
/// Titles of the tags offered by `/search_ajax.php?tag=`.
fn parse_tag_suggestions(html: &str) -> Vec<String> {
let document = Html::parse_fragment(html);
let Ok(selector) = Selector::parse("input[data-title]") else {
return vec![];
};
document
.select(&selector)
.filter_map(|input| input.value().attr("data-title"))
.map(|title| title.trim().to_string())
.filter(|title| !title.is_empty())
.collect()
}
/// The suggested tag whose name (ignoring a trailing "(franchise)") equals
/// the query, e.g. "samus aran" -> "samus aran (metroid)".
fn best_tag_title(titles: &[String], query: &str) -> Option<String> {
let wanted = slug_dashed(query);
if wanted.is_empty() {
return None;
}
let mut exact = titles.iter().filter(|title| {
let base = title.split('(').next().unwrap_or(title);
slug_dashed(base) == wanted || slug_dashed(title) == wanted
});
let first = exact.next()?;
// Several franchises share a character name: ambiguous, leave it to search.
exact.next().is_none().then(|| first.clone())
}
/// A free-form query that names an artist, category or tag goes to that
/// archive (probed on the site; unknown names 404), else keyword search.
/// The second value is false when a probe failed (rate limit, outage) and
/// the answer must not be remembered.
async fn resolve_keyword(&self, query: &str, options: &ServerOptions) -> (Target, bool) {
let slug = slug_dashed(query);
if slug.is_empty() {
return (Target::Search(query.trim().to_string()), true);
}
// Artist slugs are inconsistent on the site: "general-butch" but "biggreen".
let mut candidates = vec![Target::Artist(slug.clone())];
if slug.contains('-') {
candidates.push(Target::Artist(slug.replace('-', "")));
}
candidates.push(Target::Category(slug.clone()));
candidates.push(Target::Tag(slug));
let mut requester = requester_or_default(options, "rule34video", "resolve_keyword");
let referer = format!("{}/", self.url);
let mut conclusive = true;
for candidate in &candidates {
let Some(path) = candidate.archive_path() else {
continue;
};
let url = format!("{}/{path}/", self.url);
let headers = vec![("Referer".to_string(), referer.clone())];
match requester.get_probe(&url, headers).await {
Ok(Some(body)) => {
let items = self.get_video_items_from_html(body).len();
if items >= MIN_PROBED_ARCHIVE_ITEMS {
return (candidate.clone(), true);
}
}
Ok(None) => {}
Err(_) => conclusive = false,
}
}
// Characters are tagged "name (franchise)": ask the site's own tag search.
let suggest_url = format!(
"{}/search_ajax.php?tag={}",
self.url,
utf8_percent_encode(query.trim(), NON_ALPHANUMERIC)
);
let headers = vec![
("Referer".to_string(), referer),
("X-Requested-With".to_string(), "XMLHttpRequest".to_string()),
];
match requester.get_probe(&suggest_url, headers).await {
Ok(Some(body)) => {
let titles = Self::parse_tag_suggestions(&body);
if let Some(title) = Self::best_tag_title(&titles, query) {
return (Target::Tag(slug_dashed(&title)), conclusive);
}
}
Ok(None) => {}
Err(_) => conclusive = false,
}
(Target::Search(query.trim().to_string()), conclusive)
}
async fn resolve_target(&self, query: Option<&str>, options: &ServerOptions) -> Target {
let Some(query) = query.map(str::trim).filter(|q| !q.is_empty()) else {
return Target::Latest;
};
if let Some(target) = Self::parse_prefixed(query) {
return target;
}
if let Some(target) = self.remembered_target(query) {
return target;
}
let (target, conclusive) = self.resolve_keyword(query, options).await;
if conclusive {
self.remember_target(query, &target);
}
target
}
async fn archive(
&self,
cache: VideoCache,
target: &Target,
page: u8,
sort: &str,
options: ServerOptions,
) -> Result<Vec<VideoItem>> {
let Some(path) = target.archive_path() else {
return Ok(vec![]);
};
let sort_val = match sort {
"post_date" | "video_viewed" | "rating" | "duration" | "pseudo_random" => sort,
_ => "post_date",
};
let index = format!("rule34video:{path}:{page}:{sort_val}");
if sort_val != "pseudo_random" {
if let Some((time, items)) = cache.get(&index) {
if time.elapsed().unwrap_or_default().as_secs() < 300 {
return Ok(items.clone());
}
}
}
let mut requester = options.requester.clone().ok_or("Requester missing")?;
let url = format!(
"{}/{path}/?mode=async&function=get_block&block_id=custom_list_videos_common_videos&sort_by={sort_val}&from={page}",
self.url
);
let headers = vec![("Referer".to_string(), format!("{}/{path}/", self.url))];
let stale = || cache.get(&index).map(|(_, items)| items).unwrap_or_default();
let text = match requester.get_probe(&url, headers).await {
Ok(Some(text)) => text,
// A 404 is an unknown archive (or a page past the end).
Ok(None) => return Ok(vec![]),
Err(e) => {
eprintln!("rule34video archive fetch failed for {url}: {e}");
return Ok(stale());
}
};
let video_items = self.get_video_items_from_html(text);
let video_items = self.enrich_items(video_items, requester).await;
if video_items.is_empty() {
return Ok(stale());
}
cache.insert(index, video_items.clone());
Ok(video_items)
}
/// `/search/{q}/` takes dashes for spaces. A raw `/` splits the path (404),
/// `%20` is refused (403) and `&` cuts the query short, so keep only words.
fn search_path_segment(query: &str) -> String {
slug_dashed(query)
.split('-')
.map(|word| utf8_percent_encode(word, NON_ALPHANUMERIC).to_string())
.collect::<Vec<_>>()
.join("-")
}
/// Helper to safely extract a string between two delimiters
fn extract_between<'a>(content: &'a str, start_pat: &str, end_pat: &str) -> Option<&'a str> {
let start_idx = content.find(start_pat)? + start_pat.len();
@@ -208,7 +458,7 @@ impl Rule34videoProvider {
let url = format!(
"{}/search/{}/?mode=async&function=get_block&block_id=custom_list_videos_videos_list_search&tag_ids=&sort_by={}&from_videos={}&from_albums={}&_={}",
self.url,
query.replace(" ", "-"),
Self::search_path_segment(query),
sort_val,
page,
page,
@@ -307,30 +557,106 @@ impl Rule34videoProvider {
(formats.into_iter().map(|(_, f)| f).collect(), tags)
}
/// Artists credited on a video page: `(name, slug)` from the "Artist" group.
fn parse_artists(html: &str) -> Vec<(String, String)> {
let document = Html::parse_document(html);
let Ok(group_sel) = Selector::parse("[data-suggest-type=\"model\"] a[href*=\"/models/\"]")
else {
return vec![];
};
let name_sel = Selector::parse(".name").ok();
let mut artists: Vec<(String, String)> = Vec::new();
for link in document.select(&group_sel) {
let href = link.value().attr("href").unwrap_or_default();
let slug = href
.split("/models/")
.nth(1)
.unwrap_or_default()
.trim_matches('/')
.to_string();
let name = name_sel
.as_ref()
.and_then(|sel| link.select(sel).next())
.map(|n| n.text().collect::<String>())
.unwrap_or_else(|| link.text().collect::<String>());
let name = name.split_whitespace().collect::<Vec<_>>().join(" ");
if slug.is_empty() || name.is_empty() || artists.iter().any(|(_, s)| *s == slug) {
continue;
}
artists.push((name, slug));
}
artists
}
fn cached_detail(&self, item: &mut VideoItem) -> bool {
let Some((time, cached)) = self.details.get(&item.url) else {
return false;
};
let Some(cached) = cached.into_iter().next() else {
return false;
};
if time.elapsed().unwrap_or(DETAIL_TTL) >= DETAIL_TTL {
return false;
}
item.formats = cached.formats;
item.tags = cached.tags;
item.uploader = cached.uploader;
item.uploaderUrl = cached.uploaderUrl;
item.uploaderId = cached.uploaderId;
true
}
async fn enrich_items(&self, items: Vec<VideoItem>, requester: Requester) -> Vec<VideoItem> {
let referer = format!("{}/", self.url);
let base = self.url.clone();
// Once the site rate-limits a detail page, the rest of the batch keeps
// the listing data instead of digging the hole deeper.
let throttled = Arc::new(AtomicBool::new(false));
stream::iter(items.into_iter().map(|mut item| {
let mut requester = requester.clone();
let referer = referer.clone();
let base = base.clone();
let throttled = throttled.clone();
async move {
if item.url.is_empty() {
if item.url.is_empty() || self.cached_detail(&mut item) {
return item;
}
let html = requester
.get_with_headers(
&item.url,
vec![("Referer".to_string(), referer)],
None,
)
.await
.unwrap_or_default();
let (formats, tags) = Self::parse_detail(&html);
if throttled.load(Ordering::Relaxed) {
return item;
}
// `get_probe` hands challenges to the full fallback chain but
// reports a lasting 429 as an error rather than escalating every
// detail page to FlareSolverr.
let headers = vec![("Referer".to_string(), referer)];
let html = match requester.get_probe(&item.url, headers).await {
Ok(Some(html)) => html,
Ok(None) => return item,
Err(_) => {
throttled.store(true, Ordering::Relaxed);
return item;
}
};
let (formats, mut tags) = Self::parse_detail(&html);
if !formats.is_empty() {
item.formats = Some(formats);
}
let artists = Self::parse_artists(&html);
if let Some((name, slug)) = artists.first() {
item.uploader = Some(name.clone());
item.uploaderUrl = Some(format!("{base}/models/{slug}/"));
item.uploaderId = Some(format!("rule34video:{slug}"));
}
for (name, _) in &artists {
if !tags.iter().any(|t| t.eq_ignore_ascii_case(name)) {
tags.push(name.clone());
}
}
if !tags.is_empty() {
item.tags = Some(tags);
}
if item.formats.is_some() {
self.details.insert(item.url.clone(), vec![item.clone()]);
}
item
}
}))
@@ -344,15 +670,15 @@ impl Rule34videoProvider {
return vec![];
}
// Safely isolate the video listing section
let video_listing = match Self::extract_between(
&html,
"id=\"custom_list_videos",
"<div class=\"pagination\"",
) {
Some(content) => content,
None => return vec![],
// Isolate the video listing section. A result that fits on one page has
// no pagination block, so the listing then runs to the end.
let Some(start) = html.find("id=\"custom_list_videos") else {
return vec![];
};
let video_listing = &html[start..];
let video_listing = video_listing
.find("<div class=\"pagination\"")
.map_or(video_listing, |end| &video_listing[..end]);
let mut items = Vec::new();
// Skip the first split result as it's the preamble
@@ -449,9 +775,11 @@ impl Provider for Rule34videoProvider {
) -> Vec<VideoItem> {
let page_num = page.parse::<u8>().unwrap_or(1);
let result = match query {
Some(q) => self.query(cache, page_num, &q, &sort, options).await,
None => self.get(cache, page_num, &sort, options).await,
let target = self.resolve_target(query.as_deref(), &options).await;
let result = match &target {
Target::Latest => self.get(cache, page_num, &sort, options).await,
Target::Search(q) => self.query(cache, page_num, q, &sort, options).await,
archive => self.archive(cache, archive, page_num, &sort, options).await,
};
match result {
@@ -517,4 +845,81 @@ mod tests {
let (formats, tags) = Rule34videoProvider::parse_detail("<html></html>");
assert!(formats.is_empty() && tags.is_empty());
}
#[test]
fn parses_prefixed_queries() {
assert_eq!(
Rule34videoProvider::parse_prefixed("artist:General Butch"),
Some(Target::Artist("general-butch".into()))
);
assert_eq!(
Rule34videoProvider::parse_prefixed("category:Super Smash Bros"),
Some(Target::Category("super-smash-bros".into()))
);
assert_eq!(
Rule34videoProvider::parse_prefixed("tag:black widow (marvel)"),
Some(Target::Tag("black-widow-marvel".into()))
);
assert_eq!(Rule34videoProvider::parse_prefixed("samus aran"), None);
}
#[test]
fn picks_the_unambiguous_character_tag() {
let titles = vec![
"samus aran (metroid)".to_string(),
"samus aran's mother".to_string(),
];
assert_eq!(
Rule34videoProvider::best_tag_title(&titles, "samus aran").as_deref(),
Some("samus aran (metroid)")
);
let ambiguous = vec!["peach (mario)".to_string(), "peach (pokemon)".to_string()];
assert_eq!(Rule34videoProvider::best_tag_title(&ambiguous, "peach"), None);
}
#[test]
fn parses_tag_suggestions_and_artists() {
let suggest = r#"<div class="item checkbox-container"><input id="tag_143" type="checkbox" value="143" data-title="samus aran (metroid)"/></div>"#;
assert_eq!(
Rule34videoProvider::parse_tag_suggestions(suggest),
vec!["samus aran (metroid)".to_string()]
);
let page = r#"<div class="col vp-group" data-suggest-type="model"><div class="label">Artist</div>
<span class="chip"><a class="item btn_link" href="https://rule34video.com/models/joigang-va/">
<span class="name">JOIGang (VA)</span></a></span></div>"#;
assert_eq!(
Rule34videoProvider::parse_artists(page),
vec![("JOIGang (VA)".to_string(), "joigang-va".to_string())]
);
}
#[test]
fn parses_a_single_page_listing_without_pagination() {
let html = r#"<div id="custom_list_videos_common_videos_items">
<div class="item thumb video_7"><a class="th js-open-popup" href="https://rule34video.com/video/7/elf/">
<img class="thumb" src="https://rule34video.com/t/7.jpg"><div class="time">0:30</div>
<div class="thumb_title">Elf</div></a></div></div></div>"#;
let items = Rule34videoProvider::new().get_video_items_from_html(html.to_string());
assert_eq!(items.len(), 1);
assert_eq!(items[0].id, "7");
}
#[test]
fn search_segment_uses_dashes_and_drops_path_breaking_punctuation() {
assert_eq!(Rule34videoProvider::search_path_segment("elf girl"), "elf-girl");
assert_eq!(Rule34videoProvider::search_path_segment("2b/9s"), "2b-9s");
assert_eq!(Rule34videoProvider::search_path_segment("Tifa & Aerith"), "tifa-aerith");
assert_eq!(Rule34videoProvider::search_path_segment("Pokémon"), "pok%C3%A9mon");
}
#[test]
fn remembers_resolved_targets_by_slug() {
let provider = Rule34videoProvider::new();
provider.remember_target("Samus Aran", &Target::Tag("samus-aran-metroid".into()));
assert_eq!(
provider.remembered_target("samus aran"),
Some(Target::Tag("samus-aran-metroid".into()))
);
assert_eq!(provider.remembered_target("samus"), None);
}
}

View File

@@ -1,7 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{
Provider, report_provider_error, report_provider_error_background, requester_or_default,
Provider, probe_first_archive, report_provider_error, report_provider_error_background,
requester_or_default,
};
use crate::status::*;
use crate::util::cache::VideoCache;
@@ -517,6 +518,28 @@ impl SextbProvider {
Target::Search { slug }
}
/// A free-form query that names an actress or a studio outside the loaded
/// catalogs goes to that archive (probed on the site), else keyword search.
async fn resolve_keyword(&self, slug: &str, options: &ServerOptions) -> Option<Target> {
if slug.is_empty() {
return None;
}
let candidates = [
Target::Actress { slug: slug.to_string() },
Target::Studio { slug: slug.to_string() },
];
let urls: Vec<String> = candidates
.iter()
.map(|t| self.build_url_for_target(t, 1))
.collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
body.contains("tray-item") && !body.contains("No Video were found")
})
.await?;
candidates.into_iter().nth(index)
}
fn build_url_for_target(&self, target: &Target, page: u16) -> String {
match target {
Target::Section { section, sort } => {
@@ -1092,7 +1115,12 @@ impl SextbProvider {
per_page_limit: usize,
options: ServerOptions,
) -> Result<Vec<VideoItem>> {
let target = self.resolve_query_target(query, sort);
let mut target = self.resolve_query_target(query, sort);
if let Target::Search { slug } = &target {
if let Some(found) = self.resolve_keyword(slug, &options).await {
target = found;
}
}
let url = self.build_url_for_target(&target, page);
self.fetch_items_for_url(cache, url, per_page_limit, &options)
.await

View File

@@ -1,7 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{
Provider, build_proxy_url, report_provider_error, requester_or_default,
Provider, build_proxy_url, probe_first_archive, report_provider_error, requester_or_default,
slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
@@ -367,6 +368,26 @@ impl ThepornbunnyProvider {
}
}
/// A free-form query that names a pornstar, studio or tag goes to that
/// archive (probed on the site; unknown names redirect home), else keyword
/// search.
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let urls: Vec<String> = ["pornstar", "studio", "tag"]
.iter()
.map(|kind| format!("{BASE_URL}/{kind}/{slug}/"))
.collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{BASE_URL}/"), |body| {
body.contains("th-description")
})
.await?;
Some(Target::Archive(urls[index].trim_end_matches('/').to_string()))
}
fn resolve_option_target(options: &ServerOptions, sort: &str) -> Target {
if let Some(cat) = options.categories.as_deref() {
if cat.starts_with(BASE_URL) && cat != "all" {
@@ -476,10 +497,15 @@ impl Provider for ThepornbunnyProvider {
let page = page.parse::<u16>().unwrap_or(1).max(1);
let per_page_limit = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
let target = match query {
let mut target = match query {
Some(ref q) if !q.trim().is_empty() => Self::resolve_query_target(q.trim()),
_ => Self::resolve_option_target(&options, &sort),
};
if let Target::Search(query) = &target {
if let Some(found) = Self::resolve_keyword(query, &options).await {
target = found;
}
}
match self
.fetch_target(cache, target, page, per_page_limit, options)

View File

@@ -1,10 +1,12 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{
Provider, build_proxy_url, proxy_aylo_thumb, report_provider_error, requester_or_default,
Provider, build_proxy_url, probe_first_archive, proxy_aylo_thumb, report_provider_error,
requester_or_default, slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::requester::Requester;
use crate::util::parse_abbreviated_number;
use crate::util::time::parse_time_to_seconds;
use crate::videos::{ServerOptions, VideoFormat, VideoItem};
@@ -136,6 +138,25 @@ impl Tube8Provider {
]
}
/// A free-form query that names a pornstar or a channel goes to that
/// archive (probed on the site; unknown names 404), else keyword search.
async fn resolve_keyword(&self, requester: &mut Requester, query: &str) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let candidates = [
Target::Pornstar { slug: slug.clone() },
Target::Channel { slug },
];
let urls: Vec<String> = candidates.iter().map(|t| self.build_url(t, 1)).collect();
let (index, _) = probe_first_archive(requester, &urls, &format!("{BASE_URL}/"), |body| {
body.contains("video-box")
})
.await?;
candidates.into_iter().nth(index)
}
fn target_from_request(query: Option<&str>, sort: &str) -> Target {
if let Some(q) = query.map(str::trim).filter(|v| !v.is_empty()) {
let lower = q.to_ascii_lowercase();
@@ -439,7 +460,13 @@ impl Provider for Tube8Provider {
options: ServerOptions,
) -> Vec<VideoItem> {
let page = page.parse::<u16>().unwrap_or(1).max(1);
let target = Self::target_from_request(query.as_deref(), &sort);
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
let mut target = Self::target_from_request(query.as_deref(), &sort);
if let Target::Search { query } = &target {
if let Some(found) = self.resolve_keyword(&mut requester, query).await {
target = found;
}
}
let url = self.build_url(&target, page);
let old_items = match cache.get(&url) {
@@ -450,7 +477,6 @@ impl Provider for Tube8Provider {
None => vec![],
};
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
let text = match requester
.get_with_headers(&url, Self::html_headers(), None)
.await

View File

@@ -592,7 +592,7 @@ mod tests {
#[test]
fn decodes_cyrillic_obfuscated_video_url() {
let raw = "L2dldF9maWxlLzI1Lzg2\u{041c}D\u{0415}zNWFkND\u{0415}zODNm\u{041c}zI0ZmJhYjJlZTg3\u{041c}jRkYm\u{041c}zOWNiZTIwNjJl\u{041c}ci8y\u{041c}TgwOD\u{0410}w\u{041c}\u{0421}8y\u{041c}TgwODcwNS8y\u{041c}TgwODcwNV9ocS5tcDQvP2Q9\u{041c}czQ4NSZicj0y\u{041c}TgmdGk9\u{041c}Tc5\u{041c}TY0NjYyOQ~~";
let raw = "L2dldF9maWxlLzI1Lzg2\u{041c}D\u{0415}zNWFkND\u{0415}zODNm\u{041c}zI0ZmJhYjJlZTg3\u{041c}jRkYm\u{041c}zOWNiZTIwNjJl\u{041c}i8y\u{041c}TgwOD\u{0410}w\u{041c}\u{0421}8y\u{041c}TgwODcwNS8y\u{041c}TgwODcwNV9ocS5tcDQvP2Q9\u{041c}zQ4NSZicj0y\u{041c}TgmdGk9\u{041c}Tc5\u{041c}TY0NjYyOQ~~";
assert_eq!(
TxxxProvider::decode_video_url(raw).as_deref(),
Some("https://txxx.me/get_file/25/860135ad41383f324fbab2ee8724dbc39cbe2062e2/21808000/21808705/21808705_hq.mp4/?d=3485&br=218&ti=1791646629")

View File

@@ -1,6 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::providers::{
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::videos::{ServerOptions, VideoItem};
@@ -426,6 +428,23 @@ impl XgroovyProvider {
Target::Search(trimmed.to_string())
}
/// A free-form query that names a pornstar or a channel goes to that
/// archive (probed on the site; unknown names 404), else keyword search.
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let candidates = [Target::Pornstar(slug.clone()), Target::Channel(slug)];
let urls: Vec<String> = candidates.iter().map(|t| Self::target_url(t, 1)).collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{BASE_URL}/"), |body| {
body.contains("data-video-id")
})
.await?;
candidates.into_iter().nth(index)
}
fn resolve_option_target(&self, options: &ServerOptions) -> Target {
if let Some(cat) = options.categories.as_deref() {
if cat != "all" && !cat.is_empty() {
@@ -488,10 +507,15 @@ impl Provider for XgroovyProvider {
let page = page.parse::<u16>().unwrap_or(1).max(1);
let per_page = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
let target = match query {
let mut target = match query {
Some(q) if !q.trim().is_empty() => self.resolve_query_target(q.trim()),
_ => self.resolve_option_target(&options),
};
if let Target::Search(query) = &target {
if let Some(found) = Self::resolve_keyword(query, &options).await {
target = found;
}
}
match self
.fetch_target(cache, target, page, per_page, options)

View File

@@ -1,6 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::providers::{
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::parse_abbreviated_number;
@@ -94,6 +96,7 @@ enum Target {
Search(String),
Category(String),
Channel(String),
Pornstar(String),
}
#[derive(Debug, Clone)]
@@ -208,6 +211,7 @@ impl XhamsterProvider {
}
Target::Category(slug) => format!("{BASE_URL}/categories/{slug}"),
Target::Channel(slug) => format!("{BASE_URL}/channels/{slug}"),
Target::Pornstar(slug) => format!("{BASE_URL}/pornstars/{slug}"),
};
if page <= 1 {
base
@@ -385,6 +389,7 @@ impl XhamsterProvider {
match kind.trim().to_ascii_lowercase().as_str() {
"cat" | "category" => return Target::Category(slug),
"channel" | "channels" => return Target::Channel(slug),
"pornstar" | "pornstars" | "star" | "model" => return Target::Pornstar(slug),
_ => {}
}
}
@@ -401,6 +406,23 @@ impl XhamsterProvider {
Target::Search(trimmed.to_string())
}
/// A free-form query that names a pornstar or a channel goes to that
/// archive (probed on the site; unknown names 404), else keyword search.
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let candidates = [Target::Pornstar(slug.clone()), Target::Channel(slug)];
let urls: Vec<String> = candidates.iter().map(|t| Self::target_url(t, 1)).collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{BASE_URL}/"), |body| {
body.contains("video-thumb-info__name")
})
.await?;
candidates.into_iter().nth(index)
}
fn resolve_sort_target(sort: &str) -> Target {
match sort.trim().to_ascii_lowercase().as_str() {
"popular" | "viewed" | "most_viewed" | "mostviewed" => Target::MostViewed,
@@ -472,10 +494,15 @@ impl Provider for XhamsterProvider {
let page = page.parse::<u16>().unwrap_or(1).max(1);
let per_page = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
let target = match query {
let mut target = match query {
Some(q) if !q.trim().is_empty() => self.resolve_query_target(q.trim()),
_ => self.resolve_option_target(&options, &sort),
};
if let Target::Search(query) = &target {
if let Some(found) = Self::resolve_keyword(query, &options).await {
target = found;
}
}
match self
.fetch_target(cache, target, page, per_page, options)

View File

@@ -1,6 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::providers::{
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::parse_abbreviated_number;
@@ -86,6 +88,8 @@ enum Target {
Best,
Search(String),
Archive(String),
/// Channel / pornstar profile, served as JSON by `/channels/{slug}/videos/new/{n}`.
Channel(String),
}
#[derive(Debug, Clone)]
@@ -238,6 +242,9 @@ impl XvideosProvider {
format!("{BASE_URL}/?k={encoded}&p={}", page - 1)
}
}
Target::Channel(slug) => {
format!("{BASE_URL}/channels/{slug}/videos/new/{}", page.saturating_sub(1))
}
Target::Archive(base_url) => {
let base = base_url.trim_end_matches('/');
if page <= 1 {
@@ -528,9 +535,8 @@ impl XvideosProvider {
let slug = value.replace(' ', "_");
return Target::Archive(format!("{BASE_URL}/c/{slug}"));
}
"uploader" | "channel" | "profile" => {
let slug = value.replace(' ', "_").to_ascii_lowercase();
return Target::Archive(format!("{BASE_URL}/{slug}"));
"uploader" | "channel" | "profile" | "pornstar" | "star" | "model" => {
return Target::Channel(slug_dashed(value));
}
_ => {}
}
@@ -545,6 +551,83 @@ impl XvideosProvider {
Target::Search(trimmed.to_string())
}
/// A free-form query that names a channel / pornstar profile goes to its
/// archive (probed on the site; unknown names land on an empty profile),
/// else keyword search.
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let target = Target::Channel(slug);
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
probe_first_archive(
&mut requester,
&[Self::target_url(&target, 1)],
&format!("{BASE_URL}/"),
|body| body.contains("\"videos\":[{"),
)
.await
.map(|_| target)
}
/// Parses the JSON served for a channel / profile video list.
fn parse_channel_json(json: &str, limit: usize) -> Result<Vec<VideoItem>> {
let root: serde_json::Value = serde_json::from_str(json)
.map_err(|e| Error::from(format!("channel json: {e}")))?;
let Some(videos) = root.get("videos").and_then(|v| v.as_array()) else {
return Ok(vec![]);
};
let text = |v: &serde_json::Value, key: &str| -> String {
v.get(key).and_then(|x| x.as_str()).unwrap_or_default().trim().to_string()
};
let mut items = Vec::new();
for video in videos {
let eid = text(video, "eid");
let thumb = text(video, "i");
let title = {
let full = text(video, "tf");
if full.is_empty() { text(video, "t") } else { full }
};
if eid.is_empty() || thumb.is_empty() || title.is_empty() {
continue;
}
let id = video
.get("id")
.map(|x| x.to_string().trim_matches('"').to_string())
.unwrap_or_else(|| eid.clone());
let mut item = VideoItem::new(
id,
Self::decode_html(&title),
format!("{BASE_URL}/video.{eid}/x"),
CHANNEL_ID.to_string(),
thumb,
Self::parse_duration(&text(video, "d")),
);
item.views = Self::parse_views(&text(video, "n"));
item.rating = text(video, "r").trim_end_matches('%').parse::<f32>().ok();
let preview = text(video, "ipu");
if !preview.is_empty() {
item.preview = Some(preview);
}
let name = text(video, "pn");
let slug = text(video, "pu");
if !name.is_empty() {
item.uploader = Some(name);
let slug = slug.trim_matches('/').to_string();
if !slug.is_empty() {
item.uploaderUrl = Some(format!("{BASE_URL}/{slug}"));
item.uploaderId = Some(format!("{CHANNEL_ID}:{slug}"));
}
}
items.push(item);
if items.len() >= limit {
break;
}
}
Ok(items)
}
fn resolve_option_target(options: &ServerOptions, sort: &str) -> Target {
if let Some(cat) = options.categories.as_deref() {
if cat != "all" && !cat.is_empty() {
@@ -602,7 +685,11 @@ impl XvideosProvider {
return Ok(vec![]);
}
let items = Self::parse_listing(&html, per_page)?;
let items = if matches!(target, Target::Channel(_)) {
Self::parse_channel_json(&html, per_page)?
} else {
Self::parse_listing(&html, per_page)?
};
if !items.is_empty() {
cache.insert(cache_key, items.clone());
}
@@ -625,10 +712,15 @@ impl Provider for XvideosProvider {
let page = page.parse::<u16>().unwrap_or(1).max(1);
let per_page = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
let target = match query {
let mut target = match query {
Some(q) if !q.trim().is_empty() => Self::resolve_query_target(q.trim()),
_ => Self::resolve_option_target(&options, &sort),
};
if let Target::Search(query) = &target {
if let Some(found) = Self::resolve_keyword(query, &options).await {
target = found;
}
}
match self
.fetch_target(cache, target, page, per_page, options)

View File

@@ -1,8 +1,11 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::providers::{
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::requester::Requester;
use crate::util::parse_abbreviated_number;
use crate::util::time::parse_time_to_seconds;
use crate::videos::{ServerOptions, VideoItem};
@@ -248,6 +251,33 @@ impl YoupornProvider {
}
}
/// A free-form query that names a pornstar, channel or amateur goes to that
/// archive (probed on the site; unknown names 404), else keyword search.
async fn resolve_keyword(
&self,
requester: &mut Requester,
query: &str,
sort: &str,
) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let sort = Self::normalized_sort(sort).to_string();
let candidates = [
Target::Pornstar { slug: slug.clone(), sort: sort.clone() },
Target::Channel { slug: slug.clone(), sort: sort.clone() },
Target::Amateur { slug, sort },
];
let urls: Vec<String> = candidates.iter().map(|t| self.build_url(t, 1)).collect();
let referer = format!("{}/", self.url.trim_end_matches('/'));
let (index, _) = probe_first_archive(requester, &urls, &referer, |body| {
body.contains("js_video-box")
})
.await?;
candidates.into_iter().nth(index)
}
fn build_url(&self, target: &Target, page: u8) -> String {
match target {
Target::Latest { sort } => format!(
@@ -543,7 +573,13 @@ impl Provider for YoupornProvider {
) -> Vec<VideoItem> {
let query = query.unwrap_or_default();
let page = page.parse::<u8>().unwrap_or(1);
let target = self.target_from_query(&query, &sort);
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
let mut target = self.target_from_query(&query, &sort);
if let Target::Search { query } = &target {
if let Some(found) = self.resolve_keyword(&mut requester, query, &sort).await {
target = found;
}
}
let video_url = self.build_url(&target, page);
let old_items = match cache.get(&video_url) {
@@ -554,7 +590,6 @@ impl Provider for YoupornProvider {
None => vec![],
};
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
let referer = format!("{}/", self.url.trim_end_matches('/'));
let text = match requester
.get_with_headers(&video_url, Self::html_headers(&referer), Some(Version::HTTP_11))

View File

@@ -229,63 +229,90 @@ impl ArchivebateProxy {
embed_url: &str,
requester: &mut Requester,
) -> Option<String> {
let response = requester
.get_raw_with_headers_timeout(
embed_url,
Self::html_headers(detail_url),
Some(StdDuration::from_secs(8)),
None,
)
.await
.ok()?;
if !response.status().is_success() {
return None;
// The embed host is sometimes slow on a first (cold) request and fast on
// the next, so one timed-out attempt is retried.
for _ in 0..2 {
let Ok(response) = requester
.get_raw_with_headers_timeout(
embed_url,
Self::html_headers(detail_url),
Some(StdDuration::from_secs(10)),
None,
)
.await
else {
continue;
};
if !response.status().is_success() {
return None;
}
let html = response.text().await.ok()?;
return Self::extract_mixdrop_media_url(&html);
}
let html = response.text().await.ok()?;
Self::extract_mixdrop_media_url(&html)
None
}
}
/// A resolved Mixdrop media file plus the embed page it came from (the CDN is
/// played with that page as `Referer`).
#[derive(Debug, Clone)]
pub struct ResolvedMedia {
pub media_url: String,
pub embed_url: String,
}
/// Resolves an archivebate watch page (`https://archivebate.com/watch/{id}`)
/// to the real media URL: watch page -> Mixdrop embed -> unpacked
/// `MDCore.wurl`. Shared by the lazy `/proxy/archivebate` redirect and by the
/// provider, which hands the app the media URL directly.
pub async fn resolve_media(requester: &mut Requester, detail_url: &str) -> Option<ResolvedMedia> {
let detail_html = requester
.get_with_headers(
detail_url,
ArchivebateProxy::html_headers("https://archivebate.com/"),
Some(Version::HTTP_11),
)
.await
.unwrap_or_default();
if detail_html.is_empty() {
return None;
}
let mut embeds: Vec<String> = Vec::new();
if let Some(iframe_url) = ArchivebateProxy::first_iframe_source_from_html(&detail_html)
.map(|value| ArchivebateProxy::absolute_url(&value))
{
if ArchivebateProxy::is_mixdrop_host(&iframe_url) {
embeds.push(iframe_url);
}
}
if let Some(embed_url) = ArchivebateProxy::download_fid_from_detail_html(&detail_html)
.map(|value| ArchivebateProxy::absolute_url(&value))
.and_then(|value| ArchivebateProxy::mixdrop_embed_url_from_download_url(&value))
{
embeds.push(embed_url);
}
for embed_url in embeds {
if let Some(media_url) =
ArchivebateProxy::resolve_mixdrop_media_from_embed(detail_url, &embed_url, requester).await
{
return Some(ResolvedMedia { media_url, embed_url });
}
}
None
}
impl crate::proxies::Proxy for ArchivebateProxy {
async fn get_video_url(&self, url: String, requester: web::types::State<Requester>) -> String {
let Some(detail_url) = Self::normalize_detail_request(&url) else {
return String::new();
};
let mut requester = requester.get_ref().clone();
let detail_html = requester
.get_with_headers(
&detail_url,
Self::html_headers("https://archivebate.com/"),
Some(Version::HTTP_11),
)
resolve_media(&mut requester, &detail_url)
.await
.unwrap_or_default();
if detail_html.is_empty() {
return String::new();
}
if let Some(iframe_url) = Self::first_iframe_source_from_html(&detail_html).map(|value| Self::absolute_url(&value)) {
if Self::is_mixdrop_host(&iframe_url) {
if let Some(media_url) =
Self::resolve_mixdrop_media_from_embed(&detail_url, &iframe_url, &mut requester).await
{
return media_url;
}
}
}
if let Some(download_fid) = Self::download_fid_from_detail_html(&detail_html).map(|value| Self::absolute_url(&value)) {
if let Some(embed_url) = Self::mixdrop_embed_url_from_download_url(&download_fid) {
if let Some(media_url) =
Self::resolve_mixdrop_media_from_embed(&detail_url, &embed_url, &mut requester).await
{
return media_url;
}
}
}
String::new()
.map(|resolved| resolved.media_url)
.unwrap_or_default()
}
}

196
src/proxies/eporner.rs Normal file
View File

@@ -0,0 +1,196 @@
use std::collections::HashMap;
use std::rc::Rc;
use std::sync::{Mutex, OnceLock};
use std::time::{Duration, Instant};
use futures::StreamExt;
use http_body_util::BodyExt;
use ntex::http::Method;
use ntex::http::body::{Body, SizedStream};
use ntex::http::header::{ACCEPT_RANGES, CONTENT_LENGTH, CONTENT_RANGE, CONTENT_TYPE, RANGE};
use ntex::util::Bytes;
use ntex::web::{self, HttpRequest};
use wreq::Client;
use wreq::redirect::Policy;
use wreq_util::Emulation;
use crate::util::eporner_resolve::resolve_ranked;
use crate::util::requester::Requester;
/// The signed CDN links expire two hours after resolving; renew well before.
const LINK_TTL: Duration = Duration::from_secs(90 * 60);
const LINK_CACHE_SIZE: usize = 2000;
/// `{alnum_id}/{height}` -> (resolved at, signed CDN URL).
fn links() -> &'static Mutex<HashMap<String, (Instant, String)>> {
static LINKS: OnceLock<Mutex<HashMap<String, (Instant, String)>>> = OnceLock::new();
LINKS.get_or_init(|| Mutex::new(HashMap::new()))
}
/// Redirects are not followed so the CDN's `na.mp4` decoy shows up as a 302.
fn cdn_client() -> &'static Client {
static CLIENT: OnceLock<Client> = OnceLock::new();
CLIENT.get_or_init(|| {
Client::builder()
.tls_cert_verification(false)
.emulation(Emulation::Firefox151)
.redirect(Policy::none())
.build()
.expect("eporner CDN client")
})
}
/// `{alnum_id}[/{height}][.mp4]`.
fn parse_endpoint(endpoint: &str) -> Option<(String, Option<u32>)> {
let endpoint = endpoint.trim_matches('/').trim_end_matches(".mp4");
let mut parts = endpoint.split('/');
let alnum_id = parts
.next()
.filter(|id| !id.is_empty() && id.chars().all(|c| c.is_ascii_alphanumeric()))?;
let height = parts.next().and_then(|h| h.parse::<u32>().ok());
Some((alnum_id.to_string(), height))
}
async fn resolve_link(
requester: &web::types::State<Requester>,
alnum_id: &str,
height: Option<u32>,
) -> Option<String> {
let page_url = format!("https://www.eporner.com/video-{alnum_id}/");
let mut requester = requester.get_ref().clone();
let ranked = resolve_ranked(&mut requester, alnum_id, &page_url).await?;
height
.and_then(|h| ranked.iter().find(|(height, _)| *height == h))
.or_else(|| ranked.first())
.map(|(_, format)| format.url.clone())
}
async fn cached_link(
requester: &web::types::State<Requester>,
alnum_id: &str,
height: Option<u32>,
fresh: bool,
) -> Option<String> {
let key = format!("{alnum_id}/{}", height.unwrap_or(0));
if !fresh {
if let Ok(links) = links().lock() {
if let Some((time, url)) = links.get(&key) {
if time.elapsed() < LINK_TTL {
return Some(url.clone());
}
}
}
}
let url = resolve_link(requester, alnum_id, height).await?;
if let Ok(mut links) = links().lock() {
if links.len() >= LINK_CACHE_SIZE {
links.retain(|_, (time, _)| time.elapsed() < LINK_TTL);
if links.len() >= LINK_CACHE_SIZE {
links.clear();
}
}
links.insert(key, (Instant::now(), url.clone()));
}
Some(url)
}
/// Streams the mp4 through this server. The signed CDN links only serve the
/// IP that resolved them (checked from other networks: they answer every
/// other client with a 302 to `static.eporner.com/na.mp4`, a one-minute
/// "watch it on eporner.com" clip), so a redirect to them only ever works for
/// a player that shares the server's public IP.
pub async fn serve_media(
req: HttpRequest,
requester: web::types::State<Requester>,
) -> Result<impl web::Responder, web::Error> {
let endpoint = req.match_info().query("endpoint").to_string();
let Some((alnum_id, height)) = parse_endpoint(&endpoint) else {
return Ok(web::HttpResponse::BadRequest().finish());
};
let range = req
.headers()
.get(RANGE)
.and_then(|value| value.to_str().ok())
.map(str::to_string);
let head = req.method() == Method::HEAD;
let mut upstream = None;
for fresh in [false, true] {
let Some(url) = cached_link(&requester, &alnum_id, height, fresh).await else {
break;
};
let mut request = if head {
cdn_client().head(&url)
} else {
cdn_client().get(&url)
};
request = request.header("Referer", "https://www.eporner.com/");
if let Some(range) = &range {
request = request.header("Range", range);
}
match request.send().await {
// A redirect is the decoy for an expired or foreign link.
Ok(response) if !response.status().is_redirection() => {
upstream = Some(response);
break;
}
_ => continue,
}
}
let Some(upstream) = upstream else {
return Ok(web::HttpResponse::BadGateway().finish());
};
let status = upstream.status();
let headers = upstream.headers().clone();
let mut builder = web::HttpResponse::build(status);
builder.header(
CONTENT_TYPE,
headers
.get(CONTENT_TYPE)
.and_then(|value| value.to_str().ok())
.unwrap_or("video/mp4"),
);
builder.header(ACCEPT_RANGES, "bytes");
if let Some(value) = headers.get(CONTENT_RANGE.as_str()).and_then(|v| v.to_str().ok()) {
builder.header(CONTENT_RANGE, value);
}
if head {
// `finish()` would send `Content-Length: 0` over the upstream length.
return Ok(builder.body(Body::None));
}
let length = headers
.get(CONTENT_LENGTH.as_str())
.and_then(|v| v.to_str().ok())
.and_then(|v| v.parse::<u64>().ok());
let stream = Box::pin(upstream.into_data_stream().map(|chunk| {
chunk
.map(|bytes| Bytes::copy_from_slice(bytes.as_ref()))
.map_err(|e| std::io::Error::other(e.to_string()))
}));
let Some(length) = length else {
return Ok(builder.streaming(stream));
};
// A sized body keeps `Content-Length` on the (206) responses; a chunked
// one is what AVPlayer refuses to seek in.
let stream = stream.map(|chunk| chunk.map_err(|e| Rc::new(e) as Rc<dyn std::error::Error>));
Ok(builder.body(Body::from_message(SizedStream::new(length, stream))))
}
#[cfg(test)]
mod tests {
use super::parse_endpoint;
#[test]
fn parses_id_and_optional_height() {
assert_eq!(
parse_endpoint("exfeTXHuL1x/720.mp4"),
Some(("exfeTXHuL1x".to_string(), Some(720)))
);
assert_eq!(
parse_endpoint("/exfeTXHuL1x/"),
Some(("exfeTXHuL1x".to_string(), None))
);
assert_eq!(parse_endpoint("../etc"), None);
}
}

71
src/proxies/hanime.rs Normal file
View File

@@ -0,0 +1,71 @@
use ntex::http::Method;
use ntex::http::header::LOCATION;
use ntex::web::{self, HttpRequest};
use crate::util::hanime_resolve::{fetch_sources, pick_source, source_url};
use crate::util::requester::Requester;
/// `{slug}[/{height}][.m3u8]`.
fn parse_endpoint(endpoint: &str) -> Option<(String, Option<u32>)> {
let endpoint = endpoint.trim_matches('/').trim_end_matches(".m3u8");
let mut parts = endpoint.split('/');
let slug = parts
.next()
.filter(|slug| {
!slug.is_empty()
&& slug
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_')
})?;
let height = parts.next().and_then(|h| h.parse::<u32>().ok());
Some((slug.to_string(), height))
}
/// hanime's stream links expire, so the channel hands out this stable URL and a
/// fresh handshake happens on every play: redirects to the current playlist.
pub async fn redirect_to_media(
req: HttpRequest,
requester: web::types::State<Requester>,
) -> Result<impl web::Responder, web::Error> {
let endpoint = req.match_info().query("endpoint").to_string();
let Some((slug, height)) = parse_endpoint(&endpoint) else {
return Ok(web::HttpResponse::BadRequest().finish());
};
if req.method() == Method::HEAD {
return Ok(web::HttpResponse::Ok().finish());
}
let mut requester = requester.get_ref().clone();
let Ok(sources) = fetch_sources(&mut requester, &slug).await else {
return Ok(web::HttpResponse::BadGateway().finish());
};
let Some(source) = pick_source(&sources, height) else {
return Ok(web::HttpResponse::NotFound().finish());
};
Ok(web::HttpResponse::Found()
.header(LOCATION, source_url(source))
.finish())
}
#[cfg(test)]
mod tests {
use super::parse_endpoint;
#[test]
fn parses_slug_and_height() {
assert_eq!(
parse_endpoint("natsu-zuma-2/720.m3u8"),
Some(("natsu-zuma-2".to_string(), Some(720)))
);
assert_eq!(
parse_endpoint("/natsu-zuma-2"),
Some(("natsu-zuma-2".to_string(), None))
);
}
#[test]
fn rejects_unsafe_slugs() {
assert_eq!(parse_endpoint("../etc/720.m3u8"), None);
assert_eq!(parse_endpoint(""), None);
}
}

View File

@@ -24,7 +24,9 @@ pub mod animeidhentai;
pub mod archivebate;
pub mod clapdat;
pub mod doodstream;
pub mod eporner;
pub mod fikfapthumb;
pub mod hanime;
pub mod hanimecdn;
pub mod hanimethumb;
pub mod heavyfetish;

View File

@@ -20,6 +20,40 @@ use crate::proxies::thaiporntv::ThaipornTvProxy;
use crate::proxies::*;
use crate::util::requester::Requester;
/// Proxies that answer with a `302` to the real media instead of streaming it.
/// The Hot Tub app cannot follow a redirect inside `formats[].url`, so for app
/// clients a format pointing at one of these is moved to `video.url` (see
/// `move_redirect_formats_to_url`). Keep in sync with the `proxy2redirect`
/// routes below; `redirect_proxy_list_matches_routes` enforces it.
pub const REDIRECT_PROXIES: &[&str] = &[
"archivebate",
"clapdat",
"doodstream",
"lulustream",
"sxyprn",
"javtiful",
"spankbang",
"porndish",
"hqporner",
"heavyfetish",
"vjav",
"pornhd3x",
"vidara",
"pimpbunny",
"thaiporntv",
"tube8",
"supjav",
];
/// True for a `/proxy/<redirect proxy>/...` URL (absolute or host-relative).
pub fn is_redirect_proxy_url(url: &str) -> bool {
let Some(rest) = url.split_once("/proxy/").map(|(_, rest)| rest) else {
return false;
};
let name = rest.split('/').next().unwrap_or_default();
REDIRECT_PROXIES.contains(&name)
}
pub fn config(cfg: &mut web::ServiceConfig) {
cfg.service(
web::resource("/archivebate/{endpoint}*")
@@ -123,6 +157,11 @@ pub fn config(cfg: &mut web::ServiceConfig) {
.route(web::post().to(crate::proxies::fikfapthumb::get_image))
.route(web::get().to(crate::proxies::fikfapthumb::get_image)),
)
.service(
web::resource("/hanime/{endpoint}*")
.route(web::get().to(crate::proxies::hanime::redirect_to_media))
.route(web::head().to(crate::proxies::hanime::redirect_to_media)),
)
.service(
web::resource("/hanime-cdn/{endpoint}*")
.route(web::post().to(crate::proxies::hanimecdn::get_image))
@@ -194,6 +233,11 @@ pub fn config(cfg: &mut web::ServiceConfig) {
.route(web::post().to(crate::proxies::melonstube::serve))
.route(web::get().to(crate::proxies::melonstube::serve)),
);
cfg.service(
web::resource("/eporner/{endpoint}*")
.route(web::get().to(crate::proxies::eporner::serve_media))
.route(web::head().to(crate::proxies::eporner::serve_media)),
);
cfg.service(
web::resource("/xiaoshenke/{endpoint}*")
.route(web::post().to(crate::proxies::xiaoshenke::serve_media))
@@ -239,3 +283,39 @@ fn get_proxy(proxy: &str) -> Option<AnyProxy> {
_ => None,
}
}
#[cfg(test)]
mod redirect_proxy_tests {
use super::*;
#[test]
fn recognises_redirect_proxy_urls() {
assert!(is_redirect_proxy_url("http://h/proxy/archivebate/archivebate.com/watch/1"));
assert!(is_redirect_proxy_url("/proxy/hqporner/x"));
assert!(!is_redirect_proxy_url("http://h/proxy/noodlemagazine/x"));
assert!(!is_redirect_proxy_url("https://cdn.example/video.mp4"));
}
#[test]
fn redirect_proxy_list_matches_routes() {
let source = include_str!("proxy.rs");
let config_body = source
.split("pub fn config(")
.nth(1)
.and_then(|rest| rest.split("async fn proxy2redirect(").next())
.expect("config body");
let mut from_routes: Vec<String> = Vec::new();
for block in config_body.split("web::resource(\"/").skip(1) {
let name = block.split("/{endpoint}").next().unwrap_or_default();
let until_next = block.split("web::resource(").next().unwrap_or(block);
if until_next.contains("proxy2redirect") {
from_routes.push(name.to_string());
}
}
let mut listed: Vec<String> = REDIRECT_PROXIES.iter().map(|s| s.to_string()).collect();
from_routes.sort();
from_routes.dedup();
listed.sort();
assert_eq!(listed, from_routes);
}
}

View File

@@ -12,7 +12,7 @@
//! always compiled regardless of provider selection.
use crate::util::requester::Requester;
use crate::videos::VideoFormat;
use crate::videos::{ServerOptions, VideoFormat};
use regex::Regex;
use std::sync::OnceLock;
use wreq::Version;
@@ -101,11 +101,18 @@ fn calc_hash(hash: &str) -> Option<String> {
/// when playback doesn't carry the header the CDN wants. Each format
/// therefore declares an explicit `Referer` so the client's own default (or
/// lack thereof) can't accidentally trip this check.
pub async fn resolve_formats(
///
/// The signed CDN URLs also expire two hours after this call (the leading
/// number in `/{expiry}_{ip}_{n}/` is that deadline) and only serve the IP
/// embedded there; anything else gets the same `na.mp4` decoy.
/// Anything handed to a client should therefore be a
/// [`resolve_proxied_formats`] URL, streamed through this server.
/// Returns `(height, format)`, best first.
pub async fn resolve_ranked(
requester: &mut Requester,
alnum_id: &str,
page_url: &str,
) -> Option<Vec<VideoFormat>> {
) -> Option<Vec<(u32, VideoFormat)>> {
let detail_html = requester
.get_with_headers(page_url, html_headers(page_url), Some(Version::HTTP_11))
.await
@@ -151,25 +158,56 @@ pub async fn resolve_formats(
return None;
}
ranked.sort_by(|a, b| b.0.cmp(&a.0));
Some(ranked.into_iter().map(|(_, f)| f).collect())
Some(ranked)
}
/// Resolves eporner.com direct mp4 formats for an arbitrary eporner detail
/// page URL, extracting the alnum id from the URL itself. For use by
/// providers that only have a raw destination URL (e.g. a meta-search
/// aggregator's out-link), not a pre-parsed listing card.
pub async fn resolve_public_formats(
/// Resolves the qualities a video offers, but points each one at the
/// `/proxy/eporner/{alnum_id}/{height}.mp4` streaming proxy: the CDN URLs
/// expire after two hours and only serve the IP that resolved them.
pub async fn resolve_proxied_formats(
requester: &mut Requester,
options: &ServerOptions,
alnum_id: &str,
page_url: &str,
) -> Option<Vec<VideoFormat>> {
let alnum_id = alnum_id_from_href(page_url)?;
resolve_formats(requester, &alnum_id, page_url).await
let ranked = resolve_ranked(requester, alnum_id, page_url).await?;
let base = options.public_url_base.as_deref().unwrap_or("");
Some(proxied_formats(base, alnum_id, ranked))
}
/// Builds the URLs like `providers::build_proxy_url`, which this always
/// compiled module cannot rely on in single-provider builds.
fn proxied_formats(
public_url_base: &str,
alnum_id: &str,
ranked: Vec<(u32, VideoFormat)>,
) -> Vec<VideoFormat> {
let base = public_url_base.trim_end_matches('/');
ranked
.into_iter()
.map(|(height, mut format)| {
format.url = format!("{base}/proxy/eporner/{alnum_id}/{height}.mp4");
format
})
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn proxied_formats_point_at_the_streaming_proxy() {
let ranked = vec![(
720,
VideoFormat::new("https://vid.cdn/x-720p.mp4".into(), "720p".into(), "mp4".into()),
)];
let formats = proxied_formats("https://hottub.example/", "exfeTXHuL1x", ranked);
assert_eq!(formats[0].url, "https://hottub.example/proxy/eporner/exfeTXHuL1x/720.mp4");
// Streamed, not redirected: app clients must keep it in `formats`.
assert!(!crate::proxy::is_redirect_proxy_url(&formats[0].url));
}
#[test]
fn calc_hash_matches_site_algorithm() {
// Reverse-engineered from vjs.js; verified against a live page/xhr pair.

208
src/util/hanime_resolve.rs Normal file
View File

@@ -0,0 +1,208 @@
//! hanime.tv stream resolution, shared by the `hanime` channel and its
//! `/proxy/hanime/` redirect proxy.
//!
//! Lives here (always compiled) rather than in `providers/hanime.rs` because
//! `HOT_TUB_PROVIDER=<id>` single-provider builds only compile the selected
//! provider, while the proxy routes are always registered.
use aes_gcm::aead::{Aead, KeyInit, Payload};
use aes_gcm::{Aes256Gcm, Key, Nonce};
use base64::Engine;
use serde_json::json;
use sha2::{Digest, Sha256};
use std::time::{SystemTime, UNIX_EPOCH};
use crate::util::requester::Requester;
const HANDSHAKE_URL: &str = "https://auth.hanime.tv/api/v11/handshake";
pub const SITE_ORIGIN: &str = "https://hanime.tv";
/// Envelope encryption used for the handshake request/response bodies.
const HANDSHAKE_KEY_SEED: &[u8] = b"htv-insecure-handshake-v1";
const HANDSHAKE_AAD: &[u8] = b"htv-insecure-v1";
/// Pieces of the `x-signature` pre-image, as baked into the site's wasm module.
const SIGNATURE_SECRET: &str = "Xkdi29";
const SIGNATURE_SALT: &str = "mn2";
type Result<T> = std::result::Result<T, String>;
fn err(message: impl Into<String>) -> String {
message.into()
}
#[derive(serde::Deserialize, Debug)]
pub struct HanimeHandshakePayload {
#[serde(default)]
pub sources: Vec<HanimeSource>,
}
#[derive(serde::Deserialize, Debug, Clone)]
pub struct HanimeSource {
#[serde(default)]
pub src: String,
#[serde(default)]
pub height: u32,
#[serde(default)]
pub label: String,
/// `"normal"` for the free streams, `"promotion"` for the premium-only teaser
/// entries which always carry an empty `src`.
#[serde(default)]
pub kind: String,
}
fn b64() -> base64::engine::general_purpose::GeneralPurpose {
base64::engine::general_purpose::URL_SAFE_NO_PAD
}
fn b64_decode(value: &str) -> Result<Vec<u8>> {
b64()
.decode(value.trim_end_matches('='))
.map_err(|e| err(format!("base64url decode failed: {e}")))
}
fn handshake_cipher() -> Aes256Gcm {
let digest = Sha256::digest(HANDSHAKE_KEY_SEED);
let key = Key::<Aes256Gcm>::from_slice(digest.as_slice());
Aes256Gcm::new(key)
}
/// `{"v":1,"alg":"AES-256-GCM","iv":..,"tag":..,"data":..}`, base64url encoded.
fn seal_envelope(plaintext: &[u8]) -> Result<String> {
let mut iv = [0u8; 12];
rand::fill(&mut iv);
let sealed = handshake_cipher()
.encrypt(
Nonce::from_slice(&iv),
Payload {
msg: plaintext,
aad: HANDSHAKE_AAD,
},
)
.map_err(|e| err(format!("handshake encrypt failed: {e}")))?;
if sealed.len() < 16 {
return Err(err("handshake ciphertext too short"));
}
let (data, tag) = sealed.split_at(sealed.len() - 16);
let envelope = json!({
"v": 1,
"alg": "AES-256-GCM",
"iv": b64().encode(iv),
"tag": b64().encode(tag),
"data": b64().encode(data),
});
Ok(b64().encode(envelope.to_string().as_bytes()))
}
fn open_envelope(token: &str) -> Result<Vec<u8>> {
let envelope: serde_json::Value = serde_json::from_slice(&b64_decode(token)?)
.map_err(|e| err(format!("handshake envelope is not JSON: {e}")))?;
let field = |name: &str| -> Result<Vec<u8>> {
let raw = envelope
.get(name)
.and_then(|v| v.as_str())
.ok_or_else(|| err(format!("handshake envelope missing {name}")))?;
b64_decode(raw)
};
let iv = field("iv")?;
let mut ciphertext = field("data")?;
ciphertext.extend_from_slice(&field("tag")?);
handshake_cipher()
.decrypt(
Nonce::from_slice(&iv),
Payload {
msg: &ciphertext,
aad: HANDSHAKE_AAD,
},
)
.map_err(|e| err(format!("handshake decrypt failed: {e}")))
}
fn signature_for(timestamp: u64) -> String {
let pre_image =
format!("{timestamp},{SIGNATURE_SECRET},{SITE_ORIGIN},{SIGNATURE_SALT},{timestamp}");
hex::encode(Sha256::digest(pre_image.as_bytes()))
}
/// Returns the free `sources` for a slug, best quality first. The auth host
/// occasionally refuses one of a burst of parallel connections, so retry once.
pub async fn fetch_sources(requester: &mut Requester, slug: &str) -> Result<Vec<HanimeSource>> {
match handshake(requester, slug).await {
Ok(sources) => Ok(sources),
Err(_) => {
ntex::time::sleep(ntex::time::Millis(500)).await;
handshake(requester, slug).await
}
}
}
async fn handshake(requester: &mut Requester, slug: &str) -> Result<Vec<HanimeSource>> {
let timestamp = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs();
let token = seal_envelope(
json!({
"timestamp_unix": timestamp,
"directive": "htv_player_handshake",
"slug": slug,
})
.to_string()
.as_bytes(),
)?;
let response = requester
.post_json(
HANDSHAKE_URL,
&json!({ "token": token }),
vec![
("Origin".to_string(), SITE_ORIGIN.to_string()),
("Referer".to_string(), format!("{SITE_ORIGIN}/")),
("Accept".to_string(), "application/json".to_string()),
("x-signature-version".to_string(), "web2".to_string()),
("x-signature".to_string(), signature_for(timestamp)),
("x-time".to_string(), timestamp.to_string()),
],
)
.await
.map_err(|e| err(format!("handshake request failed: {e}")))?;
let status = response.status();
if !status.is_success() {
return Err(err(format!("handshake returned HTTP {status}")));
}
// The interesting part of the answer travels in a response header, not the body.
let encrypted = response
.headers()
.get("x-token")
.and_then(|v| v.to_str().ok())
.map(|v| v.to_string())
.ok_or_else(|| err("handshake response is missing x-token"))?;
let payload: HanimeHandshakePayload = serde_json::from_slice(&open_envelope(&encrypted)?)
.map_err(|e| err(format!("Failed to parse handshake payload: {e}")))?;
let mut sources: Vec<HanimeSource> = payload
.sources
.into_iter()
.filter(|s| s.kind == "normal" && !s.src.is_empty())
.collect();
sources.sort_by(|a, b| b.height.cmp(&a.height));
match sources.is_empty() {
true => Err(err("handshake returned no playable sources")),
false => Ok(sources),
}
}
pub fn source_url(source: &HanimeSource) -> String {
match source.src.starts_with("http") {
true => source.src.clone(),
false => format!("{SITE_ORIGIN}{}", source.src),
}
}
/// The source of the requested height, else the best one.
pub fn pick_source(sources: &[HanimeSource], height: Option<u32>) -> Option<&HanimeSource> {
height
.and_then(|h| sources.iter().find(|s| s.height == h))
.or_else(|| sources.first())
}

View File

@@ -6,6 +6,7 @@ pub mod discord;
pub mod eporner_resolve;
pub mod flaresolverr;
pub mod flow_debug;
pub mod hanime_resolve;
pub mod hoster_proxy;
pub mod proxy;
pub mod requester;

View File

@@ -222,6 +222,15 @@ impl Requester {
body.contains("Just a moment...")
|| body.contains("cf-turnstile-response")
|| body.contains("Enable JavaScript and cookies to continue")
|| Self::looks_like_ddos_guard_challenge(body)
}
/// DDoS-Guard interstitials (rule34video) are short JS-check pages, never real content.
fn looks_like_ddos_guard_challenge(body: &str) -> bool {
body.len() < 20_000
&& (body.contains("/.well-known/ddos-guard/")
|| body.contains("check.ddos-guard.net")
|| body.contains("DDoS-Guard</title>"))
}
fn jina_mirror_url(url: &str) -> Option<String> {
@@ -661,6 +670,74 @@ impl Requester {
Ok(response)
}
fn path_of(url: &str) -> Option<String> {
url::Url::parse(url).ok().map(|u| u.path().to_string())
}
/// `Url` percent-encodes a non-ASCII path ("/tags/pok%C3%A9mon/") while the
/// response URI keeps it raw ("/tags/pokémon/"): compare them decoded.
fn same_path(asked: &str, landed: &str) -> bool {
let normalize = |path: &str| {
percent_encoding::percent_decode_str(path)
.decode_utf8_lossy()
.trim_end_matches('/')
.to_lowercase()
};
normalize(asked) == normalize(landed)
}
/// Existence probe for archive pages (pornstar / studio / tag / category).
///
/// Reads the raw response so a real 404/410 answers immediately with
/// `Ok(None)` instead of escalating through the Jina/FlareSolverr chain
/// (minutes per miss). A Cloudflare challenge (403/503/"Just a moment...")
/// is handed to the full chain so FlareSolverr solves it, and 429/5xx are
/// retried once.
pub async fn get_probe(
&mut self,
url: &str,
headers: Vec<(String, String)>,
) -> Result<Option<String>, AnyErr> {
let mut last_err = String::new();
for attempt in 0..2 {
if attempt > 0 {
// An immediate retry of a 429 is all but certain to hit it again.
tokio::time::sleep(std::time::Duration::from_millis(1500)).await;
}
match self.get_raw_with_headers(url, headers.clone()).await {
Ok(response) => {
let status = response.status().as_u16();
// An unknown archive often redirects to the home page or a
// search page instead of answering 404: a redirect that
// lands on a different path is a miss.
let redirected_away = Self::path_of(url)
.zip(Some(response.uri().path().to_string()))
.is_some_and(|(asked, landed)| !Self::same_path(&asked, &landed));
let body = response.text().await.unwrap_or_default();
if status == 404 || status == 410 {
return Ok(None);
}
if status == 403
|| status == 503
|| Self::looks_like_cloudflare_challenge(&body)
{
return self.get_with_headers(url, headers, None).await.map(Some);
}
if redirected_away {
return Ok(None);
}
if status >= 500 || status == 429 {
last_err = format!("HTTP {status}");
continue;
}
return Ok(Some(body));
}
Err(e) => last_err = e.to_string(),
}
}
Err(format!("probe failed for {url}: {last_err}").into())
}
pub async fn get(
&mut self,
url: &str,
@@ -830,7 +907,9 @@ impl Requester {
rate_limit_retries,
MAX_RATE_LIMIT_RETRIES
);
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
// 1s, 2s, 4s: a site-wide limit needs longer than a second to lift.
tokio::time::sleep(std::time::Duration::from_secs(1 << (rate_limit_retries - 1)))
.await;
continue;
} else {
crate::flow_debug!(
@@ -964,6 +1043,14 @@ impl Requester {
mod tests {
use super::Requester;
#[test]
fn probe_paths_compare_percent_decoded() {
let asked = Requester::path_of("https://rule34video.com/tags/pokémon-species/").unwrap();
assert!(Requester::same_path(&asked, "/tags/pokémon-species/"));
assert!(Requester::same_path("/Models/X", "/models/x/"));
assert!(!Requester::same_path(&asked, "/search/"));
}
#[test]
fn new_requesters_share_cookie_jar() {
let a = Requester::new();
@@ -979,4 +1066,88 @@ mod tests {
assert!(cookie_header.contains("shared_cookie=1"));
}
async fn serve(listener: tokio::net::TcpListener, reply: impl Fn(&str) -> String + Send + 'static) {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
loop {
let Ok((mut sock, _)) = listener.accept().await else { return };
let mut buf = vec![0u8; 16384];
let mut len = 0;
// read headers, then the declared body
loop {
let Ok(n) = sock.read(&mut buf[len..]).await else { break };
if n == 0 { break; }
len += n;
let text = String::from_utf8_lossy(&buf[..len]).to_string();
if let Some(head_end) = text.find("\r\n\r\n") {
let want = text
.lines()
.find_map(|l| l.to_ascii_lowercase().strip_prefix("content-length:").map(|v| v.trim().parse::<usize>().unwrap_or(0)))
.unwrap_or(0);
if len >= head_end + 4 + want { break; }
}
}
let req = String::from_utf8_lossy(&buf[..len]).to_string();
let _ = sock.write_all(reply(&req).as_bytes()).await;
let _ = sock.shutdown().await;
}
}
fn http(status: &str, ctype: &str, extra: &str, body: &str) -> String {
format!(
"HTTP/1.1 {status}\r\ncontent-type: {ctype}\r\n{extra}content-length: {}\r\nconnection: close\r\n\r\n{body}",
body.len()
)
}
/// DDoS-Guard answers either a bare `403 Access denied.` (blocked fingerprint/IP) or a
/// short JS-check page. Both must end up at FlareSolverr and return the real listing.
#[tokio::test(flavor = "multi_thread")]
async fn ddos_guard_blocks_are_solved_through_flaresolverr() {
// distinct hosts: the cookie jar is process-wide, so a solved cookie would leak across passes
for (interstitial, host) in [(false, "127.0.0.1"), (true, "localhost")] {
let origin = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let flare = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let origin_port = origin.local_addr().unwrap().port();
let flare_port = flare.local_addr().unwrap().port();
tokio::spawn(serve(origin, move |req| {
if req.contains("ddg_ok=1") {
http("200 OK", "text/html", "", "<div class=\"item thumb video_1\">real listing</div>")
} else if interstitial {
http(
"200 OK",
"text/html",
"server: ddos-guard\r\n",
"<html><head><title>DDoS-Guard</title><script src=\"/.well-known/ddos-guard/check?context=free_splash\"></script></head></html>",
)
} else {
http("403 Forbidden", "text/plain", "server: ddos-guard\r\n", "Access denied.")
}
}));
tokio::spawn(serve(flare, move |req| {
let body = if req.contains("sessions.create") {
r#"{"status":"ok","message":"","session":"mock"}"#.to_string()
} else if req.contains("sessions.destroy") {
r#"{"status":"ok","message":""}"#.to_string()
} else {
format!(
r#"{{"status":"ok","message":"","startTimestamp":0,"endTimestamp":1,"version":"mock","solution":{{"url":"http://127.0.0.1:{origin_port}/","status":200,"response":"<div class=\"item thumb video_1\">real listing</div>","headers":{{}},"userAgent":"Mozilla/5.0 mock","cookies":[{{"name":"ddg_ok","value":"1","domain":"","path":"/"}}]}}}}"#
)
};
http("200 OK", "application/json", "", &body)
}));
unsafe { std::env::set_var("FLARE_URL", format!("http://127.0.0.1:{flare_port}/v1")) };
let mut requester = Requester::new();
let body = requester
.get(&format!("http://{host}:{origin_port}/?mode=async"), None)
.await
.expect("requester should fall through to FlareSolverr");
assert!(
body.contains("real listing"),
"interstitial={interstitial}: got {body:?}"
);
}
}
}