Compare commits
28 Commits
665fb2a0ce
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
cf0ae15f93 | ||
|
|
aa00abdd5b | ||
|
|
e5a2afcc70 | ||
|
|
6ba65f1457 | ||
|
|
b131910ea2 | ||
|
|
921ad93149 | ||
|
|
ad2abe0057 | ||
|
|
0aa8929f29 | ||
|
|
6c69b46395 | ||
|
|
f900987434 | ||
|
|
30271b9b7c | ||
|
|
a4364e5be3 | ||
|
|
edf6732f8e | ||
|
|
697836b914 | ||
|
|
5598216381 | ||
|
|
4c29d27b3e | ||
|
|
a486bc5502 | ||
|
|
270b327f73 | ||
|
|
813e2ec2ca | ||
|
|
26ef830f6e | ||
|
|
e34801f6f8 | ||
|
|
2646ca5659 | ||
|
|
981ec29ad8 | ||
|
|
76b9d4e058 | ||
|
|
59ea0495a6 | ||
|
|
1f95549167 | ||
|
|
1e56fe2da5 | ||
|
|
bdde02fa8c |
@@ -59,7 +59,7 @@ This is the current implementation inventory as of this snapshot of the repo. Us
|
||||
| `redtube` | `mainstream-tube` | no | no | Mainstream archive. |
|
||||
| `ripnsfw` | `onlyfans` | no | yes | Leaked-creator aggregator for ripnsfw.com — the entire catalogue (~2.5k rows) is a single client-side-rendered page whose data source is a published Google Sheet exported as CSV (`GET /data.csv`, no auth/CF gate); the provider fetches and parses that CSV directly instead of scraping any HTML, replicating the site's own JS `parseCSV`/`csvSplit` column layout (`THUMB,NEW,NOME,DOODSTREAM,LULUSTREAM,<link3>,DOOD,<guests>,SITE(bunkr links),TITLE,ID,#(tags),POST,,DATE,SIZE`). Feed/search/pagination/sort are all done in-memory over the fetched rows (no native site pagination exists — `perPage`-sized pages are sliced locally), with a 180s in-process cache to avoid re-fetching the CSV on every request. `tag:`/`category:`/`cat:` and `model:`/`uploader:` query prefixes filter by the parsed tag list / model name; bare keywords substring-match model name, guests, and tags. `video.url` is the site's client-rendered `https://ripnsfw.com/model/{slug}/{id}` route — the host has no server-side SPA fallback, so this 404s to any plain HTTP client (its custom 404 page stashes the path and JS-redirects to `/`, which restores the route — works in a real browser/webview only); `check.py`'s `_CF_PROTECTED_HOSTS` allowlist includes `ripnsfw.com` for this reason. Media is never hosted on ripnsfw.com itself: each row carries a Doodstream and/or Lulustream embed link, resolved into `formats[]` (Lulustream first, matching the site's own default-player order) via the **existing, previously-unused** `src/proxies/doodstream.rs` / `src/proxies/lulustream.rs` redirect proxies (`build_proxy_url(&options, "doodstream"/"lulustream", &strip_url_scheme(link))`) — no new proxy code was written. Both proxies resolve correctly to real signed CDN URLs (verified against live data), but neither CDN's final edge is reachable from this repo's dev sandbox: Lulustream's `*.tnmr.org` edge 403s datacenter/proxy IPs regardless of UA or TLS impersonation, and Doodstream's final `*.dood.video` edge resolves to a loopback/refused address from non-residential egress — both are IP-reputation-style CDN blocks, not code defects (same class of sandbox limitation already documented for `camsoda`/`hentaihaven`/`animeidhentai`), so `check.py` allowlists both by suffix (`_CF_PROTECTED_SUFFIXES`). No thumbnail proxy needed (`cdn.ripnsfw.com/thumbs/...` is directly hotlinkable, no CF/referer gate). No `/api/uploaders` (model pages exist but have no dedicated profile/stats schema confirmed). |
|
||||
| `rule34gen` | `ai` | no | no | AI group example. |
|
||||
| `rule34video` | `hentai-animation` | no | no | Hentai group example. |
|
||||
| `rule34video` | `hentai-animation` | no | no | Hentai group example. **Archives:** artists are `/models/{slug}/`, franchises/genres `/categories/{slug}/`, characters and everything else `/tags/{slug}/`; each is paged and sorted through `?mode=async&function=get_block&block_id=custom_list_videos_common_videos&sort_by={sort}&from={page}` (unknown slugs 404). Prefix queries `artist:`/`model:`/`creator:`, `category:`/`franchise:`/`series:`, `tag:`/`character:`; a bare query is probed as artist, then category, then tag, then resolved through `/search_ajax.php?tag=` (characters are tagged `name (franchise)`; an unambiguous exact base-name match wins), else keyword search. Cards are enriched from the detail page, which also yields the credited artist as `uploader` (`rule34video:{slug}`) and artist names as tags. |
|
||||
| `sextb` | `jav` | no | no | JAV family provider. sextb.net is Cloudflare-gated, so listing/pagination markdown is fetched through a hand-built Jina mirror URL (`mirror_url()`); the target is already an absolute `https://sextb.net/...` URL, so the scheme must be stripped before prepending the `https://r.jina.ai/http://` prefix — concatenating them directly (the pre-fix bug) produces a malformed `.../http://https://...` URL that Jina rejects outright. |
|
||||
| `shooshtime` | `onlyfans` | no | yes | Redirect proxy plus dedicated media route. Listing cards are `div.card` (card→`a.title` for the link/title, `a.image img` for thumb/preview, `div.duration`, `div.list div.box` for views+date, `div.name a.text` for uploader) — the site redesigned from an older `div.item.thumb`/`span.description` layout, so re-verify these selectors against `{BASE_URL}/videos/` (the real default-sort URL, not the bare homepage) if the feed ever goes empty again. |
|
||||
| `spankbang` | `mainstream-tube` | no | yes | Best template for redirect proxy plus anti-bot fetches. |
|
||||
@@ -83,7 +83,7 @@ This is the current implementation inventory as of this snapshot of the repo. Us
|
||||
| `fullporner` | `mainstream-tube` | no | no | HTML scraper for fullporner.com; thumbnail IDs derived from `/thumb/{id}.jpg` URLs and used to build direct `xiaoshenke.net/vid/{id}/720` media redirect URLs (Referer + User-Agent headers required); supports cat:/category:/pornstar:/star: shortcut queries; no proxy needed. |
|
||||
| `cumlouder` | `mainstream-tube` | no | no | HTML scraper for cumlouder.com (Spanish/amateur tube, same platform family as superporn). Cards are `a.muestra-escena[href^="/porn-video/"]` with lazy thumb in `img[data-src]` (protocol-relative `//th0cdn.cumlouder.com/...`, direct, no Referer/proxy), `h2` title, `.vistas` views, `.minutos` duration (`06:14 m`); relative date only ("2 years ago"), so `uploadedAt` is not set. Default feed `/porn/` (page N≥2 is the bare `/N/` path, not `/porn/N/`); category `/porn-videos/{slug}/{N}/`; tag `/tag/{slug}/{N}/`; series `/series/{slug}/{N}/`. **The site's own `/search/` (POST `form[search]` or `?q=`) is behind a Cloudflare managed challenge — 403 even in a real browser — so keyword queries are routed instead: a query matching a hardcoded category (72, scraped from `/categories/`) goes to the category archive, anything else to `/tag/{slug}/` (unknown tags 404 → empty list).** Pornstar pages live at `/girl/{slug}/{N}/` (same cards/pagination; unknown slug = fast 404). A free-form keyword that is not a category is tried as a pornstar page first, then as a tag archive; `girl:`/`pornstar:`/`star:`/`model:` force the pornstar page. Supports `cat:`/`category:`, `tag:`, `series:` prefixes and the `categories` option. A Cloudflare challenge on the raw listing fetch is handed to the requester's FlareSolverr chain. `video.url` is the detail page, which yt-dlp resolves natively via the generic html5 extractor (the page embeds a signed `m5cdnst.cumlouder.com/*.mp4?secure=…` `<source>`, no Referer needed), so no `formats`/enrichment. The origin intermittently answers a first, uncached request with a Cloudflare 522 (a retry succeeds) — the listing fetch retries up to 3×, and `check.py`/yt-dlp may show a one-off `HTTP=-1`/522 on a cold detail page that is not a provider fault. No proxy needed. |
|
||||
| `thepornbunny` | `mainstream-tube` | no | yes | KVS-style HTML scraper for thepornbunny.com; 24 items per site page; thumbnails at `https://www.thepornbunny.com/images/thumb/{id}.webp` from `data-original` attribute (no proxy needed); studio exposed as uploader; pornstar names in tags; `/proxy/thepornbunny/{slug}` fetches the video page, extracts `generate_mp4(enc_data, key, rnd, video_id)` args, decrypts `enc_data` via PBKDF2-HMAC-SHA512+AES-256-CBC to get an OK.ru session key, calls `api.ok.ru/fb.do?method=video.get&session_key=KEY&vids=RND` to get signed CDN URLs, and returns 302 to the best-quality okcdn.ru/vkuser.net MP4 URL (no special client headers needed); supports sort: new/popular/rated, 20 hardcoded categories via `categories` option, and tag:/category:/studio:/pornstar: query shortcuts. |
|
||||
| `eporner` | `mainstream-tube` | no | no | HTML scraper for eporner.com (5M+ videos); card selector `div.mb[data-id]` with inline duration/rating/views/uploader; thumbnails at `static-eu-cdn.eporner.com` (no proxy needed); pagination uses `/{N}/` suffix (page 1 = no suffix, page 2 = `/2/`); supports sort: new/popular/rated/best; 65 hardcoded categories, plus `cat:`/`category:`, `tag:`, `pornstar:`/`star:`, `uploader:`/`profile:` query-prefix routing to the matching archive, a bare-keyword category-name match, and a background-loaded pornstar name→URL map from `/pornstar-list/` (falls back to `/tag/{slug}` keyword search otherwise). `video.url` is the detail page — **not** yt-dlp-resolvable and not directly playable on its own: opening it bare serves a small restricted/preview clip. Real playback needs per-card enrichment (bounded `buffer_unordered(6)`): fetch the detail page for its 32-hex `hash`, derive the site's own hashed token (`calc_hash` — split into four 8-hex chunks, each parsed as u32 and base36-encoded, concatenated; reverse-engineered from the player JS), then call the site's own `GET /xhr/video/{alnum_id}?hash={calc}&device=generic&domain=www.eporner.com&fallback=false` JSON API for direct per-resolution mp4 URLs into `formats[]`. Separately from the hash dance, the resolved CDN mp4 URLs enforce their own Referer check: a request with no Referer or one pointing at eporner.com gets the real file, but any other non-empty Referer gets a 302 to a static `.../na.mp4` ("not available") decoy clip — the exact "this video is only available at eporner.com"-style short clip a client sees if its default (or app-injected) Referer doesn't match, so every format explicitly declares `http_headers: {Referer: "https://www.eporner.com/"}` rather than relying on the client's default. No proxy needed (formats are direct CDN URLs, just header-gated). **2026-09-21:** the `calc_hash`/hash-regex/`alnum_id_from_href`/`resolve_formats` logic moved to `src/util/eporner_resolve.rs` so `melonstube` (a meta-search aggregator that redirects out to eporner.com for ~34% of its own cards) can reuse the exact same resolution without a cross-provider `use`, which would break either provider's `HOT_TUB_PROVIDER=<id>` single-provider build (only the one selected provider module gets compiled; `src/util/*` always does). `eporner.rs` itself now just calls `crate::util::eporner_resolve::resolve_formats(...)`. |
|
||||
| `eporner` | `mainstream-tube` | no | yes | HTML scraper for eporner.com (5M+ videos); card selector `div.mb[data-id]` with inline duration/rating/views/uploader; thumbnails at `static-eu-cdn.eporner.com` (no proxy needed); pagination uses `/{N}/` suffix (page 1 = no suffix, page 2 = `/2/`); supports sort: new/popular/rated/best; 65 hardcoded categories, plus `cat:`/`category:`, `tag:`, `pornstar:`/`star:`, `uploader:`/`profile:` query-prefix routing to the matching archive, a bare-keyword category-name match, and a background-loaded pornstar name→URL map from `/pornstar-list/` (falls back to `/tag/{slug}` keyword search otherwise). `video.url` is the detail page — **not** yt-dlp-resolvable and not directly playable on its own: opening it bare serves a small restricted/preview clip. Real playback needs per-card enrichment (bounded `buffer_unordered(6)`): fetch the detail page for its 32-hex `hash`, derive the site's own hashed token (`calc_hash` — split into four 8-hex chunks, each parsed as u32 and base36-encoded, concatenated; reverse-engineered from the player JS), then call the site's own `GET /xhr/video/{alnum_id}?hash={calc}&device=generic&domain=www.eporner.com&fallback=false` JSON API for direct per-resolution mp4 URLs into `formats[]`. Separately from the hash dance, the resolved CDN mp4 URLs enforce their own Referer check: a request with no Referer or one pointing at eporner.com gets the real file, but any other non-empty Referer gets a 302 to a static `.../na.mp4` ("not available") decoy clip — the exact "this video is only available at eporner.com"-style short clip a client sees if its default (or app-injected) Referer doesn't match, so every format explicitly declares `http_headers: {Referer: "https://www.eporner.com/"}` rather than relying on the client's default. No proxy needed (formats are direct CDN URLs, just header-gated). **2026-09-21:** the `calc_hash`/hash-regex/`alnum_id_from_href`/`resolve_formats` logic moved to `src/util/eporner_resolve.rs` so `melonstube` (a meta-search aggregator that redirects out to eporner.com for ~34% of its own cards) can reuse the exact same resolution without a cross-provider `use`, which would break either provider's `HOT_TUB_PROVIDER=<id>` single-provider build (only the one selected provider module gets compiled; `src/util/*` always does). `eporner.rs` itself now just calls `crate::util::eporner_resolve::resolve_formats(...)`. **2026-10-22:** the signed CDN URLs expire exactly two hours after the `/xhr/video` call (the leading number in `/{expiry}_{ip}_{n}/`) and an expired link gets the same `na.mp4` decoy, so favourites, history and long-open lists played the one-minute clip. The links are also **IP-locked**: checked from foreign networks (check-host.net nodes) a fresh link answers 302 → `na.mp4`, while the resolving IP gets the file, so a redirect only works for a player behind the server's own public IP (a Jina fetch had misleadingly suggested otherwise). Formats therefore point at the streaming proxy `/proxy/eporner/{alnum_id}/{height}.mp4`: it resolves (cached 90 min per id/height, re-resolved when the CDN answers with a redirect), forwards `Range`, and returns a sized body so `206` responses keep `Content-Length` for AVPlayer. Not a redirect proxy, so app clients keep these in `formats`. All eporner playback bandwidth now flows through the server. melonstube's eporner destinations use the same `resolve_proxied_formats`. |
|
||||
| `xnxx` | `mainstream-tube` | no | no | HTML scraper for xnxx.com (10M+ videos); unified card parser handles two formats: `div.thumb-block[data-eid]` (search) and `div.thumb-block.video[data-video='{"id":...}']` (hits); eid extracted from `/video-{eid}/{slug}` URL path; thumbnails at `thumb-cdn77.xnxx-cdn.com` and `thumbs-gcore.xnxx-cdn.com` (no proxy, no Referer needed); 0-indexed pagination (page 1 = `/hits`, page N = `/hits/{N-1}`); default feed is `/hits` (most-viewed — xnxx has no chronological listing); search via `/search/{slug}` (works for keywords and tags); supports `tag:`, `cat:`, `category:` query shortcuts; yt-dlp resolves `video.url` natively (XNXX extractor, returns 4-7 HLS formats); no proxy needed. |
|
||||
| `xgroovy` | `mainstream-tube` | no | no | KVS (Kernel Video Sharing)-style HTML scraper for xgroovy.com; card selector `div.item[data-video-id]` with `data-views`/`data-rating` as clean numeric attributes directly on the card (no text-parsing needed, unlike xhamster); title/url from `a.popito[href] > strong.title`; thumbnails at `i.xgroovy.com` (no proxy, no Referer needed) from `img.thumb[data-jpg|src]`, preview clip from `img[data-preview]`; duration rendered as free text ("N min"/"N sec", not colon-separated) via a bespoke `parse_duration` helper; path-segment pagination (`/new/` = page 1, `/new/2/` = page 2, unlike xhamster's `?page=N` query string); feeds: `/new/` (default), categories via `/categories/{slug}/` (73 hardcoded categories as `categories` option), pornstars via `/pornstars/{slug}/`, channels via `/channels/{slug}/`; search via `/search/{dashed-query}/` (multi-word queries are lowercased and dash-joined); uploader kind inferred from URL path (`/pornstars/` vs `/channels/`) via `div.author-link > a`; supports `cat:`/`category:` query shortcuts plus static category name matching; yt-dlp resolves `video.url` via the generic html5-embed extractor (no dedicated xgroovy extractor, but resolves cleanly to a 1080p format); no proxy needed. |
|
||||
| `xhamster` | `mainstream-tube` | no | no | HTML scraper for xhamster.com; card selector `div[data-video-type="video"]` with `data-video-id`; thumbnails via `img[data-role="thumb-preview-img"]` at `ic-vt-nss.xhcdn.com` (no proxy, no Referer needed); pagination via `?page=N` query param (browse feeds use infinite-scroll so only search reliably returns different content per page); feeds: `/newest` (default), `/most-viewed`, `/best`; categories via `/categories/{slug}`; channels via `/channels/{slug}`; 43 hardcoded categories as `categories` option; uploader type inferred from URL path (`/channels/` → channel, `/creators/` → creator, `/pornstars/` → pornstar); supports `cat:`/`category:` and `channel:` query shortcuts, plus static category name matching; preview mp4 clips from `data-previewvideo` attribute; yt-dlp resolves `video.url` natively (xHamster extractor, 28 formats); no proxy needed. |
|
||||
@@ -126,6 +126,7 @@ These resolve a provider-specific input into a `302 Location`.
|
||||
These return binary media or images, sometimes rewriting manifests or forwarding cookies/referers.
|
||||
|
||||
- `/proxy/shooshtime-media/{endpoint}*`
|
||||
- `/proxy/eporner/{alnum_id}/{height}.mp4` (streams the IP-locked signed mp4 with Range passthrough; link cached 90 min)
|
||||
- `/proxy/noodlemagazine/{endpoint}*`
|
||||
- `/proxy/noodlemagazine-thumb/{endpoint}*`
|
||||
- `/proxy/hanime-cdn/{endpoint}*`
|
||||
|
||||
@@ -26,6 +26,7 @@ These rules cut across every step below. They are intentionally grouped at the t
|
||||
|
||||
### Output Shape
|
||||
|
||||
- **Verified Hot Tub app clients cannot follow redirects in `formats[].url`.** Redirect proxies (`REDIRECT_PROXIES` in `src/proxy.rs`) are for `video.url` in the app response; `api.rs` moves such formats to `url` for verified app clients as a safety net. Other clients keep the redirect-proxy URLs in `formats`.
|
||||
- Use `formats` for real media URLs (HLS, multi-quality, signed token), and keep `video.url` on a stable page URL. Never set `video.url` to a tokenized stream that needs a `Referer` to play — the `url` field has no header contract.
|
||||
- If thumbnails need a `Referer` and the player has no per-thumb header mechanism, proxy them via `/proxy/<id>-thumb/...`. The same applies to manifests with relative URIs.
|
||||
- Keep filter `id` values stable and machine-targeted. Display `title`s change; ids route.
|
||||
|
||||
42
docs/query-resolution-todo.md
Normal file
42
docs/query-resolution-todo.md
Normal file
@@ -0,0 +1,42 @@
|
||||
# Seamless query resolution todo
|
||||
|
||||
Goal: a free-form query ("gal ritchie", "family strokes") should reach the matching
|
||||
pornstar / studio / channel / tag archive **without** a `pornstar:` / `studio:` / `tag:` prefix,
|
||||
by asking the site (probe) and falling back to keyword search. Prefixes stay as explicit overrides.
|
||||
|
||||
Pattern (see `hooporn`, `cumlouder`): after known-category matching, probe page 1 of each entity
|
||||
archive for the slug with `Requester::get_probe` (404 = instant `None`, Cloudflare challenge goes to
|
||||
FlareSolverr, 5xx/429 retried once), use the first archive that yields cards (page N stays in that
|
||||
archive), else keyword search. A probe is only valid after checking what the site answers for an
|
||||
unknown slug (404 vs soft-404 that lists unrelated videos).
|
||||
|
||||
Status: `[ ]` open, `[x]` done and verified live (feed of the archive + unknown name falls back),
|
||||
`[-]` not applicable (reason).
|
||||
|
||||
| | Provider | Prefix kinds today | Plan / evidence |
|
||||
|---|---|---|---|
|
||||
| [x] | `hooporn` | cat, pornstar, studio | reference implementation |
|
||||
| [x] | `cumlouder` | cat, tag, series, girl | reference implementation (girl page, then tag) |
|
||||
| [x] | `fullporner` | cat, pornstar | probes `/pornstar/{slug}` (unknown = 404 with a suggestion grid, so status decides); `pornstar:` slug now dashed |
|
||||
| [x] | `fullpornerorg` | pornstar | probes `/videos/pornstar/{slug}/` (404 for unknown) |
|
||||
| [x] | `tube8` | tag, cat, channel, pornstar | probes pornstar then channel |
|
||||
| [x] | `youporn` | tag, channel, pornstar, amateur | probes pornstar, channel, amateur (plus existing background shortcut map) |
|
||||
| [x] | `porn4fans` | tag, cat, model | probes `/models/{slug}/` then `/tags/{slug}/` (both 404 when unknown) |
|
||||
| [x] | `xvideos` | tag, cat, uploader/channel/profile | probes `/channels/{slug}/videos/new/0` (JSON; pornstar profiles redirect there). **Also fixed**: the old `channel:`/`uploader:` prefix returned nothing (landing page cards are `/prof-video-click/` links); channels are now read from the JSON video list |
|
||||
| [x] | `xhamster` | cat, channel | added `Pornstar` target; probes pornstar then channel |
|
||||
| [x] | `xgroovy` | cat, pornstar, channel | probes pornstar then channel |
|
||||
| [x] | `thepornbunny` | tag, cat, studio, pornstar | probes pornstar, studio, tag (unknown slugs redirect home: `get_probe` now treats a redirect to another path as a miss) |
|
||||
| [x] | `pornhub` | channel, pornstar, model, user | background catalogs already matched listed names; unknown names are probed (`pornstar`, `model`, `channels`) and cached in `tag_map` (misses remembered) |
|
||||
| [x] | `clapdat` | tag, user | probes tag then user |
|
||||
| [x] | `fikfap` | tag, user | single-word query that is an existing profile (profile endpoint) goes to its posts, empty profile falls back to search |
|
||||
| [x] | `erome` | uploader, user | single-word query that is an existing `/{name}` with albums goes to the uploader |
|
||||
| [x] | `redgifs` | tag, user | single-word query that is an existing creator (users endpoint) goes to the creator, empty falls back |
|
||||
| [x] | `sextb` | (catalog matched) | catalog (actress/studio/genre) match first, then probes `/actress/{slug}` and `/studio/{slug}` for names outside the catalog |
|
||||
| [x] | `coomer` | u:, @service:user | single-word query that is an existing creator id (`/api/v1/{service}/user/{id}/profile`) goes to the creator. **Also fixed**: creator feeds are a bare JSON array, which the parser rejected (every creator feed was empty) |
|
||||
| [-] | `ripnsfw`, `hsex`, `omgxxx`, `eporner`, `borntobefuck`, `archivebate` | various | already seamless: bare names match background-loaded creator / studio / tag catalogs (or the whole catalog is filtered client-side) |
|
||||
| [-] | `perfectgirls`, `okxxx`, `pornhat`, `perverzija`, `porntrex` | sites/models/stars | bare names already resolve through the background-loaded `tag_map`; these clones build their own `wreq` client with no `ServerOptions`, so a live probe needs a separate refactor onto the shared requester |
|
||||
| [-] | `homoxxx` | models, pornstars, channels, categories, tags | legacy provider (own client, unpaginated archives, no `/models/` index on the site); not worth a probe until it is rewritten |
|
||||
| [-] | `kwiky`, `animeidhentai`, `hentaitv`, `xnxx`, `blowjobspro`, `txxx` | tag/cat | search already is the entity lookup, or the site has no pornstar/studio archives to probe (txxx: the API rejects `models`/`channels` sections) |
|
||||
| [-] | `fyptt` | cat | by design: bare category words are common search terms and must not be hijacked |
|
||||
| [-] | `notfans` | tag, category | tag URLs are hashed ids, so an unknown name cannot be probed; search covers it |
|
||||
| [-] | `allpornstream`, `jable`, `wowxxx`, `hornyleak`, `hotbunny`, `hentaimama`, `tikporn`, `camsoda` | actor/producer, model, studio, ... | Cloudflare-gated or down from this host (403/522, FlareSolverr wedged), so a probe cannot be verified here; each would add one or two FlareSolverr solves per search. Candidates for the same pattern once checked from the production network |
|
||||
82
src/api.rs
82
src/api.rs
@@ -714,6 +714,15 @@ async fn videos_post(
|
||||
.collect();
|
||||
}
|
||||
|
||||
// The Hot Tub app cannot follow a redirect in `formats[].url`: for verified
|
||||
// app clients, formats that point at a redirecting `/proxy/...` become the
|
||||
// item's `url` instead.
|
||||
if clientversion.is_verified_hottub() {
|
||||
for video in video_items.iter_mut() {
|
||||
move_redirect_formats_to_url(video);
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(literal_query) = literal_query.as_deref() {
|
||||
#[cfg(feature = "debug")]
|
||||
let before = video_items.len();
|
||||
@@ -794,6 +803,30 @@ async fn videos_post(
|
||||
Ok(web::HttpResponse::Ok().json(&videos))
|
||||
}
|
||||
|
||||
/// Moves formats that point at a redirecting proxy to `video.url`.
|
||||
///
|
||||
/// If direct (non-redirecting) formats remain they are kept and `url` is left
|
||||
/// alone; otherwise `url` becomes the first (best-listed) redirecting proxy URL
|
||||
/// and `formats` is dropped entirely.
|
||||
pub(crate) fn move_redirect_formats_to_url(video: &mut VideoItem) {
|
||||
let Some(formats) = video.formats.take() else {
|
||||
return;
|
||||
};
|
||||
let (redirecting, direct): (Vec<_>, Vec<_>) = formats
|
||||
.into_iter()
|
||||
.partition(|format| crate::proxy::is_redirect_proxy_url(&format.url));
|
||||
if redirecting.is_empty() {
|
||||
video.formats = Some(direct);
|
||||
return;
|
||||
}
|
||||
if direct.is_empty() {
|
||||
video.url = redirecting[0].url.clone();
|
||||
video.formats = None;
|
||||
} else {
|
||||
video.formats = Some(direct);
|
||||
}
|
||||
}
|
||||
|
||||
async fn uploaders_post(
|
||||
uploader_request: web::types::Json<UploadersRequest>,
|
||||
cache: web::types::State<VideoCache>,
|
||||
@@ -1082,4 +1115,53 @@ mod tests {
|
||||
assert_eq!(item.uploader.as_deref(), Some("FreeusePorn"));
|
||||
assert_eq!(item.formats.as_ref().map(|formats| formats.len()), Some(2));
|
||||
}
|
||||
|
||||
fn item_with_formats(urls: &[&str]) -> VideoItem {
|
||||
let mut item = VideoItem::new(
|
||||
"id".to_string(),
|
||||
"t".to_string(),
|
||||
"https://site.example/watch/1".to_string(),
|
||||
"c".to_string(),
|
||||
"https://thumb".to_string(),
|
||||
10,
|
||||
);
|
||||
item.formats = Some(
|
||||
urls.iter()
|
||||
.map(|u| crate::videos::VideoFormat::new(u.to_string(), "q".to_string(), "mp4".to_string()))
|
||||
.collect(),
|
||||
);
|
||||
item
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn redirect_only_formats_move_to_url() {
|
||||
let mut item = item_with_formats(&[
|
||||
"http://h/proxy/hqporner/1080",
|
||||
"http://h/proxy/hqporner/720",
|
||||
]);
|
||||
move_redirect_formats_to_url(&mut item);
|
||||
assert_eq!(item.url, "http://h/proxy/hqporner/1080");
|
||||
assert!(item.formats.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn direct_formats_are_kept_and_redirect_ones_dropped() {
|
||||
let mut item = item_with_formats(&[
|
||||
"http://h/proxy/hqporner/1080",
|
||||
"https://cdn.example/v.mp4",
|
||||
]);
|
||||
move_redirect_formats_to_url(&mut item);
|
||||
assert_eq!(item.url, "https://site.example/watch/1");
|
||||
let formats = item.formats.expect("direct format kept");
|
||||
assert_eq!(formats.len(), 1);
|
||||
assert_eq!(formats[0].url, "https://cdn.example/v.mp4");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn media_proxy_formats_are_untouched() {
|
||||
let mut item = item_with_formats(&["http://h/proxy/noodlemagazine/x.mp4"]);
|
||||
move_redirect_formats_to_url(&mut item);
|
||||
assert_eq!(item.url, "https://site.example/watch/1");
|
||||
assert_eq!(item.formats.map(|f| f.len()), Some(1));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -799,6 +799,39 @@ impl ArchivebateProvider {
|
||||
.map(|value| value.to_string())
|
||||
}
|
||||
|
||||
/// The proxied format needs no network, so an item always plays through
|
||||
/// it even when the best-effort header enrichment below times out.
|
||||
fn with_default_format(item: VideoItem, options: &ServerOptions) -> VideoItem {
|
||||
let format_url = Self::proxied_video(options, &item.url);
|
||||
if format_url.is_empty() || item.formats.is_some() {
|
||||
return item;
|
||||
}
|
||||
let mut item = item;
|
||||
item.formats = Some(vec![VideoFormat::new(
|
||||
format_url,
|
||||
"source".to_string(),
|
||||
"mp4".to_string(),
|
||||
)]);
|
||||
item
|
||||
}
|
||||
|
||||
/// For verified Hot Tub app clients: no `formats` at all. The item's `url`
|
||||
/// is the lazy `/proxy/archivebate` redirect, which the app plays directly
|
||||
/// (it resolves the media at playback time).
|
||||
fn proxy_as_item_url(videos: Vec<VideoItem>, options: &ServerOptions) -> Vec<VideoItem> {
|
||||
videos
|
||||
.into_iter()
|
||||
.map(|mut video| {
|
||||
let proxy_url = Self::proxied_video(options, &video.url);
|
||||
if !proxy_url.is_empty() {
|
||||
video.url = proxy_url;
|
||||
}
|
||||
video.formats = None;
|
||||
video
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
async fn enrich_video(&self, item: VideoItem, options: &ServerOptions) -> VideoItem {
|
||||
let page_url = item.url.clone();
|
||||
let format_url = Self::proxied_video(options, &page_url);
|
||||
@@ -1148,11 +1181,15 @@ impl ArchivebateProvider {
|
||||
platform: candidate.platform.clone(),
|
||||
gender: candidate.gender.clone(),
|
||||
};
|
||||
let items = self
|
||||
// A candidate whose profile page is gone (the search index lists
|
||||
// deleted profiles) must not abort the whole search: try the next.
|
||||
match self
|
||||
.fetch_items_for_target(cache.clone(), &target, 1, per_page, &options)
|
||||
.await?;
|
||||
if !items.is_empty() {
|
||||
return Ok(items);
|
||||
.await
|
||||
{
|
||||
Ok(items) if !items.is_empty() => return Ok(items),
|
||||
Ok(_) => {}
|
||||
Err(_) => {}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1188,6 +1225,14 @@ impl Provider for ArchivebateProvider {
|
||||
if videos.is_empty() {
|
||||
return videos;
|
||||
}
|
||||
let is_app_client = options
|
||||
.client_version
|
||||
.as_ref()
|
||||
.map(ClientVersion::is_verified_hottub)
|
||||
.unwrap_or(false);
|
||||
if is_app_client {
|
||||
return Self::proxy_as_item_url(videos, &options);
|
||||
}
|
||||
stream::iter(videos.into_iter().map(|video| {
|
||||
let provider = self.clone();
|
||||
let options = options.clone();
|
||||
@@ -1199,11 +1244,11 @@ impl Provider for ArchivebateProvider {
|
||||
.await;
|
||||
match timeout_result {
|
||||
Ok(enriched) => enriched,
|
||||
Err(_) => video,
|
||||
Err(_) => Self::with_default_format(video, &options),
|
||||
}
|
||||
}
|
||||
}))
|
||||
.buffer_unordered(4)
|
||||
.buffered(4)
|
||||
.collect::<Vec<_>>()
|
||||
.await
|
||||
}
|
||||
@@ -1218,3 +1263,52 @@ impl Provider for ArchivebateProvider {
|
||||
Some(self.build_channel(clientversion))
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn options() -> ServerOptions {
|
||||
ServerOptions {
|
||||
client_version: None,
|
||||
featured: None,
|
||||
category: None,
|
||||
sites: None,
|
||||
filter: None,
|
||||
language: None,
|
||||
public_url_base: Some("http://hottub.example".to_string()),
|
||||
requester: None,
|
||||
network: None,
|
||||
stars: None,
|
||||
categories: None,
|
||||
duration: None,
|
||||
sort: None,
|
||||
sexuality: None,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn app_clients_get_the_proxy_url_and_no_formats() {
|
||||
let mut item = VideoItem::new(
|
||||
"archivebate:1".to_string(),
|
||||
"t".to_string(),
|
||||
"https://archivebate.com/watch/1".to_string(),
|
||||
CHANNEL_ID.to_string(),
|
||||
"https://cdn.example/t.jpg".to_string(),
|
||||
10,
|
||||
);
|
||||
item.formats = Some(vec![VideoFormat::new(
|
||||
"x".to_string(),
|
||||
"source".to_string(),
|
||||
"mp4".to_string(),
|
||||
)]);
|
||||
|
||||
let out = ArchivebateProvider::proxy_as_item_url(vec![item], &options());
|
||||
assert_eq!(out.len(), 1);
|
||||
assert_eq!(
|
||||
out[0].url,
|
||||
"http://hottub.example/proxy/archivebate/archivebate.com/watch/1"
|
||||
);
|
||||
assert!(out[0].formats.is_none());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{
|
||||
Provider, build_proxy_url, report_provider_error, requester_or_default, strip_url_scheme,
|
||||
Provider, build_proxy_url, probe_first_archive, report_provider_error, requester_or_default,
|
||||
slug_dashed, strip_url_scheme,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::time::parse_time_to_seconds;
|
||||
@@ -149,6 +150,29 @@ impl ClapdatProvider {
|
||||
}
|
||||
}
|
||||
|
||||
/// A free-form query that names a tag or a user goes to that archive
|
||||
/// (probed on the site; unknown names 404), else keyword search.
|
||||
async fn resolve_keyword(&self, query: &str, options: &ServerOptions) -> Option<Target> {
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let candidates = [
|
||||
Target::Tag { slug: slug.clone() },
|
||||
Target::User { username: slug },
|
||||
];
|
||||
let urls: Vec<String> = candidates
|
||||
.iter()
|
||||
.filter_map(|t| self.listing_url(t, 1))
|
||||
.collect();
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
|
||||
body.contains("/video/")
|
||||
})
|
||||
.await?;
|
||||
candidates.into_iter().nth(index)
|
||||
}
|
||||
|
||||
fn listing_url(&self, target: &Target, page: u16) -> Option<String> {
|
||||
let page = page.max(1);
|
||||
match target {
|
||||
@@ -491,7 +515,12 @@ impl Provider for ClapdatProvider {
|
||||
sort
|
||||
};
|
||||
let query_value = query.unwrap_or_default();
|
||||
let target = self.resolve_target(&query_value, &sort_value);
|
||||
let mut target = self.resolve_target(&query_value, &sort_value);
|
||||
if let Target::Search { query } = &target {
|
||||
if let Some(found) = self.resolve_keyword(query, &options).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
|
||||
let stubs = match &target {
|
||||
Target::Search { query } => match self
|
||||
|
||||
@@ -8,7 +8,7 @@ use wreq::Version;
|
||||
use crate::{
|
||||
DbPool,
|
||||
api::ClientVersion,
|
||||
providers::{Provider, requester_or_default},
|
||||
providers::{Provider, probe_first_archive_with, requester_or_default},
|
||||
status::{Channel, ChannelOption, FilterOption},
|
||||
util::cache::VideoCache,
|
||||
videos::{ServerOptions, VideoFormat, VideoItem},
|
||||
@@ -147,6 +147,43 @@ impl CoomerProvider {
|
||||
}
|
||||
}
|
||||
|
||||
/// A single-word query that is an existing creator id (the profile
|
||||
/// endpoint answers 404 otherwise) goes to that creator's posts, trying the
|
||||
/// filtered service first, else keyword search.
|
||||
async fn resolve_keyword(
|
||||
query: &str,
|
||||
service_filter: Option<&str>,
|
||||
options: &ServerOptions,
|
||||
) -> Option<Target> {
|
||||
let name = query.trim().trim_start_matches('@');
|
||||
if name.is_empty()
|
||||
|| !name
|
||||
.chars()
|
||||
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-' || c == '.')
|
||||
{
|
||||
return None;
|
||||
}
|
||||
let mut services = vec![Service::Onlyfans, Service::Fansly, Service::Candfans];
|
||||
if let Some(preferred) = Self::resolve_service(service_filter) {
|
||||
services.retain(|s| s.as_str() != preferred.as_str());
|
||||
services.insert(0, preferred);
|
||||
}
|
||||
let urls: Vec<String> = services
|
||||
.iter()
|
||||
.map(|s| format!("{BASE_URL}/api/v1/{}/user/{name}/profile", s.as_str()))
|
||||
.collect();
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
let (index, _) =
|
||||
probe_first_archive_with(&mut requester, &urls, Self::json_headers(), |body| {
|
||||
body.contains("\"id\"")
|
||||
})
|
||||
.await?;
|
||||
Some(Target::Uploader {
|
||||
service: services[index].clone(),
|
||||
username: name.to_string(),
|
||||
})
|
||||
}
|
||||
|
||||
fn json_headers() -> Vec<(String, String)> {
|
||||
vec![
|
||||
("Referer".to_string(), REFERER.to_string()),
|
||||
@@ -290,13 +327,16 @@ impl CoomerProvider {
|
||||
)));
|
||||
}
|
||||
}
|
||||
let posts: ApiPostsResponse = serde_json::from_str(&body).map_err(|err| {
|
||||
let posts: ApiPostsPayload = serde_json::from_str(&body).map_err(|err| {
|
||||
Error::from(format!(
|
||||
"json parse failed for {url}: {err}; body[:200]={}",
|
||||
&body.chars().take(200).collect::<String>()
|
||||
))
|
||||
})?;
|
||||
Ok(posts.posts)
|
||||
Ok(match posts {
|
||||
ApiPostsPayload::List(posts) => posts,
|
||||
ApiPostsPayload::Wrapped(wrapped) => wrapped.posts,
|
||||
})
|
||||
}
|
||||
|
||||
async fn fetch_items(
|
||||
@@ -543,6 +583,14 @@ struct ApiError {
|
||||
error: String,
|
||||
}
|
||||
|
||||
/// The global feed wraps posts in `{posts: [...]}`; a creator's feed is a bare array.
|
||||
#[derive(Debug, Deserialize)]
|
||||
#[serde(untagged)]
|
||||
enum ApiPostsPayload {
|
||||
List(Vec<ApiPost>),
|
||||
Wrapped(ApiPostsResponse),
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct ApiPostsResponse {
|
||||
#[serde(default)]
|
||||
@@ -601,8 +649,13 @@ impl Provider for CoomerProvider {
|
||||
.clamp(1, 64);
|
||||
|
||||
let service_filter = options.categories.as_deref().or(options.sites.as_deref());
|
||||
let target =
|
||||
let mut target =
|
||||
Self::resolve_target(&sort_value, query.as_deref(), service_filter);
|
||||
if let Target::Search { query } = &target {
|
||||
if let Some(found) = Self::resolve_keyword(query, service_filter, &options).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
|
||||
// Bump the ignored `Sort` variant warning off by referencing it.
|
||||
let _ = Sort::New;
|
||||
@@ -849,4 +902,18 @@ mod tests {
|
||||
let post: ApiPost = serde_json::from_str(json).expect("parse");
|
||||
assert!(CoomerProvider::build_video_item(post).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_bare_array_and_wrapped_post_payloads() {
|
||||
let bare = r#"[{"id":"1","user":"u","service":"onlyfans","title":"t","file":{},"attachments":[]}]"#;
|
||||
match serde_json::from_str::<ApiPostsPayload>(bare).expect("bare array") {
|
||||
ApiPostsPayload::List(posts) => assert_eq!(posts.len(), 1),
|
||||
other => panic!("expected list, got {other:?}"),
|
||||
}
|
||||
let wrapped = r#"{"posts":[{"id":"1"}],"count":1}"#;
|
||||
match serde_json::from_str::<ApiPostsPayload>(wrapped).expect("wrapped") {
|
||||
ApiPostsPayload::Wrapped(w) => assert_eq!(w.posts.len(), 1),
|
||||
other => panic!("expected wrapped, got {other:?}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -440,15 +440,26 @@ impl EpornerProvider {
|
||||
Ok(cards)
|
||||
}
|
||||
|
||||
async fn build_item(requester: &mut Requester, card: EpornerCard) -> VideoItem {
|
||||
async fn build_item(
|
||||
requester: &mut Requester,
|
||||
options: &ServerOptions,
|
||||
card: EpornerCard,
|
||||
) -> VideoItem {
|
||||
// The hash/xhr resolution + Referer handling live in
|
||||
// `crate::util::eporner_resolve` (shared with other providers that
|
||||
// redirect out to eporner.com, e.g. `melonstube`) rather than here,
|
||||
// since single-provider builds can't depend on another provider's
|
||||
// module — see that module's doc comment.
|
||||
let formats =
|
||||
crate::util::eporner_resolve::resolve_formats(requester, &card.alnum_id, &card.page_url)
|
||||
.await;
|
||||
// Formats point at the eporner streaming proxy: the CDN links expire
|
||||
// after two hours and only serve the server's IP; anything else plays
|
||||
// a one-minute "watch on eporner.com" clip.
|
||||
let formats = crate::util::eporner_resolve::resolve_proxied_formats(
|
||||
requester,
|
||||
options,
|
||||
&card.alnum_id,
|
||||
&card.page_url,
|
||||
)
|
||||
.await;
|
||||
|
||||
let mut item = VideoItem::new(
|
||||
card.id,
|
||||
@@ -648,7 +659,8 @@ impl EpornerProvider {
|
||||
let cards = Self::parse_list_cards(&html)?;
|
||||
let items = stream::iter(cards.into_iter().take(per_page.max(1)).map(|card| {
|
||||
let mut req = requester.clone();
|
||||
async move { Self::build_item(&mut req, card).await }
|
||||
let options = &options;
|
||||
async move { Self::build_item(&mut req, options, card).await }
|
||||
}))
|
||||
.buffer_unordered(ENRICH_CONCURRENCY)
|
||||
.collect::<Vec<_>>()
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{Provider, report_provider_error, requester_or_default};
|
||||
use crate::providers::{Provider, probe_first_archive, report_provider_error, requester_or_default};
|
||||
use crate::status::*;
|
||||
use crate::util::parse_abbreviated_number;
|
||||
use crate::videos::{ServerOptions, VideoItem};
|
||||
@@ -164,6 +164,26 @@ impl EromeProvider {
|
||||
(!slug.is_empty()).then_some(slug)
|
||||
}
|
||||
|
||||
/// A single-word query that is an existing uploader name (`/{name}` answers
|
||||
/// 404 otherwise) goes to that uploader's albums, else keyword search.
|
||||
async fn resolve_keyword(&self, query: &str, options: &ServerOptions) -> Option<Target> {
|
||||
let slug = query.trim().trim_start_matches('@');
|
||||
if slug.is_empty()
|
||||
|| !slug
|
||||
.chars()
|
||||
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-')
|
||||
{
|
||||
return None;
|
||||
}
|
||||
let url = format!("{}/{slug}", self.url);
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
probe_first_archive(&mut requester, &[url], &format!("{}/", self.url), |body| {
|
||||
body.contains("album-link")
|
||||
})
|
||||
.await
|
||||
.map(|_| Target::UploaderPosts { slug: slug.to_string() })
|
||||
}
|
||||
|
||||
fn resolve_target(&self, query: &str, options: &ServerOptions, sort: &str) -> Target {
|
||||
if let Some(site) = options.sites.as_deref() {
|
||||
let normalized = site.trim();
|
||||
@@ -367,7 +387,12 @@ impl Provider for EromeProvider {
|
||||
};
|
||||
let query_value = query.unwrap_or_default();
|
||||
let page_value = page.parse::<u16>().unwrap_or(1);
|
||||
let target = self.resolve_target(&query_value, &options, &sort_value);
|
||||
let mut target = self.resolve_target(&query_value, &options, &sort_value);
|
||||
if let Target::Search { query, .. } = &target {
|
||||
if let Some(found) = self.resolve_keyword(query, &options).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
let url = self.build_url_for_target(&target, page_value);
|
||||
|
||||
match self.fetch_page(&options, &url).await {
|
||||
|
||||
@@ -639,6 +639,24 @@ impl Provider for FikfapProvider {
|
||||
.map(str::trim)
|
||||
.filter(|value| !value.is_empty());
|
||||
let target = Self::pick_target(normalized_query, &sort, &options);
|
||||
// A single-word query that is an existing profile name goes to that
|
||||
// creator's posts (checked against the profile endpoint); a profile
|
||||
// without posts falls back to the keyword search.
|
||||
if let Target::Search(query) = &target {
|
||||
if !query.contains(char::is_whitespace) {
|
||||
if let Ok(Some(profile)) = self.fetch_user_profile(query, &options).await {
|
||||
let user = Target::User(profile.username);
|
||||
if let Ok(items) = self
|
||||
.fetch_target_items(user, page, per_page, &options)
|
||||
.await
|
||||
{
|
||||
if !items.is_empty() {
|
||||
return items;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
match self.fetch_target_items(target, page, per_page, &options).await {
|
||||
Ok(items) => items,
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{Provider, report_provider_error, requester_or_default};
|
||||
use crate::providers::{
|
||||
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
use crate::util::time::parse_time_to_seconds;
|
||||
@@ -134,13 +136,13 @@ impl FullpornerProvider {
|
||||
}
|
||||
if let Some(slug) = q.strip_prefix("pornstar:") {
|
||||
return Target::Pornstar {
|
||||
slug: slug.trim().to_string(),
|
||||
slug: slug_dashed(slug),
|
||||
page,
|
||||
};
|
||||
}
|
||||
if let Some(slug) = q.strip_prefix("star:") {
|
||||
return Target::Pornstar {
|
||||
slug: slug.trim().to_string(),
|
||||
slug: slug_dashed(slug),
|
||||
page,
|
||||
};
|
||||
}
|
||||
@@ -153,6 +155,25 @@ impl FullpornerProvider {
|
||||
Target::Latest { page }
|
||||
}
|
||||
|
||||
/// A free-form query that names a pornstar goes to her archive (probed on
|
||||
/// the site; unknown names 404), otherwise it stays a keyword search.
|
||||
async fn resolve_keyword(query: &str, page: u32, options: &ServerOptions) -> Option<Target> {
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let probe = Target::Pornstar { slug: slug.clone(), page: 1 };
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
probe_first_archive(
|
||||
&mut requester,
|
||||
&[Self::build_listing_url(&probe)],
|
||||
&format!("{BASE_URL}/"),
|
||||
|body| body.contains("class=\"video-card\""),
|
||||
)
|
||||
.await
|
||||
.map(|_| Target::Pornstar { slug, page })
|
||||
}
|
||||
|
||||
fn selector(value: &str) -> Result<Selector> {
|
||||
Selector::parse(value)
|
||||
.map_err(|e| Error::from(format!("selector `{value}` failed: {e}")))
|
||||
@@ -394,7 +415,10 @@ impl Provider for FullpornerProvider {
|
||||
.filter(|q| !q.is_empty())
|
||||
.map(ToOwned::to_owned);
|
||||
|
||||
let target = Self::pick_target(normalized_query.as_deref(), page);
|
||||
let mut target = Self::pick_target(normalized_query.as_deref(), page);
|
||||
if let Target::Search { query, .. } = &target {
|
||||
target = Self::resolve_keyword(query, page, &options).await.unwrap_or(target);
|
||||
}
|
||||
|
||||
match Self::fetch_page(&target, per_page, &options).await {
|
||||
Ok(items) => items,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{
|
||||
use crate::providers::{probe_first_archive, slug_dashed,
|
||||
Provider, report_provider_error, report_provider_error_background, requester_or_default,
|
||||
};
|
||||
use crate::status::*;
|
||||
@@ -109,12 +109,12 @@ impl FullpornerorgProvider {
|
||||
if let Some(q) = query.map(str::trim).filter(|q| !q.is_empty()) {
|
||||
if let Some(slug) = q.strip_prefix("pornstar:") {
|
||||
return Target::Pornstar {
|
||||
slug: slug.trim().to_string(),
|
||||
slug: slug_dashed(slug),
|
||||
};
|
||||
}
|
||||
if let Some(slug) = q.strip_prefix("star:") {
|
||||
return Target::Pornstar {
|
||||
slug: slug.trim().to_string(),
|
||||
slug: slug_dashed(slug),
|
||||
};
|
||||
}
|
||||
return Target::Search {
|
||||
@@ -153,6 +153,25 @@ impl FullpornerorgProvider {
|
||||
}
|
||||
}
|
||||
|
||||
/// A free-form query that names a pornstar goes to her archive (probed on
|
||||
/// the site; unknown names 404), otherwise it stays a keyword search.
|
||||
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let target = Target::Pornstar { slug };
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
probe_first_archive(
|
||||
&mut requester,
|
||||
&[Self::build_listing_url(&target, 1)],
|
||||
&format!("{BASE_URL}/"),
|
||||
|body| body.contains("loop-video"),
|
||||
)
|
||||
.await
|
||||
.map(|_| target)
|
||||
}
|
||||
|
||||
fn html_headers(referer: &str) -> Vec<(String, String)> {
|
||||
vec![
|
||||
("User-Agent".to_string(), BROWSER_UA.to_string()),
|
||||
@@ -446,7 +465,10 @@ impl Provider for FullpornerorgProvider {
|
||||
.filter(|q| !q.is_empty())
|
||||
.map(ToOwned::to_owned);
|
||||
|
||||
let target = Self::pick_target(normalized_query.as_deref());
|
||||
let mut target = Self::pick_target(normalized_query.as_deref());
|
||||
if let Target::Search { query } = &target {
|
||||
target = Self::resolve_keyword(query, &options).await.unwrap_or(target);
|
||||
}
|
||||
|
||||
match Self::fetch_page(target, page, per_page, &options).await {
|
||||
Ok(items) => items,
|
||||
|
||||
@@ -1,13 +1,8 @@
|
||||
use aes_gcm::aead::{Aead, KeyInit, Payload};
|
||||
use aes_gcm::{Aes256Gcm, Key, Nonce};
|
||||
use async_trait::async_trait;
|
||||
use base64::Engine;
|
||||
use error_chain::error_chain;
|
||||
use futures::future::join_all;
|
||||
use serde_json::json;
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::sync::{Arc, Mutex, OnceLock};
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
use std::time::SystemTime;
|
||||
use std::vec;
|
||||
|
||||
use crate::DbPool;
|
||||
@@ -16,6 +11,7 @@ use crate::db;
|
||||
use crate::providers::{Provider, report_provider_error, report_provider_error_background};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
use crate::util::hanime_resolve::{self as resolve, HanimeSource, SITE_ORIGIN};
|
||||
use crate::videos::{self, ServerOptions, VideoItem};
|
||||
|
||||
pub const CHANNEL_METADATA: crate::providers::ProviderChannelMetadata =
|
||||
@@ -34,14 +30,6 @@ error_chain! {
|
||||
/// Full catalogue dump. Ignores every query parameter, so filtering, sorting and
|
||||
/// pagination all happen client side. ~860 KB compressed, CDN cached for an hour.
|
||||
const INDEX_URL: &str = "https://guest.freeanimehentai.net/api/v11/search_hvs";
|
||||
const HANDSHAKE_URL: &str = "https://auth.hanime.tv/api/v11/handshake";
|
||||
const SITE_ORIGIN: &str = "https://hanime.tv";
|
||||
/// Envelope encryption used for the handshake request/response bodies.
|
||||
const HANDSHAKE_KEY_SEED: &[u8] = b"htv-insecure-handshake-v1";
|
||||
const HANDSHAKE_AAD: &[u8] = b"htv-insecure-v1";
|
||||
/// Pieces of the `x-signature` pre-image, as baked into the site's wasm module.
|
||||
const SIGNATURE_SECRET: &str = "Xkdi29";
|
||||
const SIGNATURE_SALT: &str = "mn2";
|
||||
const INDEX_TTL_SECS: u64 = 3600;
|
||||
|
||||
type IndexCache = OnceLock<Mutex<Option<(SystemTime, Arc<Vec<HanimeSearchResult>>)>>>;
|
||||
@@ -76,99 +64,6 @@ struct HanimeSearchResult {
|
||||
released_at_unix: u64,
|
||||
}
|
||||
|
||||
#[derive(serde::Deserialize, Debug)]
|
||||
struct HanimeHandshakePayload {
|
||||
#[serde(default)]
|
||||
sources: Vec<HanimeSource>,
|
||||
}
|
||||
|
||||
#[derive(serde::Deserialize, Debug, Clone)]
|
||||
struct HanimeSource {
|
||||
#[serde(default)]
|
||||
src: String,
|
||||
#[serde(default)]
|
||||
height: u32,
|
||||
#[serde(default)]
|
||||
label: String,
|
||||
/// `"normal"` for the free streams, `"promotion"` for the premium-only teaser
|
||||
/// entries which always carry an empty `src`.
|
||||
#[serde(default)]
|
||||
kind: String,
|
||||
}
|
||||
|
||||
fn b64() -> base64::engine::general_purpose::GeneralPurpose {
|
||||
base64::engine::general_purpose::URL_SAFE_NO_PAD
|
||||
}
|
||||
|
||||
fn b64_decode(value: &str) -> Result<Vec<u8>> {
|
||||
b64()
|
||||
.decode(value.trim_end_matches('='))
|
||||
.map_err(|e| Error::from(format!("base64url decode failed: {e}")))
|
||||
}
|
||||
|
||||
fn handshake_cipher() -> Aes256Gcm {
|
||||
let digest = Sha256::digest(HANDSHAKE_KEY_SEED);
|
||||
let key = Key::<Aes256Gcm>::from_slice(digest.as_slice());
|
||||
Aes256Gcm::new(key)
|
||||
}
|
||||
|
||||
/// `{"v":1,"alg":"AES-256-GCM","iv":..,"tag":..,"data":..}`, base64url encoded.
|
||||
fn seal_envelope(plaintext: &[u8]) -> Result<String> {
|
||||
let mut iv = [0u8; 12];
|
||||
rand::fill(&mut iv);
|
||||
let sealed = handshake_cipher()
|
||||
.encrypt(
|
||||
Nonce::from_slice(&iv),
|
||||
Payload {
|
||||
msg: plaintext,
|
||||
aad: HANDSHAKE_AAD,
|
||||
},
|
||||
)
|
||||
.map_err(|e| Error::from(format!("handshake encrypt failed: {e}")))?;
|
||||
if sealed.len() < 16 {
|
||||
return Err(Error::from("handshake ciphertext too short"));
|
||||
}
|
||||
let (data, tag) = sealed.split_at(sealed.len() - 16);
|
||||
let envelope = json!({
|
||||
"v": 1,
|
||||
"alg": "AES-256-GCM",
|
||||
"iv": b64().encode(iv),
|
||||
"tag": b64().encode(tag),
|
||||
"data": b64().encode(data),
|
||||
});
|
||||
Ok(b64().encode(envelope.to_string().as_bytes()))
|
||||
}
|
||||
|
||||
fn open_envelope(token: &str) -> Result<Vec<u8>> {
|
||||
let envelope: serde_json::Value = serde_json::from_slice(&b64_decode(token)?)
|
||||
.map_err(|e| Error::from(format!("handshake envelope is not JSON: {e}")))?;
|
||||
let field = |name: &str| -> Result<Vec<u8>> {
|
||||
let raw = envelope
|
||||
.get(name)
|
||||
.and_then(|v| v.as_str())
|
||||
.ok_or_else(|| Error::from(format!("handshake envelope missing {name}")))?;
|
||||
b64_decode(raw)
|
||||
};
|
||||
let iv = field("iv")?;
|
||||
let mut ciphertext = field("data")?;
|
||||
ciphertext.extend_from_slice(&field("tag")?);
|
||||
handshake_cipher()
|
||||
.decrypt(
|
||||
Nonce::from_slice(&iv),
|
||||
Payload {
|
||||
msg: &ciphertext,
|
||||
aad: HANDSHAKE_AAD,
|
||||
},
|
||||
)
|
||||
.map_err(|e| Error::from(format!("handshake decrypt failed: {e}")))
|
||||
}
|
||||
|
||||
fn signature_for(timestamp: u64) -> String {
|
||||
let pre_image =
|
||||
format!("{timestamp},{SIGNATURE_SECRET},{SITE_ORIGIN},{SIGNATURE_SALT},{timestamp}");
|
||||
hex::encode(Sha256::digest(pre_image.as_bytes()))
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct HanimeProvider;
|
||||
|
||||
@@ -242,6 +137,25 @@ impl HanimeProvider {
|
||||
}
|
||||
}
|
||||
|
||||
/// Fresh free sources for a slug, best quality first.
|
||||
async fn fetch_sources(&self, slug: &str, options: &ServerOptions) -> Result<Vec<HanimeSource>> {
|
||||
let mut requester =
|
||||
crate::providers::requester_or_default(options, module_path!(), "missing_requester");
|
||||
resolve::fetch_sources(&mut requester, slug)
|
||||
.await
|
||||
.map_err(Error::from)
|
||||
}
|
||||
|
||||
/// The stream URLs expire, so players get a stable redirect that does a fresh
|
||||
/// handshake on every play instead of the signed link itself.
|
||||
fn proxied_url(options: &ServerOptions, slug: &str, source: &HanimeSource) -> String {
|
||||
crate::providers::build_proxy_url(
|
||||
options,
|
||||
"hanime",
|
||||
&format!("{slug}/{}.m3u8", source.height),
|
||||
)
|
||||
}
|
||||
|
||||
fn db_key(slug: &str) -> String {
|
||||
format!("https://hanime.tv/videos/hentai/{slug}")
|
||||
}
|
||||
@@ -319,86 +233,6 @@ impl HanimeProvider {
|
||||
Ok(items)
|
||||
}
|
||||
|
||||
/// Returns the free `sources` for a slug, best quality first. The auth host
|
||||
/// occasionally refuses one of a burst of parallel connections, so retry once.
|
||||
async fn fetch_sources(&self, slug: &str, options: &ServerOptions) -> Result<Vec<HanimeSource>> {
|
||||
match self.handshake(slug, options).await {
|
||||
Ok(sources) => Ok(sources),
|
||||
Err(_) => {
|
||||
ntex::time::sleep(ntex::time::Millis(500)).await;
|
||||
self.handshake(slug, options).await
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn handshake(&self, slug: &str, options: &ServerOptions) -> Result<Vec<HanimeSource>> {
|
||||
let timestamp = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.unwrap_or_default()
|
||||
.as_secs();
|
||||
let token = seal_envelope(
|
||||
json!({
|
||||
"timestamp_unix": timestamp,
|
||||
"directive": "htv_player_handshake",
|
||||
"slug": slug,
|
||||
})
|
||||
.to_string()
|
||||
.as_bytes(),
|
||||
)?;
|
||||
|
||||
let mut requester =
|
||||
crate::providers::requester_or_default(options, module_path!(), "missing_requester");
|
||||
let response = requester
|
||||
.post_json(
|
||||
HANDSHAKE_URL,
|
||||
&json!({ "token": token }),
|
||||
vec![
|
||||
("Origin".to_string(), SITE_ORIGIN.to_string()),
|
||||
("Referer".to_string(), format!("{SITE_ORIGIN}/")),
|
||||
("Accept".to_string(), "application/json".to_string()),
|
||||
("x-signature-version".to_string(), "web2".to_string()),
|
||||
("x-signature".to_string(), signature_for(timestamp)),
|
||||
("x-time".to_string(), timestamp.to_string()),
|
||||
],
|
||||
)
|
||||
.await
|
||||
.map_err(|e| Error::from(format!("handshake request failed: {e}")))?;
|
||||
|
||||
let status = response.status();
|
||||
if !status.is_success() {
|
||||
return Err(Error::from(format!("handshake returned HTTP {status}")));
|
||||
}
|
||||
|
||||
// The interesting part of the answer travels in a response header, not the body.
|
||||
let encrypted = response
|
||||
.headers()
|
||||
.get("x-token")
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.map(|v| v.to_string())
|
||||
.ok_or_else(|| Error::from("handshake response is missing x-token"))?;
|
||||
|
||||
let payload: HanimeHandshakePayload = serde_json::from_slice(&open_envelope(&encrypted)?)
|
||||
.map_err(|e| Error::from(format!("Failed to parse handshake payload: {e}")))?;
|
||||
|
||||
let mut sources: Vec<HanimeSource> = payload
|
||||
.sources
|
||||
.into_iter()
|
||||
.filter(|s| s.kind == "normal" && !s.src.is_empty())
|
||||
.collect();
|
||||
sources.sort_by(|a, b| b.height.cmp(&a.height));
|
||||
match sources.is_empty() {
|
||||
true => Err(Error::from("handshake returned no playable sources")),
|
||||
false => Ok(sources),
|
||||
}
|
||||
}
|
||||
|
||||
fn source_url(source: &HanimeSource) -> String {
|
||||
match source.src.starts_with("http") {
|
||||
true => source.src.clone(),
|
||||
false => format!("{SITE_ORIGIN}{}", source.src),
|
||||
}
|
||||
}
|
||||
|
||||
fn quality_label(source: &HanimeSource) -> String {
|
||||
match source.label.trim_end_matches('p') {
|
||||
"" => source.height.to_string(),
|
||||
@@ -452,15 +286,15 @@ impl HanimeProvider {
|
||||
.iter()
|
||||
.map(|source| {
|
||||
videos::VideoFormat::new(
|
||||
Self::source_url(source),
|
||||
Self::proxied_url(&options, &hit.slug, source),
|
||||
Self::quality_label(source),
|
||||
"m3u8".to_string(),
|
||||
)
|
||||
})
|
||||
.collect();
|
||||
let stream_url = Self::source_url(&sources[0]);
|
||||
let stream_url = Self::proxied_url(&options, &hit.slug, &sources[0]);
|
||||
let duration = self
|
||||
.fetch_duration(&stream_url, &options)
|
||||
.fetch_duration(&resolve::source_url(&sources[0]), &options)
|
||||
.await
|
||||
.unwrap_or_default();
|
||||
if let Ok(mut conn) = pool.get() {
|
||||
@@ -490,11 +324,8 @@ impl HanimeProvider {
|
||||
.and_then(|mut conn| db::get_video(&mut conn, db_key.clone()).ok().flatten());
|
||||
|
||||
match db_result {
|
||||
Some(video_url) if video_url.contains("m3u8") || video_url.contains("/hls/") => {
|
||||
let duration = self
|
||||
.fetch_duration(&video_url, &options)
|
||||
.await
|
||||
.unwrap_or_default();
|
||||
Some(video_url) if video_url.contains("/proxy/hanime/") => {
|
||||
let duration = 0;
|
||||
let formats = vec![videos::VideoFormat::new(
|
||||
video_url.clone(),
|
||||
"720".to_string(),
|
||||
|
||||
@@ -430,40 +430,13 @@ impl HoopornProvider {
|
||||
.await
|
||||
}
|
||||
|
||||
/// Fetches a listing page. A real 404 (unknown archive, page past the end)
|
||||
/// is `None` and answers immediately; a Cloudflare challenge is handed to
|
||||
/// the requester's full chain (FlareSolverr).
|
||||
/// Fetches a listing page; `None` for a real 404 (see `Requester::get_probe`).
|
||||
async fn fetch_html(requester: &mut Requester, url: &str) -> Result<Option<String>> {
|
||||
let headers = vec![("Referer".to_string(), format!("{BASE_URL}/"))];
|
||||
let mut last_err = String::new();
|
||||
for _ in 0..2 {
|
||||
match requester.get_raw_with_headers(url, headers.clone()).await {
|
||||
Ok(response) => {
|
||||
let status = response.status().as_u16();
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
if status == 404 {
|
||||
return Ok(None);
|
||||
}
|
||||
if status == 403
|
||||
|| status == 503
|
||||
|| Requester::looks_like_cloudflare_challenge(&body)
|
||||
{
|
||||
return requester
|
||||
.get_with_headers(url, headers, None)
|
||||
.await
|
||||
.map(Some)
|
||||
.map_err(|e| Error::from(format!("fetch failed for {url}: {e}")));
|
||||
}
|
||||
if status >= 500 || status == 429 {
|
||||
last_err = format!("HTTP {status}");
|
||||
continue;
|
||||
}
|
||||
return Ok(Some(body));
|
||||
}
|
||||
Err(e) => last_err = e.to_string(),
|
||||
}
|
||||
}
|
||||
Err(Error::from(format!("fetch failed for {url}: {last_err}")))
|
||||
requester
|
||||
.get_probe(url, headers)
|
||||
.await
|
||||
.map_err(|e| Error::from(format!("fetch failed for {url}: {e}")))
|
||||
}
|
||||
|
||||
async fn fetch(target: &Target, page: u32, options: &ServerOptions) -> Result<Vec<VideoItem>> {
|
||||
|
||||
@@ -494,11 +494,18 @@ impl MelonstubeProvider {
|
||||
Self::parse_kvs_formats(&html, &origin)
|
||||
}
|
||||
|
||||
/// Through the eporner streaming proxy: its CDN links expire after two
|
||||
/// hours and only serve the server's IP (else a "watch on eporner.com" clip).
|
||||
async fn resolve_eporner(
|
||||
requester: &mut Requester,
|
||||
options: &ServerOptions,
|
||||
detail_url: &str,
|
||||
) -> Option<Vec<VideoFormat>> {
|
||||
crate::util::eporner_resolve::resolve_public_formats(requester, detail_url).await
|
||||
let alnum_id = crate::util::eporner_resolve::alnum_id_from_href(detail_url)?;
|
||||
crate::util::eporner_resolve::resolve_proxied_formats(
|
||||
requester, options, &alnum_id, detail_url,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
fn xhamster_mirror_mp4_regex() -> &'static Regex {
|
||||
@@ -632,10 +639,11 @@ impl MelonstubeProvider {
|
||||
async fn resolve_source_formats(
|
||||
kind: SourceKind,
|
||||
requester: &mut Requester,
|
||||
options: &ServerOptions,
|
||||
detail_url: &str,
|
||||
) -> Option<Vec<VideoFormat>> {
|
||||
match kind {
|
||||
SourceKind::Eporner => Self::resolve_eporner(requester, detail_url).await,
|
||||
SourceKind::Eporner => Self::resolve_eporner(requester, options, detail_url).await,
|
||||
SourceKind::Kvs => Self::resolve_kvs(requester, detail_url, false).await,
|
||||
SourceKind::KvsCfGated => Self::resolve_kvs(requester, detail_url, true).await,
|
||||
SourceKind::XhamsterMirror => Self::resolve_xhamster_mirror(requester, detail_url).await,
|
||||
@@ -904,6 +912,7 @@ impl MelonstubeProvider {
|
||||
if !pending.is_empty() {
|
||||
let resolved = stream::iter(pending.into_iter().map(|(idx, kind, dest_url)| {
|
||||
let mut req = requester.clone();
|
||||
let options = &options;
|
||||
async move {
|
||||
// Bounded so one slow/hung third-party destination (e.g.
|
||||
// an unreachable FlareSolverr fallback) can't stall the
|
||||
@@ -911,7 +920,7 @@ impl MelonstubeProvider {
|
||||
// its plain destination URL with no formats.
|
||||
let formats = tokio::time::timeout(
|
||||
std::time::Duration::from_secs(ENRICH_TIMEOUT_SECS),
|
||||
Self::resolve_source_formats(kind, &mut req, &dest_url),
|
||||
Self::resolve_source_formats(kind, &mut req, options, &dest_url),
|
||||
)
|
||||
.await
|
||||
.ok()
|
||||
|
||||
@@ -612,6 +612,66 @@ pub fn report_provider_error_background(provider_name: &str, context: &str, msg:
|
||||
});
|
||||
}
|
||||
|
||||
/// `Some(slug)` joined with `-` ("Gal Ritchie" -> "gal-ritchie") for archive URLs.
|
||||
pub fn slug_dashed(value: &str) -> String {
|
||||
let mut out = String::new();
|
||||
let mut last_dash = true;
|
||||
for c in value.trim().chars() {
|
||||
if c.is_alphanumeric() {
|
||||
out.extend(c.to_lowercase());
|
||||
last_dash = false;
|
||||
} else if !last_dash {
|
||||
out.push('-');
|
||||
last_dash = true;
|
||||
}
|
||||
}
|
||||
out.trim_matches('-').to_string()
|
||||
}
|
||||
|
||||
/// Same as [`slug_dashed`] without separators ("Family Strokes" -> "familystrokes").
|
||||
pub fn slug_joined(value: &str) -> String {
|
||||
slug_dashed(value).replace('-', "")
|
||||
}
|
||||
|
||||
/// Probes candidate archive URLs (pornstar / studio / channel / tag pages) in
|
||||
/// order and returns the index and body of the first that exists and
|
||||
/// `has_items` accepts. A miss (404, empty page, transport error) moves on to
|
||||
/// the next candidate; the caller falls back to keyword search on `None`.
|
||||
/// Uses `Requester::get_probe`, so an unknown archive answers instantly and a
|
||||
/// Cloudflare challenge is solved through FlareSolverr.
|
||||
pub async fn probe_first_archive(
|
||||
requester: &mut Requester,
|
||||
urls: &[String],
|
||||
referer: &str,
|
||||
has_items: impl Fn(&str) -> bool,
|
||||
) -> Option<(usize, String)> {
|
||||
probe_first_archive_with(
|
||||
requester,
|
||||
urls,
|
||||
vec![("Referer".to_string(), referer.to_string())],
|
||||
has_items,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
/// [`probe_first_archive`] with caller-supplied request headers (JSON APIs
|
||||
/// that demand a special `Accept`, extra tokens, ...).
|
||||
pub async fn probe_first_archive_with(
|
||||
requester: &mut Requester,
|
||||
urls: &[String],
|
||||
headers: Vec<(String, String)>,
|
||||
has_items: impl Fn(&str) -> bool,
|
||||
) -> Option<(usize, String)> {
|
||||
for (index, url) in urls.iter().enumerate() {
|
||||
if let Ok(Some(body)) = requester.get_probe(url, headers.clone()).await {
|
||||
if has_items(&body) {
|
||||
return Some((index, body));
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
pub fn requester_or_default(
|
||||
options: &ServerOptions,
|
||||
provider_name: &str,
|
||||
@@ -1777,3 +1837,16 @@ mod tests {
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod slug_tests {
|
||||
use super::{slug_dashed, slug_joined};
|
||||
|
||||
#[test]
|
||||
fn slugs_names_for_archive_urls() {
|
||||
assert_eq!(slug_dashed(" Gal Ritchie "), "gal-ritchie");
|
||||
assert_eq!(slug_dashed("Mia Khalifa!!"), "mia-khalifa");
|
||||
assert_eq!(slug_joined("Family Strokes"), "familystrokes");
|
||||
assert_eq!(slug_dashed("???"), "");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,6 +2,7 @@ use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{
|
||||
Provider, report_provider_error, report_provider_error_background, requester_or_default,
|
||||
probe_first_archive,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
@@ -260,6 +261,34 @@ impl Porn4fansProvider {
|
||||
Target::Latest
|
||||
}
|
||||
|
||||
/// A free-form query that names a model or a tag goes to that archive
|
||||
/// (probed on the site; unknown names 404), else keyword search.
|
||||
async fn resolve_keyword(&self, query: &str, options: &ServerOptions) -> Option<Target> {
|
||||
let slug = Self::slugify(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let candidates = [
|
||||
Target::Model { slug: slug.clone() },
|
||||
Target::Tag { slug },
|
||||
];
|
||||
let urls: Vec<String> = candidates
|
||||
.iter()
|
||||
.map(|t| match t {
|
||||
Target::Model { slug } => format!("{}/models/{slug}/", self.url),
|
||||
Target::Tag { slug } => format!("{}/tags/{slug}/", self.url),
|
||||
_ => String::new(),
|
||||
})
|
||||
.collect();
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
let (index, _) =
|
||||
probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
|
||||
body.contains("custom_list_videos")
|
||||
})
|
||||
.await?;
|
||||
candidates.into_iter().nth(index)
|
||||
}
|
||||
|
||||
/// Build the async `get_block` URL plus the Referer to send with it.
|
||||
fn build_block_url(&self, target: &Target, sort: &str, page: u32) -> (String, String) {
|
||||
let sort_by = Self::sort_by(sort);
|
||||
@@ -788,7 +817,12 @@ impl Provider for Porn4fansProvider {
|
||||
options: ServerOptions,
|
||||
) -> Vec<VideoItem> {
|
||||
let page = page.parse::<u32>().unwrap_or(1).max(1);
|
||||
let target = self.target_from_request(query.as_deref(), options.categories.as_deref());
|
||||
let mut target = self.target_from_request(query.as_deref(), options.categories.as_deref());
|
||||
if let Target::Search { query } = &target {
|
||||
if let Some(found) = self.resolve_keyword(query, &options).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
|
||||
match self
|
||||
.get_videos_for_target(cache, &target, &sort, page, options)
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{Provider, proxy_aylo_thumb, report_provider_error, requester_or_default};
|
||||
use crate::providers::{
|
||||
Provider, probe_first_archive, proxy_aylo_thumb, report_provider_error, requester_or_default,
|
||||
slug_dashed,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
use crate::util::parse_abbreviated_number;
|
||||
@@ -45,6 +48,8 @@ error_chain! {
|
||||
pub struct PornhubProvider {
|
||||
url: String,
|
||||
tag_map: Arc<RwLock<HashMap<String, TagInfo>>>,
|
||||
/// Queries already probed on the site that are neither a pornstar, model nor channel.
|
||||
probe_misses: Arc<RwLock<std::collections::HashSet<String>>>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
@@ -91,6 +96,7 @@ impl PornhubProvider {
|
||||
let provider = Self {
|
||||
url: BASE_URL.to_string(),
|
||||
tag_map: Arc::new(RwLock::new(HashMap::new())),
|
||||
probe_misses: Arc::new(RwLock::new(std::collections::HashSet::new())),
|
||||
};
|
||||
provider.spawn_initial_load();
|
||||
provider
|
||||
@@ -588,6 +594,43 @@ impl PornhubProvider {
|
||||
values.push(normalized.to_string());
|
||||
}
|
||||
|
||||
/// A free-form query that is not in the loaded catalogs may still name a
|
||||
/// pornstar, model or channel. Probe the site (unknown names 404 or
|
||||
/// redirect away) and remember the hit in `tag_map` so later pages and
|
||||
/// queries resolve without another request.
|
||||
async fn learn_query_target(&self, query: &str, options: &ServerOptions) {
|
||||
let key = query.trim().to_ascii_lowercase();
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() || self.probe_misses.read().unwrap().contains(&key) {
|
||||
return;
|
||||
}
|
||||
let kinds = [
|
||||
QueryTargetKind::Pornstar,
|
||||
QueryTargetKind::Model,
|
||||
QueryTargetKind::Channel,
|
||||
];
|
||||
let urls: Vec<String> = kinds
|
||||
.iter()
|
||||
.map(|kind| format!("{}/{}/{slug}/videos", self.url, kind.path_segment()))
|
||||
.collect();
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "learn_query_target");
|
||||
let hit = probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
|
||||
body.contains("pcVideoListItem")
|
||||
})
|
||||
.await;
|
||||
match hit {
|
||||
Some((index, _)) => {
|
||||
self.tag_map.write().unwrap().insert(
|
||||
key,
|
||||
TagInfo { kind: kinds[index], slug, title: query.trim().to_string() },
|
||||
);
|
||||
}
|
||||
None => {
|
||||
self.probe_misses.write().unwrap().insert(key);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn fetch_listing(
|
||||
&self,
|
||||
cache: VideoCache,
|
||||
@@ -596,6 +639,11 @@ impl PornhubProvider {
|
||||
query: Option<&str>,
|
||||
options: ServerOptions,
|
||||
) -> Result<Vec<VideoItem>> {
|
||||
if let Some(q) = query.map(str::trim).filter(|q| !q.is_empty()) {
|
||||
if self.parse_query_target(q).is_none() {
|
||||
self.learn_query_target(q, &options).await;
|
||||
}
|
||||
}
|
||||
if query.is_some() && self.tag_map.read().unwrap().is_empty() {
|
||||
let _ = Self::load_tags(&self.url, Arc::clone(&self.tag_map)).await;
|
||||
}
|
||||
|
||||
@@ -560,6 +560,24 @@ impl Provider for RedgifsProvider {
|
||||
|
||||
let sort = Sort::from_sort_id(&sort);
|
||||
let target = Self::pick_target(normalized_query, sort, &options);
|
||||
// A single-word query that is an existing creator name goes to that
|
||||
// creator's gifs (checked against the users endpoint); a creator
|
||||
// without gifs falls back to the keyword search.
|
||||
if let Target::Query(text, _) = &target {
|
||||
if !text.contains(char::is_whitespace) {
|
||||
if let Ok(Some(profile)) = self.fetch_user_profile(text, &options).await {
|
||||
let creator = Target::Creator(profile.username);
|
||||
if let Ok(items) = self
|
||||
.fetch_target_items(creator, page, per_page, &options)
|
||||
.await
|
||||
{
|
||||
if !items.is_empty() {
|
||||
return items;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
match self.fetch_target_items(target, page, per_page, &options).await {
|
||||
Ok(items) => items,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::Provider;
|
||||
use crate::providers::{Provider, requester_or_default, slug_dashed};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
use crate::util::discord::send_discord_error_report;
|
||||
@@ -12,10 +12,23 @@ use async_trait::async_trait;
|
||||
use error_chain::error_chain;
|
||||
use futures::stream::{self, StreamExt};
|
||||
use htmlentity::entity::{ICodedDataTrait, decode};
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
use scraper::{Html, Selector};
|
||||
use std::collections::HashMap;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
|
||||
use percent_encoding::{NON_ALPHANUMERIC, utf8_percent_encode};
|
||||
use std::vec;
|
||||
|
||||
const ENRICH_CONCURRENCY: usize = 6;
|
||||
const ENRICH_CONCURRENCY: usize = 4;
|
||||
/// The signed `v-acctoken` format URLs last ~12h; keep details well inside that.
|
||||
const DETAIL_TTL: Duration = Duration::from_secs(2 * 60 * 60);
|
||||
const DETAIL_CACHE_SIZE: usize = 5000;
|
||||
const TARGET_TTL: Duration = Duration::from_secs(6 * 60 * 60);
|
||||
const TARGET_CACHE_SIZE: usize = 2000;
|
||||
/// A probed archive smaller than this loses to keyword search: "elf girl" is a
|
||||
/// one-video tag, while the search finds hundreds.
|
||||
const MIN_PROBED_ARCHIVE_ITEMS: usize = 4;
|
||||
|
||||
pub const CHANNEL_METADATA: crate::providers::ProviderChannelMetadata =
|
||||
crate::providers::ProviderChannelMetadata {
|
||||
@@ -36,15 +49,66 @@ error_chain! {
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
/// What a request resolves to. Artists live under `/models/`, franchises and
|
||||
/// genres under `/categories/`, characters and everything else under `/tags/`.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
enum Target {
|
||||
Latest,
|
||||
Search(String),
|
||||
Artist(String),
|
||||
Category(String),
|
||||
Tag(String),
|
||||
}
|
||||
|
||||
impl Target {
|
||||
/// Archive path (`models/{slug}`, ...) for the archive targets.
|
||||
fn archive_path(&self) -> Option<String> {
|
||||
match self {
|
||||
Target::Artist(slug) => Some(format!("models/{slug}")),
|
||||
Target::Category(slug) => Some(format!("categories/{slug}")),
|
||||
Target::Tag(slug) => Some(format!("tags/{slug}")),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub struct Rule34videoProvider {
|
||||
url: String,
|
||||
/// Enriched items (formats, tags, artist) by video URL. Every listing costs
|
||||
/// ~23 detail pages, again for the prefetched next page; refetching them
|
||||
/// on each request is what runs into the site's 429 rate limit.
|
||||
details: VideoCache,
|
||||
/// Bare query -> resolved target, so later pages skip the archive probes.
|
||||
targets: Arc<Mutex<HashMap<String, (Instant, Target)>>>,
|
||||
}
|
||||
|
||||
impl Rule34videoProvider {
|
||||
pub fn new() -> Self {
|
||||
let mut details = VideoCache::new();
|
||||
details.max_size(DETAIL_CACHE_SIZE);
|
||||
Rule34videoProvider {
|
||||
url: "https://rule34video.com".to_string(),
|
||||
details,
|
||||
targets: Arc::new(Mutex::new(HashMap::new())),
|
||||
}
|
||||
}
|
||||
|
||||
fn remembered_target(&self, query: &str) -> Option<Target> {
|
||||
let targets = self.targets.lock().ok()?;
|
||||
let (time, target) = targets.get(&slug_dashed(query))?;
|
||||
(time.elapsed() < TARGET_TTL).then(|| target.clone())
|
||||
}
|
||||
|
||||
fn remember_target(&self, query: &str, target: &Target) {
|
||||
if let Ok(mut targets) = self.targets.lock() {
|
||||
if targets.len() >= TARGET_CACHE_SIZE {
|
||||
targets.retain(|_, (time, _)| time.elapsed() < TARGET_TTL);
|
||||
if targets.len() >= TARGET_CACHE_SIZE {
|
||||
targets.clear();
|
||||
}
|
||||
}
|
||||
targets.insert(slug_dashed(query), (Instant::now(), target.clone()));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -93,6 +157,192 @@ impl Rule34videoProvider {
|
||||
}
|
||||
}
|
||||
|
||||
/// Explicit `artist:` / `category:` / `tag:` style prefixes (the server has
|
||||
/// already stripped a leading `#`).
|
||||
fn parse_prefixed(query: &str) -> Option<Target> {
|
||||
let (kind, value) = query.trim().split_once(':')?;
|
||||
let slug = slug_dashed(value);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
match kind.trim().to_ascii_lowercase().as_str() {
|
||||
"artist" | "artists" | "model" | "models" | "author" | "creator" | "uploader" => {
|
||||
Some(Target::Artist(slug))
|
||||
}
|
||||
"cat" | "category" | "categories" | "franchise" | "series" | "game" => {
|
||||
Some(Target::Category(slug))
|
||||
}
|
||||
"tag" | "tags" | "character" | "char" => Some(Target::Tag(slug)),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Titles of the tags offered by `/search_ajax.php?tag=`.
|
||||
fn parse_tag_suggestions(html: &str) -> Vec<String> {
|
||||
let document = Html::parse_fragment(html);
|
||||
let Ok(selector) = Selector::parse("input[data-title]") else {
|
||||
return vec![];
|
||||
};
|
||||
document
|
||||
.select(&selector)
|
||||
.filter_map(|input| input.value().attr("data-title"))
|
||||
.map(|title| title.trim().to_string())
|
||||
.filter(|title| !title.is_empty())
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The suggested tag whose name (ignoring a trailing "(franchise)") equals
|
||||
/// the query, e.g. "samus aran" -> "samus aran (metroid)".
|
||||
fn best_tag_title(titles: &[String], query: &str) -> Option<String> {
|
||||
let wanted = slug_dashed(query);
|
||||
if wanted.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let mut exact = titles.iter().filter(|title| {
|
||||
let base = title.split('(').next().unwrap_or(title);
|
||||
slug_dashed(base) == wanted || slug_dashed(title) == wanted
|
||||
});
|
||||
let first = exact.next()?;
|
||||
// Several franchises share a character name: ambiguous, leave it to search.
|
||||
exact.next().is_none().then(|| first.clone())
|
||||
}
|
||||
|
||||
/// A free-form query that names an artist, category or tag goes to that
|
||||
/// archive (probed on the site; unknown names 404), else keyword search.
|
||||
/// The second value is false when a probe failed (rate limit, outage) and
|
||||
/// the answer must not be remembered.
|
||||
async fn resolve_keyword(&self, query: &str, options: &ServerOptions) -> (Target, bool) {
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() {
|
||||
return (Target::Search(query.trim().to_string()), true);
|
||||
}
|
||||
// Artist slugs are inconsistent on the site: "general-butch" but "biggreen".
|
||||
let mut candidates = vec![Target::Artist(slug.clone())];
|
||||
if slug.contains('-') {
|
||||
candidates.push(Target::Artist(slug.replace('-', "")));
|
||||
}
|
||||
candidates.push(Target::Category(slug.clone()));
|
||||
candidates.push(Target::Tag(slug));
|
||||
let mut requester = requester_or_default(options, "rule34video", "resolve_keyword");
|
||||
let referer = format!("{}/", self.url);
|
||||
let mut conclusive = true;
|
||||
for candidate in &candidates {
|
||||
let Some(path) = candidate.archive_path() else {
|
||||
continue;
|
||||
};
|
||||
let url = format!("{}/{path}/", self.url);
|
||||
let headers = vec![("Referer".to_string(), referer.clone())];
|
||||
match requester.get_probe(&url, headers).await {
|
||||
Ok(Some(body)) => {
|
||||
let items = self.get_video_items_from_html(body).len();
|
||||
if items >= MIN_PROBED_ARCHIVE_ITEMS {
|
||||
return (candidate.clone(), true);
|
||||
}
|
||||
}
|
||||
Ok(None) => {}
|
||||
Err(_) => conclusive = false,
|
||||
}
|
||||
}
|
||||
|
||||
// Characters are tagged "name (franchise)": ask the site's own tag search.
|
||||
let suggest_url = format!(
|
||||
"{}/search_ajax.php?tag={}",
|
||||
self.url,
|
||||
utf8_percent_encode(query.trim(), NON_ALPHANUMERIC)
|
||||
);
|
||||
let headers = vec![
|
||||
("Referer".to_string(), referer),
|
||||
("X-Requested-With".to_string(), "XMLHttpRequest".to_string()),
|
||||
];
|
||||
match requester.get_probe(&suggest_url, headers).await {
|
||||
Ok(Some(body)) => {
|
||||
let titles = Self::parse_tag_suggestions(&body);
|
||||
if let Some(title) = Self::best_tag_title(&titles, query) {
|
||||
return (Target::Tag(slug_dashed(&title)), conclusive);
|
||||
}
|
||||
}
|
||||
Ok(None) => {}
|
||||
Err(_) => conclusive = false,
|
||||
}
|
||||
(Target::Search(query.trim().to_string()), conclusive)
|
||||
}
|
||||
|
||||
async fn resolve_target(&self, query: Option<&str>, options: &ServerOptions) -> Target {
|
||||
let Some(query) = query.map(str::trim).filter(|q| !q.is_empty()) else {
|
||||
return Target::Latest;
|
||||
};
|
||||
if let Some(target) = Self::parse_prefixed(query) {
|
||||
return target;
|
||||
}
|
||||
if let Some(target) = self.remembered_target(query) {
|
||||
return target;
|
||||
}
|
||||
let (target, conclusive) = self.resolve_keyword(query, options).await;
|
||||
if conclusive {
|
||||
self.remember_target(query, &target);
|
||||
}
|
||||
target
|
||||
}
|
||||
|
||||
async fn archive(
|
||||
&self,
|
||||
cache: VideoCache,
|
||||
target: &Target,
|
||||
page: u8,
|
||||
sort: &str,
|
||||
options: ServerOptions,
|
||||
) -> Result<Vec<VideoItem>> {
|
||||
let Some(path) = target.archive_path() else {
|
||||
return Ok(vec![]);
|
||||
};
|
||||
let sort_val = match sort {
|
||||
"post_date" | "video_viewed" | "rating" | "duration" | "pseudo_random" => sort,
|
||||
_ => "post_date",
|
||||
};
|
||||
let index = format!("rule34video:{path}:{page}:{sort_val}");
|
||||
if sort_val != "pseudo_random" {
|
||||
if let Some((time, items)) = cache.get(&index) {
|
||||
if time.elapsed().unwrap_or_default().as_secs() < 300 {
|
||||
return Ok(items.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let mut requester = options.requester.clone().ok_or("Requester missing")?;
|
||||
let url = format!(
|
||||
"{}/{path}/?mode=async&function=get_block&block_id=custom_list_videos_common_videos&sort_by={sort_val}&from={page}",
|
||||
self.url
|
||||
);
|
||||
let headers = vec![("Referer".to_string(), format!("{}/{path}/", self.url))];
|
||||
let stale = || cache.get(&index).map(|(_, items)| items).unwrap_or_default();
|
||||
let text = match requester.get_probe(&url, headers).await {
|
||||
Ok(Some(text)) => text,
|
||||
// A 404 is an unknown archive (or a page past the end).
|
||||
Ok(None) => return Ok(vec![]),
|
||||
Err(e) => {
|
||||
eprintln!("rule34video archive fetch failed for {url}: {e}");
|
||||
return Ok(stale());
|
||||
}
|
||||
};
|
||||
let video_items = self.get_video_items_from_html(text);
|
||||
let video_items = self.enrich_items(video_items, requester).await;
|
||||
if video_items.is_empty() {
|
||||
return Ok(stale());
|
||||
}
|
||||
cache.insert(index, video_items.clone());
|
||||
Ok(video_items)
|
||||
}
|
||||
|
||||
/// `/search/{q}/` takes dashes for spaces. A raw `/` splits the path (404),
|
||||
/// `%20` is refused (403) and `&` cuts the query short, so keep only words.
|
||||
fn search_path_segment(query: &str) -> String {
|
||||
slug_dashed(query)
|
||||
.split('-')
|
||||
.map(|word| utf8_percent_encode(word, NON_ALPHANUMERIC).to_string())
|
||||
.collect::<Vec<_>>()
|
||||
.join("-")
|
||||
}
|
||||
|
||||
/// Helper to safely extract a string between two delimiters
|
||||
fn extract_between<'a>(content: &'a str, start_pat: &str, end_pat: &str) -> Option<&'a str> {
|
||||
let start_idx = content.find(start_pat)? + start_pat.len();
|
||||
@@ -208,7 +458,7 @@ impl Rule34videoProvider {
|
||||
let url = format!(
|
||||
"{}/search/{}/?mode=async&function=get_block&block_id=custom_list_videos_videos_list_search&tag_ids=&sort_by={}&from_videos={}&from_albums={}&_={}",
|
||||
self.url,
|
||||
query.replace(" ", "-"),
|
||||
Self::search_path_segment(query),
|
||||
sort_val,
|
||||
page,
|
||||
page,
|
||||
@@ -307,30 +557,106 @@ impl Rule34videoProvider {
|
||||
(formats.into_iter().map(|(_, f)| f).collect(), tags)
|
||||
}
|
||||
|
||||
/// Artists credited on a video page: `(name, slug)` from the "Artist" group.
|
||||
fn parse_artists(html: &str) -> Vec<(String, String)> {
|
||||
let document = Html::parse_document(html);
|
||||
let Ok(group_sel) = Selector::parse("[data-suggest-type=\"model\"] a[href*=\"/models/\"]")
|
||||
else {
|
||||
return vec![];
|
||||
};
|
||||
let name_sel = Selector::parse(".name").ok();
|
||||
let mut artists: Vec<(String, String)> = Vec::new();
|
||||
for link in document.select(&group_sel) {
|
||||
let href = link.value().attr("href").unwrap_or_default();
|
||||
let slug = href
|
||||
.split("/models/")
|
||||
.nth(1)
|
||||
.unwrap_or_default()
|
||||
.trim_matches('/')
|
||||
.to_string();
|
||||
let name = name_sel
|
||||
.as_ref()
|
||||
.and_then(|sel| link.select(sel).next())
|
||||
.map(|n| n.text().collect::<String>())
|
||||
.unwrap_or_else(|| link.text().collect::<String>());
|
||||
let name = name.split_whitespace().collect::<Vec<_>>().join(" ");
|
||||
if slug.is_empty() || name.is_empty() || artists.iter().any(|(_, s)| *s == slug) {
|
||||
continue;
|
||||
}
|
||||
artists.push((name, slug));
|
||||
}
|
||||
artists
|
||||
}
|
||||
|
||||
fn cached_detail(&self, item: &mut VideoItem) -> bool {
|
||||
let Some((time, cached)) = self.details.get(&item.url) else {
|
||||
return false;
|
||||
};
|
||||
let Some(cached) = cached.into_iter().next() else {
|
||||
return false;
|
||||
};
|
||||
if time.elapsed().unwrap_or(DETAIL_TTL) >= DETAIL_TTL {
|
||||
return false;
|
||||
}
|
||||
item.formats = cached.formats;
|
||||
item.tags = cached.tags;
|
||||
item.uploader = cached.uploader;
|
||||
item.uploaderUrl = cached.uploaderUrl;
|
||||
item.uploaderId = cached.uploaderId;
|
||||
true
|
||||
}
|
||||
|
||||
async fn enrich_items(&self, items: Vec<VideoItem>, requester: Requester) -> Vec<VideoItem> {
|
||||
let referer = format!("{}/", self.url);
|
||||
let base = self.url.clone();
|
||||
// Once the site rate-limits a detail page, the rest of the batch keeps
|
||||
// the listing data instead of digging the hole deeper.
|
||||
let throttled = Arc::new(AtomicBool::new(false));
|
||||
stream::iter(items.into_iter().map(|mut item| {
|
||||
let mut requester = requester.clone();
|
||||
let referer = referer.clone();
|
||||
let base = base.clone();
|
||||
let throttled = throttled.clone();
|
||||
async move {
|
||||
if item.url.is_empty() {
|
||||
if item.url.is_empty() || self.cached_detail(&mut item) {
|
||||
return item;
|
||||
}
|
||||
let html = requester
|
||||
.get_with_headers(
|
||||
&item.url,
|
||||
vec![("Referer".to_string(), referer)],
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.unwrap_or_default();
|
||||
let (formats, tags) = Self::parse_detail(&html);
|
||||
if throttled.load(Ordering::Relaxed) {
|
||||
return item;
|
||||
}
|
||||
// `get_probe` hands challenges to the full fallback chain but
|
||||
// reports a lasting 429 as an error rather than escalating every
|
||||
// detail page to FlareSolverr.
|
||||
let headers = vec![("Referer".to_string(), referer)];
|
||||
let html = match requester.get_probe(&item.url, headers).await {
|
||||
Ok(Some(html)) => html,
|
||||
Ok(None) => return item,
|
||||
Err(_) => {
|
||||
throttled.store(true, Ordering::Relaxed);
|
||||
return item;
|
||||
}
|
||||
};
|
||||
let (formats, mut tags) = Self::parse_detail(&html);
|
||||
if !formats.is_empty() {
|
||||
item.formats = Some(formats);
|
||||
}
|
||||
let artists = Self::parse_artists(&html);
|
||||
if let Some((name, slug)) = artists.first() {
|
||||
item.uploader = Some(name.clone());
|
||||
item.uploaderUrl = Some(format!("{base}/models/{slug}/"));
|
||||
item.uploaderId = Some(format!("rule34video:{slug}"));
|
||||
}
|
||||
for (name, _) in &artists {
|
||||
if !tags.iter().any(|t| t.eq_ignore_ascii_case(name)) {
|
||||
tags.push(name.clone());
|
||||
}
|
||||
}
|
||||
if !tags.is_empty() {
|
||||
item.tags = Some(tags);
|
||||
}
|
||||
if item.formats.is_some() {
|
||||
self.details.insert(item.url.clone(), vec![item.clone()]);
|
||||
}
|
||||
item
|
||||
}
|
||||
}))
|
||||
@@ -344,15 +670,15 @@ impl Rule34videoProvider {
|
||||
return vec![];
|
||||
}
|
||||
|
||||
// Safely isolate the video listing section
|
||||
let video_listing = match Self::extract_between(
|
||||
&html,
|
||||
"id=\"custom_list_videos",
|
||||
"<div class=\"pagination\"",
|
||||
) {
|
||||
Some(content) => content,
|
||||
None => return vec![],
|
||||
// Isolate the video listing section. A result that fits on one page has
|
||||
// no pagination block, so the listing then runs to the end.
|
||||
let Some(start) = html.find("id=\"custom_list_videos") else {
|
||||
return vec![];
|
||||
};
|
||||
let video_listing = &html[start..];
|
||||
let video_listing = video_listing
|
||||
.find("<div class=\"pagination\"")
|
||||
.map_or(video_listing, |end| &video_listing[..end]);
|
||||
|
||||
let mut items = Vec::new();
|
||||
// Skip the first split result as it's the preamble
|
||||
@@ -449,9 +775,11 @@ impl Provider for Rule34videoProvider {
|
||||
) -> Vec<VideoItem> {
|
||||
let page_num = page.parse::<u8>().unwrap_or(1);
|
||||
|
||||
let result = match query {
|
||||
Some(q) => self.query(cache, page_num, &q, &sort, options).await,
|
||||
None => self.get(cache, page_num, &sort, options).await,
|
||||
let target = self.resolve_target(query.as_deref(), &options).await;
|
||||
let result = match &target {
|
||||
Target::Latest => self.get(cache, page_num, &sort, options).await,
|
||||
Target::Search(q) => self.query(cache, page_num, q, &sort, options).await,
|
||||
archive => self.archive(cache, archive, page_num, &sort, options).await,
|
||||
};
|
||||
|
||||
match result {
|
||||
@@ -517,4 +845,81 @@ mod tests {
|
||||
let (formats, tags) = Rule34videoProvider::parse_detail("<html></html>");
|
||||
assert!(formats.is_empty() && tags.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_prefixed_queries() {
|
||||
assert_eq!(
|
||||
Rule34videoProvider::parse_prefixed("artist:General Butch"),
|
||||
Some(Target::Artist("general-butch".into()))
|
||||
);
|
||||
assert_eq!(
|
||||
Rule34videoProvider::parse_prefixed("category:Super Smash Bros"),
|
||||
Some(Target::Category("super-smash-bros".into()))
|
||||
);
|
||||
assert_eq!(
|
||||
Rule34videoProvider::parse_prefixed("tag:black widow (marvel)"),
|
||||
Some(Target::Tag("black-widow-marvel".into()))
|
||||
);
|
||||
assert_eq!(Rule34videoProvider::parse_prefixed("samus aran"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn picks_the_unambiguous_character_tag() {
|
||||
let titles = vec![
|
||||
"samus aran (metroid)".to_string(),
|
||||
"samus aran's mother".to_string(),
|
||||
];
|
||||
assert_eq!(
|
||||
Rule34videoProvider::best_tag_title(&titles, "samus aran").as_deref(),
|
||||
Some("samus aran (metroid)")
|
||||
);
|
||||
let ambiguous = vec!["peach (mario)".to_string(), "peach (pokemon)".to_string()];
|
||||
assert_eq!(Rule34videoProvider::best_tag_title(&ambiguous, "peach"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_tag_suggestions_and_artists() {
|
||||
let suggest = r#"<div class="item checkbox-container"><input id="tag_143" type="checkbox" value="143" data-title="samus aran (metroid)"/></div>"#;
|
||||
assert_eq!(
|
||||
Rule34videoProvider::parse_tag_suggestions(suggest),
|
||||
vec!["samus aran (metroid)".to_string()]
|
||||
);
|
||||
let page = r#"<div class="col vp-group" data-suggest-type="model"><div class="label">Artist</div>
|
||||
<span class="chip"><a class="item btn_link" href="https://rule34video.com/models/joigang-va/">
|
||||
<span class="name">JOIGang (VA)</span></a></span></div>"#;
|
||||
assert_eq!(
|
||||
Rule34videoProvider::parse_artists(page),
|
||||
vec![("JOIGang (VA)".to_string(), "joigang-va".to_string())]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_a_single_page_listing_without_pagination() {
|
||||
let html = r#"<div id="custom_list_videos_common_videos_items">
|
||||
<div class="item thumb video_7"><a class="th js-open-popup" href="https://rule34video.com/video/7/elf/">
|
||||
<img class="thumb" src="https://rule34video.com/t/7.jpg"><div class="time">0:30</div>
|
||||
<div class="thumb_title">Elf</div></a></div></div></div>"#;
|
||||
let items = Rule34videoProvider::new().get_video_items_from_html(html.to_string());
|
||||
assert_eq!(items.len(), 1);
|
||||
assert_eq!(items[0].id, "7");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn search_segment_uses_dashes_and_drops_path_breaking_punctuation() {
|
||||
assert_eq!(Rule34videoProvider::search_path_segment("elf girl"), "elf-girl");
|
||||
assert_eq!(Rule34videoProvider::search_path_segment("2b/9s"), "2b-9s");
|
||||
assert_eq!(Rule34videoProvider::search_path_segment("Tifa & Aerith"), "tifa-aerith");
|
||||
assert_eq!(Rule34videoProvider::search_path_segment("Pokémon"), "pok%C3%A9mon");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn remembers_resolved_targets_by_slug() {
|
||||
let provider = Rule34videoProvider::new();
|
||||
provider.remember_target("Samus Aran", &Target::Tag("samus-aran-metroid".into()));
|
||||
assert_eq!(
|
||||
provider.remembered_target("samus aran"),
|
||||
Some(Target::Tag("samus-aran-metroid".into()))
|
||||
);
|
||||
assert_eq!(provider.remembered_target("samus"), None);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{
|
||||
Provider, report_provider_error, report_provider_error_background, requester_or_default,
|
||||
Provider, probe_first_archive, report_provider_error, report_provider_error_background,
|
||||
requester_or_default,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
@@ -517,6 +518,28 @@ impl SextbProvider {
|
||||
Target::Search { slug }
|
||||
}
|
||||
|
||||
/// A free-form query that names an actress or a studio outside the loaded
|
||||
/// catalogs goes to that archive (probed on the site), else keyword search.
|
||||
async fn resolve_keyword(&self, slug: &str, options: &ServerOptions) -> Option<Target> {
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let candidates = [
|
||||
Target::Actress { slug: slug.to_string() },
|
||||
Target::Studio { slug: slug.to_string() },
|
||||
];
|
||||
let urls: Vec<String> = candidates
|
||||
.iter()
|
||||
.map(|t| self.build_url_for_target(t, 1))
|
||||
.collect();
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
|
||||
body.contains("tray-item") && !body.contains("No Video were found")
|
||||
})
|
||||
.await?;
|
||||
candidates.into_iter().nth(index)
|
||||
}
|
||||
|
||||
fn build_url_for_target(&self, target: &Target, page: u16) -> String {
|
||||
match target {
|
||||
Target::Section { section, sort } => {
|
||||
@@ -1092,7 +1115,12 @@ impl SextbProvider {
|
||||
per_page_limit: usize,
|
||||
options: ServerOptions,
|
||||
) -> Result<Vec<VideoItem>> {
|
||||
let target = self.resolve_query_target(query, sort);
|
||||
let mut target = self.resolve_query_target(query, sort);
|
||||
if let Target::Search { slug } = &target {
|
||||
if let Some(found) = self.resolve_keyword(slug, &options).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
let url = self.build_url_for_target(&target, page);
|
||||
self.fetch_items_for_url(cache, url, per_page_limit, &options)
|
||||
.await
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{
|
||||
Provider, build_proxy_url, report_provider_error, requester_or_default,
|
||||
Provider, build_proxy_url, probe_first_archive, report_provider_error, requester_or_default,
|
||||
slug_dashed,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
@@ -367,6 +368,26 @@ impl ThepornbunnyProvider {
|
||||
}
|
||||
}
|
||||
|
||||
/// A free-form query that names a pornstar, studio or tag goes to that
|
||||
/// archive (probed on the site; unknown names redirect home), else keyword
|
||||
/// search.
|
||||
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let urls: Vec<String> = ["pornstar", "studio", "tag"]
|
||||
.iter()
|
||||
.map(|kind| format!("{BASE_URL}/{kind}/{slug}/"))
|
||||
.collect();
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{BASE_URL}/"), |body| {
|
||||
body.contains("th-description")
|
||||
})
|
||||
.await?;
|
||||
Some(Target::Archive(urls[index].trim_end_matches('/').to_string()))
|
||||
}
|
||||
|
||||
fn resolve_option_target(options: &ServerOptions, sort: &str) -> Target {
|
||||
if let Some(cat) = options.categories.as_deref() {
|
||||
if cat.starts_with(BASE_URL) && cat != "all" {
|
||||
@@ -476,10 +497,15 @@ impl Provider for ThepornbunnyProvider {
|
||||
let page = page.parse::<u16>().unwrap_or(1).max(1);
|
||||
let per_page_limit = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
|
||||
|
||||
let target = match query {
|
||||
let mut target = match query {
|
||||
Some(ref q) if !q.trim().is_empty() => Self::resolve_query_target(q.trim()),
|
||||
_ => Self::resolve_option_target(&options, &sort),
|
||||
};
|
||||
if let Target::Search(query) = &target {
|
||||
if let Some(found) = Self::resolve_keyword(query, &options).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
|
||||
match self
|
||||
.fetch_target(cache, target, page, per_page_limit, options)
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{
|
||||
Provider, build_proxy_url, proxy_aylo_thumb, report_provider_error, requester_or_default,
|
||||
Provider, build_proxy_url, probe_first_archive, proxy_aylo_thumb, report_provider_error,
|
||||
requester_or_default, slug_dashed,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
use crate::util::requester::Requester;
|
||||
use crate::util::parse_abbreviated_number;
|
||||
use crate::util::time::parse_time_to_seconds;
|
||||
use crate::videos::{ServerOptions, VideoFormat, VideoItem};
|
||||
@@ -136,6 +138,25 @@ impl Tube8Provider {
|
||||
]
|
||||
}
|
||||
|
||||
/// A free-form query that names a pornstar or a channel goes to that
|
||||
/// archive (probed on the site; unknown names 404), else keyword search.
|
||||
async fn resolve_keyword(&self, requester: &mut Requester, query: &str) -> Option<Target> {
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let candidates = [
|
||||
Target::Pornstar { slug: slug.clone() },
|
||||
Target::Channel { slug },
|
||||
];
|
||||
let urls: Vec<String> = candidates.iter().map(|t| self.build_url(t, 1)).collect();
|
||||
let (index, _) = probe_first_archive(requester, &urls, &format!("{BASE_URL}/"), |body| {
|
||||
body.contains("video-box")
|
||||
})
|
||||
.await?;
|
||||
candidates.into_iter().nth(index)
|
||||
}
|
||||
|
||||
fn target_from_request(query: Option<&str>, sort: &str) -> Target {
|
||||
if let Some(q) = query.map(str::trim).filter(|v| !v.is_empty()) {
|
||||
let lower = q.to_ascii_lowercase();
|
||||
@@ -439,7 +460,13 @@ impl Provider for Tube8Provider {
|
||||
options: ServerOptions,
|
||||
) -> Vec<VideoItem> {
|
||||
let page = page.parse::<u16>().unwrap_or(1).max(1);
|
||||
let target = Self::target_from_request(query.as_deref(), &sort);
|
||||
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
|
||||
let mut target = Self::target_from_request(query.as_deref(), &sort);
|
||||
if let Target::Search { query } = &target {
|
||||
if let Some(found) = self.resolve_keyword(&mut requester, query).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
let url = self.build_url(&target, page);
|
||||
|
||||
let old_items = match cache.get(&url) {
|
||||
@@ -450,7 +477,6 @@ impl Provider for Tube8Provider {
|
||||
None => vec![],
|
||||
};
|
||||
|
||||
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
|
||||
let text = match requester
|
||||
.get_with_headers(&url, Self::html_headers(), None)
|
||||
.await
|
||||
|
||||
@@ -592,7 +592,7 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn decodes_cyrillic_obfuscated_video_url() {
|
||||
let raw = "L2dldF9maWxlLzI1Lzg2\u{041c}D\u{0415}zNWFkND\u{0415}zODNm\u{041c}zI0ZmJhYjJlZTg3\u{041c}jRkYm\u{041c}zOWNiZTIwNjJl\u{041c}ci8y\u{041c}TgwOD\u{0410}w\u{041c}\u{0421}8y\u{041c}TgwODcwNS8y\u{041c}TgwODcwNV9ocS5tcDQvP2Q9\u{041c}czQ4NSZicj0y\u{041c}TgmdGk9\u{041c}Tc5\u{041c}TY0NjYyOQ~~";
|
||||
let raw = "L2dldF9maWxlLzI1Lzg2\u{041c}D\u{0415}zNWFkND\u{0415}zODNm\u{041c}zI0ZmJhYjJlZTg3\u{041c}jRkYm\u{041c}zOWNiZTIwNjJl\u{041c}i8y\u{041c}TgwOD\u{0410}w\u{041c}\u{0421}8y\u{041c}TgwODcwNS8y\u{041c}TgwODcwNV9ocS5tcDQvP2Q9\u{041c}zQ4NSZicj0y\u{041c}TgmdGk9\u{041c}Tc5\u{041c}TY0NjYyOQ~~";
|
||||
assert_eq!(
|
||||
TxxxProvider::decode_video_url(raw).as_deref(),
|
||||
Some("https://txxx.me/get_file/25/860135ad41383f324fbab2ee8724dbc39cbe2062e2/21808000/21808705/21808705_hq.mp4/?d=3485&br=218&ti=1791646629")
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{Provider, report_provider_error, requester_or_default};
|
||||
use crate::providers::{
|
||||
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
use crate::videos::{ServerOptions, VideoItem};
|
||||
@@ -426,6 +428,23 @@ impl XgroovyProvider {
|
||||
Target::Search(trimmed.to_string())
|
||||
}
|
||||
|
||||
/// A free-form query that names a pornstar or a channel goes to that
|
||||
/// archive (probed on the site; unknown names 404), else keyword search.
|
||||
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let candidates = [Target::Pornstar(slug.clone()), Target::Channel(slug)];
|
||||
let urls: Vec<String> = candidates.iter().map(|t| Self::target_url(t, 1)).collect();
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{BASE_URL}/"), |body| {
|
||||
body.contains("data-video-id")
|
||||
})
|
||||
.await?;
|
||||
candidates.into_iter().nth(index)
|
||||
}
|
||||
|
||||
fn resolve_option_target(&self, options: &ServerOptions) -> Target {
|
||||
if let Some(cat) = options.categories.as_deref() {
|
||||
if cat != "all" && !cat.is_empty() {
|
||||
@@ -488,10 +507,15 @@ impl Provider for XgroovyProvider {
|
||||
let page = page.parse::<u16>().unwrap_or(1).max(1);
|
||||
let per_page = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
|
||||
|
||||
let target = match query {
|
||||
let mut target = match query {
|
||||
Some(q) if !q.trim().is_empty() => self.resolve_query_target(q.trim()),
|
||||
_ => self.resolve_option_target(&options),
|
||||
};
|
||||
if let Target::Search(query) = &target {
|
||||
if let Some(found) = Self::resolve_keyword(query, &options).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
|
||||
match self
|
||||
.fetch_target(cache, target, page, per_page, options)
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{Provider, report_provider_error, requester_or_default};
|
||||
use crate::providers::{
|
||||
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
use crate::util::parse_abbreviated_number;
|
||||
@@ -94,6 +96,7 @@ enum Target {
|
||||
Search(String),
|
||||
Category(String),
|
||||
Channel(String),
|
||||
Pornstar(String),
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
@@ -208,6 +211,7 @@ impl XhamsterProvider {
|
||||
}
|
||||
Target::Category(slug) => format!("{BASE_URL}/categories/{slug}"),
|
||||
Target::Channel(slug) => format!("{BASE_URL}/channels/{slug}"),
|
||||
Target::Pornstar(slug) => format!("{BASE_URL}/pornstars/{slug}"),
|
||||
};
|
||||
if page <= 1 {
|
||||
base
|
||||
@@ -385,6 +389,7 @@ impl XhamsterProvider {
|
||||
match kind.trim().to_ascii_lowercase().as_str() {
|
||||
"cat" | "category" => return Target::Category(slug),
|
||||
"channel" | "channels" => return Target::Channel(slug),
|
||||
"pornstar" | "pornstars" | "star" | "model" => return Target::Pornstar(slug),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
@@ -401,6 +406,23 @@ impl XhamsterProvider {
|
||||
Target::Search(trimmed.to_string())
|
||||
}
|
||||
|
||||
/// A free-form query that names a pornstar or a channel goes to that
|
||||
/// archive (probed on the site; unknown names 404), else keyword search.
|
||||
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let candidates = [Target::Pornstar(slug.clone()), Target::Channel(slug)];
|
||||
let urls: Vec<String> = candidates.iter().map(|t| Self::target_url(t, 1)).collect();
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{BASE_URL}/"), |body| {
|
||||
body.contains("video-thumb-info__name")
|
||||
})
|
||||
.await?;
|
||||
candidates.into_iter().nth(index)
|
||||
}
|
||||
|
||||
fn resolve_sort_target(sort: &str) -> Target {
|
||||
match sort.trim().to_ascii_lowercase().as_str() {
|
||||
"popular" | "viewed" | "most_viewed" | "mostviewed" => Target::MostViewed,
|
||||
@@ -472,10 +494,15 @@ impl Provider for XhamsterProvider {
|
||||
let page = page.parse::<u16>().unwrap_or(1).max(1);
|
||||
let per_page = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
|
||||
|
||||
let target = match query {
|
||||
let mut target = match query {
|
||||
Some(q) if !q.trim().is_empty() => self.resolve_query_target(q.trim()),
|
||||
_ => self.resolve_option_target(&options, &sort),
|
||||
};
|
||||
if let Target::Search(query) = &target {
|
||||
if let Some(found) = Self::resolve_keyword(query, &options).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
|
||||
match self
|
||||
.fetch_target(cache, target, page, per_page, options)
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{Provider, report_provider_error, requester_or_default};
|
||||
use crate::providers::{
|
||||
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
use crate::util::parse_abbreviated_number;
|
||||
@@ -86,6 +88,8 @@ enum Target {
|
||||
Best,
|
||||
Search(String),
|
||||
Archive(String),
|
||||
/// Channel / pornstar profile, served as JSON by `/channels/{slug}/videos/new/{n}`.
|
||||
Channel(String),
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
@@ -238,6 +242,9 @@ impl XvideosProvider {
|
||||
format!("{BASE_URL}/?k={encoded}&p={}", page - 1)
|
||||
}
|
||||
}
|
||||
Target::Channel(slug) => {
|
||||
format!("{BASE_URL}/channels/{slug}/videos/new/{}", page.saturating_sub(1))
|
||||
}
|
||||
Target::Archive(base_url) => {
|
||||
let base = base_url.trim_end_matches('/');
|
||||
if page <= 1 {
|
||||
@@ -528,9 +535,8 @@ impl XvideosProvider {
|
||||
let slug = value.replace(' ', "_");
|
||||
return Target::Archive(format!("{BASE_URL}/c/{slug}"));
|
||||
}
|
||||
"uploader" | "channel" | "profile" => {
|
||||
let slug = value.replace(' ', "_").to_ascii_lowercase();
|
||||
return Target::Archive(format!("{BASE_URL}/{slug}"));
|
||||
"uploader" | "channel" | "profile" | "pornstar" | "star" | "model" => {
|
||||
return Target::Channel(slug_dashed(value));
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
@@ -545,6 +551,83 @@ impl XvideosProvider {
|
||||
Target::Search(trimmed.to_string())
|
||||
}
|
||||
|
||||
/// A free-form query that names a channel / pornstar profile goes to its
|
||||
/// archive (probed on the site; unknown names land on an empty profile),
|
||||
/// else keyword search.
|
||||
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let target = Target::Channel(slug);
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
probe_first_archive(
|
||||
&mut requester,
|
||||
&[Self::target_url(&target, 1)],
|
||||
&format!("{BASE_URL}/"),
|
||||
|body| body.contains("\"videos\":[{"),
|
||||
)
|
||||
.await
|
||||
.map(|_| target)
|
||||
}
|
||||
|
||||
/// Parses the JSON served for a channel / profile video list.
|
||||
fn parse_channel_json(json: &str, limit: usize) -> Result<Vec<VideoItem>> {
|
||||
let root: serde_json::Value = serde_json::from_str(json)
|
||||
.map_err(|e| Error::from(format!("channel json: {e}")))?;
|
||||
let Some(videos) = root.get("videos").and_then(|v| v.as_array()) else {
|
||||
return Ok(vec![]);
|
||||
};
|
||||
let text = |v: &serde_json::Value, key: &str| -> String {
|
||||
v.get(key).and_then(|x| x.as_str()).unwrap_or_default().trim().to_string()
|
||||
};
|
||||
let mut items = Vec::new();
|
||||
for video in videos {
|
||||
let eid = text(video, "eid");
|
||||
let thumb = text(video, "i");
|
||||
let title = {
|
||||
let full = text(video, "tf");
|
||||
if full.is_empty() { text(video, "t") } else { full }
|
||||
};
|
||||
if eid.is_empty() || thumb.is_empty() || title.is_empty() {
|
||||
continue;
|
||||
}
|
||||
let id = video
|
||||
.get("id")
|
||||
.map(|x| x.to_string().trim_matches('"').to_string())
|
||||
.unwrap_or_else(|| eid.clone());
|
||||
let mut item = VideoItem::new(
|
||||
id,
|
||||
Self::decode_html(&title),
|
||||
format!("{BASE_URL}/video.{eid}/x"),
|
||||
CHANNEL_ID.to_string(),
|
||||
thumb,
|
||||
Self::parse_duration(&text(video, "d")),
|
||||
);
|
||||
item.views = Self::parse_views(&text(video, "n"));
|
||||
item.rating = text(video, "r").trim_end_matches('%').parse::<f32>().ok();
|
||||
let preview = text(video, "ipu");
|
||||
if !preview.is_empty() {
|
||||
item.preview = Some(preview);
|
||||
}
|
||||
let name = text(video, "pn");
|
||||
let slug = text(video, "pu");
|
||||
if !name.is_empty() {
|
||||
item.uploader = Some(name);
|
||||
let slug = slug.trim_matches('/').to_string();
|
||||
if !slug.is_empty() {
|
||||
item.uploaderUrl = Some(format!("{BASE_URL}/{slug}"));
|
||||
item.uploaderId = Some(format!("{CHANNEL_ID}:{slug}"));
|
||||
}
|
||||
}
|
||||
items.push(item);
|
||||
if items.len() >= limit {
|
||||
break;
|
||||
}
|
||||
}
|
||||
Ok(items)
|
||||
}
|
||||
|
||||
fn resolve_option_target(options: &ServerOptions, sort: &str) -> Target {
|
||||
if let Some(cat) = options.categories.as_deref() {
|
||||
if cat != "all" && !cat.is_empty() {
|
||||
@@ -602,7 +685,11 @@ impl XvideosProvider {
|
||||
return Ok(vec![]);
|
||||
}
|
||||
|
||||
let items = Self::parse_listing(&html, per_page)?;
|
||||
let items = if matches!(target, Target::Channel(_)) {
|
||||
Self::parse_channel_json(&html, per_page)?
|
||||
} else {
|
||||
Self::parse_listing(&html, per_page)?
|
||||
};
|
||||
if !items.is_empty() {
|
||||
cache.insert(cache_key, items.clone());
|
||||
}
|
||||
@@ -625,10 +712,15 @@ impl Provider for XvideosProvider {
|
||||
let page = page.parse::<u16>().unwrap_or(1).max(1);
|
||||
let per_page = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
|
||||
|
||||
let target = match query {
|
||||
let mut target = match query {
|
||||
Some(q) if !q.trim().is_empty() => Self::resolve_query_target(q.trim()),
|
||||
_ => Self::resolve_option_target(&options, &sort),
|
||||
};
|
||||
if let Target::Search(query) = &target {
|
||||
if let Some(found) = Self::resolve_keyword(query, &options).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
|
||||
match self
|
||||
.fetch_target(cache, target, page, per_page, options)
|
||||
|
||||
@@ -1,8 +1,11 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{Provider, report_provider_error, requester_or_default};
|
||||
use crate::providers::{
|
||||
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
use crate::util::requester::Requester;
|
||||
use crate::util::parse_abbreviated_number;
|
||||
use crate::util::time::parse_time_to_seconds;
|
||||
use crate::videos::{ServerOptions, VideoItem};
|
||||
@@ -248,6 +251,33 @@ impl YoupornProvider {
|
||||
}
|
||||
}
|
||||
|
||||
/// A free-form query that names a pornstar, channel or amateur goes to that
|
||||
/// archive (probed on the site; unknown names 404), else keyword search.
|
||||
async fn resolve_keyword(
|
||||
&self,
|
||||
requester: &mut Requester,
|
||||
query: &str,
|
||||
sort: &str,
|
||||
) -> Option<Target> {
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let sort = Self::normalized_sort(sort).to_string();
|
||||
let candidates = [
|
||||
Target::Pornstar { slug: slug.clone(), sort: sort.clone() },
|
||||
Target::Channel { slug: slug.clone(), sort: sort.clone() },
|
||||
Target::Amateur { slug, sort },
|
||||
];
|
||||
let urls: Vec<String> = candidates.iter().map(|t| self.build_url(t, 1)).collect();
|
||||
let referer = format!("{}/", self.url.trim_end_matches('/'));
|
||||
let (index, _) = probe_first_archive(requester, &urls, &referer, |body| {
|
||||
body.contains("js_video-box")
|
||||
})
|
||||
.await?;
|
||||
candidates.into_iter().nth(index)
|
||||
}
|
||||
|
||||
fn build_url(&self, target: &Target, page: u8) -> String {
|
||||
match target {
|
||||
Target::Latest { sort } => format!(
|
||||
@@ -543,7 +573,13 @@ impl Provider for YoupornProvider {
|
||||
) -> Vec<VideoItem> {
|
||||
let query = query.unwrap_or_default();
|
||||
let page = page.parse::<u8>().unwrap_or(1);
|
||||
let target = self.target_from_query(&query, &sort);
|
||||
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
|
||||
let mut target = self.target_from_query(&query, &sort);
|
||||
if let Target::Search { query } = &target {
|
||||
if let Some(found) = self.resolve_keyword(&mut requester, query, &sort).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
let video_url = self.build_url(&target, page);
|
||||
|
||||
let old_items = match cache.get(&video_url) {
|
||||
@@ -554,7 +590,6 @@ impl Provider for YoupornProvider {
|
||||
None => vec![],
|
||||
};
|
||||
|
||||
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
|
||||
let referer = format!("{}/", self.url.trim_end_matches('/'));
|
||||
let text = match requester
|
||||
.get_with_headers(&video_url, Self::html_headers(&referer), Some(Version::HTTP_11))
|
||||
|
||||
@@ -229,63 +229,90 @@ impl ArchivebateProxy {
|
||||
embed_url: &str,
|
||||
requester: &mut Requester,
|
||||
) -> Option<String> {
|
||||
let response = requester
|
||||
.get_raw_with_headers_timeout(
|
||||
embed_url,
|
||||
Self::html_headers(detail_url),
|
||||
Some(StdDuration::from_secs(8)),
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.ok()?;
|
||||
if !response.status().is_success() {
|
||||
return None;
|
||||
// The embed host is sometimes slow on a first (cold) request and fast on
|
||||
// the next, so one timed-out attempt is retried.
|
||||
for _ in 0..2 {
|
||||
let Ok(response) = requester
|
||||
.get_raw_with_headers_timeout(
|
||||
embed_url,
|
||||
Self::html_headers(detail_url),
|
||||
Some(StdDuration::from_secs(10)),
|
||||
None,
|
||||
)
|
||||
.await
|
||||
else {
|
||||
continue;
|
||||
};
|
||||
if !response.status().is_success() {
|
||||
return None;
|
||||
}
|
||||
let html = response.text().await.ok()?;
|
||||
return Self::extract_mixdrop_media_url(&html);
|
||||
}
|
||||
let html = response.text().await.ok()?;
|
||||
Self::extract_mixdrop_media_url(&html)
|
||||
None
|
||||
}
|
||||
}
|
||||
|
||||
/// A resolved Mixdrop media file plus the embed page it came from (the CDN is
|
||||
/// played with that page as `Referer`).
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct ResolvedMedia {
|
||||
pub media_url: String,
|
||||
pub embed_url: String,
|
||||
}
|
||||
|
||||
/// Resolves an archivebate watch page (`https://archivebate.com/watch/{id}`)
|
||||
/// to the real media URL: watch page -> Mixdrop embed -> unpacked
|
||||
/// `MDCore.wurl`. Shared by the lazy `/proxy/archivebate` redirect and by the
|
||||
/// provider, which hands the app the media URL directly.
|
||||
pub async fn resolve_media(requester: &mut Requester, detail_url: &str) -> Option<ResolvedMedia> {
|
||||
let detail_html = requester
|
||||
.get_with_headers(
|
||||
detail_url,
|
||||
ArchivebateProxy::html_headers("https://archivebate.com/"),
|
||||
Some(Version::HTTP_11),
|
||||
)
|
||||
.await
|
||||
.unwrap_or_default();
|
||||
if detail_html.is_empty() {
|
||||
return None;
|
||||
}
|
||||
|
||||
let mut embeds: Vec<String> = Vec::new();
|
||||
if let Some(iframe_url) = ArchivebateProxy::first_iframe_source_from_html(&detail_html)
|
||||
.map(|value| ArchivebateProxy::absolute_url(&value))
|
||||
{
|
||||
if ArchivebateProxy::is_mixdrop_host(&iframe_url) {
|
||||
embeds.push(iframe_url);
|
||||
}
|
||||
}
|
||||
if let Some(embed_url) = ArchivebateProxy::download_fid_from_detail_html(&detail_html)
|
||||
.map(|value| ArchivebateProxy::absolute_url(&value))
|
||||
.and_then(|value| ArchivebateProxy::mixdrop_embed_url_from_download_url(&value))
|
||||
{
|
||||
embeds.push(embed_url);
|
||||
}
|
||||
|
||||
for embed_url in embeds {
|
||||
if let Some(media_url) =
|
||||
ArchivebateProxy::resolve_mixdrop_media_from_embed(detail_url, &embed_url, requester).await
|
||||
{
|
||||
return Some(ResolvedMedia { media_url, embed_url });
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
impl crate::proxies::Proxy for ArchivebateProxy {
|
||||
async fn get_video_url(&self, url: String, requester: web::types::State<Requester>) -> String {
|
||||
let Some(detail_url) = Self::normalize_detail_request(&url) else {
|
||||
return String::new();
|
||||
};
|
||||
|
||||
let mut requester = requester.get_ref().clone();
|
||||
let detail_html = requester
|
||||
.get_with_headers(
|
||||
&detail_url,
|
||||
Self::html_headers("https://archivebate.com/"),
|
||||
Some(Version::HTTP_11),
|
||||
)
|
||||
resolve_media(&mut requester, &detail_url)
|
||||
.await
|
||||
.unwrap_or_default();
|
||||
if detail_html.is_empty() {
|
||||
return String::new();
|
||||
}
|
||||
|
||||
if let Some(iframe_url) = Self::first_iframe_source_from_html(&detail_html).map(|value| Self::absolute_url(&value)) {
|
||||
if Self::is_mixdrop_host(&iframe_url) {
|
||||
if let Some(media_url) =
|
||||
Self::resolve_mixdrop_media_from_embed(&detail_url, &iframe_url, &mut requester).await
|
||||
{
|
||||
return media_url;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(download_fid) = Self::download_fid_from_detail_html(&detail_html).map(|value| Self::absolute_url(&value)) {
|
||||
if let Some(embed_url) = Self::mixdrop_embed_url_from_download_url(&download_fid) {
|
||||
if let Some(media_url) =
|
||||
Self::resolve_mixdrop_media_from_embed(&detail_url, &embed_url, &mut requester).await
|
||||
{
|
||||
return media_url;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
String::new()
|
||||
.map(|resolved| resolved.media_url)
|
||||
.unwrap_or_default()
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
196
src/proxies/eporner.rs
Normal file
196
src/proxies/eporner.rs
Normal file
@@ -0,0 +1,196 @@
|
||||
use std::collections::HashMap;
|
||||
use std::rc::Rc;
|
||||
use std::sync::{Mutex, OnceLock};
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use futures::StreamExt;
|
||||
use http_body_util::BodyExt;
|
||||
use ntex::http::Method;
|
||||
use ntex::http::body::{Body, SizedStream};
|
||||
use ntex::http::header::{ACCEPT_RANGES, CONTENT_LENGTH, CONTENT_RANGE, CONTENT_TYPE, RANGE};
|
||||
use ntex::util::Bytes;
|
||||
use ntex::web::{self, HttpRequest};
|
||||
use wreq::Client;
|
||||
use wreq::redirect::Policy;
|
||||
use wreq_util::Emulation;
|
||||
|
||||
use crate::util::eporner_resolve::resolve_ranked;
|
||||
use crate::util::requester::Requester;
|
||||
|
||||
/// The signed CDN links expire two hours after resolving; renew well before.
|
||||
const LINK_TTL: Duration = Duration::from_secs(90 * 60);
|
||||
const LINK_CACHE_SIZE: usize = 2000;
|
||||
|
||||
/// `{alnum_id}/{height}` -> (resolved at, signed CDN URL).
|
||||
fn links() -> &'static Mutex<HashMap<String, (Instant, String)>> {
|
||||
static LINKS: OnceLock<Mutex<HashMap<String, (Instant, String)>>> = OnceLock::new();
|
||||
LINKS.get_or_init(|| Mutex::new(HashMap::new()))
|
||||
}
|
||||
|
||||
/// Redirects are not followed so the CDN's `na.mp4` decoy shows up as a 302.
|
||||
fn cdn_client() -> &'static Client {
|
||||
static CLIENT: OnceLock<Client> = OnceLock::new();
|
||||
CLIENT.get_or_init(|| {
|
||||
Client::builder()
|
||||
.tls_cert_verification(false)
|
||||
.emulation(Emulation::Firefox151)
|
||||
.redirect(Policy::none())
|
||||
.build()
|
||||
.expect("eporner CDN client")
|
||||
})
|
||||
}
|
||||
|
||||
/// `{alnum_id}[/{height}][.mp4]`.
|
||||
fn parse_endpoint(endpoint: &str) -> Option<(String, Option<u32>)> {
|
||||
let endpoint = endpoint.trim_matches('/').trim_end_matches(".mp4");
|
||||
let mut parts = endpoint.split('/');
|
||||
let alnum_id = parts
|
||||
.next()
|
||||
.filter(|id| !id.is_empty() && id.chars().all(|c| c.is_ascii_alphanumeric()))?;
|
||||
let height = parts.next().and_then(|h| h.parse::<u32>().ok());
|
||||
Some((alnum_id.to_string(), height))
|
||||
}
|
||||
|
||||
async fn resolve_link(
|
||||
requester: &web::types::State<Requester>,
|
||||
alnum_id: &str,
|
||||
height: Option<u32>,
|
||||
) -> Option<String> {
|
||||
let page_url = format!("https://www.eporner.com/video-{alnum_id}/");
|
||||
let mut requester = requester.get_ref().clone();
|
||||
let ranked = resolve_ranked(&mut requester, alnum_id, &page_url).await?;
|
||||
height
|
||||
.and_then(|h| ranked.iter().find(|(height, _)| *height == h))
|
||||
.or_else(|| ranked.first())
|
||||
.map(|(_, format)| format.url.clone())
|
||||
}
|
||||
|
||||
async fn cached_link(
|
||||
requester: &web::types::State<Requester>,
|
||||
alnum_id: &str,
|
||||
height: Option<u32>,
|
||||
fresh: bool,
|
||||
) -> Option<String> {
|
||||
let key = format!("{alnum_id}/{}", height.unwrap_or(0));
|
||||
if !fresh {
|
||||
if let Ok(links) = links().lock() {
|
||||
if let Some((time, url)) = links.get(&key) {
|
||||
if time.elapsed() < LINK_TTL {
|
||||
return Some(url.clone());
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
let url = resolve_link(requester, alnum_id, height).await?;
|
||||
if let Ok(mut links) = links().lock() {
|
||||
if links.len() >= LINK_CACHE_SIZE {
|
||||
links.retain(|_, (time, _)| time.elapsed() < LINK_TTL);
|
||||
if links.len() >= LINK_CACHE_SIZE {
|
||||
links.clear();
|
||||
}
|
||||
}
|
||||
links.insert(key, (Instant::now(), url.clone()));
|
||||
}
|
||||
Some(url)
|
||||
}
|
||||
|
||||
/// Streams the mp4 through this server. The signed CDN links only serve the
|
||||
/// IP that resolved them (checked from other networks: they answer every
|
||||
/// other client with a 302 to `static.eporner.com/na.mp4`, a one-minute
|
||||
/// "watch it on eporner.com" clip), so a redirect to them only ever works for
|
||||
/// a player that shares the server's public IP.
|
||||
pub async fn serve_media(
|
||||
req: HttpRequest,
|
||||
requester: web::types::State<Requester>,
|
||||
) -> Result<impl web::Responder, web::Error> {
|
||||
let endpoint = req.match_info().query("endpoint").to_string();
|
||||
let Some((alnum_id, height)) = parse_endpoint(&endpoint) else {
|
||||
return Ok(web::HttpResponse::BadRequest().finish());
|
||||
};
|
||||
let range = req
|
||||
.headers()
|
||||
.get(RANGE)
|
||||
.and_then(|value| value.to_str().ok())
|
||||
.map(str::to_string);
|
||||
let head = req.method() == Method::HEAD;
|
||||
|
||||
let mut upstream = None;
|
||||
for fresh in [false, true] {
|
||||
let Some(url) = cached_link(&requester, &alnum_id, height, fresh).await else {
|
||||
break;
|
||||
};
|
||||
let mut request = if head {
|
||||
cdn_client().head(&url)
|
||||
} else {
|
||||
cdn_client().get(&url)
|
||||
};
|
||||
request = request.header("Referer", "https://www.eporner.com/");
|
||||
if let Some(range) = &range {
|
||||
request = request.header("Range", range);
|
||||
}
|
||||
match request.send().await {
|
||||
// A redirect is the decoy for an expired or foreign link.
|
||||
Ok(response) if !response.status().is_redirection() => {
|
||||
upstream = Some(response);
|
||||
break;
|
||||
}
|
||||
_ => continue,
|
||||
}
|
||||
}
|
||||
let Some(upstream) = upstream else {
|
||||
return Ok(web::HttpResponse::BadGateway().finish());
|
||||
};
|
||||
|
||||
let status = upstream.status();
|
||||
let headers = upstream.headers().clone();
|
||||
let mut builder = web::HttpResponse::build(status);
|
||||
builder.header(
|
||||
CONTENT_TYPE,
|
||||
headers
|
||||
.get(CONTENT_TYPE)
|
||||
.and_then(|value| value.to_str().ok())
|
||||
.unwrap_or("video/mp4"),
|
||||
);
|
||||
builder.header(ACCEPT_RANGES, "bytes");
|
||||
if let Some(value) = headers.get(CONTENT_RANGE.as_str()).and_then(|v| v.to_str().ok()) {
|
||||
builder.header(CONTENT_RANGE, value);
|
||||
}
|
||||
if head {
|
||||
// `finish()` would send `Content-Length: 0` over the upstream length.
|
||||
return Ok(builder.body(Body::None));
|
||||
}
|
||||
let length = headers
|
||||
.get(CONTENT_LENGTH.as_str())
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.and_then(|v| v.parse::<u64>().ok());
|
||||
let stream = Box::pin(upstream.into_data_stream().map(|chunk| {
|
||||
chunk
|
||||
.map(|bytes| Bytes::copy_from_slice(bytes.as_ref()))
|
||||
.map_err(|e| std::io::Error::other(e.to_string()))
|
||||
}));
|
||||
let Some(length) = length else {
|
||||
return Ok(builder.streaming(stream));
|
||||
};
|
||||
// A sized body keeps `Content-Length` on the (206) responses; a chunked
|
||||
// one is what AVPlayer refuses to seek in.
|
||||
let stream = stream.map(|chunk| chunk.map_err(|e| Rc::new(e) as Rc<dyn std::error::Error>));
|
||||
Ok(builder.body(Body::from_message(SizedStream::new(length, stream))))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::parse_endpoint;
|
||||
|
||||
#[test]
|
||||
fn parses_id_and_optional_height() {
|
||||
assert_eq!(
|
||||
parse_endpoint("exfeTXHuL1x/720.mp4"),
|
||||
Some(("exfeTXHuL1x".to_string(), Some(720)))
|
||||
);
|
||||
assert_eq!(
|
||||
parse_endpoint("/exfeTXHuL1x/"),
|
||||
Some(("exfeTXHuL1x".to_string(), None))
|
||||
);
|
||||
assert_eq!(parse_endpoint("../etc"), None);
|
||||
}
|
||||
}
|
||||
71
src/proxies/hanime.rs
Normal file
71
src/proxies/hanime.rs
Normal file
@@ -0,0 +1,71 @@
|
||||
use ntex::http::Method;
|
||||
use ntex::http::header::LOCATION;
|
||||
use ntex::web::{self, HttpRequest};
|
||||
|
||||
use crate::util::hanime_resolve::{fetch_sources, pick_source, source_url};
|
||||
use crate::util::requester::Requester;
|
||||
|
||||
/// `{slug}[/{height}][.m3u8]`.
|
||||
fn parse_endpoint(endpoint: &str) -> Option<(String, Option<u32>)> {
|
||||
let endpoint = endpoint.trim_matches('/').trim_end_matches(".m3u8");
|
||||
let mut parts = endpoint.split('/');
|
||||
let slug = parts
|
||||
.next()
|
||||
.filter(|slug| {
|
||||
!slug.is_empty()
|
||||
&& slug
|
||||
.chars()
|
||||
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_')
|
||||
})?;
|
||||
let height = parts.next().and_then(|h| h.parse::<u32>().ok());
|
||||
Some((slug.to_string(), height))
|
||||
}
|
||||
|
||||
/// hanime's stream links expire, so the channel hands out this stable URL and a
|
||||
/// fresh handshake happens on every play: redirects to the current playlist.
|
||||
pub async fn redirect_to_media(
|
||||
req: HttpRequest,
|
||||
requester: web::types::State<Requester>,
|
||||
) -> Result<impl web::Responder, web::Error> {
|
||||
let endpoint = req.match_info().query("endpoint").to_string();
|
||||
let Some((slug, height)) = parse_endpoint(&endpoint) else {
|
||||
return Ok(web::HttpResponse::BadRequest().finish());
|
||||
};
|
||||
if req.method() == Method::HEAD {
|
||||
return Ok(web::HttpResponse::Ok().finish());
|
||||
}
|
||||
|
||||
let mut requester = requester.get_ref().clone();
|
||||
let Ok(sources) = fetch_sources(&mut requester, &slug).await else {
|
||||
return Ok(web::HttpResponse::BadGateway().finish());
|
||||
};
|
||||
let Some(source) = pick_source(&sources, height) else {
|
||||
return Ok(web::HttpResponse::NotFound().finish());
|
||||
};
|
||||
Ok(web::HttpResponse::Found()
|
||||
.header(LOCATION, source_url(source))
|
||||
.finish())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::parse_endpoint;
|
||||
|
||||
#[test]
|
||||
fn parses_slug_and_height() {
|
||||
assert_eq!(
|
||||
parse_endpoint("natsu-zuma-2/720.m3u8"),
|
||||
Some(("natsu-zuma-2".to_string(), Some(720)))
|
||||
);
|
||||
assert_eq!(
|
||||
parse_endpoint("/natsu-zuma-2"),
|
||||
Some(("natsu-zuma-2".to_string(), None))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_unsafe_slugs() {
|
||||
assert_eq!(parse_endpoint("../etc/720.m3u8"), None);
|
||||
assert_eq!(parse_endpoint(""), None);
|
||||
}
|
||||
}
|
||||
@@ -24,7 +24,9 @@ pub mod animeidhentai;
|
||||
pub mod archivebate;
|
||||
pub mod clapdat;
|
||||
pub mod doodstream;
|
||||
pub mod eporner;
|
||||
pub mod fikfapthumb;
|
||||
pub mod hanime;
|
||||
pub mod hanimecdn;
|
||||
pub mod hanimethumb;
|
||||
pub mod heavyfetish;
|
||||
|
||||
80
src/proxy.rs
80
src/proxy.rs
@@ -20,6 +20,40 @@ use crate::proxies::thaiporntv::ThaipornTvProxy;
|
||||
use crate::proxies::*;
|
||||
use crate::util::requester::Requester;
|
||||
|
||||
/// Proxies that answer with a `302` to the real media instead of streaming it.
|
||||
/// The Hot Tub app cannot follow a redirect inside `formats[].url`, so for app
|
||||
/// clients a format pointing at one of these is moved to `video.url` (see
|
||||
/// `move_redirect_formats_to_url`). Keep in sync with the `proxy2redirect`
|
||||
/// routes below; `redirect_proxy_list_matches_routes` enforces it.
|
||||
pub const REDIRECT_PROXIES: &[&str] = &[
|
||||
"archivebate",
|
||||
"clapdat",
|
||||
"doodstream",
|
||||
"lulustream",
|
||||
"sxyprn",
|
||||
"javtiful",
|
||||
"spankbang",
|
||||
"porndish",
|
||||
"hqporner",
|
||||
"heavyfetish",
|
||||
"vjav",
|
||||
"pornhd3x",
|
||||
"vidara",
|
||||
"pimpbunny",
|
||||
"thaiporntv",
|
||||
"tube8",
|
||||
"supjav",
|
||||
];
|
||||
|
||||
/// True for a `/proxy/<redirect proxy>/...` URL (absolute or host-relative).
|
||||
pub fn is_redirect_proxy_url(url: &str) -> bool {
|
||||
let Some(rest) = url.split_once("/proxy/").map(|(_, rest)| rest) else {
|
||||
return false;
|
||||
};
|
||||
let name = rest.split('/').next().unwrap_or_default();
|
||||
REDIRECT_PROXIES.contains(&name)
|
||||
}
|
||||
|
||||
pub fn config(cfg: &mut web::ServiceConfig) {
|
||||
cfg.service(
|
||||
web::resource("/archivebate/{endpoint}*")
|
||||
@@ -123,6 +157,11 @@ pub fn config(cfg: &mut web::ServiceConfig) {
|
||||
.route(web::post().to(crate::proxies::fikfapthumb::get_image))
|
||||
.route(web::get().to(crate::proxies::fikfapthumb::get_image)),
|
||||
)
|
||||
.service(
|
||||
web::resource("/hanime/{endpoint}*")
|
||||
.route(web::get().to(crate::proxies::hanime::redirect_to_media))
|
||||
.route(web::head().to(crate::proxies::hanime::redirect_to_media)),
|
||||
)
|
||||
.service(
|
||||
web::resource("/hanime-cdn/{endpoint}*")
|
||||
.route(web::post().to(crate::proxies::hanimecdn::get_image))
|
||||
@@ -194,6 +233,11 @@ pub fn config(cfg: &mut web::ServiceConfig) {
|
||||
.route(web::post().to(crate::proxies::melonstube::serve))
|
||||
.route(web::get().to(crate::proxies::melonstube::serve)),
|
||||
);
|
||||
cfg.service(
|
||||
web::resource("/eporner/{endpoint}*")
|
||||
.route(web::get().to(crate::proxies::eporner::serve_media))
|
||||
.route(web::head().to(crate::proxies::eporner::serve_media)),
|
||||
);
|
||||
cfg.service(
|
||||
web::resource("/xiaoshenke/{endpoint}*")
|
||||
.route(web::post().to(crate::proxies::xiaoshenke::serve_media))
|
||||
@@ -239,3 +283,39 @@ fn get_proxy(proxy: &str) -> Option<AnyProxy> {
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod redirect_proxy_tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn recognises_redirect_proxy_urls() {
|
||||
assert!(is_redirect_proxy_url("http://h/proxy/archivebate/archivebate.com/watch/1"));
|
||||
assert!(is_redirect_proxy_url("/proxy/hqporner/x"));
|
||||
assert!(!is_redirect_proxy_url("http://h/proxy/noodlemagazine/x"));
|
||||
assert!(!is_redirect_proxy_url("https://cdn.example/video.mp4"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn redirect_proxy_list_matches_routes() {
|
||||
let source = include_str!("proxy.rs");
|
||||
let config_body = source
|
||||
.split("pub fn config(")
|
||||
.nth(1)
|
||||
.and_then(|rest| rest.split("async fn proxy2redirect(").next())
|
||||
.expect("config body");
|
||||
let mut from_routes: Vec<String> = Vec::new();
|
||||
for block in config_body.split("web::resource(\"/").skip(1) {
|
||||
let name = block.split("/{endpoint}").next().unwrap_or_default();
|
||||
let until_next = block.split("web::resource(").next().unwrap_or(block);
|
||||
if until_next.contains("proxy2redirect") {
|
||||
from_routes.push(name.to_string());
|
||||
}
|
||||
}
|
||||
let mut listed: Vec<String> = REDIRECT_PROXIES.iter().map(|s| s.to_string()).collect();
|
||||
from_routes.sort();
|
||||
from_routes.dedup();
|
||||
listed.sort();
|
||||
assert_eq!(listed, from_routes);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
//! always compiled regardless of provider selection.
|
||||
|
||||
use crate::util::requester::Requester;
|
||||
use crate::videos::VideoFormat;
|
||||
use crate::videos::{ServerOptions, VideoFormat};
|
||||
use regex::Regex;
|
||||
use std::sync::OnceLock;
|
||||
use wreq::Version;
|
||||
@@ -101,11 +101,18 @@ fn calc_hash(hash: &str) -> Option<String> {
|
||||
/// when playback doesn't carry the header the CDN wants. Each format
|
||||
/// therefore declares an explicit `Referer` so the client's own default (or
|
||||
/// lack thereof) can't accidentally trip this check.
|
||||
pub async fn resolve_formats(
|
||||
///
|
||||
/// The signed CDN URLs also expire two hours after this call (the leading
|
||||
/// number in `/{expiry}_{ip}_{n}/` is that deadline) and only serve the IP
|
||||
/// embedded there; anything else gets the same `na.mp4` decoy.
|
||||
/// Anything handed to a client should therefore be a
|
||||
/// [`resolve_proxied_formats`] URL, streamed through this server.
|
||||
/// Returns `(height, format)`, best first.
|
||||
pub async fn resolve_ranked(
|
||||
requester: &mut Requester,
|
||||
alnum_id: &str,
|
||||
page_url: &str,
|
||||
) -> Option<Vec<VideoFormat>> {
|
||||
) -> Option<Vec<(u32, VideoFormat)>> {
|
||||
let detail_html = requester
|
||||
.get_with_headers(page_url, html_headers(page_url), Some(Version::HTTP_11))
|
||||
.await
|
||||
@@ -151,25 +158,56 @@ pub async fn resolve_formats(
|
||||
return None;
|
||||
}
|
||||
ranked.sort_by(|a, b| b.0.cmp(&a.0));
|
||||
Some(ranked.into_iter().map(|(_, f)| f).collect())
|
||||
Some(ranked)
|
||||
}
|
||||
|
||||
/// Resolves eporner.com direct mp4 formats for an arbitrary eporner detail
|
||||
/// page URL, extracting the alnum id from the URL itself. For use by
|
||||
/// providers that only have a raw destination URL (e.g. a meta-search
|
||||
/// aggregator's out-link), not a pre-parsed listing card.
|
||||
pub async fn resolve_public_formats(
|
||||
/// Resolves the qualities a video offers, but points each one at the
|
||||
/// `/proxy/eporner/{alnum_id}/{height}.mp4` streaming proxy: the CDN URLs
|
||||
/// expire after two hours and only serve the IP that resolved them.
|
||||
pub async fn resolve_proxied_formats(
|
||||
requester: &mut Requester,
|
||||
options: &ServerOptions,
|
||||
alnum_id: &str,
|
||||
page_url: &str,
|
||||
) -> Option<Vec<VideoFormat>> {
|
||||
let alnum_id = alnum_id_from_href(page_url)?;
|
||||
resolve_formats(requester, &alnum_id, page_url).await
|
||||
let ranked = resolve_ranked(requester, alnum_id, page_url).await?;
|
||||
let base = options.public_url_base.as_deref().unwrap_or("");
|
||||
Some(proxied_formats(base, alnum_id, ranked))
|
||||
}
|
||||
|
||||
/// Builds the URLs like `providers::build_proxy_url`, which this always
|
||||
/// compiled module cannot rely on in single-provider builds.
|
||||
fn proxied_formats(
|
||||
public_url_base: &str,
|
||||
alnum_id: &str,
|
||||
ranked: Vec<(u32, VideoFormat)>,
|
||||
) -> Vec<VideoFormat> {
|
||||
let base = public_url_base.trim_end_matches('/');
|
||||
ranked
|
||||
.into_iter()
|
||||
.map(|(height, mut format)| {
|
||||
format.url = format!("{base}/proxy/eporner/{alnum_id}/{height}.mp4");
|
||||
format
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn proxied_formats_point_at_the_streaming_proxy() {
|
||||
let ranked = vec![(
|
||||
720,
|
||||
VideoFormat::new("https://vid.cdn/x-720p.mp4".into(), "720p".into(), "mp4".into()),
|
||||
)];
|
||||
let formats = proxied_formats("https://hottub.example/", "exfeTXHuL1x", ranked);
|
||||
assert_eq!(formats[0].url, "https://hottub.example/proxy/eporner/exfeTXHuL1x/720.mp4");
|
||||
// Streamed, not redirected: app clients must keep it in `formats`.
|
||||
assert!(!crate::proxy::is_redirect_proxy_url(&formats[0].url));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn calc_hash_matches_site_algorithm() {
|
||||
// Reverse-engineered from vjs.js; verified against a live page/xhr pair.
|
||||
|
||||
208
src/util/hanime_resolve.rs
Normal file
208
src/util/hanime_resolve.rs
Normal file
@@ -0,0 +1,208 @@
|
||||
//! hanime.tv stream resolution, shared by the `hanime` channel and its
|
||||
//! `/proxy/hanime/` redirect proxy.
|
||||
//!
|
||||
//! Lives here (always compiled) rather than in `providers/hanime.rs` because
|
||||
//! `HOT_TUB_PROVIDER=<id>` single-provider builds only compile the selected
|
||||
//! provider, while the proxy routes are always registered.
|
||||
|
||||
use aes_gcm::aead::{Aead, KeyInit, Payload};
|
||||
use aes_gcm::{Aes256Gcm, Key, Nonce};
|
||||
use base64::Engine;
|
||||
use serde_json::json;
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
|
||||
use crate::util::requester::Requester;
|
||||
|
||||
const HANDSHAKE_URL: &str = "https://auth.hanime.tv/api/v11/handshake";
|
||||
pub const SITE_ORIGIN: &str = "https://hanime.tv";
|
||||
/// Envelope encryption used for the handshake request/response bodies.
|
||||
const HANDSHAKE_KEY_SEED: &[u8] = b"htv-insecure-handshake-v1";
|
||||
const HANDSHAKE_AAD: &[u8] = b"htv-insecure-v1";
|
||||
/// Pieces of the `x-signature` pre-image, as baked into the site's wasm module.
|
||||
const SIGNATURE_SECRET: &str = "Xkdi29";
|
||||
const SIGNATURE_SALT: &str = "mn2";
|
||||
|
||||
type Result<T> = std::result::Result<T, String>;
|
||||
|
||||
fn err(message: impl Into<String>) -> String {
|
||||
message.into()
|
||||
}
|
||||
|
||||
#[derive(serde::Deserialize, Debug)]
|
||||
pub struct HanimeHandshakePayload {
|
||||
#[serde(default)]
|
||||
pub sources: Vec<HanimeSource>,
|
||||
}
|
||||
|
||||
#[derive(serde::Deserialize, Debug, Clone)]
|
||||
pub struct HanimeSource {
|
||||
#[serde(default)]
|
||||
pub src: String,
|
||||
#[serde(default)]
|
||||
pub height: u32,
|
||||
#[serde(default)]
|
||||
pub label: String,
|
||||
/// `"normal"` for the free streams, `"promotion"` for the premium-only teaser
|
||||
/// entries which always carry an empty `src`.
|
||||
#[serde(default)]
|
||||
pub kind: String,
|
||||
}
|
||||
|
||||
fn b64() -> base64::engine::general_purpose::GeneralPurpose {
|
||||
base64::engine::general_purpose::URL_SAFE_NO_PAD
|
||||
}
|
||||
|
||||
fn b64_decode(value: &str) -> Result<Vec<u8>> {
|
||||
b64()
|
||||
.decode(value.trim_end_matches('='))
|
||||
.map_err(|e| err(format!("base64url decode failed: {e}")))
|
||||
}
|
||||
|
||||
fn handshake_cipher() -> Aes256Gcm {
|
||||
let digest = Sha256::digest(HANDSHAKE_KEY_SEED);
|
||||
let key = Key::<Aes256Gcm>::from_slice(digest.as_slice());
|
||||
Aes256Gcm::new(key)
|
||||
}
|
||||
|
||||
/// `{"v":1,"alg":"AES-256-GCM","iv":..,"tag":..,"data":..}`, base64url encoded.
|
||||
fn seal_envelope(plaintext: &[u8]) -> Result<String> {
|
||||
let mut iv = [0u8; 12];
|
||||
rand::fill(&mut iv);
|
||||
let sealed = handshake_cipher()
|
||||
.encrypt(
|
||||
Nonce::from_slice(&iv),
|
||||
Payload {
|
||||
msg: plaintext,
|
||||
aad: HANDSHAKE_AAD,
|
||||
},
|
||||
)
|
||||
.map_err(|e| err(format!("handshake encrypt failed: {e}")))?;
|
||||
if sealed.len() < 16 {
|
||||
return Err(err("handshake ciphertext too short"));
|
||||
}
|
||||
let (data, tag) = sealed.split_at(sealed.len() - 16);
|
||||
let envelope = json!({
|
||||
"v": 1,
|
||||
"alg": "AES-256-GCM",
|
||||
"iv": b64().encode(iv),
|
||||
"tag": b64().encode(tag),
|
||||
"data": b64().encode(data),
|
||||
});
|
||||
Ok(b64().encode(envelope.to_string().as_bytes()))
|
||||
}
|
||||
|
||||
fn open_envelope(token: &str) -> Result<Vec<u8>> {
|
||||
let envelope: serde_json::Value = serde_json::from_slice(&b64_decode(token)?)
|
||||
.map_err(|e| err(format!("handshake envelope is not JSON: {e}")))?;
|
||||
let field = |name: &str| -> Result<Vec<u8>> {
|
||||
let raw = envelope
|
||||
.get(name)
|
||||
.and_then(|v| v.as_str())
|
||||
.ok_or_else(|| err(format!("handshake envelope missing {name}")))?;
|
||||
b64_decode(raw)
|
||||
};
|
||||
let iv = field("iv")?;
|
||||
let mut ciphertext = field("data")?;
|
||||
ciphertext.extend_from_slice(&field("tag")?);
|
||||
handshake_cipher()
|
||||
.decrypt(
|
||||
Nonce::from_slice(&iv),
|
||||
Payload {
|
||||
msg: &ciphertext,
|
||||
aad: HANDSHAKE_AAD,
|
||||
},
|
||||
)
|
||||
.map_err(|e| err(format!("handshake decrypt failed: {e}")))
|
||||
}
|
||||
|
||||
fn signature_for(timestamp: u64) -> String {
|
||||
let pre_image =
|
||||
format!("{timestamp},{SIGNATURE_SECRET},{SITE_ORIGIN},{SIGNATURE_SALT},{timestamp}");
|
||||
hex::encode(Sha256::digest(pre_image.as_bytes()))
|
||||
}
|
||||
|
||||
/// Returns the free `sources` for a slug, best quality first. The auth host
|
||||
/// occasionally refuses one of a burst of parallel connections, so retry once.
|
||||
pub async fn fetch_sources(requester: &mut Requester, slug: &str) -> Result<Vec<HanimeSource>> {
|
||||
match handshake(requester, slug).await {
|
||||
Ok(sources) => Ok(sources),
|
||||
Err(_) => {
|
||||
ntex::time::sleep(ntex::time::Millis(500)).await;
|
||||
handshake(requester, slug).await
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn handshake(requester: &mut Requester, slug: &str) -> Result<Vec<HanimeSource>> {
|
||||
let timestamp = SystemTime::now()
|
||||
.duration_since(UNIX_EPOCH)
|
||||
.unwrap_or_default()
|
||||
.as_secs();
|
||||
let token = seal_envelope(
|
||||
json!({
|
||||
"timestamp_unix": timestamp,
|
||||
"directive": "htv_player_handshake",
|
||||
"slug": slug,
|
||||
})
|
||||
.to_string()
|
||||
.as_bytes(),
|
||||
)?;
|
||||
|
||||
let response = requester
|
||||
.post_json(
|
||||
HANDSHAKE_URL,
|
||||
&json!({ "token": token }),
|
||||
vec![
|
||||
("Origin".to_string(), SITE_ORIGIN.to_string()),
|
||||
("Referer".to_string(), format!("{SITE_ORIGIN}/")),
|
||||
("Accept".to_string(), "application/json".to_string()),
|
||||
("x-signature-version".to_string(), "web2".to_string()),
|
||||
("x-signature".to_string(), signature_for(timestamp)),
|
||||
("x-time".to_string(), timestamp.to_string()),
|
||||
],
|
||||
)
|
||||
.await
|
||||
.map_err(|e| err(format!("handshake request failed: {e}")))?;
|
||||
|
||||
let status = response.status();
|
||||
if !status.is_success() {
|
||||
return Err(err(format!("handshake returned HTTP {status}")));
|
||||
}
|
||||
|
||||
// The interesting part of the answer travels in a response header, not the body.
|
||||
let encrypted = response
|
||||
.headers()
|
||||
.get("x-token")
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.map(|v| v.to_string())
|
||||
.ok_or_else(|| err("handshake response is missing x-token"))?;
|
||||
|
||||
let payload: HanimeHandshakePayload = serde_json::from_slice(&open_envelope(&encrypted)?)
|
||||
.map_err(|e| err(format!("Failed to parse handshake payload: {e}")))?;
|
||||
|
||||
let mut sources: Vec<HanimeSource> = payload
|
||||
.sources
|
||||
.into_iter()
|
||||
.filter(|s| s.kind == "normal" && !s.src.is_empty())
|
||||
.collect();
|
||||
sources.sort_by(|a, b| b.height.cmp(&a.height));
|
||||
match sources.is_empty() {
|
||||
true => Err(err("handshake returned no playable sources")),
|
||||
false => Ok(sources),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn source_url(source: &HanimeSource) -> String {
|
||||
match source.src.starts_with("http") {
|
||||
true => source.src.clone(),
|
||||
false => format!("{SITE_ORIGIN}{}", source.src),
|
||||
}
|
||||
}
|
||||
|
||||
/// The source of the requested height, else the best one.
|
||||
pub fn pick_source(sources: &[HanimeSource], height: Option<u32>) -> Option<&HanimeSource> {
|
||||
height
|
||||
.and_then(|h| sources.iter().find(|s| s.height == h))
|
||||
.or_else(|| sources.first())
|
||||
}
|
||||
@@ -6,6 +6,7 @@ pub mod discord;
|
||||
pub mod eporner_resolve;
|
||||
pub mod flaresolverr;
|
||||
pub mod flow_debug;
|
||||
pub mod hanime_resolve;
|
||||
pub mod hoster_proxy;
|
||||
pub mod proxy;
|
||||
pub mod requester;
|
||||
|
||||
@@ -222,6 +222,15 @@ impl Requester {
|
||||
body.contains("Just a moment...")
|
||||
|| body.contains("cf-turnstile-response")
|
||||
|| body.contains("Enable JavaScript and cookies to continue")
|
||||
|| Self::looks_like_ddos_guard_challenge(body)
|
||||
}
|
||||
|
||||
/// DDoS-Guard interstitials (rule34video) are short JS-check pages, never real content.
|
||||
fn looks_like_ddos_guard_challenge(body: &str) -> bool {
|
||||
body.len() < 20_000
|
||||
&& (body.contains("/.well-known/ddos-guard/")
|
||||
|| body.contains("check.ddos-guard.net")
|
||||
|| body.contains("DDoS-Guard</title>"))
|
||||
}
|
||||
|
||||
fn jina_mirror_url(url: &str) -> Option<String> {
|
||||
@@ -661,6 +670,74 @@ impl Requester {
|
||||
Ok(response)
|
||||
}
|
||||
|
||||
fn path_of(url: &str) -> Option<String> {
|
||||
url::Url::parse(url).ok().map(|u| u.path().to_string())
|
||||
}
|
||||
|
||||
/// `Url` percent-encodes a non-ASCII path ("/tags/pok%C3%A9mon/") while the
|
||||
/// response URI keeps it raw ("/tags/pokémon/"): compare them decoded.
|
||||
fn same_path(asked: &str, landed: &str) -> bool {
|
||||
let normalize = |path: &str| {
|
||||
percent_encoding::percent_decode_str(path)
|
||||
.decode_utf8_lossy()
|
||||
.trim_end_matches('/')
|
||||
.to_lowercase()
|
||||
};
|
||||
normalize(asked) == normalize(landed)
|
||||
}
|
||||
|
||||
/// Existence probe for archive pages (pornstar / studio / tag / category).
|
||||
///
|
||||
/// Reads the raw response so a real 404/410 answers immediately with
|
||||
/// `Ok(None)` instead of escalating through the Jina/FlareSolverr chain
|
||||
/// (minutes per miss). A Cloudflare challenge (403/503/"Just a moment...")
|
||||
/// is handed to the full chain so FlareSolverr solves it, and 429/5xx are
|
||||
/// retried once.
|
||||
pub async fn get_probe(
|
||||
&mut self,
|
||||
url: &str,
|
||||
headers: Vec<(String, String)>,
|
||||
) -> Result<Option<String>, AnyErr> {
|
||||
let mut last_err = String::new();
|
||||
for attempt in 0..2 {
|
||||
if attempt > 0 {
|
||||
// An immediate retry of a 429 is all but certain to hit it again.
|
||||
tokio::time::sleep(std::time::Duration::from_millis(1500)).await;
|
||||
}
|
||||
match self.get_raw_with_headers(url, headers.clone()).await {
|
||||
Ok(response) => {
|
||||
let status = response.status().as_u16();
|
||||
// An unknown archive often redirects to the home page or a
|
||||
// search page instead of answering 404: a redirect that
|
||||
// lands on a different path is a miss.
|
||||
let redirected_away = Self::path_of(url)
|
||||
.zip(Some(response.uri().path().to_string()))
|
||||
.is_some_and(|(asked, landed)| !Self::same_path(&asked, &landed));
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
if status == 404 || status == 410 {
|
||||
return Ok(None);
|
||||
}
|
||||
if status == 403
|
||||
|| status == 503
|
||||
|| Self::looks_like_cloudflare_challenge(&body)
|
||||
{
|
||||
return self.get_with_headers(url, headers, None).await.map(Some);
|
||||
}
|
||||
if redirected_away {
|
||||
return Ok(None);
|
||||
}
|
||||
if status >= 500 || status == 429 {
|
||||
last_err = format!("HTTP {status}");
|
||||
continue;
|
||||
}
|
||||
return Ok(Some(body));
|
||||
}
|
||||
Err(e) => last_err = e.to_string(),
|
||||
}
|
||||
}
|
||||
Err(format!("probe failed for {url}: {last_err}").into())
|
||||
}
|
||||
|
||||
pub async fn get(
|
||||
&mut self,
|
||||
url: &str,
|
||||
@@ -830,7 +907,9 @@ impl Requester {
|
||||
rate_limit_retries,
|
||||
MAX_RATE_LIMIT_RETRIES
|
||||
);
|
||||
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
|
||||
// 1s, 2s, 4s: a site-wide limit needs longer than a second to lift.
|
||||
tokio::time::sleep(std::time::Duration::from_secs(1 << (rate_limit_retries - 1)))
|
||||
.await;
|
||||
continue;
|
||||
} else {
|
||||
crate::flow_debug!(
|
||||
@@ -964,6 +1043,14 @@ impl Requester {
|
||||
mod tests {
|
||||
use super::Requester;
|
||||
|
||||
#[test]
|
||||
fn probe_paths_compare_percent_decoded() {
|
||||
let asked = Requester::path_of("https://rule34video.com/tags/pokémon-species/").unwrap();
|
||||
assert!(Requester::same_path(&asked, "/tags/pokémon-species/"));
|
||||
assert!(Requester::same_path("/Models/X", "/models/x/"));
|
||||
assert!(!Requester::same_path(&asked, "/search/"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn new_requesters_share_cookie_jar() {
|
||||
let a = Requester::new();
|
||||
@@ -979,4 +1066,88 @@ mod tests {
|
||||
|
||||
assert!(cookie_header.contains("shared_cookie=1"));
|
||||
}
|
||||
|
||||
async fn serve(listener: tokio::net::TcpListener, reply: impl Fn(&str) -> String + Send + 'static) {
|
||||
use tokio::io::{AsyncReadExt, AsyncWriteExt};
|
||||
loop {
|
||||
let Ok((mut sock, _)) = listener.accept().await else { return };
|
||||
let mut buf = vec![0u8; 16384];
|
||||
let mut len = 0;
|
||||
// read headers, then the declared body
|
||||
loop {
|
||||
let Ok(n) = sock.read(&mut buf[len..]).await else { break };
|
||||
if n == 0 { break; }
|
||||
len += n;
|
||||
let text = String::from_utf8_lossy(&buf[..len]).to_string();
|
||||
if let Some(head_end) = text.find("\r\n\r\n") {
|
||||
let want = text
|
||||
.lines()
|
||||
.find_map(|l| l.to_ascii_lowercase().strip_prefix("content-length:").map(|v| v.trim().parse::<usize>().unwrap_or(0)))
|
||||
.unwrap_or(0);
|
||||
if len >= head_end + 4 + want { break; }
|
||||
}
|
||||
}
|
||||
let req = String::from_utf8_lossy(&buf[..len]).to_string();
|
||||
let _ = sock.write_all(reply(&req).as_bytes()).await;
|
||||
let _ = sock.shutdown().await;
|
||||
}
|
||||
}
|
||||
|
||||
fn http(status: &str, ctype: &str, extra: &str, body: &str) -> String {
|
||||
format!(
|
||||
"HTTP/1.1 {status}\r\ncontent-type: {ctype}\r\n{extra}content-length: {}\r\nconnection: close\r\n\r\n{body}",
|
||||
body.len()
|
||||
)
|
||||
}
|
||||
|
||||
/// DDoS-Guard answers either a bare `403 Access denied.` (blocked fingerprint/IP) or a
|
||||
/// short JS-check page. Both must end up at FlareSolverr and return the real listing.
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn ddos_guard_blocks_are_solved_through_flaresolverr() {
|
||||
// distinct hosts: the cookie jar is process-wide, so a solved cookie would leak across passes
|
||||
for (interstitial, host) in [(false, "127.0.0.1"), (true, "localhost")] {
|
||||
let origin = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let flare = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
|
||||
let origin_port = origin.local_addr().unwrap().port();
|
||||
let flare_port = flare.local_addr().unwrap().port();
|
||||
|
||||
tokio::spawn(serve(origin, move |req| {
|
||||
if req.contains("ddg_ok=1") {
|
||||
http("200 OK", "text/html", "", "<div class=\"item thumb video_1\">real listing</div>")
|
||||
} else if interstitial {
|
||||
http(
|
||||
"200 OK",
|
||||
"text/html",
|
||||
"server: ddos-guard\r\n",
|
||||
"<html><head><title>DDoS-Guard</title><script src=\"/.well-known/ddos-guard/check?context=free_splash\"></script></head></html>",
|
||||
)
|
||||
} else {
|
||||
http("403 Forbidden", "text/plain", "server: ddos-guard\r\n", "Access denied.")
|
||||
}
|
||||
}));
|
||||
tokio::spawn(serve(flare, move |req| {
|
||||
let body = if req.contains("sessions.create") {
|
||||
r#"{"status":"ok","message":"","session":"mock"}"#.to_string()
|
||||
} else if req.contains("sessions.destroy") {
|
||||
r#"{"status":"ok","message":""}"#.to_string()
|
||||
} else {
|
||||
format!(
|
||||
r#"{{"status":"ok","message":"","startTimestamp":0,"endTimestamp":1,"version":"mock","solution":{{"url":"http://127.0.0.1:{origin_port}/","status":200,"response":"<div class=\"item thumb video_1\">real listing</div>","headers":{{}},"userAgent":"Mozilla/5.0 mock","cookies":[{{"name":"ddg_ok","value":"1","domain":"","path":"/"}}]}}}}"#
|
||||
)
|
||||
};
|
||||
http("200 OK", "application/json", "", &body)
|
||||
}));
|
||||
|
||||
unsafe { std::env::set_var("FLARE_URL", format!("http://127.0.0.1:{flare_port}/v1")) };
|
||||
let mut requester = Requester::new();
|
||||
let body = requester
|
||||
.get(&format!("http://{host}:{origin_port}/?mode=async"), None)
|
||||
.await
|
||||
.expect("requester should fall through to FlareSolverr");
|
||||
assert!(
|
||||
body.contains("real listing"),
|
||||
"interstitial={interstitial}: got {body:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user