diff --git a/docs/provider-catalog.md b/docs/provider-catalog.md index a027c40..02ace07 100644 --- a/docs/provider-catalog.md +++ b/docs/provider-catalog.md @@ -93,7 +93,7 @@ This is the current implementation inventory as of this snapshot of the repo. Us | `xlecx` | `hentai-animation` | no | no | DLE (DataLife Engine)-style HTML scraper for xlecx.one, following the `thaiporntv` template. Latest feed `/f/cat={id}/sort={mode}/order=desc/` (page N≥2 appends `page/{N}/`); `sort` options new/popular/commented/rated map to DLE's `date`/`news_read`/`comm_num`/`rating` sort params; `category` option is the 2 hardcoded top-level archives (`videos`, `hentai-video`). Search via `/index.php?do=search&subaction=search&story={query}` (page N adds `search_start={N}`). Tag shortcut (`#slug` query) routes to `/tags/{slug}/` (paginated via `/tags/{slug}/page/{N}/`); the ~1,044-entry tag catalog is background-loaded once at startup from `/tags/` (`.tags-list1 a[href*='/tags/']`, absolute-URL anchors) into an `Arc>>` and exposed as the `filter`/"Tags" `ChannelOption` — note this option (and any `categories`) is deliberately stripped from the live `/api/status` response by `should_hide_status_option`/`decorate_channel` in `providers/mod.rs` (any `id=="filter"` option titled "Tags" is hidden site-wide), so its absence there is expected, not a bug; it is still honored by `/api/videos`. Cards parsed from `h3.thumb__title` + `img` inside standard DLE thumb blocks; id extracted from the `/{id}-{slug}.html` URL suffix. Thumbnails are direct `xlecx.one/uploads/posts/...` webp (no proxy/Referer needed). `video.url` is the detail page URL, which yt-dlp resolves natively via the generic HTML5 embed extractor; no `formats`/proxy needed. No `/api/uploaders` (no stable uploader identity on the site). | | `xxxtik` | `tiktok` | yes | no | JSON-API short-form aggregator for xxxtik.com — every post is a moderated repost of a RedGifs clip, so the real media backend is the public **RedGifs v2 API**, not xxxtik itself (new pattern; no other provider currently resolves through a third-party media API). Listing API (`xxxtik-api-iw98m.ondigitalocean.app`, found by grepping the Angular `main-es2015.js` bundle): `GET /post/new`, `/post/top/{week,month,year,all}`, `/post/tag/{name}`, `/post/creator/{username}` — all **cursor**-paginated (`?cursor={lastItemId}&limit=N`; the `cursor` is the numeric `id` of the last item from the previous batch, not an offset/count — reaching page N walks N sequential requests, mirroring `fikfap`'s `fetch_cursor_page`). `GET /search?query=Q` returns tag/profile autocomplete suggestions only (no posts), so free-text search is routed through it to resolve a `Tag`/`Creator` target before a second listing call; `tag:`/`category:`/`cat:` and `user:`/`uploader:`/`creator:` query prefixes skip that lookup. Each post's `source` field is a `redgifs.com/watch/{id}` URL; the provider fetches a short-lived anonymous bearer token from `POST api.redgifs.com/v2/auth/temporary` (cached, refreshed once on a 401) and resolves `GET api.redgifs.com/v2/gifs/{id}` (bounded to 8-way concurrency via `buffer_unordered`) for the real `media.redgifs.com/*.mp4` + poster, both fetchable with zero auth/Referer. `video.url` is the xxxtik page (`https://xxxtik.com/feed/{uuid}`, not yt-dlp-resolvable — Angular SPA, generic extractor fails), with `formats` populated from the resolved redgifs mp4; tags merge xxxtik's own tags with RedGifs' tags. 20 curated tags exposed via `categories` (xxxtik has ~62k tags total, too many to background-load). `/api/uploaders` works via `GET /user/by-username/{name}` + `/post/creator/{name}`, but xxxtik's "creator" accounts are inconsistent: some (e.g. `dlhoodninja`, `besttits`) return real posts matching their profile `_count.posts`; others with a nonzero `_count.posts` (e.g. `bigboobsgw`, count 472) return an empty `/post/creator/` list — likely synthetic curation accounts (`name@default` emails) rather than real uploaders. The provider degrades gracefully (returns the profile with `videos: []`, no error) rather than guessing which accounts are "real". No proxy needed — all media/thumb URLs are publicly fetchable with no Referer or auth. | | `txxx` | `mainstream-tube` | no | no | JSON-API scraper for txxx.me (mirror of txxx.com; Vue SPA, same KVS-based API family as the hclips/upornia/hdzog network). Listing: `GET /api/json/videos2/86400/str/{sort}/{count}/{section}.{object}.{page}.all...jsond` with sort `latest-updates`/`most-popular`/`top-rated`/`longest`; **the `.json` variant ignores the page number, `jsond` does not**. Category archive = section `categories`, object = category dir slug (numeric ids return `category_not_found`); ~80 top categories hardcoded as the `categories` option (225 exist at `/api/json/categories/86400/str.all.en.jsond`). Search: `GET /api/videos2.php?params=86400/str/relevance/{count}/search..{page}.all..&s={query}` (relevance only; sort is ignored; page works although `params.page` echoes 1). `count` follows `perPage` (max 60). The page URL `/videos/{id}/{dir}/` is **not** yt-dlp-resolvable, so every card is enriched (bounded `buffered(8)`) via `GET /api/videofile.php?video_id={id}&lifetime=8640000`, whose `video_url` is base64 of `/get_file/...` with Cyrillic look-alike letters (А Е М С …) substituted and `~` for `=`; the decoded `https://txxx.me/get_file/...` 302s to a per-client signed `*.ahcdn.com` URL (IP-pinned, so it must NOT be resolved server-side; no Referer needed). Items without formats are dropped. Heights come from the listing `file_formats` (`||_hq.mp4|1278x720|…||_tr.mp4|…`; `_tr.mp4` is the trailer and becomes `preview` via `pv`). Rating 0..5 → ×20; `likes`/`dislikes` ignored; category titles become tags; uploader = content source else member `display_name` (`txxx:{user_id}`; no profile endpoint). Thumbs `tn.txxx.tube` direct. Queries matching a category name route to its archive; `cat:`/`category:` prefix. No models/channels archives (sections `models`/`channels` are rejected by the API). No proxy needed. | -| `hooporn` | `studio-network` | no | no | HTML scraper for hooporn.com (studio/pornstar tube; Cloudflare-fronted but serves plain 200 to the shared requester). Cards `div.video-card` with lazy thumb in `img[data-src]` (`//imgs.hooporn.com/thumb/...jpg?v2`, direct, no Referer), title in `div.video-title a`, `.time` duration, `span.create` unix timestamp → `uploadedAt`; no views/rating. Feed `/home/{N}`, category `/category/{slug}/{N}` (63 hardcoded from `/categories`), pornstar `/pornstar/{slug}/{N}`, studio `/studio/{slug}/{N}` (studio slugs are concatenated, e.g. `familystrokes`), search `/search?q=…&p={N}` (24 per page; `?search=`/`/search/q/N` do not work). Prefix queries `cat:`/`category:`, `pornstar:`/`star:`/`model:`, `studio:`/`site:`; a bare keyword matching a category name routes to the archive, anything else is a search. The watch page `/watch/{24hex}` is **not** yt-dlp-resolvable (player is an iframe), and the mp4 is signed (`videos.trendyporn.com/...mp4?md5=…&expires=…`, ~1 day, 403 without the signature, no Referer/IP binding), so every card is enriched (`buffered(8)`): watch page → `iframe[src*="/video/"]` (`//vid.hooporn.com/video/{n}`) → `var video = "…"` into one `formats[]` entry (quality unknown, labelled `auto`); the watch page also yields category tags, pornstar names (as tags) and the studio (uploader `hooporn:{studio-slug}`). Items whose enrichment fails are dropped. No proxy needed. | +| `hooporn` | `studio-network` | no | no | HTML scraper for hooporn.com (studio/pornstar tube; Cloudflare-fronted but serves plain 200 to the shared requester). Cards `div.video-card` with lazy thumb in `img[data-src]` (`//imgs.hooporn.com/thumb/...jpg?v2`, direct, no Referer), title in `div.video-title a`, `.time` duration, `span.create` unix timestamp → `uploadedAt`; no views/rating. Feed `/home/{N}`, category `/category/{slug}/{N}` (63 hardcoded from `/categories`), pornstar `/pornstar/{slug}/{N}`, studio `/studio/{slug}/{N}` (studio slugs are concatenated, e.g. `familystrokes`), search `/search?q=…&p={N}` (24 per page; `?search=`/`/search/q/N` do not work). Prefix queries `cat:`/`category:`, `pornstar:`/`star:`/`model:`, `studio:`/`site:`; a bare keyword matching a category name routes to the archive; any other query is first probed on the site (`/pornstar/{slug}/1`, then `/studio/{slug-without-dashes}/1`; unknown archives are an instant 404 read through the raw response, Cloudflare challenges are handed to the requester/FlareSolverr chain) and used as that archive if it exists, otherwise it is a keyword search. The probe runs on every page, so page N of a pornstar/studio query stays in that archive. The watch page `/watch/{24hex}` is **not** yt-dlp-resolvable (player is an iframe), and the mp4 is signed (`videos.trendyporn.com/...mp4?md5=…&expires=…`, ~1 day, 403 without the signature, no Referer/IP binding), so every card is enriched (`buffered(8)`): watch page → `iframe[src*="/video/"]` (`//vid.hooporn.com/video/{n}`) → `var video = "…"` into one `formats[]` entry (quality unknown, labelled `auto`); the watch page also yields category tags, pornstar names (as tags) and the studio (uploader `hooporn:{studio-slug}`). Items whose enrichment fails are dropped. No proxy needed. | ## Proxy Routes diff --git a/src/providers/hooporn.rs b/src/providers/hooporn.rs index f327e89..35575d2 100644 --- a/src/providers/hooporn.rs +++ b/src/providers/hooporn.rs @@ -111,6 +111,9 @@ enum Target { Pornstar(String), Studio(String), Search(String), + /// Free-form query: a pornstar page or a studio page if one exists by that + /// name (probed on the site), else a keyword search. + Keyword(String), } /// Detail-page data: the player iframe plus taxonomy links. @@ -239,7 +242,7 @@ impl HoopornProvider { if let Some(slug) = Self::category_slug(q) { return Target::Category(slug.to_string()); } - return Target::Search(q.to_string()); + return Target::Keyword(q.to_string()); } if let Some(cat) = options.categories.as_deref() { @@ -260,7 +263,7 @@ impl HoopornProvider { Target::Category(slug) => format!("{BASE_URL}/category/{slug}/{page}"), Target::Pornstar(slug) => format!("{BASE_URL}/pornstar/{slug}/{page}"), Target::Studio(slug) => format!("{BASE_URL}/studio/{slug}/{page}"), - Target::Search(query) => format!( + Target::Search(query) | Target::Keyword(query) => format!( "{BASE_URL}/search?q={}&p={page}", utf8_percent_encode(query, NON_ALPHANUMERIC) ), @@ -427,17 +430,82 @@ impl HoopornProvider { .await } + /// Fetches a listing page. A real 404 (unknown archive, page past the end) + /// is `None` and answers immediately; a Cloudflare challenge is handed to + /// the requester's full chain (FlareSolverr). + async fn fetch_html(requester: &mut Requester, url: &str) -> Result> { + let headers = vec![("Referer".to_string(), format!("{BASE_URL}/"))]; + let mut last_err = String::new(); + for _ in 0..2 { + match requester.get_raw_with_headers(url, headers.clone()).await { + Ok(response) => { + let status = response.status().as_u16(); + let body = response.text().await.unwrap_or_default(); + if status == 404 { + return Ok(None); + } + if status == 403 + || status == 503 + || Requester::looks_like_cloudflare_challenge(&body) + { + return requester + .get_with_headers(url, headers, None) + .await + .map(Some) + .map_err(|e| Error::from(format!("fetch failed for {url}: {e}"))); + } + if status >= 500 || status == 429 { + last_err = format!("HTTP {status}"); + continue; + } + return Ok(Some(body)); + } + Err(e) => last_err = e.to_string(), + } + } + Err(Error::from(format!("fetch failed for {url}: {last_err}"))) + } + async fn fetch(target: &Target, page: u32, options: &ServerOptions) -> Result> { - let url = Self::build_listing_url(target, page); let mut requester = requester_or_default(options, CHANNEL_ID, "fetch"); - let html = requester - .get_with_headers( - &url, - vec![("Referer".to_string(), format!("{BASE_URL}/"))], - None, - ) - .await - .map_err(|e| Error::from(format!("fetch failed for {url}: {e}")))?; + + let html = match target { + Target::Keyword(query) => { + // Probe page 1 of the pornstar and studio archive for this + // name; the first that exists wins, otherwise keyword search. + let candidates = [ + Target::Pornstar(Self::slugify(query)), + Target::Studio(Self::slugify(query).replace('-', "")), + ]; + let mut found = None; + for candidate in candidates { + let slug_ok = match &candidate { + Target::Pornstar(s) | Target::Studio(s) => !s.is_empty(), + _ => false, + }; + if !slug_ok { + continue; + } + let first = Self::fetch_html(&mut requester, &Self::build_listing_url(&candidate, 1)).await?; + if let Some(body) = first.filter(|b| b.contains("class=\"video-card\"")) { + found = Some(if page <= 1 { + Some(body) + } else { + Self::fetch_html(&mut requester, &Self::build_listing_url(&candidate, page)).await? + }); + break; + } + } + match found { + Some(html) => html, + None => Self::fetch_html(&mut requester, &Self::build_listing_url(target, page)).await?, + } + } + _ => Self::fetch_html(&mut requester, &Self::build_listing_url(target, page)).await?, + }; + let Some(html) = html else { + return Ok(vec![]); + }; let items = Self::parse_listing(&html)?; Ok(Self::enrich(items, requester).await) } @@ -529,7 +597,7 @@ mod tests { ); assert_eq!( HoopornProvider::pick_target(Some("golf"), &o), - Target::Search("golf".into()) + Target::Keyword("golf".into()) ); assert_eq!(HoopornProvider::pick_target(None, &o), Target::Latest); }