Fix six dead providers (shooshtime, hdporn92, camsoda, sextb, supjav, hentaimama)

- shooshtime: update parse_list_videos selectors for the site's redesigned
  markup (div.card / a.title / a.image img / div.duration / div.list div.box
  / div.name a.text)
- hdporn92: mirror the site's own links.hls4||hls3||hls2 fallback and scan
  every P.A.C.K.E.R. block instead of assuming the first one is the player
  block; prefer a genuine .m3u8 candidate over a decoy .txt one
- camsoda: replace naive first-`{`/last-`}` JSON extraction with a proper
  balanced-brace scan so trailing wrapper markup can't corrupt the parse
  ("key must be a string")
- sextb: fix mirror_url() double-prefixing the scheme when building Jina
  mirror URLs (produced a malformed https://r.jina.ai/http://https://... URL)
- supjav: switch curl_cffi impersonation from chrome to safari18_0 - Cloudflare
  blocks the chrome/firefox profiles outright but passes Safari
- hentaimama: resolve every card on a listing page (not just the first
  perPage) and truncate after filtering, so a bulk "Coming soon" placeholder
  batch at the front of the feed no longer empties the whole page; report a
  provider error if literally every candidate fails

Refreshes graphify's graph and docs/provider-catalog.md with the new
technical details for all six providers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QTqf6orbHZ9rFFpVpcgzcR
This commit is contained in:
Simon
2026-09-22 13:56:36 +00:00
parent c244461d0b
commit c40b48b0db
8 changed files with 356 additions and 78 deletions

View File

@@ -22,9 +22,9 @@ This is the current implementation inventory as of this snapshot of the repo. Us
| `freeuseporn` | `fetish-kink` | no | no | Fetish archive pattern. | | `freeuseporn` | `fetish-kink` | no | no | Fetish archive pattern. |
| `hanime` | `hentai-animation` | no | yes | Uses proxied CDN/thumb handling. | | `hanime` | `hentai-animation` | no | yes | Uses proxied CDN/thumb handling. |
| `heavyfetish` | `fetish-kink` | no | no | Direct media handling. | | `heavyfetish` | `fetish-kink` | no | no | Direct media handling. |
| `hdporn92` | `mainstream-tube` | no | no | WordPress retrotube-theme aggregator for hdporn92.com (network studio releases — MyPervyFamily, MYLF, StepSiblings, …). Latest feed `/` (page 1) and `/page/N/` (page N≥2); search `/?s={query}` (page 1) and `/page/N/?s={query}` (page N≥2) — the path with the `s` parameter must come after the `page/N` segment, not before. Pagination is by `article.loop-video` cards; the post id comes from the `post-{N}` CSS class. Two-step enrichment (page URL + Referer-locked m3u8 in `formats`): detail page → `<IFRAME SRC="https://morencius.com/embed/{id}">` (case-insensitive; the page emits an uppercase `<IFRAME SRC=...>` tag) → morencius embed → embedded `eval(function(p,a,c,k,e,d){…}(p,a,c,k.split('|')))` P.A.C.K.E.R. obfuscation block → deobfuscate in pure Rust (reverse-iterate the `c`-word `|`-delimited dictionary, base-`a` numeric tokens, `\b<token>\b` regex substitution) → `"hls4":"<value>"` field in the decoded JS gives the morencius stream URL `https://morencius.com/stream/{token}/{secret}/{expiry_ts}/{file_id}/master.m3u8`; `duration:"<value>"` gives the runtime. `video.url` is the hdporn92 page (no yt-dlp extractor exists for the site); `formats[0]` is the morencius `m3u8` carrying `Referer: https://morencius.com/embed/{id}` + a desktop Chrome `User-Agent` — these headers must travel with the manifest, sub-playlists, and segment requests. **Known limitation:** the morencius m3u8 path contains an `expiry_ts` segment (~12h from page render) — the cache TTL (`cacheDuration: 1800`, same as fyptt) caps the staleness window. Real playback still requires the webview to click through the morencius ad overlay; the manifest itself resolves to a 1280x720 HLS variant via `yt-dlp -j --referer <embed-url>`. No `/api/uploaders` (no uploader identity on the listing). | | `hdporn92` | `mainstream-tube` | no | no | WordPress retrotube-theme aggregator for hdporn92.com (network studio releases — MyPervyFamily, MYLF, StepSiblings, …). Latest feed `/` (page 1) and `/page/N/` (page N≥2); search `/?s={query}` (page 1) and `/page/N/?s={query}` (page N≥2) — the path with the `s` parameter must come after the `page/N` segment, not before. Pagination is by `article.loop-video` cards; the post id comes from the `post-{N}` CSS class. Two-step enrichment (page URL + Referer-locked m3u8 in `formats`): detail page → `<IFRAME SRC="https://morencius.com/embed/{id}">` (case-insensitive; the page emits an uppercase `<IFRAME SRC=...>` tag) → morencius embed → embedded `eval(function(p,a,c,k,e,d){…}(p,a,c,k.split('|')))` P.A.C.K.E.R. obfuscation block → deobfuscate in pure Rust (reverse-iterate the `c`-word `|`-delimited dictionary, base-`a` numeric tokens, `\b<token>\b` regex substitution) → the decoded JS defines `var links={"hls2":"…","hls3":"…"[,"hls4":"…"]}` and the jwplayer setup itself picks `links.hls4||links.hls3||links.hls2`; the provider mirrors that same priority order and, within it, prefers whichever tier's URL actually ends in `.m3u8` (a lower tier has been observed serving a `.txt` decoy instead of a real playlist) falling back to the first present tier if none do; `duration:"<value>"` gives the runtime. The embed page carries more than one P.A.C.K.E.R. block (an ad/VAST setup block plus the player block) — `decode_packered_blocks` decodes all of them and the hls extractor scans each until one contains an hls2/3/4 key, since assuming the first block is the player block is not reliable. `video.url` is the hdporn92 page (no yt-dlp extractor exists for the site); `formats[0]` is the morencius `m3u8` carrying `Referer: https://morencius.com/embed/{id}` + a desktop Chrome `User-Agent` — these headers must travel with the manifest, sub-playlists, and segment requests. **Known limitation:** the morencius m3u8 path contains an `expiry_ts` segment (~12h from page render) — the cache TTL (`cacheDuration: 1800`, same as fyptt) caps the staleness window. Real playback still requires the webview to click through the morencius ad overlay; the manifest itself resolves to a 1280x720 HLS variant via `yt-dlp -j --referer <embed-url>`. No `/api/uploaders` (no uploader identity on the listing). |
| `hentaihaven` | `hentai-animation` | no | no | HTML scraper for hentaihaven.xxx (WordPress/Madara theme), Cloudflare-protected so the provider is gated behind `FLARE_URL` in `skip_reason_for_provider` (mod.rs); the shared requester clears CF directly (wreq Firefox136 emulation currently passes for the listing/search/watch/episode/`player.php` GETs) and falls back to Jina/FlareSolverr. Latest feed `/hentai/page/{N}/`, search `/?s={query}` (search is single-page — page>1 returns empty); listing/search cards link to series watch pages `https://hentaihaven.xxx/watch/{slug}/`. Per-series media resolution (the UUID exists nowhere in page HTML, so enrichment is unavoidable): watch page → episode links `…/watch/{slug}/episode-K` (in `manga-chapters-holder`) → episode page → `<iframe src="…/wp-content/plugins/player-logic/player.php?data=…">` → `player.php` → `<meta name="x-secure-token" content="sha512-…">` → decode token (strip `sha512-`, then 3× of rot13→base64-decode, then `JSON.parse`) → `{en, iv, uri, hot_domains, …}` → POST `…/wp-content/plugins/player-logic/api.php` with `action=zarat_get_data_player_ajax&a={en}&b={iv}` (urlencoded; this one POST uses a dedicated `wreq` Chrome137 client, not the shared requester) → `{"status":true,"data":{"sources":[{"src":"…m3u8"}],"isOctopus":bool}}`. A multi-episode series collapses into one `VideoItem` titled `"… (N Episodes)"` with one `m3u8` `VideoFormat` per episode (`format_note`/`format_id` = "Episode K"); each format carries `Referer`/`Origin: https://hentaihaven.xxx` + a Firefox `User-Agent`. `video.url` is the `watch/{slug}/` page (no yt-dlp extractor exists for the site, so `formats` are populated rather than relying on `video.url`). Two CDN shapes are returned: newer content-addressed `octopusmanifest.org/{uuid}/playlist.m3u8` (no token, portable across IPs) and older signed `master-lengs.org/api/v3/hh/{slug}/master.m3u8?hash=…` (~2.5h). **Gotcha:** both CDNs (same IP) aggressively per-IP rate-limit/ban with a TCP RST on 80/443 once tripped — looks like "host down" but is an IP ban; browsers play fine over HTTP/3 (QUIC) while TCP clients (curl/yt-dlp/wreq) get refused, so segment fetches can fail from a tripped IP even though the manifest URL is valid. Tags from the series "Genre(s)" block; `views` from the "Viewed … Total" counter; thumbnails (`img.hentaihaven.xxx`) load directly (no proxy/referer). Resolution is slow (each listing page = ~25 series × multi-episode player-API calls), so the provider is DB-first: it fetches the listing once for the ordered watch URLs, serves already-resolved `VideoItem`s from the `videos` SQLite table instantly (`db::upsert_video` to avoid duplicate-row staleness), and `spawn_refresh`es the whole page in the background (in-memory `VideoCache` soft-TTL 1h / hard-TTL 24h, per-listing in-flight guard). No `/api/uploaders` (no uploader identity), no proxy. | | `hentaihaven` | `hentai-animation` | no | no | HTML scraper for hentaihaven.xxx (WordPress/Madara theme), Cloudflare-protected so the provider is gated behind `FLARE_URL` in `skip_reason_for_provider` (mod.rs); the shared requester clears CF directly (wreq Firefox136 emulation currently passes for the listing/search/watch/episode/`player.php` GETs) and falls back to Jina/FlareSolverr. Latest feed `/hentai/page/{N}/`, search `/?s={query}` (search is single-page — page>1 returns empty); listing/search cards link to series watch pages `https://hentaihaven.xxx/watch/{slug}/`. Per-series media resolution (the UUID exists nowhere in page HTML, so enrichment is unavoidable): watch page → episode links `…/watch/{slug}/episode-K` (in `manga-chapters-holder`) → episode page → `<iframe src="…/wp-content/plugins/player-logic/player.php?data=…">` → `player.php` → `<meta name="x-secure-token" content="sha512-…">` → decode token (strip `sha512-`, then 3× of rot13→base64-decode, then `JSON.parse`) → `{en, iv, uri, hot_domains, …}` → POST `…/wp-content/plugins/player-logic/api.php` with `action=zarat_get_data_player_ajax&a={en}&b={iv}` (urlencoded; this one POST uses a dedicated `wreq` Chrome137 client, not the shared requester) → `{"status":true,"data":{"sources":[{"src":"…m3u8"}],"isOctopus":bool}}`. A multi-episode series collapses into one `VideoItem` titled `"… (N Episodes)"` with one `m3u8` `VideoFormat` per episode (`format_note`/`format_id` = "Episode K"); each format carries `Referer`/`Origin: https://hentaihaven.xxx` + a Firefox `User-Agent`. `video.url` is the `watch/{slug}/` page (no yt-dlp extractor exists for the site, so `formats` are populated rather than relying on `video.url`). Two CDN shapes are returned: newer content-addressed `octopusmanifest.org/{uuid}/playlist.m3u8` (no token, portable across IPs) and older signed `master-lengs.org/api/v3/hh/{slug}/master.m3u8?hash=…` (~2.5h). **Gotcha:** both CDNs (same IP) aggressively per-IP rate-limit/ban with a TCP RST on 80/443 once tripped — looks like "host down" but is an IP ban; browsers play fine over HTTP/3 (QUIC) while TCP clients (curl/yt-dlp/wreq) get refused, so segment fetches can fail from a tripped IP even though the manifest URL is valid. Tags from the series "Genre(s)" block; `views` from the "Viewed … Total" counter; thumbnails (`img.hentaihaven.xxx`) load directly (no proxy/referer). Resolution is slow (each listing page = ~25 series × multi-episode player-API calls), so the provider is DB-first: it fetches the listing once for the ordered watch URLs, serves already-resolved `VideoItem`s from the `videos` SQLite table instantly (`db::upsert_video` to avoid duplicate-row staleness), and `spawn_refresh`es the whole page in the background (in-memory `VideoCache` soft-TTL 1h / hard-TTL 24h, per-listing in-flight guard). No `/api/uploaders` (no uploader identity), no proxy. |
| `hentaimama` | `hentai-animation` | no | no | WordPress/DooPlay-theme provider for hentaimama.io (series `/tvshows/{slug}/` each holding numbered `/episodes/{slug}-episode-N/` posts). Latest feed is the `episodes` post-type archive: `/episodes/` (page N: `/episodes/page/N/`), 24 cards/page, each already carrying the numeric post id (`id="post-{N}"`/`rel="{N}"`), title (`div.season_m` `span.b`+`span.c`, joined with a space — the two spans are adjacent text nodes with no whitespace between them), thumb, and rating, so the latest feed needs only one more request per item (see media resolution below), no page-2 detail fetch. Search (`/?s=query`, page N: `/page/N/?s=query`), genre archives (`/genre/{slug}/`, 118+ curated genres background-loaded from `a.genreitem` on `/genres-filter/` into the `categories` filter option, sanitized out of `/api/status` like other providers' catalogs but honored in `/api/videos`; bare keywords matching a loaded genre title route here, otherwise `genre:`/`cat:`/`category:` prefixes), and studio archives (`/studio/{slug}/` via `studio:`/`uploader:` prefix, naive-slugified — no background catalog) all only expose **series** cards (`article.series-card` / `a.sc-poster`), not individual episodes, so each match is resolved to its single most-recent episode (fetch the series page's `a.dt-se-item` list, take the last href, then resolve that episode page) — richer results (tags, rating, exact title, `uploadedAt`) than the latest feed, at the cost of two extra requests instead of one. Media resolution (same for every path): `POST /wp-admin/admin-ajax.php` with `action=get_player_contents&a={post_id}&i=1` returns a JSON array of iframe-HTML strings; the populated slot's `src` is `hentaimama.io/?dt_embed=rtmp&p={base64}&ep={id}` where the base64 `p` param decodes directly to the storage path (e.g. `P/{file}.mp4`) on a **second, unrelated mirror host** `gdvid.info` — so the direct, non-expiring, Referer-free `https://gdvid.info/{path}` format URL is built locally from the AJAX response with no extra fetch; a `dt_embed=cu` fallback mirror (`na-01.javprovider.com`, signed/expiring `?st=&e=` token) exists but is only used as a last resort (fetch the iframe `src` page itself and pull the jwplayer `file:` value) if the primary parse fails, since it needs an extra request. `video.url` is the episode page (not yt-dlp-resolvable — the player loads via the AJAX handshake, not static HTML), so `formats[0]` carries the `gdvid.info` mp4. Thumbnails (`wp-content/uploads/...webp`) load directly, no proxy/Referer. No `/api/uploaders` (studio pages exist but have no per-provider profile schema confirmed; no background studio catalog for the same reason). No proxy needed. | | `hentaimama` | `hentai-animation` | no | no | WordPress/DooPlay-theme provider for hentaimama.io (series `/tvshows/{slug}/` each holding numbered `/episodes/{slug}-episode-N/` posts). Latest feed is the `episodes` post-type archive: `/episodes/` (page N: `/episodes/page/N/`), 24 cards/page, each already carrying the numeric post id (`id="post-{N}"`/`rel="{N}"`), title (`div.season_m` `span.b`+`span.c`, joined with a space — the two spans are adjacent text nodes with no whitespace between them), thumb, and rating, so the latest feed needs only one more request per item (see media resolution below), no page-2 detail fetch. **The site sometimes bulk-lists an entire freshly-added series as "Coming soon" placeholders** (`get_player_contents` returns `[]` for every slot, no error) — since these sort first by date, the latest feed resolves *every* card on the page (not just the first `perPage`) and truncates to `perPage` only after filtering out unresolvable placeholders, so a bulk-placeholder batch at the front doesn't wipe out the whole feed; if literally every card on a page fails, a `report_provider_error_background("get.all_candidates_failed", ...)` call surfaces it instead of returning a silent empty list. Search (`/?s=query`, page N: `/page/N/?s=query`), genre archives (`/genre/{slug}/`, 118+ curated genres background-loaded from `a.genreitem` on `/genres-filter/` into the `categories` filter option, sanitized out of `/api/status` like other providers' catalogs but honored in `/api/videos`; bare keywords matching a loaded genre title route here, otherwise `genre:`/`cat:`/`category:` prefixes), and studio archives (`/studio/{slug}/` via `studio:`/`uploader:` prefix, naive-slugified — no background catalog) all only expose **series** cards (`article.series-card` / `a.sc-poster`), not individual episodes, so each match is resolved to its single most-recent episode (fetch the series page's `a.dt-se-item` list, take the last href, then resolve that episode page) — richer results (tags, rating, exact title, `uploadedAt`) than the latest feed, at the cost of two extra requests instead of one. Media resolution (same for every path): `POST /wp-admin/admin-ajax.php` with `action=get_player_contents&a={post_id}&i=1` returns a JSON array of iframe-HTML strings; the populated slot's `src` is `hentaimama.io/?dt_embed=rtmp&p={base64}&ep={id}` where the base64 `p` param decodes directly to the storage path (e.g. `P/{file}.mp4`) on a **second, unrelated mirror host** `gdvid.info` — so the direct, non-expiring, Referer-free `https://gdvid.info/{path}` format URL is built locally from the AJAX response with no extra fetch; a `dt_embed=cu` fallback mirror (`na-01.javprovider.com`, signed/expiring `?st=&e=` token) exists but is only used as a last resort (fetch the iframe `src` page itself and pull the jwplayer `file:` value) if the primary parse fails, since it needs an extra request. `video.url` is the episode page (not yt-dlp-resolvable — the player loads via the AJAX handshake, not static HTML), so `formats[0]` carries the `gdvid.info` mp4. Thumbnails (`wp-content/uploads/...webp`) load directly, no proxy/Referer. No `/api/uploaders` (studio pages exist but have no per-provider profile schema confirmed; no background studio catalog for the same reason). No proxy needed. |
| `hentaitv` | `hentai-animation` | no | yes | Next.js hentai site (hentai.tv) backed by a clean JSON API: `GET /api/browse?page=N&sort=<Label>&genres=<ExactName>` (`{videos:[28],total,pages}`, real pagination) and `GET /api/search?q=Q` (`{videos:[...]}`, single-page — `page` is ignored, so page>1 returns empty). Unlike `animeidhentai`, browse honors both `sort` (labels `Most Recent`/`Most Viewed`/`Trending`, mapped from option ids `new`/`views`/`trending`) and `genres` (the **exact case-sensitive** stored genre name, e.g. `Big Boobs`, `incest`), so genre archives go through `/api/browse?genres=` and paginate. The 68-genre catalogue (exact names) is background-loaded from the `/browse` page HTML (`"genres":[{"name","count"}]`, not exposed by the JSON API) and powers the `categories` filter plus keyword→genre routing. Each episode JSON has `slug`, `title`/`ep`, `tags[]`, `views`, `rating` (0-10 → ×10), `duration` ("MM:SS"), `brand` (studio → `uploader`), `thumb`/`backdrop`/`cover` (relative, served from `hentai.tv/uploads/...`, no referer), and `embedUrl=https://nhplayer.com/v/{embedId}/`. `video.url` is the reachable watch page `https://hentai.tv/hentai/{slug}`; `genre:`/`cat:`/`category:` prefixes and bare keywords that exactly match a genre route to the genre archive, everything else to search. Playback shares the **same nhplayer→`r2.1hanime.com` signed-CDN backend as `animeidhentai`**: `/proxy/hentaitv/{embedId}.mp4` is a redirect proxy that replicates nhplayer's PoW+DOM challenge (`player.php`→`player-core-v2.php`→`get-video-url-v2.php`, SHA-256-first-byte-zero PoW, ≥700ms dwell, fixed fingerprint) to mint a signed `?verify=<ts>-<sig>` URL — HEAD→200, GET→302 to the CDN URL (cached 150s). The CF wall is JA3-based not IP-based, so the signed URL is verifiable from anywhere with `yt-dlp --impersonate chrome` even though plain `curl`/`wreq` get 403. `src/proxies/hentaitv.rs` is a near-copy of `src/proxies/animeidhentai.rs` (only `SITE_REFERER` differs). No `/api/uploaders` (brand is studio-only). | | `hentaitv` | `hentai-animation` | no | yes | Next.js hentai site (hentai.tv) backed by a clean JSON API: `GET /api/browse?page=N&sort=<Label>&genres=<ExactName>` (`{videos:[28],total,pages}`, real pagination) and `GET /api/search?q=Q` (`{videos:[...]}`, single-page — `page` is ignored, so page>1 returns empty). Unlike `animeidhentai`, browse honors both `sort` (labels `Most Recent`/`Most Viewed`/`Trending`, mapped from option ids `new`/`views`/`trending`) and `genres` (the **exact case-sensitive** stored genre name, e.g. `Big Boobs`, `incest`), so genre archives go through `/api/browse?genres=` and paginate. The 68-genre catalogue (exact names) is background-loaded from the `/browse` page HTML (`"genres":[{"name","count"}]`, not exposed by the JSON API) and powers the `categories` filter plus keyword→genre routing. Each episode JSON has `slug`, `title`/`ep`, `tags[]`, `views`, `rating` (0-10 → ×10), `duration` ("MM:SS"), `brand` (studio → `uploader`), `thumb`/`backdrop`/`cover` (relative, served from `hentai.tv/uploads/...`, no referer), and `embedUrl=https://nhplayer.com/v/{embedId}/`. `video.url` is the reachable watch page `https://hentai.tv/hentai/{slug}`; `genre:`/`cat:`/`category:` prefixes and bare keywords that exactly match a genre route to the genre archive, everything else to search. Playback shares the **same nhplayer→`r2.1hanime.com` signed-CDN backend as `animeidhentai`**: `/proxy/hentaitv/{embedId}.mp4` is a redirect proxy that replicates nhplayer's PoW+DOM challenge (`player.php`→`player-core-v2.php`→`get-video-url-v2.php`, SHA-256-first-byte-zero PoW, ≥700ms dwell, fixed fingerprint) to mint a signed `?verify=<ts>-<sig>` URL — HEAD→200, GET→302 to the CDN URL (cached 150s). The CF wall is JA3-based not IP-based, so the signed URL is verifiable from anywhere with `yt-dlp --impersonate chrome` even though plain `curl`/`wreq` get 403. `src/proxies/hentaitv.rs` is a near-copy of `src/proxies/animeidhentai.rs` (only `SITE_REFERER` differs). No `/api/uploaders` (brand is studio-only). |
| `homoxxx` | `gay-male` | no | no | Gay category grouping example. | | `homoxxx` | `gay-male` | no | no | Gay category grouping example. |
| `hotbunny` | `ai` | no | yes (thumbs only) | JSON-API provider for hotbunny.ai (AI-generated hentai community). Feeds: `GET /api/post/feed?limit=N` (hot), `/api/post/feed/popular`, `/api/post/feed/recents`, `/api/post/category/{cuid}?limit=N`, `/api/post/search?query=Q&limit=N`, `/api/post/author/{name}/all?limit=N`. Pagination: fetch `page × per_page` items in one call and slice (cursor-based API, but limit works up to 200 total). Key fields: `id`, `imageUrl` (relative thumbnail path on CDN), `authorName`, `likes` (→ `views`), `sharedVideos[].videoUrl` (first video path, relative to CDN), `width`/`height`, `galleries[].label` (used as title prefix). 15 hardcoded categories from `/api/categories` (stable CUIDs). `video.url` is the post page `https://hotbunny.ai/post/{id}` (React SPA, not yt-dlp-resolvable); `formats[0].url` = `https://assets.hotbunny.ai/{sharedVideos[0].videoUrl}` with `Referer: https://hotbunny.ai/` header. `assets.hotbunny.ai` is Cloudflare bot-management protected (returns 404 to plain curl/yt-dlp; iOS AVFoundation passes with real Apple TLS — no proxy wrapping needed for video). Thumbnails proxied via `/proxy/hotbunny-thumb/...` to inject the required `Referer`. Supports `uploader:`/`author:`, `cat:`/`category:`/`#tag` query shortcuts, `categories` filter option, and `sort` option (feed/popular/recent). | | `hotbunny` | `ai` | no | yes (thumbs only) | JSON-API provider for hotbunny.ai (AI-generated hentai community). Feeds: `GET /api/post/feed?limit=N` (hot), `/api/post/feed/popular`, `/api/post/feed/recents`, `/api/post/category/{cuid}?limit=N`, `/api/post/search?query=Q&limit=N`, `/api/post/author/{name}/all?limit=N`. Pagination: fetch `page × per_page` items in one call and slice (cursor-based API, but limit works up to 200 total). Key fields: `id`, `imageUrl` (relative thumbnail path on CDN), `authorName`, `likes` (→ `views`), `sharedVideos[].videoUrl` (first video path, relative to CDN), `width`/`height`, `galleries[].label` (used as title prefix). 15 hardcoded categories from `/api/categories` (stable CUIDs). `video.url` is the post page `https://hotbunny.ai/post/{id}` (React SPA, not yt-dlp-resolvable); `formats[0].url` = `https://assets.hotbunny.ai/{sharedVideos[0].videoUrl}` with `Referer: https://hotbunny.ai/` header. `assets.hotbunny.ai` is Cloudflare bot-management protected (returns 404 to plain curl/yt-dlp; iOS AVFoundation passes with real Apple TLS — no proxy wrapping needed for video). Thumbnails proxied via `/proxy/hotbunny-thumb/...` to inject the required `Referer`. Supports `uploader:`/`author:`, `cat:`/`category:`/`#tag` query shortcuts, `categories` filter option, and `sort` option (feed/popular/recent). |
@@ -60,11 +60,11 @@ This is the current implementation inventory as of this snapshot of the repo. Us
| `ripnsfw` | `onlyfans` | no | yes | Leaked-creator aggregator for ripnsfw.com — the entire catalogue (~2.5k rows) is a single client-side-rendered page whose data source is a published Google Sheet exported as CSV (`GET /data.csv`, no auth/CF gate); the provider fetches and parses that CSV directly instead of scraping any HTML, replicating the site's own JS `parseCSV`/`csvSplit` column layout (`THUMB,NEW,NOME,DOODSTREAM,LULUSTREAM,<link3>,DOOD,<guests>,SITE(bunkr links),TITLE,ID,#(tags),POST,,DATE,SIZE`). Feed/search/pagination/sort are all done in-memory over the fetched rows (no native site pagination exists — `perPage`-sized pages are sliced locally), with a 180s in-process cache to avoid re-fetching the CSV on every request. `tag:`/`category:`/`cat:` and `model:`/`uploader:` query prefixes filter by the parsed tag list / model name; bare keywords substring-match model name, guests, and tags. `video.url` is the site's client-rendered `https://ripnsfw.com/model/{slug}/{id}` route — the host has no server-side SPA fallback, so this 404s to any plain HTTP client (its custom 404 page stashes the path and JS-redirects to `/`, which restores the route — works in a real browser/webview only); `check.py`'s `_CF_PROTECTED_HOSTS` allowlist includes `ripnsfw.com` for this reason. Media is never hosted on ripnsfw.com itself: each row carries a Doodstream and/or Lulustream embed link, resolved into `formats[]` (Lulustream first, matching the site's own default-player order) via the **existing, previously-unused** `src/proxies/doodstream.rs` / `src/proxies/lulustream.rs` redirect proxies (`build_proxy_url(&options, "doodstream"/"lulustream", &strip_url_scheme(link))`) — no new proxy code was written. Both proxies resolve correctly to real signed CDN URLs (verified against live data), but neither CDN's final edge is reachable from this repo's dev sandbox: Lulustream's `*.tnmr.org` edge 403s datacenter/proxy IPs regardless of UA or TLS impersonation, and Doodstream's final `*.dood.video` edge resolves to a loopback/refused address from non-residential egress — both are IP-reputation-style CDN blocks, not code defects (same class of sandbox limitation already documented for `camsoda`/`hentaihaven`/`animeidhentai`), so `check.py` allowlists both by suffix (`_CF_PROTECTED_SUFFIXES`). No thumbnail proxy needed (`cdn.ripnsfw.com/thumbs/...` is directly hotlinkable, no CF/referer gate). No `/api/uploaders` (model pages exist but have no dedicated profile/stats schema confirmed). | | `ripnsfw` | `onlyfans` | no | yes | Leaked-creator aggregator for ripnsfw.com — the entire catalogue (~2.5k rows) is a single client-side-rendered page whose data source is a published Google Sheet exported as CSV (`GET /data.csv`, no auth/CF gate); the provider fetches and parses that CSV directly instead of scraping any HTML, replicating the site's own JS `parseCSV`/`csvSplit` column layout (`THUMB,NEW,NOME,DOODSTREAM,LULUSTREAM,<link3>,DOOD,<guests>,SITE(bunkr links),TITLE,ID,#(tags),POST,,DATE,SIZE`). Feed/search/pagination/sort are all done in-memory over the fetched rows (no native site pagination exists — `perPage`-sized pages are sliced locally), with a 180s in-process cache to avoid re-fetching the CSV on every request. `tag:`/`category:`/`cat:` and `model:`/`uploader:` query prefixes filter by the parsed tag list / model name; bare keywords substring-match model name, guests, and tags. `video.url` is the site's client-rendered `https://ripnsfw.com/model/{slug}/{id}` route — the host has no server-side SPA fallback, so this 404s to any plain HTTP client (its custom 404 page stashes the path and JS-redirects to `/`, which restores the route — works in a real browser/webview only); `check.py`'s `_CF_PROTECTED_HOSTS` allowlist includes `ripnsfw.com` for this reason. Media is never hosted on ripnsfw.com itself: each row carries a Doodstream and/or Lulustream embed link, resolved into `formats[]` (Lulustream first, matching the site's own default-player order) via the **existing, previously-unused** `src/proxies/doodstream.rs` / `src/proxies/lulustream.rs` redirect proxies (`build_proxy_url(&options, "doodstream"/"lulustream", &strip_url_scheme(link))`) — no new proxy code was written. Both proxies resolve correctly to real signed CDN URLs (verified against live data), but neither CDN's final edge is reachable from this repo's dev sandbox: Lulustream's `*.tnmr.org` edge 403s datacenter/proxy IPs regardless of UA or TLS impersonation, and Doodstream's final `*.dood.video` edge resolves to a loopback/refused address from non-residential egress — both are IP-reputation-style CDN blocks, not code defects (same class of sandbox limitation already documented for `camsoda`/`hentaihaven`/`animeidhentai`), so `check.py` allowlists both by suffix (`_CF_PROTECTED_SUFFIXES`). No thumbnail proxy needed (`cdn.ripnsfw.com/thumbs/...` is directly hotlinkable, no CF/referer gate). No `/api/uploaders` (model pages exist but have no dedicated profile/stats schema confirmed). |
| `rule34gen` | `ai` | no | no | AI group example. | | `rule34gen` | `ai` | no | no | AI group example. |
| `rule34video` | `hentai-animation` | no | no | Hentai group example. | | `rule34video` | `hentai-animation` | no | no | Hentai group example. |
| `sextb` | `jav` | no | no | JAV family provider. | | `sextb` | `jav` | no | no | JAV family provider. sextb.net is Cloudflare-gated, so listing/pagination markdown is fetched through a hand-built Jina mirror URL (`mirror_url()`); the target is already an absolute `https://sextb.net/...` URL, so the scheme must be stripped before prepending the `https://r.jina.ai/http://` prefix — concatenating them directly (the pre-fix bug) produces a malformed `.../http://https://...` URL that Jina rejects outright. |
| `shooshtime` | `onlyfans` | no | yes | Redirect proxy plus dedicated media route. | | `shooshtime` | `onlyfans` | no | yes | Redirect proxy plus dedicated media route. Listing cards are `div.card` (card→`a.title` for the link/title, `a.image img` for thumb/preview, `div.duration`, `div.list div.box` for views+date, `div.name a.text` for uploader) — the site redesigned from an older `div.item.thumb`/`span.description` layout, so re-verify these selectors against `{BASE_URL}/videos/` (the real default-sort URL, not the bare homepage) if the feed ever goes empty again. |
| `spankbang` | `mainstream-tube` | no | yes | Best template for redirect proxy plus anti-bot fetches. | | `spankbang` | `mainstream-tube` | no | yes | Best template for redirect proxy plus anti-bot fetches. |
| `thaiporntv` | `mainstream-tube` | no | yes | Decodes `data-enc` attribute for proxied HLS playback. | | `thaiporntv` | `mainstream-tube` | no | yes | Decodes `data-enc` attribute for proxied HLS playback. |
| `supjav` | `jav` | no | yes | JAV/HLS provider; detail page URLs for `video.url`, proxied HLS format URLs via `/proxy/supjav/...`. | | `supjav` | `jav` | no | yes | JAV/HLS provider; detail page URLs for `video.url`, proxied HLS format URLs via `/proxy/supjav/...`. Fetches through a `curl_cffi` Python subprocess impersonating **Safari** (`safari18_0`) — Cloudflare's bot management (`cf-mitigated: challenge`, Turnstile) blocks curl_cffi's `chrome`/`firefox` impersonation profiles outright (403) but passes Safari cleanly, the same pattern used by melonstube. |
| `sxyprn` | `mainstream-tube` | no | yes | Redirect proxy helper usage. | | `sxyprn` | `mainstream-tube` | no | yes | Redirect proxy helper usage. |
| `tnaflix` | `mainstream-tube` | no | no | Mainstream tube provider. | | `tnaflix` | `mainstream-tube` | no | no | Mainstream tube provider. |
| `tokyomotion` | `jav` | no | no | JAV/tube hybrid. | | `tokyomotion` | `jav` | no | no | JAV/tube hybrid. |
@@ -86,7 +86,7 @@ This is the current implementation inventory as of this snapshot of the repo. Us
| `xnxx` | `mainstream-tube` | no | no | HTML scraper for xnxx.com (10M+ videos); unified card parser handles two formats: `div.thumb-block[data-eid]` (search) and `div.thumb-block.video[data-video='{"id":...}']` (hits); eid extracted from `/video-{eid}/{slug}` URL path; thumbnails at `thumb-cdn77.xnxx-cdn.com` and `thumbs-gcore.xnxx-cdn.com` (no proxy, no Referer needed); 0-indexed pagination (page 1 = `/hits`, page N = `/hits/{N-1}`); default feed is `/hits` (most-viewed — xnxx has no chronological listing); search via `/search/{slug}` (works for keywords and tags); supports `tag:`, `cat:`, `category:` query shortcuts; yt-dlp resolves `video.url` natively (XNXX extractor, returns 4-7 HLS formats); no proxy needed. | | `xnxx` | `mainstream-tube` | no | no | HTML scraper for xnxx.com (10M+ videos); unified card parser handles two formats: `div.thumb-block[data-eid]` (search) and `div.thumb-block.video[data-video='{"id":...}']` (hits); eid extracted from `/video-{eid}/{slug}` URL path; thumbnails at `thumb-cdn77.xnxx-cdn.com` and `thumbs-gcore.xnxx-cdn.com` (no proxy, no Referer needed); 0-indexed pagination (page 1 = `/hits`, page N = `/hits/{N-1}`); default feed is `/hits` (most-viewed — xnxx has no chronological listing); search via `/search/{slug}` (works for keywords and tags); supports `tag:`, `cat:`, `category:` query shortcuts; yt-dlp resolves `video.url` natively (XNXX extractor, returns 4-7 HLS formats); no proxy needed. |
| `xgroovy` | `mainstream-tube` | no | no | KVS (Kernel Video Sharing)-style HTML scraper for xgroovy.com; card selector `div.item[data-video-id]` with `data-views`/`data-rating` as clean numeric attributes directly on the card (no text-parsing needed, unlike xhamster); title/url from `a.popito[href] > strong.title`; thumbnails at `i.xgroovy.com` (no proxy, no Referer needed) from `img.thumb[data-jpg|src]`, preview clip from `img[data-preview]`; duration rendered as free text ("N min"/"N sec", not colon-separated) via a bespoke `parse_duration` helper; path-segment pagination (`/new/` = page 1, `/new/2/` = page 2, unlike xhamster's `?page=N` query string); feeds: `/new/` (default), categories via `/categories/{slug}/` (73 hardcoded categories as `categories` option), pornstars via `/pornstars/{slug}/`, channels via `/channels/{slug}/`; search via `/search/{dashed-query}/` (multi-word queries are lowercased and dash-joined); uploader kind inferred from URL path (`/pornstars/` vs `/channels/`) via `div.author-link > a`; supports `cat:`/`category:` query shortcuts plus static category name matching; yt-dlp resolves `video.url` via the generic html5-embed extractor (no dedicated xgroovy extractor, but resolves cleanly to a 1080p format); no proxy needed. | | `xgroovy` | `mainstream-tube` | no | no | KVS (Kernel Video Sharing)-style HTML scraper for xgroovy.com; card selector `div.item[data-video-id]` with `data-views`/`data-rating` as clean numeric attributes directly on the card (no text-parsing needed, unlike xhamster); title/url from `a.popito[href] > strong.title`; thumbnails at `i.xgroovy.com` (no proxy, no Referer needed) from `img.thumb[data-jpg|src]`, preview clip from `img[data-preview]`; duration rendered as free text ("N min"/"N sec", not colon-separated) via a bespoke `parse_duration` helper; path-segment pagination (`/new/` = page 1, `/new/2/` = page 2, unlike xhamster's `?page=N` query string); feeds: `/new/` (default), categories via `/categories/{slug}/` (73 hardcoded categories as `categories` option), pornstars via `/pornstars/{slug}/`, channels via `/channels/{slug}/`; search via `/search/{dashed-query}/` (multi-word queries are lowercased and dash-joined); uploader kind inferred from URL path (`/pornstars/` vs `/channels/`) via `div.author-link > a`; supports `cat:`/`category:` query shortcuts plus static category name matching; yt-dlp resolves `video.url` via the generic html5-embed extractor (no dedicated xgroovy extractor, but resolves cleanly to a 1080p format); no proxy needed. |
| `xhamster` | `mainstream-tube` | no | no | HTML scraper for xhamster.com; card selector `div[data-video-type="video"]` with `data-video-id`; thumbnails via `img[data-role="thumb-preview-img"]` at `ic-vt-nss.xhcdn.com` (no proxy, no Referer needed); pagination via `?page=N` query param (browse feeds use infinite-scroll so only search reliably returns different content per page); feeds: `/newest` (default), `/most-viewed`, `/best`; categories via `/categories/{slug}`; channels via `/channels/{slug}`; 43 hardcoded categories as `categories` option; uploader type inferred from URL path (`/channels/` → channel, `/creators/` → creator, `/pornstars/` → pornstar); supports `cat:`/`category:` and `channel:` query shortcuts, plus static category name matching; preview mp4 clips from `data-previewvideo` attribute; yt-dlp resolves `video.url` natively (xHamster extractor, 28 formats); no proxy needed. | | `xhamster` | `mainstream-tube` | no | no | HTML scraper for xhamster.com; card selector `div[data-video-type="video"]` with `data-video-id`; thumbnails via `img[data-role="thumb-preview-img"]` at `ic-vt-nss.xhcdn.com` (no proxy, no Referer needed); pagination via `?page=N` query param (browse feeds use infinite-scroll so only search reliably returns different content per page); feeds: `/newest` (default), `/most-viewed`, `/best`; categories via `/categories/{slug}`; channels via `/channels/{slug}`; 43 hardcoded categories as `categories` option; uploader type inferred from URL path (`/channels/` → channel, `/creators/` → creator, `/pornstars/` → pornstar); supports `cat:`/`category:` and `channel:` query shortcuts, plus static category name matching; preview mp4 clips from `data-previewvideo` attribute; yt-dlp resolves `video.url` natively (xHamster extractor, 28 formats); no proxy needed. |
| `camsoda` | `live-cams` | no | no | Live-cam provider for camsoda.com (chaturbate-style — `live` performers streaming now, `video.url` = the room page, `is_live=true`, no `formats`). camsoda.com is hard Cloudflare-protected: direct requests and yt-dlp both get HTTP 403, and FlareSolverr was unreliable during development, so the live-browse API is reached through the shared requester's Jina mirror fallback (`r.jina.ai/http://...`, `X-Return-Format: html`); Jina rate-limits per IP, so the provider caches each fetched feed URL for 60s (and serves stale items on a 429 rather than emptying the feed), and a single-provider build (`HOT_TUB_PROVIDER=camsoda`) validates most cleanly (one fetch at a time). Endpoint (found in the non-CF static `main.js` bundle): `GET https://www.camsoda.com/api/v1/browse/react{route}?p=N` returning a body with a top-level `userList` array (Jina wraps it in `<pre>`, so the provider slices out the `{...}` and parses it with `serde_json::Value`, like the chaturbate provider). Per-cam fields: `username`→id + room URL (`/{username}`), `subjectText`→title (html-decoded, falls back to `displayName`), `displayName`→uploader, `connectionCount`→views (string or number tolerated), `thumbUrl`→thumb (direct `media.livemediahost.com` CDN, no proxy/referer), `status` (skip `offline`), `vr`/`private` surfaced as tags. Category option `category` uses verified `browse/react` routes — `all`(featured)/`girls`/`trans`/`couples`/`voyeur-cams`/`new` (`/male` is NOT a path route, camsoda gates male via `gender-hide`); `cat:`/`category:` prefixes and a bare keyword matching a category id route there too. Search: `GET browse/react/search/{dashed-query}?sortByConnection=1` (single connection-sorted result set, no real paging). Playback: `video.url` is the live room page; the room and the token-gated edge HLS (`*.livemediahost.com`) are both Cloudflare-protected, so HLS can't be resolved server-side and no `formats` are populated — yt-dlp has a `Camsoda` live extractor that resolves the room on a non-CF-blocked client, and `check.py` reports the sandbox's CF 403s as expected warnings (`www.camsoda.com` is in its CF allowlist), not errors. The earlier recorded-`/media` JSON scrape was replaced because clips were token-gated/non-playable; live cams are the site's actual product. No proxy needed. | | `camsoda` | `live-cams` | no | no | Live-cam provider for camsoda.com (chaturbate-style — `live` performers streaming now, `video.url` = the room page, `is_live=true`, no `formats`). camsoda.com is hard Cloudflare-protected: direct requests and yt-dlp both get HTTP 403, and FlareSolverr was unreliable during development, so the live-browse API is reached through the shared requester's Jina mirror fallback (`r.jina.ai/http://...`, `X-Return-Format: html`); Jina rate-limits per IP, so the provider caches each fetched feed URL for 60s (and serves stale items on a 429 rather than emptying the feed), and a single-provider build (`HOT_TUB_PROVIDER=camsoda`) validates most cleanly (one fetch at a time). Endpoint (found in the non-CF static `main.js` bundle): `GET https://www.camsoda.com/api/v1/browse/react{route}?p=N` returning a body with a top-level `userList` array (Jina wraps it in `<pre>`, so the provider scans for the first balanced top-level `{...}` object — tracking string/escape state rather than just pairing the first `{` with the last `}` in the body, since a naive last-`}` pairing pulls in any trailing markup the wrapper appends after `</pre>` and made `serde_json` fail with "key must be a string" — then parses it with `serde_json::Value`, like the chaturbate provider). Per-cam fields: `username`→id + room URL (`/{username}`), `subjectText`→title (html-decoded, falls back to `displayName`), `displayName`→uploader, `connectionCount`→views (string or number tolerated), `thumbUrl`→thumb (direct `media.livemediahost.com` CDN, no proxy/referer), `status` (skip `offline`), `vr`/`private` surfaced as tags. Category option `category` uses verified `browse/react` routes — `all`(featured)/`girls`/`trans`/`couples`/`voyeur-cams`/`new` (`/male` is NOT a path route, camsoda gates male via `gender-hide`); `cat:`/`category:` prefixes and a bare keyword matching a category id route there too. Search: `GET browse/react/search/{dashed-query}?sortByConnection=1` (single connection-sorted result set, no real paging). Playback: `video.url` is the live room page; the room and the token-gated edge HLS (`*.livemediahost.com`) are both Cloudflare-protected, so HLS can't be resolved server-side and no `formats` are populated — yt-dlp has a `Camsoda` live extractor that resolves the room on a non-CF-blocked client, and `check.py` reports the sandbox's CF 403s as expected warnings (`www.camsoda.com` is in its CF allowlist), not errors. The earlier recorded-`/media` JSON scrape was replaced because clips were token-gated/non-playable; live cams are the site's actual product. No proxy needed. |
| `xvideos` | `mainstream-tube` | no | no | HTML scraper for xvideos.com; handles two card formats: homepage (`div.thumb-block[data-id][data-eid]`) uses `p.title a[title]` + `data-pvv` on img, best-of-month page uses `div.thumb-block.video[data-video=JSON]` with `div.title a` text + `previewVideo` JSON key; thumbnails at `thumb-cdn77.xvideos-cdn.com` / `thumbs-gcore.xvideos-cdn.com` (no proxy needed); latest: `/` (page 1) / `/new/{N-1}` (page N≥2); best-of-month: `/best/{YYYY-MM}` (previous calendar month), page N: `/best/{YYYY-MM}/{N-1}`; search: `/?k={query}` / `/?k={query}&p={N-1}` (0-indexed); tag shortcuts: `/tags/{slug}/{N-1}`; category shortcuts: `/c/{Name}-{ID}/{N-1}` (38 hardcoded categories); `cat:`, `tag:`, `uploader:` query prefix routing; yt-dlp resolves `video.url` natively (XVideos extractor → HLS formats); CDN preview mp4 in `preview` field; no proxy needed. | | `xvideos` | `mainstream-tube` | no | no | HTML scraper for xvideos.com; handles two card formats: homepage (`div.thumb-block[data-id][data-eid]`) uses `p.title a[title]` + `data-pvv` on img, best-of-month page uses `div.thumb-block.video[data-video=JSON]` with `div.title a` text + `previewVideo` JSON key; thumbnails at `thumb-cdn77.xvideos-cdn.com` / `thumbs-gcore.xvideos-cdn.com` (no proxy needed); latest: `/` (page 1) / `/new/{N-1}` (page N≥2); best-of-month: `/best/{YYYY-MM}` (previous calendar month), page N: `/best/{YYYY-MM}/{N-1}`; search: `/?k={query}` / `/?k={query}&p={N-1}` (0-indexed); tag shortcuts: `/tags/{slug}/{N-1}`; category shortcuts: `/c/{Name}-{ID}/{N-1}` (38 hardcoded categories); `cat:`, `tag:`, `uploader:` query prefix routing; yt-dlp resolves `video.url` natively (XVideos extractor → HLS formats); CDN preview mp4 in `preview` field; no proxy needed. |
| `wowxxx` | `studio-network` | no | no | HTML scraper for wow.xxx premium aggregator; default feed `/latest-updates/`, page 2 `/{N}/` suffix (for example `/latest-updates/2/`), search `/search/{query}/relevance/` with the same page suffix; supports `site:`/`studio:`/`network:`/`model:`/`pornstar:`/`tag:`/`cat:` query shortcuts to direct archive routes; list cards expose preview clips (`cast.wow.xxx/preview/*.mp4`), thumbnails (`img.wow.xxx/.../medium@2x/1.jpg`), duration, rating, views, site (as uploader), and model tags; `video.url` is the detail page URL and yt-dlp resolves HTML5 MP4 formats dynamically; no proxy needed. | | `wowxxx` | `studio-network` | no | no | HTML scraper for wow.xxx premium aggregator; default feed `/latest-updates/`, page 2 `/{N}/` suffix (for example `/latest-updates/2/`), search `/search/{query}/relevance/` with the same page suffix; supports `site:`/`studio:`/`network:`/`model:`/`pornstar:`/`tag:`/`cat:` query shortcuts to direct archive routes; list cards expose preview clips (`cast.wow.xxx/preview/*.mp4`), thumbnails (`img.wow.xxx/.../medium@2x/1.jpg`), duration, rating, views, site (as uploader), and model tags; `video.url` is the detail page URL and yt-dlp resolves HTML5 MP4 formats dynamically; no proxy needed. |
| `xlecx` | `hentai-animation` | no | no | DLE (DataLife Engine)-style HTML scraper for xlecx.one, following the `thaiporntv` template. Latest feed `/f/cat={id}/sort={mode}/order=desc/` (page N≥2 appends `page/{N}/`); `sort` options new/popular/commented/rated map to DLE's `date`/`news_read`/`comm_num`/`rating` sort params; `category` option is the 2 hardcoded top-level archives (`videos`, `hentai-video`). Search via `/index.php?do=search&subaction=search&story={query}` (page N adds `search_start={N}`). Tag shortcut (`#slug` query) routes to `/tags/{slug}/` (paginated via `/tags/{slug}/page/{N}/`); the ~1,044-entry tag catalog is background-loaded once at startup from `/tags/` (`.tags-list1 a[href*='/tags/']`, absolute-URL anchors) into an `Arc<RwLock<Vec<FilterOption>>>` and exposed as the `filter`/"Tags" `ChannelOption` — note this option (and any `categories`) is deliberately stripped from the live `/api/status` response by `should_hide_status_option`/`decorate_channel` in `providers/mod.rs` (any `id=="filter"` option titled "Tags" is hidden site-wide), so its absence there is expected, not a bug; it is still honored by `/api/videos`. Cards parsed from `h3.thumb__title` + `img` inside standard DLE thumb blocks; id extracted from the `/{id}-{slug}.html` URL suffix. Thumbnails are direct `xlecx.one/uploads/posts/...` webp (no proxy/Referer needed). `video.url` is the detail page URL, which yt-dlp resolves natively via the generic HTML5 embed extractor; no `formats`/proxy needed. No `/api/uploaders` (no stable uploader identity on the site). | | `xlecx` | `hentai-animation` | no | no | DLE (DataLife Engine)-style HTML scraper for xlecx.one, following the `thaiporntv` template. Latest feed `/f/cat={id}/sort={mode}/order=desc/` (page N≥2 appends `page/{N}/`); `sort` options new/popular/commented/rated map to DLE's `date`/`news_read`/`comm_num`/`rating` sort params; `category` option is the 2 hardcoded top-level archives (`videos`, `hentai-video`). Search via `/index.php?do=search&subaction=search&story={query}` (page N adds `search_start={N}`). Tag shortcut (`#slug` query) routes to `/tags/{slug}/` (paginated via `/tags/{slug}/page/{N}/`); the ~1,044-entry tag catalog is background-loaded once at startup from `/tags/` (`.tags-list1 a[href*='/tags/']`, absolute-URL anchors) into an `Arc<RwLock<Vec<FilterOption>>>` and exposed as the `filter`/"Tags" `ChannelOption` — note this option (and any `categories`) is deliberately stripped from the live `/api/status` response by `should_hide_status_option`/`decorate_channel` in `providers/mod.rs` (any `id=="filter"` option titled "Tags" is hidden site-wide), so its absence there is expected, not a bug; it is still honored by `/api/videos`. Cards parsed from `h3.thumb__title` + `img` inside standard DLE thumb blocks; id extracted from the `/{id}-{slug}.html` URL suffix. Thumbnails are direct `xlecx.one/uploads/posts/...` webp (no proxy/Referer needed). `video.url` is the detail page URL, which yt-dlp resolves natively via the generic HTML5 embed extractor; no `formats`/proxy needed. No `/api/uploaders` (no stable uploader identity on the site). |

View File

@@ -169,11 +169,43 @@ impl CamsodaProvider {
} }
/// Extract the JSON object from a body that may be wrapped in HTML by the /// Extract the JSON object from a body that may be wrapped in HTML by the
/// Jina mirror (`<pre>{...}</pre>`) or returned raw. /// Jina mirror (`<pre>{...}</pre>`) or returned raw. Scans for the
/// balanced top-level object starting at the first `{` instead of just
/// pairing it with the last `}` in the body — the naive approach breaks
/// when the wrapper has trailing content of its own (e.g. a tracking
/// `<script>` block after `</pre>` containing an unquoted-key JS object
/// literal), which pulls that garbage into the "JSON" and makes
/// `serde_json` fail with errors like "key must be a string".
fn extract_json(body: &str) -> Option<&str> { fn extract_json(body: &str) -> Option<&str> {
let start = body.find('{')?; let start = body.find('{')?;
let end = body.rfind('}')?; let mut depth = 0i32;
(end > start).then(|| &body[start..=end]) let mut in_string = false;
let mut escaped = false;
for (offset, ch) in body[start..].char_indices() {
if in_string {
if escaped {
escaped = false;
} else if ch == '\\' {
escaped = true;
} else if ch == '"' {
in_string = false;
}
continue;
}
match ch {
'"' => in_string = true,
'{' => depth += 1,
'}' => {
depth -= 1;
if depth == 0 {
let end = start + offset + ch.len_utf8();
return Some(&body[start..end]);
}
}
_ => {}
}
}
None
} }
/// Parse the `browse/react` JSON (top-level `userList`) into live /// Parse the `browse/react` JSON (top-level `userList`) into live
@@ -411,6 +443,20 @@ mod tests {
assert!(tags.contains(&"VR".to_string())); assert!(tags.contains(&"VR".to_string()));
} }
#[test]
fn ignores_trailing_garbage_after_the_json_object() {
// A Jina/FlareSolverr wrapper can append its own markup after the
// real JSON (e.g. a tracking <script> block with an unquoted-key JS
// object literal). The naive first-`{`/last-`}` span used to pull
// that in and made serde_json fail with "key must be a string".
let body = r#"<html><body><pre>{"perPageCount":1,"totalCount":1,"userList":[
{"id":1,"username":"theowonder","displayName":"Theo Wonder","connectionCount":"69","status":"online","subjectText":"hi","thumbUrl":"https://media.livemediahost.com/thumbs/199/theowonder.jpg","vr":false}
]}</pre></body><script>ga({event:'view'})</script></html>"#;
let items = CamsodaProvider::parse_items(body).unwrap();
assert_eq!(items.len(), 1);
assert_eq!(items[0].id, "theowonder");
}
#[test] #[test]
fn picks_target() { fn picks_target() {
assert_eq!( assert_eq!(

View File

@@ -265,53 +265,99 @@ impl Hdporn92Provider {
} }
} }
/// Decodes the P.A.C.K.E.R. `eval(function(p,a,c,k,e,d){...}(...))` block in /// Decodes every P.A.C.K.E.R. `eval(function(p,a,c,k,e,d){...}(...))` block
/// the morencius embed page and returns the deobfuscated JavaScript source. /// found in the morencius embed page and returns the deobfuscated JS
fn decode_packered(source: &str) -> Option<String> { /// source for each. The embed page carries more than one such block (an
/// ad/VAST setup block plus the actual jwplayer setup block), so callers
/// must scan all of them for the data they need rather than assume the
/// first one is the relevant one.
fn decode_packered_blocks(source: &str) -> Vec<String> {
// Match: eval(function(p,a,c,k,e,d){...}('P',A,C,'K'.split('|') // Match: eval(function(p,a,c,k,e,d){...}('P',A,C,'K'.split('|')
// We capture P, A, C, and the dictionary K. // We capture P, A, C, and the dictionary K.
let re = Regex::new( let Ok(re) = Regex::new(
r#"eval\(function\(p,a,c,k,e,d\)\{.*?\}\('([\s\S]*?)',(\d+),(\d+),'([\s\S]*?)'\.split\('\|'\)\)\)"#, r#"eval\(function\(p,a,c,k,e,d\)\{.*?\}\('([\s\S]*?)',(\d+),(\d+),'([\s\S]*?)'\.split\('\|'\)\)\)"#,
) ) else {
.ok()?; return Vec::new();
let caps = re.captures(source)?; };
let p = caps.get(1)?.as_str();
let a: u32 = caps.get(2)?.as_str().parse().ok()?;
let c: usize = caps.get(3)?.as_str().parse().ok()?;
let k: Vec<&str> = caps.get(4)?.as_str().split('|').collect();
// P.A.C.K.E.R. deobfuscation: iterate c-1..=0; replace \b<token>\b with re.captures_iter(source)
// dictionary[i] (the original JS uses a `\w+` lookalike via the `e` .filter_map(|caps| {
// parameter, but in practice every token is a base-A digit string). let p = caps.get(1)?.as_str();
let mut result = p.to_string(); let a: u32 = caps.get(2)?.as_str().parse().ok()?;
for i in (0..c).rev() { let c: usize = caps.get(3)?.as_str().parse().ok()?;
let value = k.get(i).copied().unwrap_or(""); let k: Vec<&str> = caps.get(4)?.as_str().split('|').collect();
if value.is_empty() {
continue; // P.A.C.K.E.R. deobfuscation: iterate c-1..=0; replace
} // \b<token>\b with dictionary[i] (the original JS uses a
let token = to_base(i, a); // `\w+` lookalike via the `e` parameter, but in practice
// Build a word-boundary regex for the token. Base-A tokens are // every token is a base-A digit string).
// always made of [0-9a-z], so \b is a real word boundary. let mut result = p.to_string();
let token_re = Regex::new(&format!(r"\b{}\b", regex::escape(&token))).ok()?; for i in (0..c).rev() {
result = token_re.replace_all(&result, value).into_owned(); let value = k.get(i).copied().unwrap_or("");
} if value.is_empty() {
Some(result) continue;
}
let token = to_base(i, a);
// Build a word-boundary regex for the token. Base-A
// tokens are always made of [0-9a-z], so \b is a real
// word boundary.
let token_re =
Regex::new(&format!(r"\b{}\b", regex::escape(&token))).ok()?;
result = token_re.replace_all(&result, value).into_owned();
}
Some(result)
})
.collect()
} }
/// Pulls the morencius `/stream/.../master.m3u8` URL out of the decoded JS. /// Decodes the first P.A.C.K.E.R. block found in `source`. Kept for
fn extract_hls4_url(decoded_js: &str) -> Option<String> { /// callers/tests that only care about a single block; production code
// The decoded JS has: "hls4":"/stream/<token>/<secret>/<expiry>/<file_id>/master.m3u8" /// that needs to find the block with a specific key (e.g. `hls4`) should
// The value is a path (no scheme), so we prefix it. /// use [`Self::decode_packered_blocks`] and scan all of them instead.
let re = Regex::new(r#""hls4"\s*:\s*"([^"]+)""#).ok()?; fn decode_packered(source: &str) -> Option<String> {
let caps = re.captures(decoded_js)?; Self::decode_packered_blocks(source).into_iter().next()
let path = caps.get(1)?.as_str(); }
if path.starts_with("http") {
Some(path.to_string()) /// Pulls the morencius HLS URL out of the decoded JS. The player itself
} else if path.starts_with('/') { /// picks `links.hls4 || links.hls3 || links.hls2` (not every tier is
Some(format!("https://morencius.com{path}")) /// always populated, and hls4 is frequently absent), so mirror that
} else { /// priority order. At least one tier has been observed serving a `.txt`
Some(format!("https://morencius.com/{path}")) /// decoy instead of a real `.m3u8` playlist, so within that priority
} /// order prefer a candidate that actually looks like a stream URL.
fn extract_hls_url(decoded_js: &str) -> Option<String> {
let re = Regex::new(r#""(hls[234])"\s*:\s*"([^"]+)""#).ok()?;
let candidates: Vec<(String, String)> = re
.captures_iter(decoded_js)
.filter_map(|caps| {
let tier = caps.get(1)?.as_str().to_string();
let raw = caps.get(2)?.as_str().replace("\\/", "/");
Some((tier, raw))
})
.collect();
let normalize = |path: &str| {
if path.starts_with("http") {
path.to_string()
} else if path.starts_with('/') {
format!("https://morencius.com{path}")
} else {
format!("https://morencius.com/{path}")
}
};
const TIERS: [&str; 3] = ["hls4", "hls3", "hls2"];
TIERS
.iter()
.filter_map(|tier| candidates.iter().find(|(t, _)| t == tier))
.map(|(_, raw)| normalize(raw))
.find(|url| url.to_lowercase().ends_with(".m3u8"))
.or_else(|| {
TIERS
.iter()
.filter_map(|tier| candidates.iter().find(|(t, _)| t == tier))
.map(|(_, raw)| normalize(raw))
.next()
})
} }
/// Pulls the integer duration (in seconds, may be decimal) from the decoded /// Pulls the integer duration (in seconds, may be decimal) from the decoded
@@ -336,11 +382,21 @@ impl Hdporn92Provider {
// Step 2: fetch the morencius embed page (referer = the hdporn92 detail). // Step 2: fetch the morencius embed page (referer = the hdporn92 detail).
let player_html = Self::fetch_html(options, &embed_url, &stub.url).await?; let player_html = Self::fetch_html(options, &embed_url, &stub.url).await?;
let decoded = Self::decode_packered(&player_html) let decoded_blocks = Self::decode_packered_blocks(&player_html);
.ok_or_else(|| Error::from(format!("no eval block in {embed_url}")))?; if decoded_blocks.is_empty() {
let stream_path = Self::extract_hls4_url(&decoded) return Err(Error::from(format!("no eval block in {embed_url}")));
.ok_or_else(|| Error::from(format!("no hls4 url in decoded JS of {embed_url}")))?; }
let duration = Self::extract_duration(&decoded).unwrap_or(0); // A page may carry multiple P.A.C.K.E.R. blocks (e.g. an ad/VAST
// setup block); only one decodes to JS containing the player's
// hls2/3/4 stream keys, so scan all of them rather than assuming
// the first.
let decoded = decoded_blocks
.iter()
.find(|block| Self::extract_hls_url(block).is_some())
.ok_or_else(|| Error::from(format!("no hls url in decoded JS of {embed_url}")))?;
let stream_path = Self::extract_hls_url(decoded)
.ok_or_else(|| Error::from(format!("no hls url in decoded JS of {embed_url}")))?;
let duration = Self::extract_duration(decoded).unwrap_or(0);
// Build the format. morencius serves the m3u8 only when Referer is the // Build the format. morencius serves the m3u8 only when Referer is the
// embed URL — without it, the response is 403. // embed URL — without it, the response is 403.
@@ -568,7 +624,7 @@ mod tests {
.expect("deobfuscator should return Some"); .expect("deobfuscator should return Some");
// Sanity: the decoded text contains the hls4 key. // Sanity: the decoded text contains the hls4 key.
assert!(decoded.contains("\"hls4\""), "decoded: {decoded}"); assert!(decoded.contains("\"hls4\""), "decoded: {decoded}");
let url = Hdporn92Provider::extract_hls4_url(&decoded) let url = Hdporn92Provider::extract_hls_url(&decoded)
.expect("hls4 should be present after decode"); .expect("hls4 should be present after decode");
// The path was /stream/abc/secret/123/456/master.m3u8. // The path was /stream/abc/secret/123/456/master.m3u8.
assert_eq!(url, "https://morencius.com/stream/abc/secret/123/456/master.m3u8"); assert_eq!(url, "https://morencius.com/stream/abc/secret/123/456/master.m3u8");
@@ -603,18 +659,19 @@ mod tests {
"src/providers/testdata/hdporn92_morencius.html", "src/providers/testdata/hdporn92_morencius.html",
) )
.expect("test fixture should exist"); .expect("test fixture should exist");
let decoded = Hdporn92Provider::decode_packered(&html) // The page carries several P.A.C.K.E.R. blocks (ad setup, player
.expect("real morencius HTML should be deobfuscable"); // setup, ...); only the player block decodes to JS containing hls4.
// The decoded JS contains the hls4 value. let blocks = Hdporn92Provider::decode_packered_blocks(&html);
let url = Hdporn92Provider::extract_hls4_url(&decoded) assert!(!blocks.is_empty(), "real morencius HTML should be deobfuscable");
.expect("hls4 should be present after decode"); let decoded = blocks
assert!( .iter()
url.starts_with("https://morencius.com/stream/"), .find(|block| Hdporn92Provider::extract_hls_url(block).is_some())
"got {url}" .expect("one decoded block should contain an hls2/3/4 key");
); let url = Hdporn92Provider::extract_hls_url(decoded)
assert!(url.ends_with("/master.m3u8"), "got {url}"); .expect("an hls url should be present after decode");
assert!(url.starts_with("https://"), "got {url}");
// The duration is also present. // The duration is also present.
let duration = Hdporn92Provider::extract_duration(&decoded); let duration = Hdporn92Provider::extract_duration(decoded);
assert!(duration.unwrap_or(0) > 0, "duration should be > 0"); assert!(duration.unwrap_or(0) > 0, "duration should be > 0");
} }
} }

View File

@@ -570,20 +570,30 @@ impl HentaimamaProvider {
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos"); let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
let html = Self::fetch_html(&mut requester, &list_url, BASE_URL).await?; let html = Self::fetch_html(&mut requester, &list_url, BASE_URL).await?;
let candidate_count;
let items = match target { let items = match target {
Target::Latest => { Target::Latest => {
let cards = Self::parse_episode_cards(&html)?; let cards = Self::parse_episode_cards(&html)?;
stream::iter(cards.into_iter().take(per_page.max(1)).map(|entry| { candidate_count = cards.len();
// Resolve every card on the page rather than just the first
// `per_page` - the site sometimes bulk-lists a whole series'
// episodes as "Coming soon" placeholders (get_player_contents
// returns `[]`, no error), which would otherwise silently
// wipe out the entire feed if those happen to sort first.
let mut items = stream::iter(cards.into_iter().map(|entry| {
let mut req = requester.clone(); let mut req = requester.clone();
async move { Self::build_item_from_episode_card(&mut req, &entry).await } async move { Self::build_item_from_episode_card(&mut req, &entry).await }
})) }))
.buffer_unordered(6) .buffer_unordered(6)
.filter_map(|item| async move { item }) .filter_map(|item| async move { item })
.collect::<Vec<_>>() .collect::<Vec<_>>()
.await .await;
items.truncate(per_page.max(1));
items
} }
_ => { _ => {
let urls = Self::parse_series_card_urls(&html)?; let urls = Self::parse_series_card_urls(&html)?;
candidate_count = urls.len();
stream::iter(urls.into_iter().take(per_page.max(1)).map(|url| { stream::iter(urls.into_iter().take(per_page.max(1)).map(|url| {
let mut req = requester.clone(); let mut req = requester.clone();
async move { Self::resolve_series_to_items(&mut req, &url).await } async move { Self::resolve_series_to_items(&mut req, &url).await }
@@ -595,6 +605,16 @@ impl HentaimamaProvider {
} }
}; };
if items.is_empty() && candidate_count > 0 {
report_provider_error_background(
CHANNEL_ID,
"get.all_candidates_failed",
&format!(
"{candidate_count} candidate card(s) parsed from {list_url} but none resolved to a playable item"
),
);
}
if !items.is_empty() { if !items.is_empty() {
cache.insert(list_url, items.clone()); cache.insert(list_url, items.clone());
} }

View File

@@ -243,7 +243,15 @@ impl SextbProvider {
} }
fn mirror_url(&self, url: &str) -> String { fn mirror_url(&self, url: &str) -> String {
format!("{MIRROR_PREFIX}{url}") // `url` is already an absolute `https://sextb.net/...` URL, and
// MIRROR_PREFIX already supplies a scheme — concatenating them
// directly produces a malformed `.../http://https://...` target
// that Jina rejects, so strip the scheme first.
let stripped = url
.strip_prefix("https://")
.or_else(|| url.strip_prefix("http://"))
.unwrap_or(url);
format!("{MIRROR_PREFIX}{stripped}")
} }
fn slugify(value: &str) -> String { fn slugify(value: &str) -> String {
@@ -1258,6 +1266,15 @@ mod tests {
); );
} }
#[test]
fn builds_jina_mirror_url_without_double_scheme() {
let url = provider().mirror_url("https://sextb.net/censored?genre=all");
assert_eq!(
url,
"https://r.jina.ai/http://sextb.net/censored?genre=all"
);
}
#[test] #[test]
fn builds_search_page_two_url() { fn builds_search_page_two_url() {
let url = provider().build_url_for_target( let url = provider().build_url_for_target(

View File

@@ -811,13 +811,13 @@ impl ShooshtimeProvider {
fn parse_list_videos(&self, html: &str) -> Result<Vec<VideoItem>> { fn parse_list_videos(&self, html: &str) -> Result<Vec<VideoItem>> {
let document = Html::parse_document(html); let document = Html::parse_document(html);
let card_selector = Self::selector("div.item.thumb")?; let card_selector = Self::selector("div.card")?;
let video_link_selector = Self::selector("a[href*=\"/videos/\"]")?; let video_link_selector = Self::selector("a.title")?;
let image_selector = Self::selector("img")?; let image_selector = Self::selector("a.image img")?;
let duration_selector = Self::selector("span.duration")?; let duration_selector = Self::selector("div.duration")?;
let title_selector = Self::selector("span.description, span.description.secondary")?; let title_selector = Self::selector("a.title")?;
let info_selector = Self::selector("ul.info li.val")?; let info_selector = Self::selector("div.list div.box")?;
let uploader_selector = Self::selector("ul.info a[href*=\"/members/\"]")?; let uploader_selector = Self::selector("div.name a.text")?;
let mut items = Vec::new(); let mut items = Vec::new();

View File

@@ -587,7 +587,7 @@ if referer:
response = requests.get( response = requests.get(
url, url,
impersonate="chrome", impersonate="safari18_0",
timeout=30, timeout=30,
allow_redirects=True, allow_redirects=True,
headers=headers, headers=headers,

File diff suppressed because one or more lines are too long