docs and prompt

2026-04-10 22:57:27 +00:00
parent 57eb2d7063
commit f85a725883
6 changed files with 923 additions and 61 deletions
--- a/docs/README.md
+++ b/docs/README.md
@@ -0,0 +1,40 @@
+# Hottub Docs
+
+This folder is the fastest handoff path for anyone adding or repairing a channel.
+
+Start here:
+
+1. Read `architecture.md` for the server flow, request lifecycle, and core types.
+2. Read `provider-playbook.md` for the exact process to add a new provider or proxy.
+3. Use `provider-catalog.md` to find the closest existing implementation to copy.
+4. Use `docs/hottubapp/*.html` when you need the client-facing API contract for status, videos, or uploaders.
+5. Only then touch `prompts/new-channel.md`; it assumes the docs above exist.
+
+Recommended local workflow:
+
+```bash
+cargo check -q
+HOT_TUB_PROVIDER=<channel_id> cargo check -q
+HOT_TUB_PROVIDER=<channel_id> cargo run --features debug
+```
+
+Useful runtime baseline:
+
+```dotenv
+DATABASE_URL=hottub.db
+RUST_LOG=info
+PROXY=0
+BURP_URL=http://127.0.0.1:8081
+FLARE_URL=http://127.0.0.1:8191/v1
+DOMAIN=127.0.0.1:18080
+DISCORD_WEBHOOK=
+```
+
+Key facts:
+
+- Hottub is a Rust `ntex` server with providers under `src/providers/`.
+- `build.rs` controls compile-time provider registration.
+- `/api/videos` is the main provider execution path.
+- `/proxy/...` exists for sites whose direct media or thumbnails need a redirect/proxy layer.
+- Only three providers currently implement `/api/uploaders`: `hsex`, `omgxxx`, and `vjav`.
+- Uploader IDs should be namespaced like `<channel>:<site-local-id>` so `/api/uploaders` can route directly.
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -0,0 +1,313 @@
+# Hottub Architecture
+
+## Purpose
+
+Hottub is a Rust server that exposes Hot Tub compatible endpoints for channel discovery, video search, uploader lookups, and site-specific proxying. Most work in this repo is adding or repairing a provider module under `src/providers/`.
+
+## Top-Level Structure
+
+- `src/main.rs`: server bootstrap, env loading, database pool, shared requester/cache, route mounting.
+- `src/api.rs`: `/api/status`, `/api/videos`, `/api/uploaders`, `/api/test`, `/api/proxies`.
+- `src/providers/mod.rs`: provider trait, provider registry, build-time provider selection, status decoration, runtime validation, panic/error guards.
+- `src/providers/*.rs`: one module per channel/provider.
+- `src/proxy.rs`: route table for `/proxy/...`.
+- `src/proxies/*.rs`: redirect/media/thumb proxy implementations.
+- `src/videos.rs`: request/response payloads, `VideoItem`, `VideoFormat`, `ServerOptions`.
+- `src/status.rs`: status/channel/group payloads.
+- `src/uploaders.rs`: uploader request/profile payloads.
+- `src/util/requester.rs`: outbound HTTP with cookies, optional Burp proxying, Jina fallback, and FlareSolverr fallback.
+- `build.rs`: compile-time provider registry generation and single-provider build support.
+
+## Startup Flow
+
+1. `main` loads `.env` and ensures `RUST_LOG` is set.
+2. It creates the Diesel SQLite pool from `DATABASE_URL`.
+3. It creates a shared `Requester`, enables Burp proxying when `PROXY != 0`, and builds the LRU video cache.
+4. It configures provider runtime validation in `providers::configure_runtime_validation`.
+5. It spawns a background thread that forces provider initialization via `providers::init_providers_now()`.
+6. It starts an `ntex` HTTP server on `0.0.0.0:18080`.
+
+## Runtime Environment
+
+Important environment variables:
+
+- `DATABASE_URL`: required SQLite path.
+- `RUST_LOG`: defaults to `warn` if unset.
+- `PROXY`: enables Burp proxying when not equal to `0`.
+- `BURP_URL`: outbound proxy URL used when `PROXY` is enabled.
+- `FLARE_URL`: FlareSolverr endpoint used as the last HTML-fetch fallback.
+- `DOMAIN`: used by the `/` redirect target.
+- `DISCORD_WEBHOOK`: enables `/api/test` and provider error reporting.
+
+Bundled reference material:
+
+- `docs/hottubapp/📡 Status - Hot Tub Docs.html`
+- `docs/hottubapp/🎬 Videos - Hot Tub Docs.html`
+- `docs/hottubapp/👤 Uploaders - Hot Tub Docs.html`
+
+Those HTML files are useful when a provider author needs to confirm the expected client payload shape without reading Rust structs first.
+
+## Build-Time Provider Selection
+
+`build.rs` reads `HOT_TUB_PROVIDER` or `HOTTUB_PROVIDER`.
+
+- If unset, every provider in `build.rs` is compiled and registered.
+- If set, only that provider is compiled into the binary.
+- In a single-provider build, `/api/videos` remaps `"channel": "all"` to the compiled provider.
+
+Generated files in `OUT_DIR` are included by `src/providers/mod.rs`:
+
+- `provider_modules.rs`
+- `provider_registry.rs`
+- `provider_metadata_fn.rs`
+- `provider_selection.rs`
+
+This means adding a new provider always requires updating `build.rs`.
+
+## HTTP Surface
+
+### `/`
+
+Returns a `302` redirect to `hottub://source?url=<DOMAIN-or-request-host>`.
+
+### `/api/status`
+
+Builds the channel list by iterating `ALL_PROVIDERS` and calling `Provider::get_channel`.
+
+Important behavior:
+
+- The `User-Agent` is parsed into `ClientVersion`.
+- A provider can hide itself by returning `None`.
+- `providers::build_status_response` decorates channels with `groupKey`, top tags, runtime status, and sort order.
+- Some heavy status filters are intentionally removed from the client-facing response. The server still accepts them in `/api/videos`.
+
+### `/api/videos`
+
+This is the main provider execution path.
+
+Flow:
+
+1. Parse `VideosRequest`.
+2. Normalize `channel`, `sort`, `query`, `page`, and `perPage`.
+3. Build `ServerOptions`.
+4. If `query` is a full `http://` or `https://` URL, try the `yt-dlp -J` fast path first.
+5. Otherwise call `provider.get_videos(...)` through `run_provider_guarded`.
+6. For quoted queries like `"teacher"`, apply a literal substring filter after provider fetch.
+7. Spawn a background prefetch for the next page.
+8. For short videos (`duration <= 120`), populate `preview` from the main URL or first format.
+
+Important behavior:
+
+- Leading `#` is stripped from queries before provider dispatch.
+- `"all"` uses `AllProvider` in a normal build, but resolves to the single compiled provider in a single-provider build.
+- Older `Hot Tub/38` clients are patched by replacing `video.url` with the last format URL when formats exist.
+
+### `/api/uploaders`
+
+Uploader lookup is optional and provider-specific.
+
+Important behavior:
+
+- At least one of `uploaderId` or `uploaderName` is required.
+- If `uploaderId` looks like `channel:id`, the server directly targets that provider.
+- Otherwise it scans all providers and returns the best exact-name match.
+- Only `hsex`, `omgxxx`, and `vjav` currently implement `get_uploader`.
+- In practice, provider-owned uploader IDs should be namespaced, for example `vjav:12345` or `hsex:author_slug`.
+
+### `/api/test`
+
+Sends a Discord error test if `DISCORD_WEBHOOK` is configured.
+
+### `/api/proxies`
+
+Returns the background-fetched outbound proxy snapshot from `src/util/proxy.rs`.
+
+## Core Data Structures
+
+### `VideosRequest`
+
+Defined in `src/videos.rs`. Common fields used by providers:
+
+- `channel`
+- `sort`
+- `query`
+- `page`
+- `perPage`
+- `featured`
+- `category`
+- `sites`
+- `all_provider_sites`
+- `filter`
+- `language`
+- `networks`
+- `stars`
+- `categories`
+- `duration`
+- `sexuality`
+
+### `ServerOptions`
+
+The server’s normalized option bag. Providers should read from this instead of reparsing the raw API request.
+
+Important fields:
+
+- `public_url_base`: needed when generating `/proxy/...` URLs.
+- `requester`: the shared request client with cookies/debug trace/proxy state.
+- `sort`, `sites`, `filter`, `category`, `language`, `network`, `stars`, `categories`, `duration`, `sexuality`.
+
+### `VideoItem`
+
+Minimum useful fields for a provider:
+
+- `id`
+- `title`
+- `url`
+- `channel`
+- `thumb`
+- `duration`
+
+High-value optional fields:
+
+- `views`
+- `rating`
+- `uploader`
+- `uploaderUrl`
+- `uploaderId`
+- `tags`
+- `uploadedAt`
+- `formats`
+- `preview`
+- `aspectRatio`
+
+Avoid setting `embed` for new providers unless the site truly needs it.
+
+### `VideoFormat`
+
+Use `formats` when:
+
+- the site returns a better direct media URL than the page URL
+- HLS or multiple qualities exist
+- extra HTTP headers such as `Referer` are required
+
+Use `http_header` or `add_http_header` when the player endpoint needs request headers.
+
+### `Channel` and `ChannelOption`
+
+Each provider’s `get_channel` returns the status metadata exposed by `/api/status`.
+
+Typical option IDs used across the repo:
+
+- `sort`
+- `filter`
+- `sites`
+- `category`
+- `language`
+- `networks`
+- `stars`
+- `categories`
+
+Use the same IDs when possible so the server and client behavior stay consistent.
+
+### `UploaderProfile`
+
+If a provider supports `/api/uploaders`, keep the ID routable:
+
+- preferred format: `<channel>:<site-local-id>`
+- examples in the repo: `vjav:<user_id>`, `hsex:<author>`, `omgxxx:<kind>:<id>`
+
+This lets `src/api.rs` derive the owning provider immediately.
+
+## Provider Contract
+
+Defined in `src/providers/mod.rs`:
+
+- `async fn get_videos(...) -> Vec<VideoItem>`
+- `fn get_channel(clientversion: ClientVersion) -> Option<Channel>`
+- `async fn get_uploader(...) -> Result<Option<UploaderProfile>, String>` optional
+
+The server wraps provider execution in:
+
+- `run_provider_guarded` for video paths
+- `run_uploader_provider_guarded` for uploader paths
+
+Panics and reported errors trigger runtime validation and optional Discord reporting.
+
+## Runtime Validation and Error Handling
+
+`src/providers/mod.rs` includes a validation subsystem that:
+
+- runs a small sample request against a provider after failures
+- checks that enough video items exist
+- tries media URLs or format URLs with a `Range` header
+- marks repeated failures over time
+
+This means a provider that returns page URLs but no real media/formats may pass visually but still fail operationally.
+
+## Requester Behavior
+
+`src/util/requester.rs` is the standard outbound HTTP layer.
+
+Capabilities:
+
+- shared cookie jar across clones
+- optional Burp proxying via `PROXY` and `BURP_URL`
+- direct request retries for `429`
+- Jina mirror fallback for blocked HTML fetches
+- FlareSolverr fallback via `FLARE_URL`
+- raw response helpers for media validation and custom headers
+
+Use the shared requester from `ServerOptions` through `requester_or_default`. Do not instantiate a brand-new requester in normal provider fetch paths unless you have a very specific reason.
+
+FlareSolverr note:
+
+- `src/util/flaresolverr.rs` keeps a reusable session pool pattern by rotating a ready session per solve.
+- If a provider only works after anti-bot negotiation, the shared requester is the path that benefits from that solved session and cookie state.
+
+## Proxy Subsystem
+
+There are two proxy styles.
+
+### Redirect proxies
+
+These take a provider-specific endpoint and return `302 Location: <resolved-media-url>`.
+
+Examples:
+
+- `/proxy/spankbang/...`
+- `/proxy/sxyprn/...`
+- `/proxy/pornhd3x/...`
+- `/proxy/vjav/...`
+
+### Media or image proxies
+
+These actively fetch media or thumbnails and stream or rewrite the response.
+
+Examples:
+
+- `/proxy/noodlemagazine/...`
+- `/proxy/noodlemagazine-thumb/...`
+- `/proxy/shooshtime-media/...`
+- `/proxy/hanime-cdn/...`
+
+If a site only needs a referer-preserving redirect, use a redirect proxy. If manifests, relative playlist entries, cookies, or binary thumbs need rewriting, use a media/image proxy.
+
+## Best Existing Templates
+
+Use the closest existing provider instead of inventing a new style.
+
+- `src/providers/vjav.rs`: rich API-backed provider with tags, uploader support, and detail enrichment.
+- `src/providers/hsex.rs`: HTML scraping with background-loaded filters, uploader support, and direct HLS formats.
+- `src/providers/omgxxx.rs`: large filter catalogs and uploader lookup by site/network identity.
+- `src/providers/noodlemagazine.rs`: proxied media/thumbs, Jina fallback, and mirrored listing parsing.
+- `src/providers/pornhd3x.rs`: complex filter catalogs, detail enrichment, and proxy-generated playback URLs.
+- `src/providers/spankbang.rs`: anti-bot handling and a redirect-proxy-based media strategy.
+
+## Important Gotchas
+
+- New providers must export `CHANNEL_METADATA`.
+- New providers must be listed in `build.rs` or they will never compile into the registry.
+- If a provider returns proxied URLs, it usually also needs `options.public_url_base`.
+- Keep filter IDs stable. The `title` is for display; the `id` is what the provider matches on.
+- `categories` in `Channel` are not the same as `ChannelOption { id: "categories" }`.
+- `/api/status` sanitizes some options away from the client-facing payload. That does not mean the provider option is useless in `/api/videos`.
+- If a site needs per-request cookies or a solved user agent, rely on the shared requester.
--- a/docs/provider-catalog.md
+++ b/docs/provider-catalog.md
@@ -0,0 +1,95 @@
+# Provider And Proxy Catalog
+
+This is the current implementation inventory as of this snapshot of the repo. Use it to find the nearest existing pattern before adding a new channel.
+
+## Providers
+
+| Provider | Group | `/api/uploaders` | Uses local `/proxy` | Notes |
+| --- | --- | --- | --- | --- |
+| `all` | `meta-search` | no | no | Aggregates all compiled providers. |
+| `beeg` | `mainstream-tube` | no | no | Basic mainstream tube pattern. |
+| `chaturbate` | `live-cams` | no | no | Live cam channel. |
+| `freepornvideosxxx` | `studio-network` | no | no | Studio-style scraper. |
+| `freeuseporn` | `fetish-kink` | no | no | Fetish archive pattern. |
+| `hanime` | `hentai-animation` | no | yes | Uses proxied CDN/thumb handling. |
+| `heavyfetish` | `fetish-kink` | no | no | Direct media handling. |
+| `hentaihaven` | `hentai-animation` | no | no | HLS format builder pattern. |
+| `homoxxx` | `gay-male` | no | no | Gay category grouping example. |
+| `hqporner` | `studio-network` | no | yes | Uses thumb and redirect proxy helpers. |
+| `hsex` | `chinese` | yes | no | Strong template for tags, uploaders, and direct HLS formats. |
+| `hypnotube` | `fetish-kink` | no | no | Fetish/tube hybrid. |
+| `javtiful` | `jav` | no | no | JAV channel family. |
+| `missav` | `jav` | no | no | HLS format pattern. |
+| `noodlemagazine` | `mainstream-tube` | no | yes | Best template for media and thumbnail proxying. |
+| `okporn` | `mainstream-tube` | no | no | Simple mainstream archive. |
+| `okxxx` | `mainstream-tube` | no | no | Mainstream search/archive pattern. |
+| `omgxxx` | `studio-network` | yes | no | Best template for sites/networks/stars filter catalogs. |
+| `paradisehill` | `mainstream-tube` | no | no | Simple page scraper. |
+| `perfectgirls` | `studio-network` | no | no | Studio archive. |
+| `perverzija` | `studio-network` | no | no | Multi-format/HLS examples. |
+| `pimpbunny` | `onlyfans` | no | yes | Proxy-backed playback and thumbnail handling. |
+| `pmvhaven` | `pmv-compilation` | no | no | PMV grouping example. |
+| `porn00` | `mainstream-tube` | no | no | Lightweight scraper. |
+| `porn4fans` | `onlyfans` | no | no | OnlyFans-like grouping example. |
+| `porndish` | `studio-network` | no | yes | Redirect proxy plus thumb proxy usage. |
+| `pornhat` | `mainstream-tube` | no | no | Basic tube provider. |
+| `pornhd3x` | `studio-network` | no | yes | Best template for complex catalogs and redirect proxy generation. |
+| `pornhub` | `mainstream-tube` | no | no | Rich metadata and format examples. |
+| `pornmz` | `mainstream-tube` | no | no | Mainstream archive. |
+| `pornzog` | `mainstream-tube` | no | no | Basic list/detail scraper. |
+| `redtube` | `mainstream-tube` | no | no | Mainstream archive. |
+| `rule34gen` | `ai` | no | no | AI group example. |
+| `rule34video` | `hentai-animation` | no | no | Hentai group example. |
+| `sextb` | `jav` | no | no | JAV family provider. |
+| `shooshtime` | `onlyfans` | no | yes | Redirect proxy plus dedicated media route. |
+| `spankbang` | `mainstream-tube` | no | yes | Best template for redirect proxy plus anti-bot fetches. |
+| `supjav` | `jav` | no | no | JAV/HLS and uploader-id examples. |
+| `sxyprn` | `mainstream-tube` | no | yes | Redirect proxy helper usage. |
+| `tnaflix` | `mainstream-tube` | no | no | Mainstream tube provider. |
+| `tokyomotion` | `jav` | no | no | JAV/tube hybrid. |
+| `viralxxxporn` | `mainstream-tube` | no | no | Basic parser with format extraction. |
+| `vjav` | `jav` | yes | no | Best API-style template with uploaders and tag-id lookup maps. |
+| `vrporn` | `studio-network` | no | no | Multi-format direct playback. |
+| `xfree` | `tiktok` | no | no | Short-form grouping example. |
+| `xxdbx` | `onlyfans` | no | no | OnlyFans-like grouping example. |
+| `xxthots` | `onlyfans` | no | no | OnlyFans-like metadata example. |
+| `yesporn` | `mainstream-tube` | no | no | Preview format examples. |
+| `youjizz` | `mainstream-tube` | no | no | Mainstream tube provider. |
+
+## Proxy Routes
+
+### Redirect proxies
+
+These resolve a provider-specific input into a `302 Location`.
+
+- `/proxy/doodstream/{endpoint}*`
+- `/proxy/sxyprn/{endpoint}*`
+- `/proxy/javtiful/{endpoint}*`
+- `/proxy/spankbang/{endpoint}*`
+- `/proxy/porndish/{endpoint}*`
+- `/proxy/hqporner/{endpoint}*`
+- `/proxy/heavyfetish/{endpoint}*`
+- `/proxy/vjav/{endpoint}*`
+- `/proxy/pornhd3x/{endpoint}*`
+- `/proxy/shooshtime/{endpoint}*`
+- `/proxy/pimpbunny/{endpoint}*`
+
+### Media/image proxies
+
+These return binary media or images, sometimes rewriting manifests or forwarding cookies/referers.
+
+- `/proxy/shooshtime-media/{endpoint}*`
+- `/proxy/noodlemagazine/{endpoint}*`
+- `/proxy/noodlemagazine-thumb/{endpoint}*`
+- `/proxy/hanime-cdn/{endpoint}*`
+- `/proxy/hqporner-thumb/{endpoint}*`
+- `/proxy/porndish-thumb/{endpoint}*`
+- `/proxy/pornhub-thumb/{endpoint}*`
+
+## Best Copy Sources By Problem
+
+- Need uploader support: copy `hsex`, `omgxxx`, or `vjav`.
+- Need proxied media: copy `noodlemagazine`.
+- Need proxied redirect-only playback: copy `spankbang` or `pornhd3x`.
+- Need big background-loaded filter catalogs: copy `pornhd3x` or `omgxxx`.
+- Need tag title to site-ID lookup maps: copy `vjav` or `hsex`.
--- a/docs/provider-playbook.md
+++ b/docs/provider-playbook.md
@@ -0,0 +1,349 @@
+# New Provider Playbook
+
+This is the implementation checklist for adding a working channel with the least guessing.
+
+## Definition Of Done
+
+A provider is not done when it compiles. It is done when:
+
+1. `/api/status` shows the channel with sensible options and grouping.
+2. `/api/videos` returns real items for the default feed.
+3. Search works.
+4. Pagination works.
+5. Thumbnails load.
+6. `video.url` or at least one `formats[*].url` resolves to playable media.
+7. If the site needs proxying, the `/proxy/...` route works.
+8. `HOT_TUB_PROVIDER=<id> cargo check -q` passes.
+
+## Files To Touch
+
+Always:
+
+- `build.rs`
+- `src/providers/<channel_id>.rs`
+
+Sometimes:
+
+- `src/proxy.rs`
+- `src/proxies/<channel_id>.rs`
+- `src/proxies/<channel_id>thumb.rs`
+- `prompts/new-channel.md` if you are improving the handoff prompt
+- `docs/provider-catalog.md` if you add a new provider or proxy
+
+## Step 1: Pick The Closest Template
+
+Do not start from an empty file.
+
+Choose the nearest match:
+
+- API-first site with tags/uploader metadata: copy `vjav.rs`
+- HTML site with background-loaded tags/uploaders: copy `hsex.rs`
+- Site with multiple large catalogs like sites/networks/stars: copy `omgxxx.rs`
+- Site whose media or thumbs need local proxying: copy `noodlemagazine.rs`, `pornhd3x.rs`, `spankbang.rs`, or `porndish.rs`
+- Very simple archive/search site: copy a small provider from `mainstream-tube`
+
+Before writing code, confirm the site shape:
+
+1. home or latest feed URL
+2. search URL and page 2 URL
+3. detail page URL shape
+4. player request or manifest request
+5. thumbnail host and whether it needs referer/cookies
+6. tag/category/uploader/studio routes if they exist
+7. whether the site exposes JSON endpoints that are easier than HTML scraping
+
+Use browser/network tooling for this if needed. Do not guess URL patterns from one page.
+
+## Step 2: Register The Provider
+
+Add the provider to `build.rs`:
+
+- `id`: channel id used by `/api/videos`
+- `module`: Rust file name
+- `ty`: provider struct name
+
+If this is missing, the server will not discover the provider.
+
+## Step 3: Define Channel Metadata
+
+Every provider should export:
+
+```rust
+pub const CHANNEL_METADATA: crate::providers::ProviderChannelMetadata =
+    crate::providers::ProviderChannelMetadata {
+        group_id: "...",
+        tags: &["...", "...", "..."],
+    };
+```
+
+Pick `group_id` from the existing set in `src/providers/mod.rs`:
+
+- `meta-search`
+- `mainstream-tube`
+- `tiktok`
+- `studio-network`
+- `amateur-homemade`
+- `onlyfans`
+- `chinese`
+- `jav`
+- `fetish-kink`
+- `hentai-animation`
+- `ai`
+- `gay-male`
+- `live-cams`
+- `pmv-compilation`
+
+## Step 4: Build The Channel Surface
+
+Implement `build_channel` or equivalent and return it from `get_channel`.
+
+Required:
+
+- `id`
+- `name`
+- `description`
+- `favicon`
+- `status`
+- `nsfw`
+
+Recommended:
+
+- `cacheDuration: Some(1800)` unless the site is unusually stable
+- use standard option IDs like `sort`, `filter`, `sites`, `category`, `stars`, `categories`
+- keep options minimal at first; only expose filters that actually work in `get_videos`
+
+The option `id` values matter more than the display `title`.
+
+## Step 5: Model Provider Routing Explicitly
+
+Create a local enum like:
+
+```rust
+enum Target {
+    Latest,
+    Search { query: String },
+    Tag { slug: String },
+    Uploader { id: String },
+}
+```
+
+Then write one function that resolves `sort`, `query`, `filter`, `sites`, and related options into a `Target`.
+
+This is easier to debug than scattering URL decisions across the provider.
+
+## Step 6: Load Filter Catalogs In The Background If Needed
+
+If the site exposes tags, uploaders, studios, networks, or stars:
+
+- store them in `Arc<RwLock<Vec<FilterOption>>>`
+- initialize them with an `All` option
+- spawn a background thread in `new()`
+- create a tiny Tokio runtime inside that thread
+- fill the lists without blocking server startup
+
+Patterns:
+
+- `hsex.rs`
+- `omgxxx.rs`
+- `pornhd3x.rs`
+- `vjav.rs`
+
+If tags or uploaders need stable IDs, keep a lookup map such as:
+
+- `HashMap<String, String>` from title to site ID
+- `HashMap<String, String>` from site ID to URL target
+
+Normalize lookup keys to lowercase trimmed strings.
+
+## Step 7: Fetch Pages Through The Shared Requester
+
+In `get_videos`, start with:
+
+```rust
+let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
+```
+
+Use it for HTML, JSON, and raw media requests.
+
+Why:
+
+- it preserves cookies
+- it carries debug trace IDs
+- it respects Burp proxying
+- it can fall back to Jina or FlareSolverr
+
+## Step 8: Parse Listing Cards First, Then Enrich Only If Needed
+
+Preferred flow:
+
+1. Fetch the archive or search page.
+2. Parse a lightweight list of stubs.
+3. Return list data directly if enough metadata is already present.
+4. Fetch detail pages or JSON endpoints only for fields the card does not expose.
+
+Use bounded concurrency for detail enrichment. Existing providers usually use `futures::stream` with `buffer_unordered`.
+
+## Step 9: Build High-Quality `VideoItem`s
+
+Always fill:
+
+- `id`
+- `title`
+- `url`
+- `channel`
+- `thumb`
+- `duration`
+
+Fill when available:
+
+- `views`
+- `rating`
+- `uploader`
+- `uploaderUrl`
+- `uploaderId`
+- `tags`
+- `uploadedAt`
+- `preview`
+- `aspectRatio`
+- `formats`
+
+Rules:
+
+- Keep `tags` as a list of displayable titles.
+- Keep uploader data as structured fields, not mashed into the title.
+- If you support uploader profiles, set `uploaderId` to a namespaced value like `<channel>:<site-local-id>`.
+- Do not include `embed` unless the provider truly needs it.
+- If direct media exists, prefer `formats` and keep `url` stable.
+
+## Step 10: Decide Whether A Proxy Is Required
+
+Use no proxy when:
+
+- page URLs are enough and the client can resolve media itself
+- or direct media URLs already work cleanly
+
+Use a redirect proxy when:
+
+- the provider must turn a detail URL into a resolved media URL
+- headers/cookies do not need full response rewriting
+
+Use a media/image proxy when:
+
+- the site requires a referer for every fetch
+- thumbnails need cookie-backed access
+- manifests contain relative URIs that must be rewritten
+- the server must stream binary content itself
+
+If a proxy is needed:
+
+1. add `src/proxies/<id>.rs`
+2. wire the route in `src/proxy.rs`
+3. generate provider URLs with `build_proxy_url(&options, "<id>", target)`
+
+## Step 11: Implement Search Correctly
+
+Check for three search modes:
+
+1. native site search endpoint
+2. tag/uploader shortcut search from preloaded filter catalogs
+3. literal client-side substring search after fetch, triggered by quoted queries
+
+Important server behavior:
+
+- `#tag` becomes `tag`
+- `"teacher"` becomes a literal post-fetch filter
+- raw URL queries may bypass the provider through the `yt-dlp` fast path
+
+Provider guidance:
+
+- if the query matches a known tag/uploader shortcut, prefer the site’s direct archive URL instead of generic search
+- otherwise fall back to the site’s keyword search
+
+## Step 12: Support Pagination Explicitly
+
+Do not assume pagination is `?page=N`.
+
+Confirm:
+
+- archive page 2 URL shape
+- search page 2 URL shape
+- tag page 2 URL shape
+- uploader page 2 URL shape
+
+If the site uses infinite scroll or an XHR endpoint, document that in code comments and hit the underlying endpoint directly.
+
+## Step 13: Only Add `/api/uploaders` When The Site Has Real Uploader Identity
+
+Uploader support is optional. Only implement it when the site exposes stable uploader pages or IDs.
+
+Use `hsex.rs`, `omgxxx.rs`, or `vjav.rs` as the template.
+
+Minimum expectations for `UploaderProfile`:
+
+- stable `id`
+- `name`
+- `channel`
+- `videoCount`
+- `totalViews`
+
+Nice to have:
+
+- `avatar`
+- `description`
+- `videos`
+- `layout`
+- per-channel stats
+
+## Validation Checklist
+
+Run all of these:
+
+```bash
+cargo check -q
+HOT_TUB_PROVIDER=<channel_id> cargo check -q
+HOT_TUB_PROVIDER=<channel_id> cargo run --features debug
+```
+
+Then hit:
+
+```bash
+curl -s http://127.0.0.1:18080/api/status \
+  -H 'User-Agent: Hot%20Tub/22c CFNetwork/1494.0.7 Darwin/23.4.0' | jq
+```
+
+```bash
+curl -s http://127.0.0.1:18080/api/videos \
+  -H 'Content-Type: application/json' \
+  -d '{"channel":"<channel_id>","sort":"new","page":1,"perPage":10}' | jq
+```
+
+Also verify:
+
+- search query works
+- page 2 works
+- tag shortcut works if implemented
+- uploader shortcut works if implemented
+- `yt-dlp '<video.url or first format url>'` resolves media
+- thumbnail URL returns an image
+- proxy route returns a `302` or working media body, whichever is expected
+- if uploaders are implemented, `/api/uploaders` works with both `uploaderId` and `uploaderName`
+
+## Common Failure Modes
+
+- Forgot `build.rs` entry.
+- Returned page URLs but no playable media/formats.
+- Used a local requester instead of the shared one and lost cookies.
+- Built `/proxy/...` URLs without `public_url_base`.
+- Put human-readable titles into filter IDs, making routing brittle.
+- Added huge option lists to the status response without background loading.
+- Implemented search but not search pagination.
+- Implemented proxies but forgot to test them independently with `curl -I`.
+
+## Best Reference Matrix
+
+- Rich uploader support: `vjav.rs`, `hsex.rs`, `omgxxx.rs`
+- Tag and uploader lookup maps: `vjav.rs`, `hsex.rs`
+- Background catalog loading: `hsex.rs`, `omgxxx.rs`, `pornhd3x.rs`
+- Redirect proxy: `spankbang.rs` plus `src/proxies/spankbang.rs`
+- Manifest or image proxy: `noodlemagazine.rs` plus `src/proxies/noodlemagazine.rs`
+- Complex detail enrichment: `pornhd3x.rs`
--- a/docs/uploaders-endpoint-plan.md
+++ b/docs/uploaders-endpoint-plan.md
@@ -1,60 +0,0 @@
-# Uploaders Endpoint Plan
-
-## Summary
-
-Implement `POST /api/uploaders` using the Hot Tub uploader profile contract and ship it framework-first. The server will expose shared uploader request/response types, a provider hook for uploader lookup, endpoint routing in `src/api.rs`, and a first real provider implementation in `hsex`.
-
-## Implementation
-
- Add dedicated uploader API types in `src/uploaders.rs`:
-  - `UploadersRequest`
-  - `UploaderProfile`
-  - `UploaderChannelStat`
-  - `UploaderVideoRef`
-  - `UploaderLayoutRow`
- Keep camelCase as the canonical serialized shape.
- Accept documented decode aliases:
-  - `uploader_id`
-  - `uploader_name`
-  - `profile_content`
-  - `profile_picture_url`
-  - `video_ids`
-  - `horizontal_videos`
- Add `POST /api/uploaders` in `src/api.rs`.
- Validate that at least one of `uploaderId` or `uploaderName` is present.
- Return:
-  - `400` for invalid request
-  - `404` for no match
-  - `500` for provider execution failure
- Add `Provider::get_uploader(...)` with a default `Ok(None)` implementation.
- Add a guarded uploader execution helper in `src/providers/mod.rs`.
- Use canonical uploader IDs in the format `<channel>:<provider-local-id>`.
- Implement the first provider-backed uploader profile in `src/providers/hsex.rs`.
-
-## Hsex Strategy
-
- Resolve uploader lookup by canonical uploader ID or exact uploader name.
- Reuse existing uploader archive discovery and archive page fetching.
- Build uploader profile metadata from uploader archive pages.
- Populate `videos` with `UploaderVideoRef` values derived from existing `VideoItem`s.
- Always return `layout`.
- When `profileContent == true`, return:
-  - `videos`
-  - `tapes: []`
-  - `playlists: []`
-  - a `"For You"` horizontal row plus the default videos row
- When `profileContent == false`, return metadata and layout only.
-
-## Tests
-
- Request alias decoding for uploader request fields.
- Response alias decoding for avatar and layout row compatibility fields.
- Endpoint helper tests for request validation and provider routing.
- Hsex uploader ID generation and uploader page parsing coverage.
-
-## Assumptions
-
- The first ship focuses on the endpoint framework and one real provider implementation.
- Providers without explicit uploader support remain unsupported by `/api/uploaders`.
- Name-based resolution uses exact display-name matching.
- `videoCount` and `totalViews` are best-effort when the upstream site does not expose authoritative profile totals.
--- a/prompts/new-channel.md
+++ b/prompts/new-channel.md
@@ -1 +1,126 @@
-write a provider for the site "vjav.com". use playwright-mcp to get the index site and get the layout of videos from this. also figure out how searches work and if it has a different layout then. also find out how more videos get loaded/more pages urls. also find if it has video sites like recommended,most views etc and include them. try to find as much information about a video on the site as possible and build the video items with all the information. put tags and uploader into lists so they can be utilized later. on query check if the search already exists in these FilterObject arrays and use them for the url instead (similar to omgxxx). after coding this, test that all provided urls work, yt-dlp can download the video.url, the thumbnails work, searches, tag-/uploader- searches work, more pages work. Keep a key-value storage for all provided Tag titles to tag IDs to do the correct crawl for queries of a tag. analzye the provider too and set the channel tags and put it in a group that matches this provider or create a new group where it fits in. do not include the embed field in the videoitems responses. when parsing uploaders, make sure that the uploaders. if unsure about the endpoints, check with the docs dir.
+Implement a new Hottub provider for `<SITE_NAME>` at `<BASE_URL>`.
+
+You are working inside the Hottub Rust server. Your job is to add a functioning provider module that can survive handoff to another model with minimal guesswork. Do not stop at code generation. Carry the work through code, validation, and documentation updates.
+
+Execution order is mandatory:
+
+1. Read the repo docs.
+2. Inspect the target site and collect evidence about routes, player/media requests, and pagination.
+3. Choose the closest existing provider/proxy as the template.
+4. Implement the provider.
+5. Validate it end to end.
+6. Update docs if the new provider adds a new pattern.
+
+Do not start coding until you know:
+
+- latest/default feed URL
+- search URL
+- page 2 URL
+- detail page URL
+- actual media request or manifest URL
+- thumbnail behavior
+- whether tag/uploader/studio pages exist
+- whether the site has a JSON API that is easier than HTML scraping
+
+Read these files first:
+
+1. `docs/README.md`
+2. `docs/architecture.md`
+3. `docs/provider-playbook.md`
+4. `docs/provider-catalog.md`
+5. `docs/hottubapp/🎬 Videos - Hot Tub Docs.html`
+6. `docs/hottubapp/📡 Status - Hot Tub Docs.html`
+7. `docs/hottubapp/👤 Uploaders - Hot Tub Docs.html`
+
+Then inspect the closest existing providers and proxies before coding. Pick the nearest template instead of starting from scratch.
+
+Template selection rules:
+
+- Use `src/providers/vjav.rs` if the target site has JSON APIs, rich tag metadata, or stable uploader identities.
+- Use `src/providers/hsex.rs` if the target site is mostly HTML and needs background-loaded tags/uploaders.
+- Use `src/providers/omgxxx.rs` if the site exposes multiple large filter catalogs like sites, networks, models, or studios.
+- Use `src/providers/noodlemagazine.rs`, `src/providers/pornhd3x.rs`, or `src/providers/spankbang.rs` if media or thumbnails require local `/proxy/...` routes.
+
+Required deliverables:
+
+1. Add a new provider file at `src/providers/<channel_id>.rs`.
+2. Register it in `build.rs`.
+3. Export `CHANNEL_METADATA` with the correct group.
+4. Implement `get_channel` with sane options and descriptions.
+5. Implement `get_videos` so the default feed works, search works, and page 2 works.
+6. If the site needs proxying, add `src/proxies/<channel_id>.rs` and wire `src/proxy.rs`.
+7. Reuse `requester_or_default(&options, CHANNEL_ID, "...")` for outbound requests.
+8. Return high-quality `VideoItem`s with the best metadata the site exposes.
+9. Do not use `embed` unless the site truly requires it.
+10. Update `docs/provider-catalog.md` if you add a new provider or proxy.
+
+Implementation requirements:
+
+- Determine the real site routing for:
+  - default/latest listing
+  - search
+  - page 2 and later
+  - tag/category shortcuts
+  - uploader/studio/model shortcuts if the site exposes them
+  - featured/trending/most-viewed or similar alternate feeds
+- Model routing explicitly with a local enum like `Target`.
+- If the site exposes tag or uploader IDs, keep a lookup map from normalized display title to site ID/URL target.
+- Put tags into `VideoItem.tags`.
+- Put uploader name/url/id into `uploader`, `uploaderUrl`, and `uploaderId` when available.
+- If uploader support is implemented, use a namespaced `uploaderId` such as `<channel>:<site-local-id>` so `/api/uploaders` can route directly.
+- If the query matches a known tag/uploader shortcut, use the direct archive URL instead of generic search.
+- If the site exposes real media URLs or HLS manifests, populate `formats`.
+- If direct playback needs a referer/cookie transform, use a local `/proxy/...` route built with `build_proxy_url(&options, "...", target)`.
+- Keep the first version small and reliable. Add extra filters only after the default feed, search, and pagination are working.
+
+Validation requirements:
+
+1. `cargo check -q`
+2. `HOT_TUB_PROVIDER=<channel_id> cargo check -q`
+3. `HOT_TUB_PROVIDER=<channel_id> cargo run --features debug`
+4. Verify `/api/status` exposes the new channel.
+5. Verify `/api/videos` returns results for:
+   - default feed
+   - search query
+   - page 2
+   - at least one tag/uploader shortcut if implemented
+6. Verify thumbnails load.
+7. Verify `yt-dlp` can resolve `video.url` or one of `formats[*].url`.
+8. If a proxy route exists, verify it directly with `curl -I` or equivalent.
+
+Testing commands to run:
+
+```bash
+curl -s http://127.0.0.1:18080/api/status \
+  -H 'User-Agent: Hot%20Tub/22c CFNetwork/1494.0.7 Darwin/23.4.0' | jq
+```
+
+```bash
+curl -s http://127.0.0.1:18080/api/videos \
+  -H 'Content-Type: application/json' \
+  -H 'User-Agent: Hot%20Tub/22c CFNetwork/1494.0.7 Darwin/23.4.0' \
+  -d '{"channel":"<channel_id>","sort":"new","page":1,"perPage":10}' | jq
+```
+
+```bash
+curl -s http://127.0.0.1:18080/api/videos \
+  -H 'Content-Type: application/json' \
+  -d '{"channel":"<channel_id>","query":"test","page":1,"perPage":10}' | jq
+```
+
+Important Hottub-specific rules:
+
+- Do not invent a new provider style if an existing provider already matches the site shape.
+- Do not forget `build.rs`; missing registration means the provider does not exist at runtime.
+- Do not create a brand-new requester in normal provider fetches unless you have a strong reason.
+- Do not assume page URLs are playable media URLs.
+- Do not expose status filters that you did not implement in `get_videos`.
+- Do not finish without checking at least one returned media URL with `yt-dlp`.
+- Do not claim pagination works unless page 2 was verified.
+
+Completion format:
+
+1. Briefly state which existing provider/proxy you used as the template and why.
+2. List the files changed.
+3. Report the exact validation commands you ran and whether they passed.
+4. Report any residual limitations or site behaviors that still need follow-up.