Compare commits

...

16 Commits

Author SHA1 Message Date
Simon
edf6732f8e Query resolution todo, slug helper tests, fix txxx decoder test fixture
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
2026-10-17 06:19:18 +00:00
Simon
697836b914 coomer: creator feeds are a bare array; single-word queries that are creator ids go to the creator 2026-10-17 06:16:24 +00:00
Simon
5598216381 sextb: probe actress/studio for names outside the loaded catalogs 2026-10-17 06:07:34 +00:00
Simon
4c29d27b3e redgifs: single-word queries that are creator names go to the creator 2026-10-17 06:05:54 +00:00
Simon
a486bc5502 erome: single-word queries that are uploader names go to the uploader 2026-10-17 06:03:58 +00:00
Simon
270b327f73 fikfap: single-word queries that are profile names go to the creator 2026-10-17 06:01:34 +00:00
Simon
813e2ec2ca clapdat: resolve tag/user names without prefix 2026-10-17 05:58:53 +00:00
Simon
26ef830f6e pornhub: probe pornstar/model/channel for unknown free-form queries 2026-10-17 05:56:31 +00:00
Simon
e34801f6f8 thepornbunny: resolve pornstar/studio/tag names; get_probe treats redirect-away as a miss 2026-10-17 05:53:43 +00:00
Simon
2646ca5659 xgroovy: resolve pornstar/channel names without prefix 2026-10-17 05:51:40 +00:00
Simon
981ec29ad8 xhamster: resolve pornstar/channel names without prefix 2026-10-17 05:50:41 +00:00
Simon
76b9d4e058 xvideos: resolve channel/pornstar names without prefix; channel archives via the JSON video list 2026-10-17 05:48:58 +00:00
Simon
59ea0495a6 porn4fans: resolve model/tag names without prefix 2026-10-17 05:40:28 +00:00
Simon
1f95549167 youporn: resolve pornstar/channel/amateur names without prefix 2026-10-17 05:38:47 +00:00
Simon
1e56fe2da5 tube8: resolve pornstar/channel names without prefix 2026-10-17 05:37:07 +00:00
Simon
bdde02fa8c Add get_probe/probe_first_archive; fullporner and fullpornerorg resolve pornstar names without prefix 2026-10-17 05:34:15 +00:00
21 changed files with 756 additions and 71 deletions

View File

@@ -0,0 +1,42 @@
# Seamless query resolution todo
Goal: a free-form query ("gal ritchie", "family strokes") should reach the matching
pornstar / studio / channel / tag archive **without** a `pornstar:` / `studio:` / `tag:` prefix,
by asking the site (probe) and falling back to keyword search. Prefixes stay as explicit overrides.
Pattern (see `hooporn`, `cumlouder`): after known-category matching, probe page 1 of each entity
archive for the slug with `Requester::get_probe` (404 = instant `None`, Cloudflare challenge goes to
FlareSolverr, 5xx/429 retried once), use the first archive that yields cards (page N stays in that
archive), else keyword search. A probe is only valid after checking what the site answers for an
unknown slug (404 vs soft-404 that lists unrelated videos).
Status: `[ ]` open, `[x]` done and verified live (feed of the archive + unknown name falls back),
`[-]` not applicable (reason).
| | Provider | Prefix kinds today | Plan / evidence |
|---|---|---|---|
| [x] | `hooporn` | cat, pornstar, studio | reference implementation |
| [x] | `cumlouder` | cat, tag, series, girl | reference implementation (girl page, then tag) |
| [x] | `fullporner` | cat, pornstar | probes `/pornstar/{slug}` (unknown = 404 with a suggestion grid, so status decides); `pornstar:` slug now dashed |
| [x] | `fullpornerorg` | pornstar | probes `/videos/pornstar/{slug}/` (404 for unknown) |
| [x] | `tube8` | tag, cat, channel, pornstar | probes pornstar then channel |
| [x] | `youporn` | tag, channel, pornstar, amateur | probes pornstar, channel, amateur (plus existing background shortcut map) |
| [x] | `porn4fans` | tag, cat, model | probes `/models/{slug}/` then `/tags/{slug}/` (both 404 when unknown) |
| [x] | `xvideos` | tag, cat, uploader/channel/profile | probes `/channels/{slug}/videos/new/0` (JSON; pornstar profiles redirect there). **Also fixed**: the old `channel:`/`uploader:` prefix returned nothing (landing page cards are `/prof-video-click/` links); channels are now read from the JSON video list |
| [x] | `xhamster` | cat, channel | added `Pornstar` target; probes pornstar then channel |
| [x] | `xgroovy` | cat, pornstar, channel | probes pornstar then channel |
| [x] | `thepornbunny` | tag, cat, studio, pornstar | probes pornstar, studio, tag (unknown slugs redirect home: `get_probe` now treats a redirect to another path as a miss) |
| [x] | `pornhub` | channel, pornstar, model, user | background catalogs already matched listed names; unknown names are probed (`pornstar`, `model`, `channels`) and cached in `tag_map` (misses remembered) |
| [x] | `clapdat` | tag, user | probes tag then user |
| [x] | `fikfap` | tag, user | single-word query that is an existing profile (profile endpoint) goes to its posts, empty profile falls back to search |
| [x] | `erome` | uploader, user | single-word query that is an existing `/{name}` with albums goes to the uploader |
| [x] | `redgifs` | tag, user | single-word query that is an existing creator (users endpoint) goes to the creator, empty falls back |
| [x] | `sextb` | (catalog matched) | catalog (actress/studio/genre) match first, then probes `/actress/{slug}` and `/studio/{slug}` for names outside the catalog |
| [x] | `coomer` | u:, @service:user | single-word query that is an existing creator id (`/api/v1/{service}/user/{id}/profile`) goes to the creator. **Also fixed**: creator feeds are a bare JSON array, which the parser rejected (every creator feed was empty) |
| [-] | `ripnsfw`, `hsex`, `omgxxx`, `eporner`, `borntobefuck`, `archivebate` | various | already seamless: bare names match background-loaded creator / studio / tag catalogs (or the whole catalog is filtered client-side) |
| [-] | `perfectgirls`, `okxxx`, `pornhat`, `perverzija`, `porntrex` | sites/models/stars | bare names already resolve through the background-loaded `tag_map`; these clones build their own `wreq` client with no `ServerOptions`, so a live probe needs a separate refactor onto the shared requester |
| [-] | `homoxxx` | models, pornstars, channels, categories, tags | legacy provider (own client, unpaginated archives, no `/models/` index on the site); not worth a probe until it is rewritten |
| [-] | `kwiky`, `animeidhentai`, `hentaitv`, `xnxx`, `blowjobspro`, `txxx` | tag/cat | search already is the entity lookup, or the site has no pornstar/studio archives to probe (txxx: the API rejects `models`/`channels` sections) |
| [-] | `fyptt` | cat | by design: bare category words are common search terms and must not be hijacked |
| [-] | `notfans` | tag, category | tag URLs are hashed ids, so an unknown name cannot be probed; search covers it |
| [-] | `allpornstream`, `jable`, `wowxxx`, `hornyleak`, `hotbunny`, `hentaimama`, `tikporn`, `camsoda` | actor/producer, model, studio, ... | Cloudflare-gated or down from this host (403/522, FlareSolverr wedged), so a probe cannot be verified here; each would add one or two FlareSolverr solves per search. Candidates for the same pattern once checked from the production network |

View File

@@ -1,7 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{
Provider, build_proxy_url, report_provider_error, requester_or_default, strip_url_scheme,
Provider, build_proxy_url, probe_first_archive, report_provider_error, requester_or_default,
slug_dashed, strip_url_scheme,
};
use crate::status::*;
use crate::util::time::parse_time_to_seconds;
@@ -149,6 +150,29 @@ impl ClapdatProvider {
}
}
/// A free-form query that names a tag or a user goes to that archive
/// (probed on the site; unknown names 404), else keyword search.
async fn resolve_keyword(&self, query: &str, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let candidates = [
Target::Tag { slug: slug.clone() },
Target::User { username: slug },
];
let urls: Vec<String> = candidates
.iter()
.filter_map(|t| self.listing_url(t, 1))
.collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
body.contains("/video/")
})
.await?;
candidates.into_iter().nth(index)
}
fn listing_url(&self, target: &Target, page: u16) -> Option<String> {
let page = page.max(1);
match target {
@@ -491,7 +515,12 @@ impl Provider for ClapdatProvider {
sort
};
let query_value = query.unwrap_or_default();
let target = self.resolve_target(&query_value, &sort_value);
let mut target = self.resolve_target(&query_value, &sort_value);
if let Target::Search { query } = &target {
if let Some(found) = self.resolve_keyword(query, &options).await {
target = found;
}
}
let stubs = match &target {
Target::Search { query } => match self

View File

@@ -8,7 +8,7 @@ use wreq::Version;
use crate::{
DbPool,
api::ClientVersion,
providers::{Provider, requester_or_default},
providers::{Provider, probe_first_archive_with, requester_or_default},
status::{Channel, ChannelOption, FilterOption},
util::cache::VideoCache,
videos::{ServerOptions, VideoFormat, VideoItem},
@@ -147,6 +147,43 @@ impl CoomerProvider {
}
}
/// A single-word query that is an existing creator id (the profile
/// endpoint answers 404 otherwise) goes to that creator's posts, trying the
/// filtered service first, else keyword search.
async fn resolve_keyword(
query: &str,
service_filter: Option<&str>,
options: &ServerOptions,
) -> Option<Target> {
let name = query.trim().trim_start_matches('@');
if name.is_empty()
|| !name
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-' || c == '.')
{
return None;
}
let mut services = vec![Service::Onlyfans, Service::Fansly, Service::Candfans];
if let Some(preferred) = Self::resolve_service(service_filter) {
services.retain(|s| s.as_str() != preferred.as_str());
services.insert(0, preferred);
}
let urls: Vec<String> = services
.iter()
.map(|s| format!("{BASE_URL}/api/v1/{}/user/{name}/profile", s.as_str()))
.collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) =
probe_first_archive_with(&mut requester, &urls, Self::json_headers(), |body| {
body.contains("\"id\"")
})
.await?;
Some(Target::Uploader {
service: services[index].clone(),
username: name.to_string(),
})
}
fn json_headers() -> Vec<(String, String)> {
vec![
("Referer".to_string(), REFERER.to_string()),
@@ -290,13 +327,16 @@ impl CoomerProvider {
)));
}
}
let posts: ApiPostsResponse = serde_json::from_str(&body).map_err(|err| {
let posts: ApiPostsPayload = serde_json::from_str(&body).map_err(|err| {
Error::from(format!(
"json parse failed for {url}: {err}; body[:200]={}",
&body.chars().take(200).collect::<String>()
))
})?;
Ok(posts.posts)
Ok(match posts {
ApiPostsPayload::List(posts) => posts,
ApiPostsPayload::Wrapped(wrapped) => wrapped.posts,
})
}
async fn fetch_items(
@@ -543,6 +583,14 @@ struct ApiError {
error: String,
}
/// The global feed wraps posts in `{posts: [...]}`; a creator's feed is a bare array.
#[derive(Debug, Deserialize)]
#[serde(untagged)]
enum ApiPostsPayload {
List(Vec<ApiPost>),
Wrapped(ApiPostsResponse),
}
#[derive(Debug, Deserialize)]
struct ApiPostsResponse {
#[serde(default)]
@@ -601,8 +649,13 @@ impl Provider for CoomerProvider {
.clamp(1, 64);
let service_filter = options.categories.as_deref().or(options.sites.as_deref());
let target =
let mut target =
Self::resolve_target(&sort_value, query.as_deref(), service_filter);
if let Target::Search { query } = &target {
if let Some(found) = Self::resolve_keyword(query, service_filter, &options).await {
target = found;
}
}
// Bump the ignored `Sort` variant warning off by referencing it.
let _ = Sort::New;
@@ -849,4 +902,18 @@ mod tests {
let post: ApiPost = serde_json::from_str(json).expect("parse");
assert!(CoomerProvider::build_video_item(post).is_none());
}
#[test]
fn parses_bare_array_and_wrapped_post_payloads() {
let bare = r#"[{"id":"1","user":"u","service":"onlyfans","title":"t","file":{},"attachments":[]}]"#;
match serde_json::from_str::<ApiPostsPayload>(bare).expect("bare array") {
ApiPostsPayload::List(posts) => assert_eq!(posts.len(), 1),
other => panic!("expected list, got {other:?}"),
}
let wrapped = r#"{"posts":[{"id":"1"}],"count":1}"#;
match serde_json::from_str::<ApiPostsPayload>(wrapped).expect("wrapped") {
ApiPostsPayload::Wrapped(w) => assert_eq!(w.posts.len(), 1),
other => panic!("expected wrapped, got {other:?}"),
}
}
}

View File

@@ -1,6 +1,6 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::providers::{Provider, probe_first_archive, report_provider_error, requester_or_default};
use crate::status::*;
use crate::util::parse_abbreviated_number;
use crate::videos::{ServerOptions, VideoItem};
@@ -164,6 +164,26 @@ impl EromeProvider {
(!slug.is_empty()).then_some(slug)
}
/// A single-word query that is an existing uploader name (`/{name}` answers
/// 404 otherwise) goes to that uploader's albums, else keyword search.
async fn resolve_keyword(&self, query: &str, options: &ServerOptions) -> Option<Target> {
let slug = query.trim().trim_start_matches('@');
if slug.is_empty()
|| !slug
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-')
{
return None;
}
let url = format!("{}/{slug}", self.url);
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
probe_first_archive(&mut requester, &[url], &format!("{}/", self.url), |body| {
body.contains("album-link")
})
.await
.map(|_| Target::UploaderPosts { slug: slug.to_string() })
}
fn resolve_target(&self, query: &str, options: &ServerOptions, sort: &str) -> Target {
if let Some(site) = options.sites.as_deref() {
let normalized = site.trim();
@@ -367,7 +387,12 @@ impl Provider for EromeProvider {
};
let query_value = query.unwrap_or_default();
let page_value = page.parse::<u16>().unwrap_or(1);
let target = self.resolve_target(&query_value, &options, &sort_value);
let mut target = self.resolve_target(&query_value, &options, &sort_value);
if let Target::Search { query, .. } = &target {
if let Some(found) = self.resolve_keyword(query, &options).await {
target = found;
}
}
let url = self.build_url_for_target(&target, page_value);
match self.fetch_page(&options, &url).await {

View File

@@ -639,6 +639,24 @@ impl Provider for FikfapProvider {
.map(str::trim)
.filter(|value| !value.is_empty());
let target = Self::pick_target(normalized_query, &sort, &options);
// A single-word query that is an existing profile name goes to that
// creator's posts (checked against the profile endpoint); a profile
// without posts falls back to the keyword search.
if let Target::Search(query) = &target {
if !query.contains(char::is_whitespace) {
if let Ok(Some(profile)) = self.fetch_user_profile(query, &options).await {
let user = Target::User(profile.username);
if let Ok(items) = self
.fetch_target_items(user, page, per_page, &options)
.await
{
if !items.is_empty() {
return items;
}
}
}
}
}
match self.fetch_target_items(target, page, per_page, &options).await {
Ok(items) => items,

View File

@@ -1,6 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::providers::{
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::time::parse_time_to_seconds;
@@ -134,13 +136,13 @@ impl FullpornerProvider {
}
if let Some(slug) = q.strip_prefix("pornstar:") {
return Target::Pornstar {
slug: slug.trim().to_string(),
slug: slug_dashed(slug),
page,
};
}
if let Some(slug) = q.strip_prefix("star:") {
return Target::Pornstar {
slug: slug.trim().to_string(),
slug: slug_dashed(slug),
page,
};
}
@@ -153,6 +155,25 @@ impl FullpornerProvider {
Target::Latest { page }
}
/// A free-form query that names a pornstar goes to her archive (probed on
/// the site; unknown names 404), otherwise it stays a keyword search.
async fn resolve_keyword(query: &str, page: u32, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let probe = Target::Pornstar { slug: slug.clone(), page: 1 };
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
probe_first_archive(
&mut requester,
&[Self::build_listing_url(&probe)],
&format!("{BASE_URL}/"),
|body| body.contains("class=\"video-card\""),
)
.await
.map(|_| Target::Pornstar { slug, page })
}
fn selector(value: &str) -> Result<Selector> {
Selector::parse(value)
.map_err(|e| Error::from(format!("selector `{value}` failed: {e}")))
@@ -394,7 +415,10 @@ impl Provider for FullpornerProvider {
.filter(|q| !q.is_empty())
.map(ToOwned::to_owned);
let target = Self::pick_target(normalized_query.as_deref(), page);
let mut target = Self::pick_target(normalized_query.as_deref(), page);
if let Target::Search { query, .. } = &target {
target = Self::resolve_keyword(query, page, &options).await.unwrap_or(target);
}
match Self::fetch_page(&target, per_page, &options).await {
Ok(items) => items,

View File

@@ -1,6 +1,6 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{
use crate::providers::{probe_first_archive, slug_dashed,
Provider, report_provider_error, report_provider_error_background, requester_or_default,
};
use crate::status::*;
@@ -109,12 +109,12 @@ impl FullpornerorgProvider {
if let Some(q) = query.map(str::trim).filter(|q| !q.is_empty()) {
if let Some(slug) = q.strip_prefix("pornstar:") {
return Target::Pornstar {
slug: slug.trim().to_string(),
slug: slug_dashed(slug),
};
}
if let Some(slug) = q.strip_prefix("star:") {
return Target::Pornstar {
slug: slug.trim().to_string(),
slug: slug_dashed(slug),
};
}
return Target::Search {
@@ -153,6 +153,25 @@ impl FullpornerorgProvider {
}
}
/// A free-form query that names a pornstar goes to her archive (probed on
/// the site; unknown names 404), otherwise it stays a keyword search.
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let target = Target::Pornstar { slug };
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
probe_first_archive(
&mut requester,
&[Self::build_listing_url(&target, 1)],
&format!("{BASE_URL}/"),
|body| body.contains("loop-video"),
)
.await
.map(|_| target)
}
fn html_headers(referer: &str) -> Vec<(String, String)> {
vec![
("User-Agent".to_string(), BROWSER_UA.to_string()),
@@ -446,7 +465,10 @@ impl Provider for FullpornerorgProvider {
.filter(|q| !q.is_empty())
.map(ToOwned::to_owned);
let target = Self::pick_target(normalized_query.as_deref());
let mut target = Self::pick_target(normalized_query.as_deref());
if let Target::Search { query } = &target {
target = Self::resolve_keyword(query, &options).await.unwrap_or(target);
}
match Self::fetch_page(target, page, per_page, &options).await {
Ok(items) => items,

View File

@@ -430,40 +430,13 @@ impl HoopornProvider {
.await
}
/// Fetches a listing page. A real 404 (unknown archive, page past the end)
/// is `None` and answers immediately; a Cloudflare challenge is handed to
/// the requester's full chain (FlareSolverr).
/// Fetches a listing page; `None` for a real 404 (see `Requester::get_probe`).
async fn fetch_html(requester: &mut Requester, url: &str) -> Result<Option<String>> {
let headers = vec![("Referer".to_string(), format!("{BASE_URL}/"))];
let mut last_err = String::new();
for _ in 0..2 {
match requester.get_raw_with_headers(url, headers.clone()).await {
Ok(response) => {
let status = response.status().as_u16();
let body = response.text().await.unwrap_or_default();
if status == 404 {
return Ok(None);
}
if status == 403
|| status == 503
|| Requester::looks_like_cloudflare_challenge(&body)
{
return requester
.get_with_headers(url, headers, None)
.await
.map(Some)
.map_err(|e| Error::from(format!("fetch failed for {url}: {e}")));
}
if status >= 500 || status == 429 {
last_err = format!("HTTP {status}");
continue;
}
return Ok(Some(body));
}
Err(e) => last_err = e.to_string(),
}
}
Err(Error::from(format!("fetch failed for {url}: {last_err}")))
requester
.get_probe(url, headers)
.await
.map_err(|e| Error::from(format!("fetch failed for {url}: {e}")))
}
async fn fetch(target: &Target, page: u32, options: &ServerOptions) -> Result<Vec<VideoItem>> {

View File

@@ -612,6 +612,66 @@ pub fn report_provider_error_background(provider_name: &str, context: &str, msg:
});
}
/// `Some(slug)` joined with `-` ("Gal Ritchie" -> "gal-ritchie") for archive URLs.
pub fn slug_dashed(value: &str) -> String {
let mut out = String::new();
let mut last_dash = true;
for c in value.trim().chars() {
if c.is_alphanumeric() {
out.extend(c.to_lowercase());
last_dash = false;
} else if !last_dash {
out.push('-');
last_dash = true;
}
}
out.trim_matches('-').to_string()
}
/// Same as [`slug_dashed`] without separators ("Family Strokes" -> "familystrokes").
pub fn slug_joined(value: &str) -> String {
slug_dashed(value).replace('-', "")
}
/// Probes candidate archive URLs (pornstar / studio / channel / tag pages) in
/// order and returns the index and body of the first that exists and
/// `has_items` accepts. A miss (404, empty page, transport error) moves on to
/// the next candidate; the caller falls back to keyword search on `None`.
/// Uses `Requester::get_probe`, so an unknown archive answers instantly and a
/// Cloudflare challenge is solved through FlareSolverr.
pub async fn probe_first_archive(
requester: &mut Requester,
urls: &[String],
referer: &str,
has_items: impl Fn(&str) -> bool,
) -> Option<(usize, String)> {
probe_first_archive_with(
requester,
urls,
vec![("Referer".to_string(), referer.to_string())],
has_items,
)
.await
}
/// [`probe_first_archive`] with caller-supplied request headers (JSON APIs
/// that demand a special `Accept`, extra tokens, ...).
pub async fn probe_first_archive_with(
requester: &mut Requester,
urls: &[String],
headers: Vec<(String, String)>,
has_items: impl Fn(&str) -> bool,
) -> Option<(usize, String)> {
for (index, url) in urls.iter().enumerate() {
if let Ok(Some(body)) = requester.get_probe(url, headers.clone()).await {
if has_items(&body) {
return Some((index, body));
}
}
}
None
}
pub fn requester_or_default(
options: &ServerOptions,
provider_name: &str,
@@ -1777,3 +1837,16 @@ mod tests {
);
}
}
#[cfg(test)]
mod slug_tests {
use super::{slug_dashed, slug_joined};
#[test]
fn slugs_names_for_archive_urls() {
assert_eq!(slug_dashed(" Gal Ritchie "), "gal-ritchie");
assert_eq!(slug_dashed("Mia Khalifa!!"), "mia-khalifa");
assert_eq!(slug_joined("Family Strokes"), "familystrokes");
assert_eq!(slug_dashed("???"), "");
}
}

View File

@@ -2,6 +2,7 @@ use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{
Provider, report_provider_error, report_provider_error_background, requester_or_default,
probe_first_archive,
};
use crate::status::*;
use crate::util::cache::VideoCache;
@@ -260,6 +261,34 @@ impl Porn4fansProvider {
Target::Latest
}
/// A free-form query that names a model or a tag goes to that archive
/// (probed on the site; unknown names 404), else keyword search.
async fn resolve_keyword(&self, query: &str, options: &ServerOptions) -> Option<Target> {
let slug = Self::slugify(query);
if slug.is_empty() {
return None;
}
let candidates = [
Target::Model { slug: slug.clone() },
Target::Tag { slug },
];
let urls: Vec<String> = candidates
.iter()
.map(|t| match t {
Target::Model { slug } => format!("{}/models/{slug}/", self.url),
Target::Tag { slug } => format!("{}/tags/{slug}/", self.url),
_ => String::new(),
})
.collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) =
probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
body.contains("custom_list_videos")
})
.await?;
candidates.into_iter().nth(index)
}
/// Build the async `get_block` URL plus the Referer to send with it.
fn build_block_url(&self, target: &Target, sort: &str, page: u32) -> (String, String) {
let sort_by = Self::sort_by(sort);
@@ -788,7 +817,12 @@ impl Provider for Porn4fansProvider {
options: ServerOptions,
) -> Vec<VideoItem> {
let page = page.parse::<u32>().unwrap_or(1).max(1);
let target = self.target_from_request(query.as_deref(), options.categories.as_deref());
let mut target = self.target_from_request(query.as_deref(), options.categories.as_deref());
if let Target::Search { query } = &target {
if let Some(found) = self.resolve_keyword(query, &options).await {
target = found;
}
}
match self
.get_videos_for_target(cache, &target, &sort, page, options)

View File

@@ -1,6 +1,9 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, proxy_aylo_thumb, report_provider_error, requester_or_default};
use crate::providers::{
Provider, probe_first_archive, proxy_aylo_thumb, report_provider_error, requester_or_default,
slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::parse_abbreviated_number;
@@ -45,6 +48,8 @@ error_chain! {
pub struct PornhubProvider {
url: String,
tag_map: Arc<RwLock<HashMap<String, TagInfo>>>,
/// Queries already probed on the site that are neither a pornstar, model nor channel.
probe_misses: Arc<RwLock<std::collections::HashSet<String>>>,
}
#[derive(Debug, Clone, Copy)]
@@ -91,6 +96,7 @@ impl PornhubProvider {
let provider = Self {
url: BASE_URL.to_string(),
tag_map: Arc::new(RwLock::new(HashMap::new())),
probe_misses: Arc::new(RwLock::new(std::collections::HashSet::new())),
};
provider.spawn_initial_load();
provider
@@ -588,6 +594,43 @@ impl PornhubProvider {
values.push(normalized.to_string());
}
/// A free-form query that is not in the loaded catalogs may still name a
/// pornstar, model or channel. Probe the site (unknown names 404 or
/// redirect away) and remember the hit in `tag_map` so later pages and
/// queries resolve without another request.
async fn learn_query_target(&self, query: &str, options: &ServerOptions) {
let key = query.trim().to_ascii_lowercase();
let slug = slug_dashed(query);
if slug.is_empty() || self.probe_misses.read().unwrap().contains(&key) {
return;
}
let kinds = [
QueryTargetKind::Pornstar,
QueryTargetKind::Model,
QueryTargetKind::Channel,
];
let urls: Vec<String> = kinds
.iter()
.map(|kind| format!("{}/{}/{slug}/videos", self.url, kind.path_segment()))
.collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "learn_query_target");
let hit = probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
body.contains("pcVideoListItem")
})
.await;
match hit {
Some((index, _)) => {
self.tag_map.write().unwrap().insert(
key,
TagInfo { kind: kinds[index], slug, title: query.trim().to_string() },
);
}
None => {
self.probe_misses.write().unwrap().insert(key);
}
}
}
async fn fetch_listing(
&self,
cache: VideoCache,
@@ -596,6 +639,11 @@ impl PornhubProvider {
query: Option<&str>,
options: ServerOptions,
) -> Result<Vec<VideoItem>> {
if let Some(q) = query.map(str::trim).filter(|q| !q.is_empty()) {
if self.parse_query_target(q).is_none() {
self.learn_query_target(q, &options).await;
}
}
if query.is_some() && self.tag_map.read().unwrap().is_empty() {
let _ = Self::load_tags(&self.url, Arc::clone(&self.tag_map)).await;
}

View File

@@ -560,6 +560,24 @@ impl Provider for RedgifsProvider {
let sort = Sort::from_sort_id(&sort);
let target = Self::pick_target(normalized_query, sort, &options);
// A single-word query that is an existing creator name goes to that
// creator's gifs (checked against the users endpoint); a creator
// without gifs falls back to the keyword search.
if let Target::Query(text, _) = &target {
if !text.contains(char::is_whitespace) {
if let Ok(Some(profile)) = self.fetch_user_profile(text, &options).await {
let creator = Target::Creator(profile.username);
if let Ok(items) = self
.fetch_target_items(creator, page, per_page, &options)
.await
{
if !items.is_empty() {
return items;
}
}
}
}
}
match self.fetch_target_items(target, page, per_page, &options).await {
Ok(items) => items,

View File

@@ -1,7 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{
Provider, report_provider_error, report_provider_error_background, requester_or_default,
Provider, probe_first_archive, report_provider_error, report_provider_error_background,
requester_or_default,
};
use crate::status::*;
use crate::util::cache::VideoCache;
@@ -517,6 +518,28 @@ impl SextbProvider {
Target::Search { slug }
}
/// A free-form query that names an actress or a studio outside the loaded
/// catalogs goes to that archive (probed on the site), else keyword search.
async fn resolve_keyword(&self, slug: &str, options: &ServerOptions) -> Option<Target> {
if slug.is_empty() {
return None;
}
let candidates = [
Target::Actress { slug: slug.to_string() },
Target::Studio { slug: slug.to_string() },
];
let urls: Vec<String> = candidates
.iter()
.map(|t| self.build_url_for_target(t, 1))
.collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
body.contains("tray-item") && !body.contains("No Video were found")
})
.await?;
candidates.into_iter().nth(index)
}
fn build_url_for_target(&self, target: &Target, page: u16) -> String {
match target {
Target::Section { section, sort } => {
@@ -1092,7 +1115,12 @@ impl SextbProvider {
per_page_limit: usize,
options: ServerOptions,
) -> Result<Vec<VideoItem>> {
let target = self.resolve_query_target(query, sort);
let mut target = self.resolve_query_target(query, sort);
if let Target::Search { slug } = &target {
if let Some(found) = self.resolve_keyword(slug, &options).await {
target = found;
}
}
let url = self.build_url_for_target(&target, page);
self.fetch_items_for_url(cache, url, per_page_limit, &options)
.await

View File

@@ -1,7 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{
Provider, build_proxy_url, report_provider_error, requester_or_default,
Provider, build_proxy_url, probe_first_archive, report_provider_error, requester_or_default,
slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
@@ -367,6 +368,26 @@ impl ThepornbunnyProvider {
}
}
/// A free-form query that names a pornstar, studio or tag goes to that
/// archive (probed on the site; unknown names redirect home), else keyword
/// search.
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let urls: Vec<String> = ["pornstar", "studio", "tag"]
.iter()
.map(|kind| format!("{BASE_URL}/{kind}/{slug}/"))
.collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{BASE_URL}/"), |body| {
body.contains("th-description")
})
.await?;
Some(Target::Archive(urls[index].trim_end_matches('/').to_string()))
}
fn resolve_option_target(options: &ServerOptions, sort: &str) -> Target {
if let Some(cat) = options.categories.as_deref() {
if cat.starts_with(BASE_URL) && cat != "all" {
@@ -476,10 +497,15 @@ impl Provider for ThepornbunnyProvider {
let page = page.parse::<u16>().unwrap_or(1).max(1);
let per_page_limit = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
let target = match query {
let mut target = match query {
Some(ref q) if !q.trim().is_empty() => Self::resolve_query_target(q.trim()),
_ => Self::resolve_option_target(&options, &sort),
};
if let Target::Search(query) = &target {
if let Some(found) = Self::resolve_keyword(query, &options).await {
target = found;
}
}
match self
.fetch_target(cache, target, page, per_page_limit, options)

View File

@@ -1,10 +1,12 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{
Provider, build_proxy_url, proxy_aylo_thumb, report_provider_error, requester_or_default,
Provider, build_proxy_url, probe_first_archive, proxy_aylo_thumb, report_provider_error,
requester_or_default, slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::requester::Requester;
use crate::util::parse_abbreviated_number;
use crate::util::time::parse_time_to_seconds;
use crate::videos::{ServerOptions, VideoFormat, VideoItem};
@@ -136,6 +138,25 @@ impl Tube8Provider {
]
}
/// A free-form query that names a pornstar or a channel goes to that
/// archive (probed on the site; unknown names 404), else keyword search.
async fn resolve_keyword(&self, requester: &mut Requester, query: &str) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let candidates = [
Target::Pornstar { slug: slug.clone() },
Target::Channel { slug },
];
let urls: Vec<String> = candidates.iter().map(|t| self.build_url(t, 1)).collect();
let (index, _) = probe_first_archive(requester, &urls, &format!("{BASE_URL}/"), |body| {
body.contains("video-box")
})
.await?;
candidates.into_iter().nth(index)
}
fn target_from_request(query: Option<&str>, sort: &str) -> Target {
if let Some(q) = query.map(str::trim).filter(|v| !v.is_empty()) {
let lower = q.to_ascii_lowercase();
@@ -439,7 +460,13 @@ impl Provider for Tube8Provider {
options: ServerOptions,
) -> Vec<VideoItem> {
let page = page.parse::<u16>().unwrap_or(1).max(1);
let target = Self::target_from_request(query.as_deref(), &sort);
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
let mut target = Self::target_from_request(query.as_deref(), &sort);
if let Target::Search { query } = &target {
if let Some(found) = self.resolve_keyword(&mut requester, query).await {
target = found;
}
}
let url = self.build_url(&target, page);
let old_items = match cache.get(&url) {
@@ -450,7 +477,6 @@ impl Provider for Tube8Provider {
None => vec![],
};
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
let text = match requester
.get_with_headers(&url, Self::html_headers(), None)
.await

View File

@@ -592,7 +592,7 @@ mod tests {
#[test]
fn decodes_cyrillic_obfuscated_video_url() {
let raw = "L2dldF9maWxlLzI1Lzg2\u{041c}D\u{0415}zNWFkND\u{0415}zODNm\u{041c}zI0ZmJhYjJlZTg3\u{041c}jRkYm\u{041c}zOWNiZTIwNjJl\u{041c}ci8y\u{041c}TgwOD\u{0410}w\u{041c}\u{0421}8y\u{041c}TgwODcwNS8y\u{041c}TgwODcwNV9ocS5tcDQvP2Q9\u{041c}czQ4NSZicj0y\u{041c}TgmdGk9\u{041c}Tc5\u{041c}TY0NjYyOQ~~";
let raw = "L2dldF9maWxlLzI1Lzg2\u{041c}D\u{0415}zNWFkND\u{0415}zODNm\u{041c}zI0ZmJhYjJlZTg3\u{041c}jRkYm\u{041c}zOWNiZTIwNjJl\u{041c}i8y\u{041c}TgwOD\u{0410}w\u{041c}\u{0421}8y\u{041c}TgwODcwNS8y\u{041c}TgwODcwNV9ocS5tcDQvP2Q9\u{041c}zQ4NSZicj0y\u{041c}TgmdGk9\u{041c}Tc5\u{041c}TY0NjYyOQ~~";
assert_eq!(
TxxxProvider::decode_video_url(raw).as_deref(),
Some("https://txxx.me/get_file/25/860135ad41383f324fbab2ee8724dbc39cbe2062e2/21808000/21808705/21808705_hq.mp4/?d=3485&br=218&ti=1791646629")

View File

@@ -1,6 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::providers::{
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::videos::{ServerOptions, VideoItem};
@@ -426,6 +428,23 @@ impl XgroovyProvider {
Target::Search(trimmed.to_string())
}
/// A free-form query that names a pornstar or a channel goes to that
/// archive (probed on the site; unknown names 404), else keyword search.
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let candidates = [Target::Pornstar(slug.clone()), Target::Channel(slug)];
let urls: Vec<String> = candidates.iter().map(|t| Self::target_url(t, 1)).collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{BASE_URL}/"), |body| {
body.contains("data-video-id")
})
.await?;
candidates.into_iter().nth(index)
}
fn resolve_option_target(&self, options: &ServerOptions) -> Target {
if let Some(cat) = options.categories.as_deref() {
if cat != "all" && !cat.is_empty() {
@@ -488,10 +507,15 @@ impl Provider for XgroovyProvider {
let page = page.parse::<u16>().unwrap_or(1).max(1);
let per_page = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
let target = match query {
let mut target = match query {
Some(q) if !q.trim().is_empty() => self.resolve_query_target(q.trim()),
_ => self.resolve_option_target(&options),
};
if let Target::Search(query) = &target {
if let Some(found) = Self::resolve_keyword(query, &options).await {
target = found;
}
}
match self
.fetch_target(cache, target, page, per_page, options)

View File

@@ -1,6 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::providers::{
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::parse_abbreviated_number;
@@ -94,6 +96,7 @@ enum Target {
Search(String),
Category(String),
Channel(String),
Pornstar(String),
}
#[derive(Debug, Clone)]
@@ -208,6 +211,7 @@ impl XhamsterProvider {
}
Target::Category(slug) => format!("{BASE_URL}/categories/{slug}"),
Target::Channel(slug) => format!("{BASE_URL}/channels/{slug}"),
Target::Pornstar(slug) => format!("{BASE_URL}/pornstars/{slug}"),
};
if page <= 1 {
base
@@ -385,6 +389,7 @@ impl XhamsterProvider {
match kind.trim().to_ascii_lowercase().as_str() {
"cat" | "category" => return Target::Category(slug),
"channel" | "channels" => return Target::Channel(slug),
"pornstar" | "pornstars" | "star" | "model" => return Target::Pornstar(slug),
_ => {}
}
}
@@ -401,6 +406,23 @@ impl XhamsterProvider {
Target::Search(trimmed.to_string())
}
/// A free-form query that names a pornstar or a channel goes to that
/// archive (probed on the site; unknown names 404), else keyword search.
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let candidates = [Target::Pornstar(slug.clone()), Target::Channel(slug)];
let urls: Vec<String> = candidates.iter().map(|t| Self::target_url(t, 1)).collect();
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{BASE_URL}/"), |body| {
body.contains("video-thumb-info__name")
})
.await?;
candidates.into_iter().nth(index)
}
fn resolve_sort_target(sort: &str) -> Target {
match sort.trim().to_ascii_lowercase().as_str() {
"popular" | "viewed" | "most_viewed" | "mostviewed" => Target::MostViewed,
@@ -472,10 +494,15 @@ impl Provider for XhamsterProvider {
let page = page.parse::<u16>().unwrap_or(1).max(1);
let per_page = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
let target = match query {
let mut target = match query {
Some(q) if !q.trim().is_empty() => self.resolve_query_target(q.trim()),
_ => self.resolve_option_target(&options, &sort),
};
if let Target::Search(query) = &target {
if let Some(found) = Self::resolve_keyword(query, &options).await {
target = found;
}
}
match self
.fetch_target(cache, target, page, per_page, options)

View File

@@ -1,6 +1,8 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::providers::{
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::parse_abbreviated_number;
@@ -86,6 +88,8 @@ enum Target {
Best,
Search(String),
Archive(String),
/// Channel / pornstar profile, served as JSON by `/channels/{slug}/videos/new/{n}`.
Channel(String),
}
#[derive(Debug, Clone)]
@@ -238,6 +242,9 @@ impl XvideosProvider {
format!("{BASE_URL}/?k={encoded}&p={}", page - 1)
}
}
Target::Channel(slug) => {
format!("{BASE_URL}/channels/{slug}/videos/new/{}", page.saturating_sub(1))
}
Target::Archive(base_url) => {
let base = base_url.trim_end_matches('/');
if page <= 1 {
@@ -528,9 +535,8 @@ impl XvideosProvider {
let slug = value.replace(' ', "_");
return Target::Archive(format!("{BASE_URL}/c/{slug}"));
}
"uploader" | "channel" | "profile" => {
let slug = value.replace(' ', "_").to_ascii_lowercase();
return Target::Archive(format!("{BASE_URL}/{slug}"));
"uploader" | "channel" | "profile" | "pornstar" | "star" | "model" => {
return Target::Channel(slug_dashed(value));
}
_ => {}
}
@@ -545,6 +551,83 @@ impl XvideosProvider {
Target::Search(trimmed.to_string())
}
/// A free-form query that names a channel / pornstar profile goes to its
/// archive (probed on the site; unknown names land on an empty profile),
/// else keyword search.
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let target = Target::Channel(slug);
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
probe_first_archive(
&mut requester,
&[Self::target_url(&target, 1)],
&format!("{BASE_URL}/"),
|body| body.contains("\"videos\":[{"),
)
.await
.map(|_| target)
}
/// Parses the JSON served for a channel / profile video list.
fn parse_channel_json(json: &str, limit: usize) -> Result<Vec<VideoItem>> {
let root: serde_json::Value = serde_json::from_str(json)
.map_err(|e| Error::from(format!("channel json: {e}")))?;
let Some(videos) = root.get("videos").and_then(|v| v.as_array()) else {
return Ok(vec![]);
};
let text = |v: &serde_json::Value, key: &str| -> String {
v.get(key).and_then(|x| x.as_str()).unwrap_or_default().trim().to_string()
};
let mut items = Vec::new();
for video in videos {
let eid = text(video, "eid");
let thumb = text(video, "i");
let title = {
let full = text(video, "tf");
if full.is_empty() { text(video, "t") } else { full }
};
if eid.is_empty() || thumb.is_empty() || title.is_empty() {
continue;
}
let id = video
.get("id")
.map(|x| x.to_string().trim_matches('"').to_string())
.unwrap_or_else(|| eid.clone());
let mut item = VideoItem::new(
id,
Self::decode_html(&title),
format!("{BASE_URL}/video.{eid}/x"),
CHANNEL_ID.to_string(),
thumb,
Self::parse_duration(&text(video, "d")),
);
item.views = Self::parse_views(&text(video, "n"));
item.rating = text(video, "r").trim_end_matches('%').parse::<f32>().ok();
let preview = text(video, "ipu");
if !preview.is_empty() {
item.preview = Some(preview);
}
let name = text(video, "pn");
let slug = text(video, "pu");
if !name.is_empty() {
item.uploader = Some(name);
let slug = slug.trim_matches('/').to_string();
if !slug.is_empty() {
item.uploaderUrl = Some(format!("{BASE_URL}/{slug}"));
item.uploaderId = Some(format!("{CHANNEL_ID}:{slug}"));
}
}
items.push(item);
if items.len() >= limit {
break;
}
}
Ok(items)
}
fn resolve_option_target(options: &ServerOptions, sort: &str) -> Target {
if let Some(cat) = options.categories.as_deref() {
if cat != "all" && !cat.is_empty() {
@@ -602,7 +685,11 @@ impl XvideosProvider {
return Ok(vec![]);
}
let items = Self::parse_listing(&html, per_page)?;
let items = if matches!(target, Target::Channel(_)) {
Self::parse_channel_json(&html, per_page)?
} else {
Self::parse_listing(&html, per_page)?
};
if !items.is_empty() {
cache.insert(cache_key, items.clone());
}
@@ -625,10 +712,15 @@ impl Provider for XvideosProvider {
let page = page.parse::<u16>().unwrap_or(1).max(1);
let per_page = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
let target = match query {
let mut target = match query {
Some(q) if !q.trim().is_empty() => Self::resolve_query_target(q.trim()),
_ => Self::resolve_option_target(&options, &sort),
};
if let Target::Search(query) = &target {
if let Some(found) = Self::resolve_keyword(query, &options).await {
target = found;
}
}
match self
.fetch_target(cache, target, page, per_page, options)

View File

@@ -1,8 +1,11 @@
use crate::DbPool;
use crate::api::ClientVersion;
use crate::providers::{Provider, report_provider_error, requester_or_default};
use crate::providers::{
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
};
use crate::status::*;
use crate::util::cache::VideoCache;
use crate::util::requester::Requester;
use crate::util::parse_abbreviated_number;
use crate::util::time::parse_time_to_seconds;
use crate::videos::{ServerOptions, VideoItem};
@@ -248,6 +251,33 @@ impl YoupornProvider {
}
}
/// A free-form query that names a pornstar, channel or amateur goes to that
/// archive (probed on the site; unknown names 404), else keyword search.
async fn resolve_keyword(
&self,
requester: &mut Requester,
query: &str,
sort: &str,
) -> Option<Target> {
let slug = slug_dashed(query);
if slug.is_empty() {
return None;
}
let sort = Self::normalized_sort(sort).to_string();
let candidates = [
Target::Pornstar { slug: slug.clone(), sort: sort.clone() },
Target::Channel { slug: slug.clone(), sort: sort.clone() },
Target::Amateur { slug, sort },
];
let urls: Vec<String> = candidates.iter().map(|t| self.build_url(t, 1)).collect();
let referer = format!("{}/", self.url.trim_end_matches('/'));
let (index, _) = probe_first_archive(requester, &urls, &referer, |body| {
body.contains("js_video-box")
})
.await?;
candidates.into_iter().nth(index)
}
fn build_url(&self, target: &Target, page: u8) -> String {
match target {
Target::Latest { sort } => format!(
@@ -543,7 +573,13 @@ impl Provider for YoupornProvider {
) -> Vec<VideoItem> {
let query = query.unwrap_or_default();
let page = page.parse::<u8>().unwrap_or(1);
let target = self.target_from_query(&query, &sort);
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
let mut target = self.target_from_query(&query, &sort);
if let Target::Search { query } = &target {
if let Some(found) = self.resolve_keyword(&mut requester, query, &sort).await {
target = found;
}
}
let video_url = self.build_url(&target, page);
let old_items = match cache.get(&video_url) {
@@ -554,7 +590,6 @@ impl Provider for YoupornProvider {
None => vec![],
};
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
let referer = format!("{}/", self.url.trim_end_matches('/'));
let text = match requester
.get_with_headers(&video_url, Self::html_headers(&referer), Some(Version::HTTP_11))

View File

@@ -661,6 +661,60 @@ impl Requester {
Ok(response)
}
fn path_of(url: &str) -> Option<String> {
url::Url::parse(url).ok().map(|u| u.path().to_string())
}
/// Existence probe for archive pages (pornstar / studio / tag / category).
///
/// Reads the raw response so a real 404/410 answers immediately with
/// `Ok(None)` instead of escalating through the Jina/FlareSolverr chain
/// (minutes per miss). A Cloudflare challenge (403/503/"Just a moment...")
/// is handed to the full chain so FlareSolverr solves it, and 429/5xx are
/// retried once.
pub async fn get_probe(
&mut self,
url: &str,
headers: Vec<(String, String)>,
) -> Result<Option<String>, AnyErr> {
let mut last_err = String::new();
for _ in 0..2 {
match self.get_raw_with_headers(url, headers.clone()).await {
Ok(response) => {
let status = response.status().as_u16();
// An unknown archive often redirects to the home page or a
// search page instead of answering 404: a redirect that
// lands on a different path is a miss.
let redirected_away = Self::path_of(url)
.zip(Some(response.uri().path().to_string()))
.is_some_and(|(asked, landed)| {
!asked.trim_end_matches('/').eq_ignore_ascii_case(landed.trim_end_matches('/'))
});
let body = response.text().await.unwrap_or_default();
if status == 404 || status == 410 {
return Ok(None);
}
if status == 403
|| status == 503
|| Self::looks_like_cloudflare_challenge(&body)
{
return self.get_with_headers(url, headers, None).await.map(Some);
}
if redirected_away {
return Ok(None);
}
if status >= 500 || status == 429 {
last_err = format!("HTTP {status}");
continue;
}
return Ok(Some(body));
}
Err(e) => last_err = e.to_string(),
}
}
Err(format!("probe failed for {url}: {last_err}").into())
}
pub async fn get(
&mut self,
url: &str,