Compare commits
16 Commits
665fb2a0ce
...
edf6732f8e
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
edf6732f8e | ||
|
|
697836b914 | ||
|
|
5598216381 | ||
|
|
4c29d27b3e | ||
|
|
a486bc5502 | ||
|
|
270b327f73 | ||
|
|
813e2ec2ca | ||
|
|
26ef830f6e | ||
|
|
e34801f6f8 | ||
|
|
2646ca5659 | ||
|
|
981ec29ad8 | ||
|
|
76b9d4e058 | ||
|
|
59ea0495a6 | ||
|
|
1f95549167 | ||
|
|
1e56fe2da5 | ||
|
|
bdde02fa8c |
42
docs/query-resolution-todo.md
Normal file
42
docs/query-resolution-todo.md
Normal file
@@ -0,0 +1,42 @@
|
||||
# Seamless query resolution todo
|
||||
|
||||
Goal: a free-form query ("gal ritchie", "family strokes") should reach the matching
|
||||
pornstar / studio / channel / tag archive **without** a `pornstar:` / `studio:` / `tag:` prefix,
|
||||
by asking the site (probe) and falling back to keyword search. Prefixes stay as explicit overrides.
|
||||
|
||||
Pattern (see `hooporn`, `cumlouder`): after known-category matching, probe page 1 of each entity
|
||||
archive for the slug with `Requester::get_probe` (404 = instant `None`, Cloudflare challenge goes to
|
||||
FlareSolverr, 5xx/429 retried once), use the first archive that yields cards (page N stays in that
|
||||
archive), else keyword search. A probe is only valid after checking what the site answers for an
|
||||
unknown slug (404 vs soft-404 that lists unrelated videos).
|
||||
|
||||
Status: `[ ]` open, `[x]` done and verified live (feed of the archive + unknown name falls back),
|
||||
`[-]` not applicable (reason).
|
||||
|
||||
| | Provider | Prefix kinds today | Plan / evidence |
|
||||
|---|---|---|---|
|
||||
| [x] | `hooporn` | cat, pornstar, studio | reference implementation |
|
||||
| [x] | `cumlouder` | cat, tag, series, girl | reference implementation (girl page, then tag) |
|
||||
| [x] | `fullporner` | cat, pornstar | probes `/pornstar/{slug}` (unknown = 404 with a suggestion grid, so status decides); `pornstar:` slug now dashed |
|
||||
| [x] | `fullpornerorg` | pornstar | probes `/videos/pornstar/{slug}/` (404 for unknown) |
|
||||
| [x] | `tube8` | tag, cat, channel, pornstar | probes pornstar then channel |
|
||||
| [x] | `youporn` | tag, channel, pornstar, amateur | probes pornstar, channel, amateur (plus existing background shortcut map) |
|
||||
| [x] | `porn4fans` | tag, cat, model | probes `/models/{slug}/` then `/tags/{slug}/` (both 404 when unknown) |
|
||||
| [x] | `xvideos` | tag, cat, uploader/channel/profile | probes `/channels/{slug}/videos/new/0` (JSON; pornstar profiles redirect there). **Also fixed**: the old `channel:`/`uploader:` prefix returned nothing (landing page cards are `/prof-video-click/` links); channels are now read from the JSON video list |
|
||||
| [x] | `xhamster` | cat, channel | added `Pornstar` target; probes pornstar then channel |
|
||||
| [x] | `xgroovy` | cat, pornstar, channel | probes pornstar then channel |
|
||||
| [x] | `thepornbunny` | tag, cat, studio, pornstar | probes pornstar, studio, tag (unknown slugs redirect home: `get_probe` now treats a redirect to another path as a miss) |
|
||||
| [x] | `pornhub` | channel, pornstar, model, user | background catalogs already matched listed names; unknown names are probed (`pornstar`, `model`, `channels`) and cached in `tag_map` (misses remembered) |
|
||||
| [x] | `clapdat` | tag, user | probes tag then user |
|
||||
| [x] | `fikfap` | tag, user | single-word query that is an existing profile (profile endpoint) goes to its posts, empty profile falls back to search |
|
||||
| [x] | `erome` | uploader, user | single-word query that is an existing `/{name}` with albums goes to the uploader |
|
||||
| [x] | `redgifs` | tag, user | single-word query that is an existing creator (users endpoint) goes to the creator, empty falls back |
|
||||
| [x] | `sextb` | (catalog matched) | catalog (actress/studio/genre) match first, then probes `/actress/{slug}` and `/studio/{slug}` for names outside the catalog |
|
||||
| [x] | `coomer` | u:, @service:user | single-word query that is an existing creator id (`/api/v1/{service}/user/{id}/profile`) goes to the creator. **Also fixed**: creator feeds are a bare JSON array, which the parser rejected (every creator feed was empty) |
|
||||
| [-] | `ripnsfw`, `hsex`, `omgxxx`, `eporner`, `borntobefuck`, `archivebate` | various | already seamless: bare names match background-loaded creator / studio / tag catalogs (or the whole catalog is filtered client-side) |
|
||||
| [-] | `perfectgirls`, `okxxx`, `pornhat`, `perverzija`, `porntrex` | sites/models/stars | bare names already resolve through the background-loaded `tag_map`; these clones build their own `wreq` client with no `ServerOptions`, so a live probe needs a separate refactor onto the shared requester |
|
||||
| [-] | `homoxxx` | models, pornstars, channels, categories, tags | legacy provider (own client, unpaginated archives, no `/models/` index on the site); not worth a probe until it is rewritten |
|
||||
| [-] | `kwiky`, `animeidhentai`, `hentaitv`, `xnxx`, `blowjobspro`, `txxx` | tag/cat | search already is the entity lookup, or the site has no pornstar/studio archives to probe (txxx: the API rejects `models`/`channels` sections) |
|
||||
| [-] | `fyptt` | cat | by design: bare category words are common search terms and must not be hijacked |
|
||||
| [-] | `notfans` | tag, category | tag URLs are hashed ids, so an unknown name cannot be probed; search covers it |
|
||||
| [-] | `allpornstream`, `jable`, `wowxxx`, `hornyleak`, `hotbunny`, `hentaimama`, `tikporn`, `camsoda` | actor/producer, model, studio, ... | Cloudflare-gated or down from this host (403/522, FlareSolverr wedged), so a probe cannot be verified here; each would add one or two FlareSolverr solves per search. Candidates for the same pattern once checked from the production network |
|
||||
@@ -1,7 +1,8 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{
|
||||
Provider, build_proxy_url, report_provider_error, requester_or_default, strip_url_scheme,
|
||||
Provider, build_proxy_url, probe_first_archive, report_provider_error, requester_or_default,
|
||||
slug_dashed, strip_url_scheme,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::time::parse_time_to_seconds;
|
||||
@@ -149,6 +150,29 @@ impl ClapdatProvider {
|
||||
}
|
||||
}
|
||||
|
||||
/// A free-form query that names a tag or a user goes to that archive
|
||||
/// (probed on the site; unknown names 404), else keyword search.
|
||||
async fn resolve_keyword(&self, query: &str, options: &ServerOptions) -> Option<Target> {
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let candidates = [
|
||||
Target::Tag { slug: slug.clone() },
|
||||
Target::User { username: slug },
|
||||
];
|
||||
let urls: Vec<String> = candidates
|
||||
.iter()
|
||||
.filter_map(|t| self.listing_url(t, 1))
|
||||
.collect();
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
|
||||
body.contains("/video/")
|
||||
})
|
||||
.await?;
|
||||
candidates.into_iter().nth(index)
|
||||
}
|
||||
|
||||
fn listing_url(&self, target: &Target, page: u16) -> Option<String> {
|
||||
let page = page.max(1);
|
||||
match target {
|
||||
@@ -491,7 +515,12 @@ impl Provider for ClapdatProvider {
|
||||
sort
|
||||
};
|
||||
let query_value = query.unwrap_or_default();
|
||||
let target = self.resolve_target(&query_value, &sort_value);
|
||||
let mut target = self.resolve_target(&query_value, &sort_value);
|
||||
if let Target::Search { query } = &target {
|
||||
if let Some(found) = self.resolve_keyword(query, &options).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
|
||||
let stubs = match &target {
|
||||
Target::Search { query } => match self
|
||||
|
||||
@@ -8,7 +8,7 @@ use wreq::Version;
|
||||
use crate::{
|
||||
DbPool,
|
||||
api::ClientVersion,
|
||||
providers::{Provider, requester_or_default},
|
||||
providers::{Provider, probe_first_archive_with, requester_or_default},
|
||||
status::{Channel, ChannelOption, FilterOption},
|
||||
util::cache::VideoCache,
|
||||
videos::{ServerOptions, VideoFormat, VideoItem},
|
||||
@@ -147,6 +147,43 @@ impl CoomerProvider {
|
||||
}
|
||||
}
|
||||
|
||||
/// A single-word query that is an existing creator id (the profile
|
||||
/// endpoint answers 404 otherwise) goes to that creator's posts, trying the
|
||||
/// filtered service first, else keyword search.
|
||||
async fn resolve_keyword(
|
||||
query: &str,
|
||||
service_filter: Option<&str>,
|
||||
options: &ServerOptions,
|
||||
) -> Option<Target> {
|
||||
let name = query.trim().trim_start_matches('@');
|
||||
if name.is_empty()
|
||||
|| !name
|
||||
.chars()
|
||||
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-' || c == '.')
|
||||
{
|
||||
return None;
|
||||
}
|
||||
let mut services = vec![Service::Onlyfans, Service::Fansly, Service::Candfans];
|
||||
if let Some(preferred) = Self::resolve_service(service_filter) {
|
||||
services.retain(|s| s.as_str() != preferred.as_str());
|
||||
services.insert(0, preferred);
|
||||
}
|
||||
let urls: Vec<String> = services
|
||||
.iter()
|
||||
.map(|s| format!("{BASE_URL}/api/v1/{}/user/{name}/profile", s.as_str()))
|
||||
.collect();
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
let (index, _) =
|
||||
probe_first_archive_with(&mut requester, &urls, Self::json_headers(), |body| {
|
||||
body.contains("\"id\"")
|
||||
})
|
||||
.await?;
|
||||
Some(Target::Uploader {
|
||||
service: services[index].clone(),
|
||||
username: name.to_string(),
|
||||
})
|
||||
}
|
||||
|
||||
fn json_headers() -> Vec<(String, String)> {
|
||||
vec![
|
||||
("Referer".to_string(), REFERER.to_string()),
|
||||
@@ -290,13 +327,16 @@ impl CoomerProvider {
|
||||
)));
|
||||
}
|
||||
}
|
||||
let posts: ApiPostsResponse = serde_json::from_str(&body).map_err(|err| {
|
||||
let posts: ApiPostsPayload = serde_json::from_str(&body).map_err(|err| {
|
||||
Error::from(format!(
|
||||
"json parse failed for {url}: {err}; body[:200]={}",
|
||||
&body.chars().take(200).collect::<String>()
|
||||
))
|
||||
})?;
|
||||
Ok(posts.posts)
|
||||
Ok(match posts {
|
||||
ApiPostsPayload::List(posts) => posts,
|
||||
ApiPostsPayload::Wrapped(wrapped) => wrapped.posts,
|
||||
})
|
||||
}
|
||||
|
||||
async fn fetch_items(
|
||||
@@ -543,6 +583,14 @@ struct ApiError {
|
||||
error: String,
|
||||
}
|
||||
|
||||
/// The global feed wraps posts in `{posts: [...]}`; a creator's feed is a bare array.
|
||||
#[derive(Debug, Deserialize)]
|
||||
#[serde(untagged)]
|
||||
enum ApiPostsPayload {
|
||||
List(Vec<ApiPost>),
|
||||
Wrapped(ApiPostsResponse),
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct ApiPostsResponse {
|
||||
#[serde(default)]
|
||||
@@ -601,8 +649,13 @@ impl Provider for CoomerProvider {
|
||||
.clamp(1, 64);
|
||||
|
||||
let service_filter = options.categories.as_deref().or(options.sites.as_deref());
|
||||
let target =
|
||||
let mut target =
|
||||
Self::resolve_target(&sort_value, query.as_deref(), service_filter);
|
||||
if let Target::Search { query } = &target {
|
||||
if let Some(found) = Self::resolve_keyword(query, service_filter, &options).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
|
||||
// Bump the ignored `Sort` variant warning off by referencing it.
|
||||
let _ = Sort::New;
|
||||
@@ -849,4 +902,18 @@ mod tests {
|
||||
let post: ApiPost = serde_json::from_str(json).expect("parse");
|
||||
assert!(CoomerProvider::build_video_item(post).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_bare_array_and_wrapped_post_payloads() {
|
||||
let bare = r#"[{"id":"1","user":"u","service":"onlyfans","title":"t","file":{},"attachments":[]}]"#;
|
||||
match serde_json::from_str::<ApiPostsPayload>(bare).expect("bare array") {
|
||||
ApiPostsPayload::List(posts) => assert_eq!(posts.len(), 1),
|
||||
other => panic!("expected list, got {other:?}"),
|
||||
}
|
||||
let wrapped = r#"{"posts":[{"id":"1"}],"count":1}"#;
|
||||
match serde_json::from_str::<ApiPostsPayload>(wrapped).expect("wrapped") {
|
||||
ApiPostsPayload::Wrapped(w) => assert_eq!(w.posts.len(), 1),
|
||||
other => panic!("expected wrapped, got {other:?}"),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{Provider, report_provider_error, requester_or_default};
|
||||
use crate::providers::{Provider, probe_first_archive, report_provider_error, requester_or_default};
|
||||
use crate::status::*;
|
||||
use crate::util::parse_abbreviated_number;
|
||||
use crate::videos::{ServerOptions, VideoItem};
|
||||
@@ -164,6 +164,26 @@ impl EromeProvider {
|
||||
(!slug.is_empty()).then_some(slug)
|
||||
}
|
||||
|
||||
/// A single-word query that is an existing uploader name (`/{name}` answers
|
||||
/// 404 otherwise) goes to that uploader's albums, else keyword search.
|
||||
async fn resolve_keyword(&self, query: &str, options: &ServerOptions) -> Option<Target> {
|
||||
let slug = query.trim().trim_start_matches('@');
|
||||
if slug.is_empty()
|
||||
|| !slug
|
||||
.chars()
|
||||
.all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-')
|
||||
{
|
||||
return None;
|
||||
}
|
||||
let url = format!("{}/{slug}", self.url);
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
probe_first_archive(&mut requester, &[url], &format!("{}/", self.url), |body| {
|
||||
body.contains("album-link")
|
||||
})
|
||||
.await
|
||||
.map(|_| Target::UploaderPosts { slug: slug.to_string() })
|
||||
}
|
||||
|
||||
fn resolve_target(&self, query: &str, options: &ServerOptions, sort: &str) -> Target {
|
||||
if let Some(site) = options.sites.as_deref() {
|
||||
let normalized = site.trim();
|
||||
@@ -367,7 +387,12 @@ impl Provider for EromeProvider {
|
||||
};
|
||||
let query_value = query.unwrap_or_default();
|
||||
let page_value = page.parse::<u16>().unwrap_or(1);
|
||||
let target = self.resolve_target(&query_value, &options, &sort_value);
|
||||
let mut target = self.resolve_target(&query_value, &options, &sort_value);
|
||||
if let Target::Search { query, .. } = &target {
|
||||
if let Some(found) = self.resolve_keyword(query, &options).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
let url = self.build_url_for_target(&target, page_value);
|
||||
|
||||
match self.fetch_page(&options, &url).await {
|
||||
|
||||
@@ -639,6 +639,24 @@ impl Provider for FikfapProvider {
|
||||
.map(str::trim)
|
||||
.filter(|value| !value.is_empty());
|
||||
let target = Self::pick_target(normalized_query, &sort, &options);
|
||||
// A single-word query that is an existing profile name goes to that
|
||||
// creator's posts (checked against the profile endpoint); a profile
|
||||
// without posts falls back to the keyword search.
|
||||
if let Target::Search(query) = &target {
|
||||
if !query.contains(char::is_whitespace) {
|
||||
if let Ok(Some(profile)) = self.fetch_user_profile(query, &options).await {
|
||||
let user = Target::User(profile.username);
|
||||
if let Ok(items) = self
|
||||
.fetch_target_items(user, page, per_page, &options)
|
||||
.await
|
||||
{
|
||||
if !items.is_empty() {
|
||||
return items;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
match self.fetch_target_items(target, page, per_page, &options).await {
|
||||
Ok(items) => items,
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{Provider, report_provider_error, requester_or_default};
|
||||
use crate::providers::{
|
||||
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
use crate::util::time::parse_time_to_seconds;
|
||||
@@ -134,13 +136,13 @@ impl FullpornerProvider {
|
||||
}
|
||||
if let Some(slug) = q.strip_prefix("pornstar:") {
|
||||
return Target::Pornstar {
|
||||
slug: slug.trim().to_string(),
|
||||
slug: slug_dashed(slug),
|
||||
page,
|
||||
};
|
||||
}
|
||||
if let Some(slug) = q.strip_prefix("star:") {
|
||||
return Target::Pornstar {
|
||||
slug: slug.trim().to_string(),
|
||||
slug: slug_dashed(slug),
|
||||
page,
|
||||
};
|
||||
}
|
||||
@@ -153,6 +155,25 @@ impl FullpornerProvider {
|
||||
Target::Latest { page }
|
||||
}
|
||||
|
||||
/// A free-form query that names a pornstar goes to her archive (probed on
|
||||
/// the site; unknown names 404), otherwise it stays a keyword search.
|
||||
async fn resolve_keyword(query: &str, page: u32, options: &ServerOptions) -> Option<Target> {
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let probe = Target::Pornstar { slug: slug.clone(), page: 1 };
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
probe_first_archive(
|
||||
&mut requester,
|
||||
&[Self::build_listing_url(&probe)],
|
||||
&format!("{BASE_URL}/"),
|
||||
|body| body.contains("class=\"video-card\""),
|
||||
)
|
||||
.await
|
||||
.map(|_| Target::Pornstar { slug, page })
|
||||
}
|
||||
|
||||
fn selector(value: &str) -> Result<Selector> {
|
||||
Selector::parse(value)
|
||||
.map_err(|e| Error::from(format!("selector `{value}` failed: {e}")))
|
||||
@@ -394,7 +415,10 @@ impl Provider for FullpornerProvider {
|
||||
.filter(|q| !q.is_empty())
|
||||
.map(ToOwned::to_owned);
|
||||
|
||||
let target = Self::pick_target(normalized_query.as_deref(), page);
|
||||
let mut target = Self::pick_target(normalized_query.as_deref(), page);
|
||||
if let Target::Search { query, .. } = &target {
|
||||
target = Self::resolve_keyword(query, page, &options).await.unwrap_or(target);
|
||||
}
|
||||
|
||||
match Self::fetch_page(&target, per_page, &options).await {
|
||||
Ok(items) => items,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{
|
||||
use crate::providers::{probe_first_archive, slug_dashed,
|
||||
Provider, report_provider_error, report_provider_error_background, requester_or_default,
|
||||
};
|
||||
use crate::status::*;
|
||||
@@ -109,12 +109,12 @@ impl FullpornerorgProvider {
|
||||
if let Some(q) = query.map(str::trim).filter(|q| !q.is_empty()) {
|
||||
if let Some(slug) = q.strip_prefix("pornstar:") {
|
||||
return Target::Pornstar {
|
||||
slug: slug.trim().to_string(),
|
||||
slug: slug_dashed(slug),
|
||||
};
|
||||
}
|
||||
if let Some(slug) = q.strip_prefix("star:") {
|
||||
return Target::Pornstar {
|
||||
slug: slug.trim().to_string(),
|
||||
slug: slug_dashed(slug),
|
||||
};
|
||||
}
|
||||
return Target::Search {
|
||||
@@ -153,6 +153,25 @@ impl FullpornerorgProvider {
|
||||
}
|
||||
}
|
||||
|
||||
/// A free-form query that names a pornstar goes to her archive (probed on
|
||||
/// the site; unknown names 404), otherwise it stays a keyword search.
|
||||
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let target = Target::Pornstar { slug };
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
probe_first_archive(
|
||||
&mut requester,
|
||||
&[Self::build_listing_url(&target, 1)],
|
||||
&format!("{BASE_URL}/"),
|
||||
|body| body.contains("loop-video"),
|
||||
)
|
||||
.await
|
||||
.map(|_| target)
|
||||
}
|
||||
|
||||
fn html_headers(referer: &str) -> Vec<(String, String)> {
|
||||
vec![
|
||||
("User-Agent".to_string(), BROWSER_UA.to_string()),
|
||||
@@ -446,7 +465,10 @@ impl Provider for FullpornerorgProvider {
|
||||
.filter(|q| !q.is_empty())
|
||||
.map(ToOwned::to_owned);
|
||||
|
||||
let target = Self::pick_target(normalized_query.as_deref());
|
||||
let mut target = Self::pick_target(normalized_query.as_deref());
|
||||
if let Target::Search { query } = &target {
|
||||
target = Self::resolve_keyword(query, &options).await.unwrap_or(target);
|
||||
}
|
||||
|
||||
match Self::fetch_page(target, page, per_page, &options).await {
|
||||
Ok(items) => items,
|
||||
|
||||
@@ -430,40 +430,13 @@ impl HoopornProvider {
|
||||
.await
|
||||
}
|
||||
|
||||
/// Fetches a listing page. A real 404 (unknown archive, page past the end)
|
||||
/// is `None` and answers immediately; a Cloudflare challenge is handed to
|
||||
/// the requester's full chain (FlareSolverr).
|
||||
/// Fetches a listing page; `None` for a real 404 (see `Requester::get_probe`).
|
||||
async fn fetch_html(requester: &mut Requester, url: &str) -> Result<Option<String>> {
|
||||
let headers = vec![("Referer".to_string(), format!("{BASE_URL}/"))];
|
||||
let mut last_err = String::new();
|
||||
for _ in 0..2 {
|
||||
match requester.get_raw_with_headers(url, headers.clone()).await {
|
||||
Ok(response) => {
|
||||
let status = response.status().as_u16();
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
if status == 404 {
|
||||
return Ok(None);
|
||||
}
|
||||
if status == 403
|
||||
|| status == 503
|
||||
|| Requester::looks_like_cloudflare_challenge(&body)
|
||||
{
|
||||
return requester
|
||||
.get_with_headers(url, headers, None)
|
||||
.await
|
||||
.map(Some)
|
||||
.map_err(|e| Error::from(format!("fetch failed for {url}: {e}")));
|
||||
}
|
||||
if status >= 500 || status == 429 {
|
||||
last_err = format!("HTTP {status}");
|
||||
continue;
|
||||
}
|
||||
return Ok(Some(body));
|
||||
}
|
||||
Err(e) => last_err = e.to_string(),
|
||||
}
|
||||
}
|
||||
Err(Error::from(format!("fetch failed for {url}: {last_err}")))
|
||||
requester
|
||||
.get_probe(url, headers)
|
||||
.await
|
||||
.map_err(|e| Error::from(format!("fetch failed for {url}: {e}")))
|
||||
}
|
||||
|
||||
async fn fetch(target: &Target, page: u32, options: &ServerOptions) -> Result<Vec<VideoItem>> {
|
||||
|
||||
@@ -612,6 +612,66 @@ pub fn report_provider_error_background(provider_name: &str, context: &str, msg:
|
||||
});
|
||||
}
|
||||
|
||||
/// `Some(slug)` joined with `-` ("Gal Ritchie" -> "gal-ritchie") for archive URLs.
|
||||
pub fn slug_dashed(value: &str) -> String {
|
||||
let mut out = String::new();
|
||||
let mut last_dash = true;
|
||||
for c in value.trim().chars() {
|
||||
if c.is_alphanumeric() {
|
||||
out.extend(c.to_lowercase());
|
||||
last_dash = false;
|
||||
} else if !last_dash {
|
||||
out.push('-');
|
||||
last_dash = true;
|
||||
}
|
||||
}
|
||||
out.trim_matches('-').to_string()
|
||||
}
|
||||
|
||||
/// Same as [`slug_dashed`] without separators ("Family Strokes" -> "familystrokes").
|
||||
pub fn slug_joined(value: &str) -> String {
|
||||
slug_dashed(value).replace('-', "")
|
||||
}
|
||||
|
||||
/// Probes candidate archive URLs (pornstar / studio / channel / tag pages) in
|
||||
/// order and returns the index and body of the first that exists and
|
||||
/// `has_items` accepts. A miss (404, empty page, transport error) moves on to
|
||||
/// the next candidate; the caller falls back to keyword search on `None`.
|
||||
/// Uses `Requester::get_probe`, so an unknown archive answers instantly and a
|
||||
/// Cloudflare challenge is solved through FlareSolverr.
|
||||
pub async fn probe_first_archive(
|
||||
requester: &mut Requester,
|
||||
urls: &[String],
|
||||
referer: &str,
|
||||
has_items: impl Fn(&str) -> bool,
|
||||
) -> Option<(usize, String)> {
|
||||
probe_first_archive_with(
|
||||
requester,
|
||||
urls,
|
||||
vec![("Referer".to_string(), referer.to_string())],
|
||||
has_items,
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
/// [`probe_first_archive`] with caller-supplied request headers (JSON APIs
|
||||
/// that demand a special `Accept`, extra tokens, ...).
|
||||
pub async fn probe_first_archive_with(
|
||||
requester: &mut Requester,
|
||||
urls: &[String],
|
||||
headers: Vec<(String, String)>,
|
||||
has_items: impl Fn(&str) -> bool,
|
||||
) -> Option<(usize, String)> {
|
||||
for (index, url) in urls.iter().enumerate() {
|
||||
if let Ok(Some(body)) = requester.get_probe(url, headers.clone()).await {
|
||||
if has_items(&body) {
|
||||
return Some((index, body));
|
||||
}
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
pub fn requester_or_default(
|
||||
options: &ServerOptions,
|
||||
provider_name: &str,
|
||||
@@ -1777,3 +1837,16 @@ mod tests {
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod slug_tests {
|
||||
use super::{slug_dashed, slug_joined};
|
||||
|
||||
#[test]
|
||||
fn slugs_names_for_archive_urls() {
|
||||
assert_eq!(slug_dashed(" Gal Ritchie "), "gal-ritchie");
|
||||
assert_eq!(slug_dashed("Mia Khalifa!!"), "mia-khalifa");
|
||||
assert_eq!(slug_joined("Family Strokes"), "familystrokes");
|
||||
assert_eq!(slug_dashed("???"), "");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,6 +2,7 @@ use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{
|
||||
Provider, report_provider_error, report_provider_error_background, requester_or_default,
|
||||
probe_first_archive,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
@@ -260,6 +261,34 @@ impl Porn4fansProvider {
|
||||
Target::Latest
|
||||
}
|
||||
|
||||
/// A free-form query that names a model or a tag goes to that archive
|
||||
/// (probed on the site; unknown names 404), else keyword search.
|
||||
async fn resolve_keyword(&self, query: &str, options: &ServerOptions) -> Option<Target> {
|
||||
let slug = Self::slugify(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let candidates = [
|
||||
Target::Model { slug: slug.clone() },
|
||||
Target::Tag { slug },
|
||||
];
|
||||
let urls: Vec<String> = candidates
|
||||
.iter()
|
||||
.map(|t| match t {
|
||||
Target::Model { slug } => format!("{}/models/{slug}/", self.url),
|
||||
Target::Tag { slug } => format!("{}/tags/{slug}/", self.url),
|
||||
_ => String::new(),
|
||||
})
|
||||
.collect();
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
let (index, _) =
|
||||
probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
|
||||
body.contains("custom_list_videos")
|
||||
})
|
||||
.await?;
|
||||
candidates.into_iter().nth(index)
|
||||
}
|
||||
|
||||
/// Build the async `get_block` URL plus the Referer to send with it.
|
||||
fn build_block_url(&self, target: &Target, sort: &str, page: u32) -> (String, String) {
|
||||
let sort_by = Self::sort_by(sort);
|
||||
@@ -788,7 +817,12 @@ impl Provider for Porn4fansProvider {
|
||||
options: ServerOptions,
|
||||
) -> Vec<VideoItem> {
|
||||
let page = page.parse::<u32>().unwrap_or(1).max(1);
|
||||
let target = self.target_from_request(query.as_deref(), options.categories.as_deref());
|
||||
let mut target = self.target_from_request(query.as_deref(), options.categories.as_deref());
|
||||
if let Target::Search { query } = &target {
|
||||
if let Some(found) = self.resolve_keyword(query, &options).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
|
||||
match self
|
||||
.get_videos_for_target(cache, &target, &sort, page, options)
|
||||
|
||||
@@ -1,6 +1,9 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{Provider, proxy_aylo_thumb, report_provider_error, requester_or_default};
|
||||
use crate::providers::{
|
||||
Provider, probe_first_archive, proxy_aylo_thumb, report_provider_error, requester_or_default,
|
||||
slug_dashed,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
use crate::util::parse_abbreviated_number;
|
||||
@@ -45,6 +48,8 @@ error_chain! {
|
||||
pub struct PornhubProvider {
|
||||
url: String,
|
||||
tag_map: Arc<RwLock<HashMap<String, TagInfo>>>,
|
||||
/// Queries already probed on the site that are neither a pornstar, model nor channel.
|
||||
probe_misses: Arc<RwLock<std::collections::HashSet<String>>>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
@@ -91,6 +96,7 @@ impl PornhubProvider {
|
||||
let provider = Self {
|
||||
url: BASE_URL.to_string(),
|
||||
tag_map: Arc::new(RwLock::new(HashMap::new())),
|
||||
probe_misses: Arc::new(RwLock::new(std::collections::HashSet::new())),
|
||||
};
|
||||
provider.spawn_initial_load();
|
||||
provider
|
||||
@@ -588,6 +594,43 @@ impl PornhubProvider {
|
||||
values.push(normalized.to_string());
|
||||
}
|
||||
|
||||
/// A free-form query that is not in the loaded catalogs may still name a
|
||||
/// pornstar, model or channel. Probe the site (unknown names 404 or
|
||||
/// redirect away) and remember the hit in `tag_map` so later pages and
|
||||
/// queries resolve without another request.
|
||||
async fn learn_query_target(&self, query: &str, options: &ServerOptions) {
|
||||
let key = query.trim().to_ascii_lowercase();
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() || self.probe_misses.read().unwrap().contains(&key) {
|
||||
return;
|
||||
}
|
||||
let kinds = [
|
||||
QueryTargetKind::Pornstar,
|
||||
QueryTargetKind::Model,
|
||||
QueryTargetKind::Channel,
|
||||
];
|
||||
let urls: Vec<String> = kinds
|
||||
.iter()
|
||||
.map(|kind| format!("{}/{}/{slug}/videos", self.url, kind.path_segment()))
|
||||
.collect();
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "learn_query_target");
|
||||
let hit = probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
|
||||
body.contains("pcVideoListItem")
|
||||
})
|
||||
.await;
|
||||
match hit {
|
||||
Some((index, _)) => {
|
||||
self.tag_map.write().unwrap().insert(
|
||||
key,
|
||||
TagInfo { kind: kinds[index], slug, title: query.trim().to_string() },
|
||||
);
|
||||
}
|
||||
None => {
|
||||
self.probe_misses.write().unwrap().insert(key);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn fetch_listing(
|
||||
&self,
|
||||
cache: VideoCache,
|
||||
@@ -596,6 +639,11 @@ impl PornhubProvider {
|
||||
query: Option<&str>,
|
||||
options: ServerOptions,
|
||||
) -> Result<Vec<VideoItem>> {
|
||||
if let Some(q) = query.map(str::trim).filter(|q| !q.is_empty()) {
|
||||
if self.parse_query_target(q).is_none() {
|
||||
self.learn_query_target(q, &options).await;
|
||||
}
|
||||
}
|
||||
if query.is_some() && self.tag_map.read().unwrap().is_empty() {
|
||||
let _ = Self::load_tags(&self.url, Arc::clone(&self.tag_map)).await;
|
||||
}
|
||||
|
||||
@@ -560,6 +560,24 @@ impl Provider for RedgifsProvider {
|
||||
|
||||
let sort = Sort::from_sort_id(&sort);
|
||||
let target = Self::pick_target(normalized_query, sort, &options);
|
||||
// A single-word query that is an existing creator name goes to that
|
||||
// creator's gifs (checked against the users endpoint); a creator
|
||||
// without gifs falls back to the keyword search.
|
||||
if let Target::Query(text, _) = &target {
|
||||
if !text.contains(char::is_whitespace) {
|
||||
if let Ok(Some(profile)) = self.fetch_user_profile(text, &options).await {
|
||||
let creator = Target::Creator(profile.username);
|
||||
if let Ok(items) = self
|
||||
.fetch_target_items(creator, page, per_page, &options)
|
||||
.await
|
||||
{
|
||||
if !items.is_empty() {
|
||||
return items;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
match self.fetch_target_items(target, page, per_page, &options).await {
|
||||
Ok(items) => items,
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{
|
||||
Provider, report_provider_error, report_provider_error_background, requester_or_default,
|
||||
Provider, probe_first_archive, report_provider_error, report_provider_error_background,
|
||||
requester_or_default,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
@@ -517,6 +518,28 @@ impl SextbProvider {
|
||||
Target::Search { slug }
|
||||
}
|
||||
|
||||
/// A free-form query that names an actress or a studio outside the loaded
|
||||
/// catalogs goes to that archive (probed on the site), else keyword search.
|
||||
async fn resolve_keyword(&self, slug: &str, options: &ServerOptions) -> Option<Target> {
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let candidates = [
|
||||
Target::Actress { slug: slug.to_string() },
|
||||
Target::Studio { slug: slug.to_string() },
|
||||
];
|
||||
let urls: Vec<String> = candidates
|
||||
.iter()
|
||||
.map(|t| self.build_url_for_target(t, 1))
|
||||
.collect();
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{}/", self.url), |body| {
|
||||
body.contains("tray-item") && !body.contains("No Video were found")
|
||||
})
|
||||
.await?;
|
||||
candidates.into_iter().nth(index)
|
||||
}
|
||||
|
||||
fn build_url_for_target(&self, target: &Target, page: u16) -> String {
|
||||
match target {
|
||||
Target::Section { section, sort } => {
|
||||
@@ -1092,7 +1115,12 @@ impl SextbProvider {
|
||||
per_page_limit: usize,
|
||||
options: ServerOptions,
|
||||
) -> Result<Vec<VideoItem>> {
|
||||
let target = self.resolve_query_target(query, sort);
|
||||
let mut target = self.resolve_query_target(query, sort);
|
||||
if let Target::Search { slug } = &target {
|
||||
if let Some(found) = self.resolve_keyword(slug, &options).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
let url = self.build_url_for_target(&target, page);
|
||||
self.fetch_items_for_url(cache, url, per_page_limit, &options)
|
||||
.await
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{
|
||||
Provider, build_proxy_url, report_provider_error, requester_or_default,
|
||||
Provider, build_proxy_url, probe_first_archive, report_provider_error, requester_or_default,
|
||||
slug_dashed,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
@@ -367,6 +368,26 @@ impl ThepornbunnyProvider {
|
||||
}
|
||||
}
|
||||
|
||||
/// A free-form query that names a pornstar, studio or tag goes to that
|
||||
/// archive (probed on the site; unknown names redirect home), else keyword
|
||||
/// search.
|
||||
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let urls: Vec<String> = ["pornstar", "studio", "tag"]
|
||||
.iter()
|
||||
.map(|kind| format!("{BASE_URL}/{kind}/{slug}/"))
|
||||
.collect();
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{BASE_URL}/"), |body| {
|
||||
body.contains("th-description")
|
||||
})
|
||||
.await?;
|
||||
Some(Target::Archive(urls[index].trim_end_matches('/').to_string()))
|
||||
}
|
||||
|
||||
fn resolve_option_target(options: &ServerOptions, sort: &str) -> Target {
|
||||
if let Some(cat) = options.categories.as_deref() {
|
||||
if cat.starts_with(BASE_URL) && cat != "all" {
|
||||
@@ -476,10 +497,15 @@ impl Provider for ThepornbunnyProvider {
|
||||
let page = page.parse::<u16>().unwrap_or(1).max(1);
|
||||
let per_page_limit = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
|
||||
|
||||
let target = match query {
|
||||
let mut target = match query {
|
||||
Some(ref q) if !q.trim().is_empty() => Self::resolve_query_target(q.trim()),
|
||||
_ => Self::resolve_option_target(&options, &sort),
|
||||
};
|
||||
if let Target::Search(query) = &target {
|
||||
if let Some(found) = Self::resolve_keyword(query, &options).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
|
||||
match self
|
||||
.fetch_target(cache, target, page, per_page_limit, options)
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{
|
||||
Provider, build_proxy_url, proxy_aylo_thumb, report_provider_error, requester_or_default,
|
||||
Provider, build_proxy_url, probe_first_archive, proxy_aylo_thumb, report_provider_error,
|
||||
requester_or_default, slug_dashed,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
use crate::util::requester::Requester;
|
||||
use crate::util::parse_abbreviated_number;
|
||||
use crate::util::time::parse_time_to_seconds;
|
||||
use crate::videos::{ServerOptions, VideoFormat, VideoItem};
|
||||
@@ -136,6 +138,25 @@ impl Tube8Provider {
|
||||
]
|
||||
}
|
||||
|
||||
/// A free-form query that names a pornstar or a channel goes to that
|
||||
/// archive (probed on the site; unknown names 404), else keyword search.
|
||||
async fn resolve_keyword(&self, requester: &mut Requester, query: &str) -> Option<Target> {
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let candidates = [
|
||||
Target::Pornstar { slug: slug.clone() },
|
||||
Target::Channel { slug },
|
||||
];
|
||||
let urls: Vec<String> = candidates.iter().map(|t| self.build_url(t, 1)).collect();
|
||||
let (index, _) = probe_first_archive(requester, &urls, &format!("{BASE_URL}/"), |body| {
|
||||
body.contains("video-box")
|
||||
})
|
||||
.await?;
|
||||
candidates.into_iter().nth(index)
|
||||
}
|
||||
|
||||
fn target_from_request(query: Option<&str>, sort: &str) -> Target {
|
||||
if let Some(q) = query.map(str::trim).filter(|v| !v.is_empty()) {
|
||||
let lower = q.to_ascii_lowercase();
|
||||
@@ -439,7 +460,13 @@ impl Provider for Tube8Provider {
|
||||
options: ServerOptions,
|
||||
) -> Vec<VideoItem> {
|
||||
let page = page.parse::<u16>().unwrap_or(1).max(1);
|
||||
let target = Self::target_from_request(query.as_deref(), &sort);
|
||||
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
|
||||
let mut target = Self::target_from_request(query.as_deref(), &sort);
|
||||
if let Target::Search { query } = &target {
|
||||
if let Some(found) = self.resolve_keyword(&mut requester, query).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
let url = self.build_url(&target, page);
|
||||
|
||||
let old_items = match cache.get(&url) {
|
||||
@@ -450,7 +477,6 @@ impl Provider for Tube8Provider {
|
||||
None => vec![],
|
||||
};
|
||||
|
||||
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
|
||||
let text = match requester
|
||||
.get_with_headers(&url, Self::html_headers(), None)
|
||||
.await
|
||||
|
||||
@@ -592,7 +592,7 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn decodes_cyrillic_obfuscated_video_url() {
|
||||
let raw = "L2dldF9maWxlLzI1Lzg2\u{041c}D\u{0415}zNWFkND\u{0415}zODNm\u{041c}zI0ZmJhYjJlZTg3\u{041c}jRkYm\u{041c}zOWNiZTIwNjJl\u{041c}ci8y\u{041c}TgwOD\u{0410}w\u{041c}\u{0421}8y\u{041c}TgwODcwNS8y\u{041c}TgwODcwNV9ocS5tcDQvP2Q9\u{041c}czQ4NSZicj0y\u{041c}TgmdGk9\u{041c}Tc5\u{041c}TY0NjYyOQ~~";
|
||||
let raw = "L2dldF9maWxlLzI1Lzg2\u{041c}D\u{0415}zNWFkND\u{0415}zODNm\u{041c}zI0ZmJhYjJlZTg3\u{041c}jRkYm\u{041c}zOWNiZTIwNjJl\u{041c}i8y\u{041c}TgwOD\u{0410}w\u{041c}\u{0421}8y\u{041c}TgwODcwNS8y\u{041c}TgwODcwNV9ocS5tcDQvP2Q9\u{041c}zQ4NSZicj0y\u{041c}TgmdGk9\u{041c}Tc5\u{041c}TY0NjYyOQ~~";
|
||||
assert_eq!(
|
||||
TxxxProvider::decode_video_url(raw).as_deref(),
|
||||
Some("https://txxx.me/get_file/25/860135ad41383f324fbab2ee8724dbc39cbe2062e2/21808000/21808705/21808705_hq.mp4/?d=3485&br=218&ti=1791646629")
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{Provider, report_provider_error, requester_or_default};
|
||||
use crate::providers::{
|
||||
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
use crate::videos::{ServerOptions, VideoItem};
|
||||
@@ -426,6 +428,23 @@ impl XgroovyProvider {
|
||||
Target::Search(trimmed.to_string())
|
||||
}
|
||||
|
||||
/// A free-form query that names a pornstar or a channel goes to that
|
||||
/// archive (probed on the site; unknown names 404), else keyword search.
|
||||
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let candidates = [Target::Pornstar(slug.clone()), Target::Channel(slug)];
|
||||
let urls: Vec<String> = candidates.iter().map(|t| Self::target_url(t, 1)).collect();
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{BASE_URL}/"), |body| {
|
||||
body.contains("data-video-id")
|
||||
})
|
||||
.await?;
|
||||
candidates.into_iter().nth(index)
|
||||
}
|
||||
|
||||
fn resolve_option_target(&self, options: &ServerOptions) -> Target {
|
||||
if let Some(cat) = options.categories.as_deref() {
|
||||
if cat != "all" && !cat.is_empty() {
|
||||
@@ -488,10 +507,15 @@ impl Provider for XgroovyProvider {
|
||||
let page = page.parse::<u16>().unwrap_or(1).max(1);
|
||||
let per_page = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
|
||||
|
||||
let target = match query {
|
||||
let mut target = match query {
|
||||
Some(q) if !q.trim().is_empty() => self.resolve_query_target(q.trim()),
|
||||
_ => self.resolve_option_target(&options),
|
||||
};
|
||||
if let Target::Search(query) = &target {
|
||||
if let Some(found) = Self::resolve_keyword(query, &options).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
|
||||
match self
|
||||
.fetch_target(cache, target, page, per_page, options)
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{Provider, report_provider_error, requester_or_default};
|
||||
use crate::providers::{
|
||||
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
use crate::util::parse_abbreviated_number;
|
||||
@@ -94,6 +96,7 @@ enum Target {
|
||||
Search(String),
|
||||
Category(String),
|
||||
Channel(String),
|
||||
Pornstar(String),
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
@@ -208,6 +211,7 @@ impl XhamsterProvider {
|
||||
}
|
||||
Target::Category(slug) => format!("{BASE_URL}/categories/{slug}"),
|
||||
Target::Channel(slug) => format!("{BASE_URL}/channels/{slug}"),
|
||||
Target::Pornstar(slug) => format!("{BASE_URL}/pornstars/{slug}"),
|
||||
};
|
||||
if page <= 1 {
|
||||
base
|
||||
@@ -385,6 +389,7 @@ impl XhamsterProvider {
|
||||
match kind.trim().to_ascii_lowercase().as_str() {
|
||||
"cat" | "category" => return Target::Category(slug),
|
||||
"channel" | "channels" => return Target::Channel(slug),
|
||||
"pornstar" | "pornstars" | "star" | "model" => return Target::Pornstar(slug),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
@@ -401,6 +406,23 @@ impl XhamsterProvider {
|
||||
Target::Search(trimmed.to_string())
|
||||
}
|
||||
|
||||
/// A free-form query that names a pornstar or a channel goes to that
|
||||
/// archive (probed on the site; unknown names 404), else keyword search.
|
||||
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let candidates = [Target::Pornstar(slug.clone()), Target::Channel(slug)];
|
||||
let urls: Vec<String> = candidates.iter().map(|t| Self::target_url(t, 1)).collect();
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
let (index, _) = probe_first_archive(&mut requester, &urls, &format!("{BASE_URL}/"), |body| {
|
||||
body.contains("video-thumb-info__name")
|
||||
})
|
||||
.await?;
|
||||
candidates.into_iter().nth(index)
|
||||
}
|
||||
|
||||
fn resolve_sort_target(sort: &str) -> Target {
|
||||
match sort.trim().to_ascii_lowercase().as_str() {
|
||||
"popular" | "viewed" | "most_viewed" | "mostviewed" => Target::MostViewed,
|
||||
@@ -472,10 +494,15 @@ impl Provider for XhamsterProvider {
|
||||
let page = page.parse::<u16>().unwrap_or(1).max(1);
|
||||
let per_page = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
|
||||
|
||||
let target = match query {
|
||||
let mut target = match query {
|
||||
Some(q) if !q.trim().is_empty() => self.resolve_query_target(q.trim()),
|
||||
_ => self.resolve_option_target(&options, &sort),
|
||||
};
|
||||
if let Target::Search(query) = &target {
|
||||
if let Some(found) = Self::resolve_keyword(query, &options).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
|
||||
match self
|
||||
.fetch_target(cache, target, page, per_page, options)
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{Provider, report_provider_error, requester_or_default};
|
||||
use crate::providers::{
|
||||
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
use crate::util::parse_abbreviated_number;
|
||||
@@ -86,6 +88,8 @@ enum Target {
|
||||
Best,
|
||||
Search(String),
|
||||
Archive(String),
|
||||
/// Channel / pornstar profile, served as JSON by `/channels/{slug}/videos/new/{n}`.
|
||||
Channel(String),
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
@@ -238,6 +242,9 @@ impl XvideosProvider {
|
||||
format!("{BASE_URL}/?k={encoded}&p={}", page - 1)
|
||||
}
|
||||
}
|
||||
Target::Channel(slug) => {
|
||||
format!("{BASE_URL}/channels/{slug}/videos/new/{}", page.saturating_sub(1))
|
||||
}
|
||||
Target::Archive(base_url) => {
|
||||
let base = base_url.trim_end_matches('/');
|
||||
if page <= 1 {
|
||||
@@ -528,9 +535,8 @@ impl XvideosProvider {
|
||||
let slug = value.replace(' ', "_");
|
||||
return Target::Archive(format!("{BASE_URL}/c/{slug}"));
|
||||
}
|
||||
"uploader" | "channel" | "profile" => {
|
||||
let slug = value.replace(' ', "_").to_ascii_lowercase();
|
||||
return Target::Archive(format!("{BASE_URL}/{slug}"));
|
||||
"uploader" | "channel" | "profile" | "pornstar" | "star" | "model" => {
|
||||
return Target::Channel(slug_dashed(value));
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
@@ -545,6 +551,83 @@ impl XvideosProvider {
|
||||
Target::Search(trimmed.to_string())
|
||||
}
|
||||
|
||||
/// A free-form query that names a channel / pornstar profile goes to its
|
||||
/// archive (probed on the site; unknown names land on an empty profile),
|
||||
/// else keyword search.
|
||||
async fn resolve_keyword(query: &str, options: &ServerOptions) -> Option<Target> {
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let target = Target::Channel(slug);
|
||||
let mut requester = requester_or_default(options, CHANNEL_ID, "resolve_keyword");
|
||||
probe_first_archive(
|
||||
&mut requester,
|
||||
&[Self::target_url(&target, 1)],
|
||||
&format!("{BASE_URL}/"),
|
||||
|body| body.contains("\"videos\":[{"),
|
||||
)
|
||||
.await
|
||||
.map(|_| target)
|
||||
}
|
||||
|
||||
/// Parses the JSON served for a channel / profile video list.
|
||||
fn parse_channel_json(json: &str, limit: usize) -> Result<Vec<VideoItem>> {
|
||||
let root: serde_json::Value = serde_json::from_str(json)
|
||||
.map_err(|e| Error::from(format!("channel json: {e}")))?;
|
||||
let Some(videos) = root.get("videos").and_then(|v| v.as_array()) else {
|
||||
return Ok(vec![]);
|
||||
};
|
||||
let text = |v: &serde_json::Value, key: &str| -> String {
|
||||
v.get(key).and_then(|x| x.as_str()).unwrap_or_default().trim().to_string()
|
||||
};
|
||||
let mut items = Vec::new();
|
||||
for video in videos {
|
||||
let eid = text(video, "eid");
|
||||
let thumb = text(video, "i");
|
||||
let title = {
|
||||
let full = text(video, "tf");
|
||||
if full.is_empty() { text(video, "t") } else { full }
|
||||
};
|
||||
if eid.is_empty() || thumb.is_empty() || title.is_empty() {
|
||||
continue;
|
||||
}
|
||||
let id = video
|
||||
.get("id")
|
||||
.map(|x| x.to_string().trim_matches('"').to_string())
|
||||
.unwrap_or_else(|| eid.clone());
|
||||
let mut item = VideoItem::new(
|
||||
id,
|
||||
Self::decode_html(&title),
|
||||
format!("{BASE_URL}/video.{eid}/x"),
|
||||
CHANNEL_ID.to_string(),
|
||||
thumb,
|
||||
Self::parse_duration(&text(video, "d")),
|
||||
);
|
||||
item.views = Self::parse_views(&text(video, "n"));
|
||||
item.rating = text(video, "r").trim_end_matches('%').parse::<f32>().ok();
|
||||
let preview = text(video, "ipu");
|
||||
if !preview.is_empty() {
|
||||
item.preview = Some(preview);
|
||||
}
|
||||
let name = text(video, "pn");
|
||||
let slug = text(video, "pu");
|
||||
if !name.is_empty() {
|
||||
item.uploader = Some(name);
|
||||
let slug = slug.trim_matches('/').to_string();
|
||||
if !slug.is_empty() {
|
||||
item.uploaderUrl = Some(format!("{BASE_URL}/{slug}"));
|
||||
item.uploaderId = Some(format!("{CHANNEL_ID}:{slug}"));
|
||||
}
|
||||
}
|
||||
items.push(item);
|
||||
if items.len() >= limit {
|
||||
break;
|
||||
}
|
||||
}
|
||||
Ok(items)
|
||||
}
|
||||
|
||||
fn resolve_option_target(options: &ServerOptions, sort: &str) -> Target {
|
||||
if let Some(cat) = options.categories.as_deref() {
|
||||
if cat != "all" && !cat.is_empty() {
|
||||
@@ -602,7 +685,11 @@ impl XvideosProvider {
|
||||
return Ok(vec![]);
|
||||
}
|
||||
|
||||
let items = Self::parse_listing(&html, per_page)?;
|
||||
let items = if matches!(target, Target::Channel(_)) {
|
||||
Self::parse_channel_json(&html, per_page)?
|
||||
} else {
|
||||
Self::parse_listing(&html, per_page)?
|
||||
};
|
||||
if !items.is_empty() {
|
||||
cache.insert(cache_key, items.clone());
|
||||
}
|
||||
@@ -625,10 +712,15 @@ impl Provider for XvideosProvider {
|
||||
let page = page.parse::<u16>().unwrap_or(1).max(1);
|
||||
let per_page = per_page.parse::<usize>().unwrap_or(10).clamp(1, 60);
|
||||
|
||||
let target = match query {
|
||||
let mut target = match query {
|
||||
Some(q) if !q.trim().is_empty() => Self::resolve_query_target(q.trim()),
|
||||
_ => Self::resolve_option_target(&options, &sort),
|
||||
};
|
||||
if let Target::Search(query) = &target {
|
||||
if let Some(found) = Self::resolve_keyword(query, &options).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
|
||||
match self
|
||||
.fetch_target(cache, target, page, per_page, options)
|
||||
|
||||
@@ -1,8 +1,11 @@
|
||||
use crate::DbPool;
|
||||
use crate::api::ClientVersion;
|
||||
use crate::providers::{Provider, report_provider_error, requester_or_default};
|
||||
use crate::providers::{
|
||||
Provider, probe_first_archive, report_provider_error, requester_or_default, slug_dashed,
|
||||
};
|
||||
use crate::status::*;
|
||||
use crate::util::cache::VideoCache;
|
||||
use crate::util::requester::Requester;
|
||||
use crate::util::parse_abbreviated_number;
|
||||
use crate::util::time::parse_time_to_seconds;
|
||||
use crate::videos::{ServerOptions, VideoItem};
|
||||
@@ -248,6 +251,33 @@ impl YoupornProvider {
|
||||
}
|
||||
}
|
||||
|
||||
/// A free-form query that names a pornstar, channel or amateur goes to that
|
||||
/// archive (probed on the site; unknown names 404), else keyword search.
|
||||
async fn resolve_keyword(
|
||||
&self,
|
||||
requester: &mut Requester,
|
||||
query: &str,
|
||||
sort: &str,
|
||||
) -> Option<Target> {
|
||||
let slug = slug_dashed(query);
|
||||
if slug.is_empty() {
|
||||
return None;
|
||||
}
|
||||
let sort = Self::normalized_sort(sort).to_string();
|
||||
let candidates = [
|
||||
Target::Pornstar { slug: slug.clone(), sort: sort.clone() },
|
||||
Target::Channel { slug: slug.clone(), sort: sort.clone() },
|
||||
Target::Amateur { slug, sort },
|
||||
];
|
||||
let urls: Vec<String> = candidates.iter().map(|t| self.build_url(t, 1)).collect();
|
||||
let referer = format!("{}/", self.url.trim_end_matches('/'));
|
||||
let (index, _) = probe_first_archive(requester, &urls, &referer, |body| {
|
||||
body.contains("js_video-box")
|
||||
})
|
||||
.await?;
|
||||
candidates.into_iter().nth(index)
|
||||
}
|
||||
|
||||
fn build_url(&self, target: &Target, page: u8) -> String {
|
||||
match target {
|
||||
Target::Latest { sort } => format!(
|
||||
@@ -543,7 +573,13 @@ impl Provider for YoupornProvider {
|
||||
) -> Vec<VideoItem> {
|
||||
let query = query.unwrap_or_default();
|
||||
let page = page.parse::<u8>().unwrap_or(1);
|
||||
let target = self.target_from_query(&query, &sort);
|
||||
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
|
||||
let mut target = self.target_from_query(&query, &sort);
|
||||
if let Target::Search { query } = &target {
|
||||
if let Some(found) = self.resolve_keyword(&mut requester, query, &sort).await {
|
||||
target = found;
|
||||
}
|
||||
}
|
||||
let video_url = self.build_url(&target, page);
|
||||
|
||||
let old_items = match cache.get(&video_url) {
|
||||
@@ -554,7 +590,6 @@ impl Provider for YoupornProvider {
|
||||
None => vec![],
|
||||
};
|
||||
|
||||
let mut requester = requester_or_default(&options, CHANNEL_ID, "get_videos");
|
||||
let referer = format!("{}/", self.url.trim_end_matches('/'));
|
||||
let text = match requester
|
||||
.get_with_headers(&video_url, Self::html_headers(&referer), Some(Version::HTTP_11))
|
||||
|
||||
@@ -661,6 +661,60 @@ impl Requester {
|
||||
Ok(response)
|
||||
}
|
||||
|
||||
fn path_of(url: &str) -> Option<String> {
|
||||
url::Url::parse(url).ok().map(|u| u.path().to_string())
|
||||
}
|
||||
|
||||
/// Existence probe for archive pages (pornstar / studio / tag / category).
|
||||
///
|
||||
/// Reads the raw response so a real 404/410 answers immediately with
|
||||
/// `Ok(None)` instead of escalating through the Jina/FlareSolverr chain
|
||||
/// (minutes per miss). A Cloudflare challenge (403/503/"Just a moment...")
|
||||
/// is handed to the full chain so FlareSolverr solves it, and 429/5xx are
|
||||
/// retried once.
|
||||
pub async fn get_probe(
|
||||
&mut self,
|
||||
url: &str,
|
||||
headers: Vec<(String, String)>,
|
||||
) -> Result<Option<String>, AnyErr> {
|
||||
let mut last_err = String::new();
|
||||
for _ in 0..2 {
|
||||
match self.get_raw_with_headers(url, headers.clone()).await {
|
||||
Ok(response) => {
|
||||
let status = response.status().as_u16();
|
||||
// An unknown archive often redirects to the home page or a
|
||||
// search page instead of answering 404: a redirect that
|
||||
// lands on a different path is a miss.
|
||||
let redirected_away = Self::path_of(url)
|
||||
.zip(Some(response.uri().path().to_string()))
|
||||
.is_some_and(|(asked, landed)| {
|
||||
!asked.trim_end_matches('/').eq_ignore_ascii_case(landed.trim_end_matches('/'))
|
||||
});
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
if status == 404 || status == 410 {
|
||||
return Ok(None);
|
||||
}
|
||||
if status == 403
|
||||
|| status == 503
|
||||
|| Self::looks_like_cloudflare_challenge(&body)
|
||||
{
|
||||
return self.get_with_headers(url, headers, None).await.map(Some);
|
||||
}
|
||||
if redirected_away {
|
||||
return Ok(None);
|
||||
}
|
||||
if status >= 500 || status == 429 {
|
||||
last_err = format!("HTTP {status}");
|
||||
continue;
|
||||
}
|
||||
return Ok(Some(body));
|
||||
}
|
||||
Err(e) => last_err = e.to_string(),
|
||||
}
|
||||
}
|
||||
Err(format!("probe failed for {url}: {last_err}").into())
|
||||
}
|
||||
|
||||
pub async fn get(
|
||||
&mut self,
|
||||
url: &str,
|
||||
|
||||
Reference in New Issue
Block a user