simon/hottub
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix hypnotube search sessions; check.py confirms HEAD failures with GET

Browse Source 
The site serves one results page per searchgate POST, so later searches
and page 2 returned nothing on the shared cookie session. Use an isolated
session per (query, page). check.py now retries a refused HEAD (403/404/
405/501) with a ranged GET before reporting a URL as unreachable.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUYdCh2nCVsQYgJAT7gwSi
This commit is contained in:
Simon
2026-10-10 08:24:15 +00:00
parent 78acac2150
commit a1e6021766
3 changed files with 50 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
check.py
View File

@@ -149,13 +149,15 @@ def _is_cf_protected(url: str) -> bool:
def http_ok(url: str, headers: dict | None = None) -> tuple[bool, int]:
"""Return (ok, http_status). Tries HEAD then ranged GET on 405."""
"""Return (ok, http_status). Tries HEAD, then a ranged GET when HEAD is refused."""
h = {"User-Agent": _BROWSER_UA, **(headers or {})}
try:
r = requests.head(url, headers=h, timeout=HTTP_TIMEOUT, allow_redirects=True)
if r.status_code in (200, 206):
return True, r.status_code
if r.status_code == 405:
# Some servers mishandle HEAD (405/501, or even 404/403 while GET is 200);
# confirm with a ranged GET before reporting the URL as broken.
if r.status_code in (403, 404, 405, 501):
r2 = requests.get(
url,
headers={**h, "Range": "bytes=0-1023"},