Why HTML Scraping Fails on Early-Stage D2C Brands: Building a Hybrid DOM and OCR Pipeline

1. Preventing Amazon CDN Layout Degradation

Scrapers using libraries like requests, playwright, or curl_cffi accumulate session cookies across successive HTTP calls. As cookie headers grow, Amazon's traffic management servers flag the session and return minimalist dynamic layouts that lack productDetails_techSpec_section_1 and seller metadata containers.

Clearing session cookies before each GET request resets the session state, prompting the CDN to treat each call as an initial desktop visit:

class AmazonFetcher:
    def __init__(self):
        self.session = requests.Session(impersonate="chrome120")
        self.session.headers.update(DEFAULT_HEADERS)

    def get(self, url: str) -> Optional[str]:
        self._sleep_polite()
        for attempt in range(1, MAX_RETRIES + 1):
            try:
                # Reset cookies to force full server-rendered desktop DOM
                self.session.cookies.clear()
                resp = self.session.get(url, timeout=REQUEST_TIMEOUT)
                if resp.status_code == 200:
                    return resp.text
                elif resp.status_code in (429, 503):
                    backoff = self._calculate_backoff(attempt)
                    time.sleep(backoff)
            except Exception:
                time.sleep(self._calculate_backoff(attempt))
        return None

Pairing cookie flushes with Chrome TLS fingerprint impersonation (curl_cffi) maintains access to server-rendered DOM tables without requiring headless browser overhead.

2. Rate Limiting with Jittered Exponential Backoff

To handle 429 and 503 response codes without creating synchronized request spikes across workers, backoff intervals incorporate uniform random jitter:

Adding jitter desynchronizes retry attempts across asynchronous workers and prevents recurring request collisions against CDN rate limit thresholds.

3. In-Memory Packaging OCR via ONNX Runtime

Commercial cloud vision APIs charge per image, which becomes expensive when scanning four to six packaging images per ASIN. Local Tesseract installations require external system binaries and struggle with curved surfaces, low-contrast typography, and rotated labels.

The pipeline uses RapidOCR (PaddleOCR models executed through ONNX Runtime). Running locally on standard x86 CPU threads, inference completes in roughly 600 milliseconds per image.

Images stream directly into RAM as byte arrays and convert to NumPy arrays through OpenCV. No intermediate image files touch the local disk:

def _process_single_image(self, img_url: str) -> str:
    resp = self.fetcher.session.get(img_url, timeout=15)
    if resp.status_code != 200:
        return ""

    # Decode bytes directly in RAM
    img_arr = np.frombuffer(resp.content, dtype=np.uint8)
    img = cv2.imdecode(img_arr, cv2.IMREAD_COLOR)
    if img is None:
        return ""

    # Local ONNX inference on CPU
    result, _ = self.ocr_engine(img)
    if not result:
        return ""

    # Keep text lines with confidence >= 0.45
    high_confidence_text = [
        line[1] for line in result if float(line[2]) >= 0.45
    ]
    return "\n".join(high_confidence_text)

4. Conditional OCR Triggering

Running OCR across every image on every listing creates unnecessary computational overhead. The pipeline uses an asymmetric trigger:

• When HTML parsing extracts both a valid telephone number and an email address from the DOM, OCR processing is bypassed entirely.
• When either the phone number or the email address is missing from the DOM, the pipeline downloads secondary product photos and executes the OCR routine.

In live batch runs, this conditional check bypassed OCR on approximately 70% of qualified listings.

5. Cleaning Hidden Directional Unicode Characters

Amazon India pages include hidden directional formatting characters (\u200e, \u200f, \xa0, and \u202a through \u202e) inside brand names, phone strings, and seller addresses. These characters do not render visually in browsers, but they break regular expressions and trigger UnicodeEncodeError crashes on Windows systems running cp1252 console encoding.

All raw text passes through an upfront sanitizer before validation:

CLEAN_CHARS_REGEX = re.compile(r'[\u200e\u200f\xa0\u202a-\u202e\t\r]')

def sanitize_text(raw_text: str) -> str:
    if not raw_text:
        return ""
    return CLEAN_CHARS_REGEX.sub(' ', raw_text).strip()

Benchmark Results Across 15 Live Listings

Conclusion

Marketplace scrapers that rely exclusively on HTML elements miss valuable B2B contact data when sellers omit specifications. In regulated categories like food, cosmetics, and health supplements, packaging photos contain verified legal contact details mandated by law. Combining session cookie resets with local in-memory ONNX OCR provides a practical, low-latency method to extract complete contact records without third-party vision APIs.