An exception occurred during handling of failed request

[HJK181]

Which package is this bug report for? If unsure which one to select, leave blank

@crawlee/http (HttpCrawler)

Issue description

My HttpCrawler fails with

DEBUG HttpCrawler: Crawled 18/547 pages, 0 failed requests, desired concurrency 2.
ERROR HttpCrawler: An exception occurred during handling of failed request. This places the crawler and its underlying storages into an unknown state and crawling will be terminated. 
  Handling request failure of https://cc-stage-shopware.kleerly.de/epson/stylus/ (3FHkwFLmzy6kAAL) timed out after 300 seconds.
      at Timeout._onTimeout (/home/myuser/node_modules/@apify/timeout/cjs/index.cjs:64:68)
      at listOnTimeout (node:internal/timers:581:17)
      at process.processTimers (node:internal/timers:519:7)
ERROR HttpCrawler:AutoscaledPool: runTaskFunction failed.
  Handling request failure of https://cc-stage-shopware.kleerly.de/epson/stylus/ (3FHkwFLmzy6kAAL) timed out after 300 seconds.
      at Timeout._onTimeout (/home/myuser/node_modules/@apify/timeout/cjs/index.cjs:64:68)
      at listOnTimeout (node:internal/timers:581:17)
      at process.processTimers (node:internal/timers:519:7)

The only kind of errors logged before this happened were (19 times according to SDK_CRAWLER_STATISTICS_6.json):

ERROR Request failed with Detected a session error, rotating session... 
Proxy responded with 500 Unable to connect: 0 bytes.

Below is the first 100 bytes of the proxy response body:

Code sample

const crawler = new HttpCrawler(
      {
        maxConcurrency: 2,
        maxRequestsPerMinute: 180,
        ...options,
        proxyConfiguration,
        useSessionPool: true,
        persistCookiesPerSession: true,
        additionalMimeTypes: ["text/plain", "application/pdf"],
        async requestHandler({ pushData, request, response }) {
          await pushData({
            url: request.url,
            statusCode: response.statusCode,
          });
        },
        async failedRequestHandler({ pushData, request, response }) {
          log.error(`Request for URL "${request.url}" failed.`);
          await pushData({
            url: request.url,
            statusCode: response?.statusCode ?? 0,
          });
        },
        async errorHandler({ request }, { message }) {
          log.error(`Request failed with ${message}`);
          if (!request.noRetry) {
            const baseWaitTime = Math.pow(2, request.retryCount) * 1000;
            const jitter = baseWaitTime * (Math.random() - 0.5);
            const waitTime = baseWaitTime + jitter;
            await new Promise((resolve) => setTimeout(resolve, waitTime));
          }
        },
      },
      config,
    );
crawler.run(urls)

Package version

[email protected]

Node.js version

v20.18.0

Operating system

Docker image From apify/actor-node-playwright-chrome

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response

[B4nan]

ERROR Request failed with Detected a session error, rotating session...

This sounds like you use bad proxies, it means the connection didn't go through to the target site. Such errors are retried separately, but will also go through the errorHandler.

Handling request failure of https://cc-stage-shopware.kleerly.de/epson/stylus/ (3FHkwFLmzy6kAAL) timed out after 300 seconds.

I can't open that URL myself, are you sure its valid?

Handling request failure of https://cc-stage-shopware.kleerly.de/epson/stylus/ (3FHkwFLmzy6kAAL) timed out after 300 seconds.

This means that your errorHandler timed out, my guess is your waiting logic ends up waiting for more than 300s which is the internal timeout for the error handler. It is computed dynamically based on requestHandlerTimeoutSecs, its value is multiplied by two, and there is a minimum for this timeout set to 300s.

[B4nan]

Retries are executed as regular requests, respecting all the limits and other options. The request gets back to the queue just like any other request. Maybe you were just hitting too high concurrency, without proxies it could happen since the HTTP crawler can be really fast.

You could always just increase the request handler timeout, that will in turn increase the internal timeout too. But you should make sure you don't wait for too long there if you want to have a custom wait at that level.

Or there is CRAWLEE_INTERNAL_TIMEOUT env var to set its value explicitly.

[HJK181]

Ok, thank you very much for your help.

[B4nan]

were executed in bursts instead of some delay between

Btw I can imagine this happened because your requests were too fast, you got 180 of them within just a few seconds, then the maxRequestsPerMinute kicked in, things were waiting for a minute to accommodate the 180 RPM, then all retries started. With proxies it shouldn't really matter, as the requests will be slower, and they will act as separate users from the point of the target server.

[HJK181]

My first proxy is just a locally running Tinyproxy instance as the crawler runs in a Kubernetes pod it might pass the bot protection. Only if it's getting blocked the request will be routed through a proxy service. So I might still hit that bursts...

I'm using concurrency 2, is the following assumption true?

the crawler will burst up to 2 requests and only start to crawl the next URL if one of the previous requests have finished?

[B4nan]

Yes, concurrency 2 means there are never more than 2 request handlers running "in parallel".