Is it possible to pass in a custom transport?

[tleyden]

I'm trying to add response caching via hishel transports, but am not seeing a way to customize the transport used by the Crawlee client as it is created internally in _get_client():

    def _get_client(self, proxy_url: str | None) -> httpx.AsyncClient:
        """Helper to get a HTTP client for the given proxy URL.

        If the client for the given proxy URL doesn't exist, it will be created and stored.
        """
        if proxy_url not in self._client_by_proxy_url:
            # Prepare a default kwargs for the new client.
            kwargs: dict[str, Any] = {
                'transport': _HttpxTransport(
                    proxy=proxy_url,
                    http1=self._http1,
                    http2=self._http2,
                ),
                'proxy': proxy_url,
                'http1': self._http1,
                'http2': self._http2,
            }

            # Update the default kwargs with any additional user-provided kwargs.
            kwargs.update(self._async_client_kwargs)

            client = httpx.AsyncClient(**kwargs)
            self._client_by_proxy_url[proxy_url] = client

        return self._client_by_proxy_url[proxy_url]

Is there a way to customize the httpx client transport that I'm not seeing?

Or instead of using a 3rd party library, does Crawlee have a native method for storing long term persistent caches of responses?

Somewhat related question, if its not possible to customize the transport. Is overriding HttpxHttpClient._get_client() the recommended way to use a custom httpx client, or is there a cleaner way?

    hishel_client = await _create_hishel_client(cache_path)
    class HishelCacheClient(HttpxHttpClient):
        def _get_client(self, proxy_url: str | None) -> httpx.AsyncClient:
            return hishel_client
    http_client = HishelCacheClient()

    crawler = BeautifulSoupCrawler(
        http_client=http_client,
    )

[Mantisus]

Hey @tleyden , thank you for your interest in crawlee.

Indeed, there's currently no way to pass a custom transport to HttpxHttpClient. I'm not entirely sure if we should add such capability. Since a transport can significantly affect client behavior, making it unpredictable, adding this feature requires careful consideration. Perhaps @janbuchar or @vdusek have some thoughts on this.

Regarding your proposed solution. Please note that it completely disables proxy usage. You might want to keep the current _get_client implementation and only modify the transport definition.

[janbuchar]

Indeed, there's currently no way to pass a custom transport to HttpxHttpClient. I'm not entirely sure if we should add such capability. Since a transport can significantly affect client behavior, making it unpredictable, adding this feature requires careful consideration. Perhaps @janbuchar or @vdusek have some thoughts on this.

I agree with @Mantisus here. Unless there is significant demand from the community, I don't think we should allow custom transports. Perhaps adding a _create_httpx_client method that you could override in a subclass may help?

If you want to take the reins completely, you're free to implement your own subclass of BaseHttpClient and use HttpxHttpClient as inspiration.

[tleyden]

Thank you @janbuchar and @Mantisus for the help!

I ended up overriding the transport with this code:

    hishel_transport: hishel.AsyncCacheTransport = _create_hishel_transport(cache_path)
    crawlee_client = HttpxHttpClient(transport=hishel_transport)

    crawler = BeautifulSoupCrawler(
        http_client=crawlee_client,
    )

and will keep in mind the risks of using a custom transport.

[janbuchar]

@tleyden cool, I'll just repeat that your client will be missing proxy settings in this case. Glad you solved the problem to your satisfaction 🙂