refactor!: Refactor storage creation and caching, configuration and services (#1386)

### Description

This is a collection of closely related changes that are hard to
separate from one another. The main purpose is to enable flexible
storage use across the code base without unexpected limitations and
limit unexpected side effects in global services.

#### Top-level changes:
- There can be multiple crawlers with different storage clients,
configurations, or event managers. (Previously, this would cause
`ServiceConflictError`)
- `StorageInstanceManager` allows for similar but different storage
instances to be used at the same time(Previously, similar storage
instances could be incorrectly retrieved instead of creating a new
storage instance).
- Differently configured storages can be used at the same time, even the
storages that are using the same `StorageClient` and are different only
by using different `Configuration`.
- `Crawler` can no longer cause side effects in the global
service_locator (apart from adding new instances to
`StorageInstanceManager`).
- Global `service_locator` can be used at the same time as local
instances of `ServiceLocator` (for example, each Crawler has its own
`ServiceLocator` instance, which does not interfere with the global
service_locator.)
- Services in `ServiceLocator` can be set only once. Any attempt to
reset them will throw an Error. Not setting the services and using them
is possible. That will set services in `ServiceLocator` to some implicit
default, and it will log warnings as implicit services can lead to
hard-to-predict code. The preferred way is to set services explicitly.
Either manually or through some helper code, for example, through
`Actor`. [See related
PR](apify/apify-sdk-python#576)

#### Implementation notes:
- Storage caching now supports all relevant ways to distinguish storage
instances. Apart from generic parameters like `name`, `id`,
`storage_type`, `storage_client_type`, there is also an
`additional_cache_key`. This can be used by the `StorageClient` to
define a unique way to distinguish between two similar but different
instances. For example, `FileSystemStorageClient` depends on
`Configuration.storage_dir`, which is included in the custom cache key
for `FileSystemStorageClient`, but this is not true for
`MemoryStorageClient` as the `storage_dir` is not relevant for it, see
example:
(This `additional_cache_key` could possibly be used for caching of NDU
in #1401)
```python
storage_client = FileSystemStorageClient()
d1= await Dataset.open(storage_client=storage_client, configuration=Configuration(storage_dir="path1"))
d2= await Dataset.open(storage_client=storage_client, configuration=Configuration(storage_dir="path2"))
d3= await Dataset.open(storage_client=storage_client, configuration=Configuration(storage_dir="path1"))

assert d2 is not d1
assert d3 is d1

storage_client_2 =MemoryStorageClient()
d4= await Dataset.open(storage_client=storage_client_2, configuration=Configuration(storage_dir="path1"))
d5= await Dataset.open(storage_client=storage_client_2, configuration=Configuration(storage_dir="path2"))
assert d4 is d5
```
- Each crawler will create its own instance of `ServiceLocator`. It will
either use explicitly passed services(configuration, storage client,
event_manager) to crawler init or services from the global
`service_locator` as implicit defaults. This allows multiple differently
configured crawlers to work in the same code. For example:
```python
custom_configuration_1 = Configuration()
custom_event_manager_1 = LocalEventManager.from_config(custom_configuration_1)
custom_storage_client_1 = MemoryStorageClient()

custom_configuration_2 = Configuration()
custom_event_manager_2 = LocalEventManager.from_config(custom_configuration_2)
custom_storage_client_2 = MemoryStorageClient()

crawler_1 = BasicCrawler(
    configuration=custom_configuration_1,
    event_manager=custom_event_manager_1,
    storage_client=custom_storage_client_1,
)

crawler_2 = BasicCrawler(
    configuration=custom_configuration_2,
    event_manager=custom_event_manager_2,
    storage_client=custom_storage_client_2,
  )

# use crawlers without runtime crash...
```
- `ServiceLocator` is now way more strict when it comes to setting the
services. Previously, it allowed changing services until some service
had `_was_retrieved` flag set to `True`. Then it would throw a runtime
error. This led to hard-to-predict code as the global `service_locator`
could be changed as a side effect from many places. Now the services in
`ServiceLocator` can be set only once, and the side effects of
attempting to change the services are limited as much as possible. Such
side effects are also accompanied by warning messages to draw attention
to code that could cause RuntimeError.
### Issues

Closes: #1379
Connected to: 
- #1354  (through necessary changes in `StorageInstanceManagaer`)
- apify/apify-sdk-python#513 (through
necessary changes in `StorageInstanceManagaer` and storage
clients/configuration related changes in `service_locator`)

### Testing

- New unit tests were added.
- Tested on the `Apify` platform together with SDK changes in [related
PR](apify/apify-sdk-python#576)

---------

Co-authored-by: Vlada Dusek <[email protected]>

Author: Pijukatel

docs/guides/code_examples/service_locator/service_storage_configuration.py

@@ -1,7 +1,9 @@
 import asyncio
 from datetime import timedelta
 
+from crawlee import service_locator
 from crawlee.configuration import Configuration
+from crawlee.storage_clients import MemoryStorageClient
 from crawlee.storages import Dataset
 
 
@@ -11,10 +13,16 @@ async def main() -> None:
         headless=False,
         persist_state_interval=timedelta(seconds=30),
     )
+    # Set the custom configuration as the global default configuration.
+    service_locator.set_configuration(configuration)
 
-    # Pass the configuration to the dataset (or other storage) when opening it.
-    dataset = await Dataset.open(
-        configuration=configuration,
+    # Use the global defaults when creating the dataset (or other storage).
+    dataset_1 = await Dataset.open()
+
+    # Or set explicitly specific configuration if
+    # you do not want to rely on global defaults.
+    dataset_2 = await Dataset.open(
+        storage_client=MemoryStorageClient(), configuration=configuration
     )
 
 

docs/upgrading/upgrading_to_v1.md

@@ -189,6 +189,97 @@ The interface for custom storage clients has been simplified:
 - Collection storage clients have been removed.
 - The number of methods that have to be implemented have been reduced.
 
+## ServiceLocator changes
+
+### ServiceLocator is stricter with registering services
+You can register the services just once, and you can no longer override already registered services.
+
+**Before (v0.6):**
+```python
+from crawlee import service_locator
+from crawlee.storage_clients import MemoryStorageClient
+
+service_locator.set_storage_client(MemoryStorageClient())
+service_locator.set_storage_client(MemoryStorageClient())
+```
+**Now (v1.0):**
+
+```python
+from crawlee import service_locator
+from crawlee.storage_clients import MemoryStorageClient
+
+service_locator.set_storage_client(MemoryStorageClient())
+service_locator.set_storage_client(MemoryStorageClient())  # Raises an error
+```
+
+### BasicCrawler has its own instance of ServiceLocator to track its own services
+Explicitly passed services to the crawler can be different the global ones accessible in `crawlee.service_locator`. `BasicCrawler` no longer causes the global services in `service_locator` to be set to the crawler's explicitly passed services.
+
+**Before (v0.6):**
+```python
+from crawlee import service_locator
+from crawlee.crawlers import BasicCrawler
+from crawlee.storage_clients import MemoryStorageClient
+from crawlee.storages import Dataset
+
+
+async def main() -> None:
+    custom_storage_client = MemoryStorageClient()
+    crawler = BasicCrawler(storage_client=custom_storage_client)
+
+    assert service_locator.get_storage_client() is custom_storage_client
+    assert await crawler.get_dataset() is await Dataset.open()
+```
+**Now (v1.0):**
+
+```python
+from crawlee import service_locator
+from crawlee.crawlers import BasicCrawler
+from crawlee.storage_clients import MemoryStorageClient
+from crawlee.storages import Dataset
+
+
+async def main() -> None:
+    custom_storage_client = MemoryStorageClient()
+    crawler = BasicCrawler(storage_client=custom_storage_client)
+
+    assert service_locator.get_storage_client() is not custom_storage_client
+    assert await crawler.get_dataset() is not await Dataset.open()
+```
+
+This allows two crawlers with different services at the same time.
+
+**Now (v1.0):**
+
+```python
+from crawlee.crawlers import BasicCrawler
+from crawlee.storage_clients import MemoryStorageClient, FileSystemStorageClient
+from crawlee.configuration import Configuration
+from crawlee.events import LocalEventManager
+
+custom_configuration_1 = Configuration()
+custom_event_manager_1 = LocalEventManager.from_config(custom_configuration_1)
+custom_storage_client_1 = MemoryStorageClient()
+
+custom_configuration_2 = Configuration()
+custom_event_manager_2 = LocalEventManager.from_config(custom_configuration_2)
+custom_storage_client_2 = FileSystemStorageClient()
+
+crawler_1 = BasicCrawler(
+    configuration=custom_configuration_1,
+    event_manager=custom_event_manager_1,
+    storage_client=custom_storage_client_1,
+)
+
+crawler_2 = BasicCrawler(
+    configuration=custom_configuration_2,
+    event_manager=custom_event_manager_2,
+    storage_client=custom_storage_client_2,
+  )
+
+# use crawlers without runtime crash...
+```
+
 ## Other smaller updates
 
 There are more smaller updates.

pyproject.toml

@@ -40,7 +40,7 @@ dependencies = [
     "protego>=0.5.0",
     "psutil>=6.0.0",
     "pydantic-settings>=2.2.0,!=2.7.0,!=2.7.1,!=2.8.0",
-    "pydantic>=2.8.0,!=2.10.0,!=2.10.1,!=2.10.2",
+    "pydantic>=2.11.0",
     "pyee>=9.0.0",
     "tldextract>=5.1.0",
     "typing-extensions>=4.1.0",

src/crawlee/_autoscaling/snapshotter.py

@@ -113,7 +113,7 @@ def from_config(cls, config: Configuration | None = None) -> Snapshotter:
         Args:
             config: The `Configuration` instance. Uses the global (default) one if not provided.
         """
-        config = service_locator.get_configuration()
+        config = config or service_locator.get_configuration()
 
         # Compute the maximum memory size based on the provided configuration. If `memory_mbytes` is provided,
         # it uses that value. Otherwise, it calculates the `max_memory_size` as a proportion of the system's

src/crawlee/_service_locator.py

@@ -11,6 +11,10 @@
 if TYPE_CHECKING:
     from crawlee.storages._storage_instance_manager import StorageInstanceManager
 
+from logging import getLogger
+
+logger = getLogger(__name__)
+
 
 @docs_group('Configuration')
 class ServiceLocator:
@@ -19,23 +23,24 @@ class ServiceLocator:
     All services are initialized to its default value lazily.
     """
 
-    def __init__(self) -> None:
-        self._configuration: Configuration | None = None
-        self._event_manager: EventManager | None = None
-        self._storage_client: StorageClient | None = None
-        self._storage_instance_manager: StorageInstanceManager | None = None
+    global_storage_instance_manager: StorageInstanceManager | None = None
 
-        # Flags to check if the services were already set.
-        self._configuration_was_retrieved = False
-        self._event_manager_was_retrieved = False
-        self._storage_client_was_retrieved = False
+    def __init__(
+        self,
+        configuration: Configuration | None = None,
+        event_manager: EventManager | None = None,
+        storage_client: StorageClient | None = None,
+    ) -> None:
+        self._configuration = configuration
+        self._event_manager = event_manager
+        self._storage_client = storage_client
 
     def get_configuration(self) -> Configuration:
         """Get the configuration."""
         if self._configuration is None:
+            logger.warning('No configuration set, implicitly creating and using default Configuration.')
             self._configuration = Configuration()
 
-        self._configuration_was_retrieved = True
         return self._configuration
 
     def set_configuration(self, configuration: Configuration) -> None:
@@ -47,21 +52,25 @@ def set_configuration(self, configuration: Configuration) -> None:
         Raises:
             ServiceConflictError: If the configuration has already been retrieved before.
         """
-        if self._configuration_was_retrieved:
+        if self._configuration is configuration:
+            # Same instance, no need to anything
+            return
+        if self._configuration:
             raise ServiceConflictError(Configuration, configuration, self._configuration)
 
         self._configuration = configuration
 
     def get_event_manager(self) -> EventManager:
         """Get the event manager."""
         if self._event_manager is None:
-            self._event_manager = (
-                LocalEventManager().from_config(config=self._configuration)
-                if self._configuration
-                else LocalEventManager.from_config()
-            )
+            logger.warning('No event manager set, implicitly creating and using default LocalEventManager.')
+            if self._configuration is None:
+                logger.warning(
+                    'Implicit creation of event manager will implicitly set configuration as side effect. '
+                    'It is advised to explicitly first set the configuration instead.'
+                )
+            self._event_manager = LocalEventManager().from_config(config=self._configuration)
 
-        self._event_manager_was_retrieved = True
         return self._event_manager
 
     def set_event_manager(self, event_manager: EventManager) -> None:
@@ -73,17 +82,25 @@ def set_event_manager(self, event_manager: EventManager) -> None:
         Raises:
             ServiceConflictError: If the event manager has already been retrieved before.
         """
-        if self._event_manager_was_retrieved:
+        if self._event_manager is event_manager:
+            # Same instance, no need to anything
+            return
+        if self._event_manager:
             raise ServiceConflictError(EventManager, event_manager, self._event_manager)
 
         self._event_manager = event_manager
 
     def get_storage_client(self) -> StorageClient:
         """Get the storage client."""
         if self._storage_client is None:
+            logger.warning('No storage client set, implicitly creating and using default FileSystemStorageClient.')
+            if self._configuration is None:
+                logger.warning(
+                    'Implicit creation of storage client will implicitly set configuration as side effect. '
+                    'It is advised to explicitly first set the configuration instead.'
+                )
             self._storage_client = FileSystemStorageClient()
 
-        self._storage_client_was_retrieved = True
         return self._storage_client
 
     def set_storage_client(self, storage_client: StorageClient) -> None:
@@ -95,21 +112,24 @@ def set_storage_client(self, storage_client: StorageClient) -> None:
         Raises:
             ServiceConflictError: If the storage client has already been retrieved before.
         """
-        if self._storage_client_was_retrieved:
+        if self._storage_client is storage_client:
+            # Same instance, no need to anything
+            return
+        if self._storage_client:
             raise ServiceConflictError(StorageClient, storage_client, self._storage_client)
 
         self._storage_client = storage_client
 
     @property
     def storage_instance_manager(self) -> StorageInstanceManager:
-        """Get the storage instance manager."""
-        if self._storage_instance_manager is None:
+        """Get the storage instance manager. It is global manager shared by all instances of ServiceLocator."""
+        if ServiceLocator.global_storage_instance_manager is None:
             # Import here to avoid circular imports.
             from crawlee.storages._storage_instance_manager import StorageInstanceManager  # noqa: PLC0415
 
-            self._storage_instance_manager = StorageInstanceManager()
+            ServiceLocator.global_storage_instance_manager = StorageInstanceManager()
 
-        return self._storage_instance_manager
+        return ServiceLocator.global_storage_instance_manager
 
 
 service_locator = ServiceLocator()

src/crawlee/configuration.py

@@ -28,7 +28,7 @@ class Configuration(BaseSettings):
     Settings can also be configured via environment variables, prefixed with `CRAWLEE_`.
     """
 
-    model_config = SettingsConfigDict(populate_by_name=True)
+    model_config = SettingsConfigDict(validate_by_name=True, validate_by_alias=True)
 
     internal_timeout: Annotated[timedelta | None, Field(alias='crawlee_internal_timeout')] = None
     """Timeout for the internal asynchronous operations."""