Asynchronous document processing for an important number of files

[glypt]

Hi,
I would like to use docling with accurate OCR (I already browsed all the options I need to use) and I would like to do that asynchronously. I would like to use the numbers of cores available on the machine probably using a ThreadPoolExecutor. And then launch the pipeline processing of each document on each thread. I already tried quickly but I seem to have a race, I only protected the tokenizer with a lock which is shared among all the document converter instances. What else should I pay attention to? Do you have an idea from where the race condition should come, are the DocumentConverter and HybridChunker thread-safe?

[dosubot[bot]]

Hi @glypt! I'm Dosu and I’m helping the docling team.

Neither DocumentConverter nor HybridChunker are fully thread-safe for concurrent document processing. DocumentConverter uses a lock to protect pipeline initialization, but once a pipeline/model/tokenizer is created, it may be shared across threads, which can cause race conditions—especially since components like HybridChunker and HuggingFaceTokenizer are not inherently thread-safe and use cached properties or internal state that can be mutated during processing. Protecting just the tokenizer with a lock helps, but it's not enough: you should avoid sharing any pipeline, model, or chunker instance (including HybridChunker) across threads. Instead, instantiate separate objects per thread, or use thread-local storage to ensure each thread has its own instances. This approach also applies to other ML components in the pipeline, such as OCR and layout models, which may not be thread-safe if shared across threads. For large-scale or distributed workloads, consider using docling-serve, which is designed for parallel document conversion at scale (source).

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[glypt]

@dosu, following your approach, there might be some bottlenecks, the amount of memory used per thread to store the tokenizer, the ocr models, etc. might in the end degrade performance compare to a sequential processing of all the files.

[dosubot[bot]]

You're right—duplicating models and tokenizers per thread can cause significant memory usage, which may offset the gains from parallelism and even degrade performance compared to sequential processing, especially with large models like OCR and layout detection. This is a common trade-off in ML workloads: thread safety requires isolation, but isolation increases memory pressure.

For docling, true parallelism with threads is limited by Python's GIL and by global locks in the PDF backends, so you often don't get linear speedup with more threads, but you do get higher memory usage. Even with separate converter/model instances per thread and limiting threads to CPU cores, you can still hit deadlocks or resource exhaustion, especially with OCR enabled and libraries like PyTorch that aren't fully thread-safe in multi-threaded scenarios. In practice, using ProcessPoolExecutor (process-based parallelism) is safer for heavy workloads, as it avoids GIL and thread safety issues, but each process will still load its own copy of the models, so total memory usage scales with the number of processes (source).

If you use docling-serve, you can control memory usage by tuning the number of worker processes and threads. Within a process, you can set the DOCLING_SERVE_ENG_LOC_SHARE_MODELS environment variable to True to share models among threads, reducing per-thread memory usage, but this may reintroduce some thread safety risks if the underlying models aren't fully read-only (source). There is no built-in support for sharing models across processes, so overall memory usage still grows with the number of processes.

To mitigate memory bottlenecks: limit the number of concurrent workers to match available CPU cores, disable non-essential features (like picture classification or code enrichment), process large files in smaller batches or with page_range, and use the PyPdfium backend for lower memory usage. For large-scale jobs, docling-serve with multiple workers is the most robust approach, but you’ll need to balance worker count and memory limits for your hardware (source).

Let me know if you want code or config examples for any of these approaches!

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[dosubot[bot]]

You're right—instantiating separate models and tokenizers per thread can increase memory usage and, if you have many threads or large models (like OCR), this can become a bottleneck and even degrade performance compared to sequential processing. This is a common trade-off when aiming for thread safety in ML pipelines.

To mitigate memory usage while maintaining parallelism, consider these strategies:

• Use the most memory-efficient PDF backend (pypdfium2) and avoid keeping parsed pages in memory by setting generate_parsed_pages=False (now default) source.
• Disable non-essential features like generate_page_images, generate_picture_images, and enrichment steps (picture classification, code extraction) source.
• Lower images_scale to 1.0 unless you need high-res images source.
• Process large files in smaller batches using page_range or page_batch_size to avoid holding too much in memory at once source.
• Explicitly set num_threads and device in pipeline options to match your hardware and avoid resource contention source.

For parallelism, using ThreadPoolExecutor is only safe if each thread has its own pipeline/model/tokenizer instance, but this does increase memory usage. If memory is a concern, consider using ProcessPoolExecutor instead—each process will have its own memory space, which avoids Python GIL and thread safety issues, but still increases total memory usage. For large-scale jobs, docling-serve is designed for scalable, distributed processing and can be configured to share models within a process (to save memory) or isolate them per worker (for safety) source.

If you hit memory limits, try reducing the number of parallel workers, disabling enrichment, or splitting jobs into smaller batches. Enabling pipeline profiling (profile_pipeline_timings=True) can help you pinpoint bottlenecks source.

Let me know if you want more details on configuring docling-serve or tuning for your specific workload!

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[glypt]

Is there a plan to work on the thread safe version of those functionalities (document converter and hybridchunker)?