conver md to docx，out of memory

[bianhy]

Explain the problem.
My markdown file contains over a thousand images, and when converting to docx, the CPU usage exceeds 100%. How can I solve this problem?

pandoc -f markdown -t docx e5ddd1c0ff661aa222c766b34bea0ab8.md -o e5ddd1c0ff661aa222c766b34bea0ab8.docx 2>&1

Pandoc version?
pandoc.exe 3.7.0.2
Features: +server +lua
Scripting engine: Lua 5.4

[jgm]

Possibly related: #10973
What format are the images?

[jgm]

I just tried a conversion with 1000+ small jpg images and it finished quickly with 53M in use. Is there any way you can provide a zip or something so I can attempt to reproduce the problem? or a more detailed description, at least? How large are the images, and what format?

[bianhy]

我刚刚尝试转换了1000多张小型jpg图片，很快就完成了，占用了53M空间。请问您能提供一个zip文件或其他压缩包，让我可以尝试重现这个问题吗？或者至少提供更详细的描述？图片有多大？是什么格式？

input.md

[bianhy]

[jgm]

I see, all of the images are remote. This may have to do with the part of pandoc that fetches remote resources, rather than the determination of image sizes. It would be worth downloading these and trying it with paths to local images.

[jgm]

profiling says that most of the time is taken by openURL.
I also tried downloading locally and then pandoc was quite fast. So the bottleneck is downloading 1000 images. It may be that something can be improved there.

[jgm]

It looks like a lot of time is spent in certificate validation. Try with --no-check-certificate.

[jgm]

I do see one code improvement that could be made. We have
https://github.com/jgm/pandoc/blob/main/src/Text/Pandoc/Class/IO.hs#L145-L146
in openURL. Perhaps we could just fetch this once and cache it in CommonState.

Or better yet cache the tlsManager created at https://github.com/jgm/pandoc/blob/main/src/Text/Pandoc/Class/IO.hs#L164-L165

[jgm]

The docs say: "Creating a new Manager is a relatively expensive operation, you are advised to share a single Manager between requests instead."

[bianhy]

I see, all of the images are remote. This may have to do with the part of pandoc that fetches remote resources, rather than the determination of image sizes. It would be worth downloading these and trying it with paths to local images.

I tried using the path of the local image. Very fast and uses very little memory. Thank you.

[jgm]

Thanks for your report; it did lead to a change to pandoc that fixes a memory leak in cases like yours, with many remote requests.