Outage Report: RESTART Checkpoint Hangs App

[kevincox]

We would like to share a major production outage we experienced that was caused by litestream and was very difficult to diagnose. We hope that this report can help other litestream users and be used to improve litestream in the future.

We were running litestream v0.3.13 which is the most recent release. (I see that there are some development changes that may improve the situation described here.)

The symptom of the outage was that our app could not start any database operations. The core error was SQLITE_BUSY but due to the 100% failure rate we also ended up seeing lots of other high level errors such as "Timed out waiting for acquire database connection" as our 5s busy_timeout was quickly backing up, and we started seeing timeouts at the connection pool level which obscured the underlying error. This was very perplexing to us because we only have one write connection in the app so the only possible source of other write transactions were litestream or human operators. Neither of these factors seemed at first glance that they should be holding extremely long write locks.

The TL;DR of the outage is fairly simple now that we understand it:

1. Our app has many long-lived read transactions.
2. Very occasionally (never until this outage) this causes the WAL to exceed max-checkpoint-page-count.
3. Litestream triggers a RESTART checkpoint.
4. This checkpoint blocks all writes while waiting on all readers to finish which could take a very long time.
5. Our app was unable to perform writes until the checkpoint completed.

Litestream's behaviour is not terrible in isolation. It will trigger RESTART checkpoints once the WAL exceeds the (undocumented?) max-checkpoint-page-count and TRUNCATE checkpoints once the WAL exceeds the (hardcoded) DefaultTruncatePageN. As far as we can tell this behaviour is completely undocumented. https://litestream.io/tips/#busy-timeout suggests that litestream may execute "short write locks" but never mentions indefinitely long writer locks.

RESTART checkpoints can also be triggered by setting validation-interval which is again undocumented.

We built our app with very strict rules on the write connection (as there is only one with sqlite WAL) but were very loose with read transactions as there can be any number of them, and they don't block readers or writers. We knew that long-lived read transactions would block WAL truncation and decided that this wasn't an issue for our use case. We had ample disk space buffer and WAL size monitoring size to ensure that we didn't run out of disk space.

Our plan falls apart when an unexpected FULL (or higher) checkpoint is triggered. This causes a "priority inversion" where our write connection is blocked on readers. These readers may be active for tens of minutes which is a completely unacceptable amount of time for our app to not be able to write anything. (The acceptable time is in the hundreds of milliseconds range.)

Our current workaround is to patch litestream to only ever trigger PASSIVE checkpoints (as it does most of the time). We have strong evidence that we have entered situations that would have triggered the outage again but successfully recovered with no degradation after running this patch. Long-term our team has decided to move away from litestream due to lack of trust caused by this hidden behaviour.

Suggestions:

1. Document the cases where litestream may do anything beyond "short write locks". Notably including any time it will execute a checkpoint of FULL or stronger.
2. Document and make configurable max-checkpoint-page-count and DefaultTruncatePageN. Explain how these values can be set to never trigger anything but PASSIVE checkpoints (example set arbitrarily high or support explicit "disabled" values).
3. In a future breaking release consider turning off non-PASSIVE checkpoints by default. It is unknowable to litestream how long other types of checkpoints will hold a full-database lock as it depends on other clients of the database. This indefinite "exclusive" lock is very risky to trigger them without user opt-in. The documentation can then explain the settings available to limit WAL size and the associated costs (block all writes until all current transactions complete).

Another related note: while we were reading the documentation I noticed that https://litestream.io/tips/#disable-autocheckpoints-for-high-write-load-servers suggests setting PRAGMA wal_autocheckpoint = 0. I would exercise caution before applying this setting as if litestream is not running (and you don't have any other processes that would trigger a checkpoint) it will cause infinite WAL growth. This may make sense if you need to preserve all incremental writes, but I don't think it should be framed as the correct solution without explaining the tradeoffs. For users who don't require every incremental write a much safer recommendation would be to set this value to something comfortably above max-checkpoint-page-count but also comfortably within your available disk space. This ensures that even if litestream is not running your app will eventually maintain the WAL size on its own.

[corylanou]

AI-Assisted Analysis of Current Codebase

⚠️ Note: This analysis was performed by AI and has not been confirmed for accuracy. Manual verification and team discussion needed to determine the best path forward.

Current Status (as of latest main branch)

After analyzing the current codebase, here's what I found regarding the checkpoint issues reported in this issue:

🟡 Partially Addressed - Core Issue Currently Disabled

✅ What Has Changed Since v0.3.13:

1. Automatic checkpoint triggering is COMMENTED OUT - The problematic checkpoint logic that caused the outage is disabled (db.go lines 577-602)
2. Configuration options added:
   • is now configurable via YAML
   • is now configurable via YAML
3. Default behavior remains PASSIVE for most checkpoint operations

⚠️ Still Problematic:

1. The dangerous code still exists (just commented out):

   • TRUNCATE checkpoint trigger at threshold
   • RESTART checkpoint trigger at threshold
   • If re-enabled, would cause the same blocking behavior

2. Missing configuration options:

   • is NOT configurable (hardcoded at 500,000 pages)
   • No option to force PASSIVE-only checkpoints
   • No way to explicitly disable thresholds (e.g., set to "disabled")

3. ** appears to be dead code**:

   • Exists in YAML config structure but unused in actual code
   • No evidence it triggers RESTART checkpoints as originally reported

4. Manual forced checkpoints still exist:

   • CRC64 function forces RESTART checkpoint (db.go:1466)

📋 Original Suggestions Not Yet Implemented:

• ❌ Document cases where non-PASSIVE checkpoints occur
• ❌ Make configurable
• ❌ Support explicit "disabled" values for thresholds
• ❌ Option to enforce PASSIVE-only mode
• ❌ Consider making PASSIVE-only the default

🎯 Recommendation:

The team should discuss whether to:

1. Properly remove the commented checkpoint code if it's not needed
2. Re-implement with safeguards (e.g., PASSIVE-only mode option)
3. Document current behavior clearly, especially the disabled features
4. Implement the remaining suggestions from this issue

The current state is safer than v0.3.13 (since automatic aggressive checkpoints are disabled), but the underlying architectural concern remains unresolved. The commented code suggests this was a temporary fix rather than a final solution.

This needs team review to determine the appropriate path forward.