feat: Add litestream-test harness for comprehensive database testing

[corylanou]

Summary

This PR introduces a comprehensive testing harness (litestream-test) for validating Litestream's behavior with large databases, various write patterns, and edge cases. This is the first implementation based on discussions with @benbjohnson about the need for thorough testing, particularly for databases larger than 1GB and various operational scenarios.

Background

Based on requirements outlined in [internal planning documents], we need comprehensive testing for:

• Databases larger than 1GB (critical SQLite lock page edge case at 0x40000000)
• Various write patterns and frequencies
• Database shrinking scenarios (grow then delete data)
• Interruption and recovery scenarios (checkpoint during Litestream downtime)
• Multi-level compaction validation
• LTX file continuity checking

Implementation

New Binary: /cmd/litestream-test/

Four primary commands implemented:

1. populate - Quickly create test databases to target sizes

   • Configurable page sizes (critical for 1GB lock page testing)
   • Multiple tables with indexes
   • Batch inserts for efficiency
   • Support for databases from MB to GB scale

2. load - Generate continuous workload on databases

   • Write patterns: constant, burst, random, wave
   • Configurable read/write ratios
   • Multiple concurrent workers
   • Real-time statistics reporting

3. validate - Verify replication integrity

   • Quick check, integrity check, checksum comparison
   • LTX file continuity checking
   • Full data validation between source and restored DBs
   • Integration with existing Litestream restore

4. shrink - Test database shrinking scenarios

   • Configurable delete percentage
   • Optional VACUUM and checkpoint operations
   • All SQLite checkpoint modes supported (PASSIVE, FULL, RESTART, TRUNCATE)

Testing

Comprehensive testing was performed using the demo script available here:
Test Harness Demo Script (Gist)

Test Results

✅ Database Creation

• Created databases from 5MB to 50MB successfully
• Verified correct page size configuration (4KB, 8KB tested)
• Tables with proper indexes created

✅ Write Patterns

• Burst pattern: Correctly generated bursts (205 writes in 5s, then 0, then 204)
• Random pattern: Generated variable rates as expected
• Wave pattern: Smooth oscillating pattern confirmed
• Load generation successfully added 600+ rows during tests

✅ Page Size Configuration

• Successfully created database with 8KB pages (verified: 8192 bytes)
• Ready for 1GB lock page boundary testing

✅ Shrink Operations

• Deleted 40% of data (10,239 rows)
• FULL checkpoint executed successfully
• Database size reduced from 66MB to 45MB

Example Usage

# Create a 1GB database for lock page testing
./bin/litestream-test populate -db /tmp/test.db -target-size 1GB -page-size 8192

# Generate continuous load
./bin/litestream-test load -db /tmp/test.db -write-rate 100 -duration 1m -pattern burst

# Test shrinking
./bin/litestream-test shrink -db /tmp/test.db -delete-percentage 50 -checkpoint

# Validate replication
./bin/litestream-test validate -source-db /tmp/test.db -replica-url s3://bucket/test

Key Features

• ✅ Handles SQLite lock page at 1GB boundary (page calculation included)
• ✅ Supports interruption/recovery test scenarios
• ✅ Uses crypto/rand for secure random data generation
• ✅ Comprehensive error handling and structured logging
• ✅ Compatible with existing Litestream binaries

Next Steps

This is the first pass of the testing framework. Future enhancements could include:

• Automated test suites for CI/CD
• Multi-database concurrent testing
• Network failure simulation
• Performance regression detection
• VFS testing capabilities

Notes

• All code follows existing Litestream patterns and conventions
• Uses same dependencies (go-sqlite3, slog)
• Passes all pre-commit hooks (goimports, go-vet, staticcheck)
• No changes to existing Litestream code

cc: @benbjohnson @corylanou