Skip to content

Conversation

corylanou
Copy link
Collaborator

Summary

This PR implements directory replication support, allowing Litestream to replicate entire directories of SQLite databases. This is particularly useful for multi-tenant applications where each tenant has their own database file.

Closes #42

Features

  • 📁 Scan directories for SQLite databases with configurable file patterns (*.db, *.sqlite, etc.)
  • 🔄 Support for recursive directory scanning to find databases in subdirectories
  • ✅ Validate files by checking SQLite headers (prevents replicating non-SQLite files)
  • 🎯 Mix single database and directory configurations in the same config file

Configuration Example

dbs:
  # Replicate all databases in a directory
  - directory: /var/lib/myapp/databases
    pattern: "*.db"        # optional, defaults to "*.db"
    recursive: true        # optional, defaults to false
    replica:
      type: s3
      bucket: my-bucket
      path: backups

  # Can still use single database configs alongside
  - path: /var/lib/critical/main.db
    replica:
      type: s3
      bucket: critical-backups
      path: main

Implementation Details

  • Extended DBConfig with Directory, Pattern, and Recursive fields
  • Added NewDBsFromDirectoryConfig() to create multiple DB instances from a directory
  • Added FindSQLiteDatabases() for directory scanning with pattern matching
  • Added IsSQLiteDatabase() for file validation via header check
  • Updated ReplicateCommand to handle directory configurations
  • Added comprehensive test coverage in main_test.go

Testing

  • ✅ All existing tests pass
  • ✅ Added tests for directory scanning functionality
  • ✅ Added tests for SQLite file detection
  • ✅ Added tests for configuration validation
  • go vet passes
  • staticcheck passes

Example Configuration

See etc/litestream-directory-example.yml for comprehensive examples including multi-tenant and mixed configurations.

Documentation Follow-up

A follow-up issue should be created in the litestream.io repository to document:

  • New configuration options (directory, pattern, recursive)
  • Use cases for multi-tenant applications
  • Examples of mixed single/directory configurations
  • Current limitations (no dynamic discovery without restart, shared config for all DBs in directory)

Future Enhancements

  • Dynamic database discovery (add/remove databases without restart)
  • Per-database override configurations within a directory
  • Automatic cleanup of removed databases from replicas

🤖 Generated with Claude Code

Co-Authored-By: Claude [email protected]

@corylanou corylanou marked this pull request as draft September 5, 2025 22:56
This implements the ability to replicate entire directories of SQLite databases,
perfect for multi-tenant applications where each tenant has their own database file.

Features:
- Scan directories for SQLite databases with configurable file patterns
- Support for recursive directory scanning
- Validate files by checking SQLite headers
- Mix single database and directory configurations in the same config file

Configuration example:
```yaml
dbs:
  - directory: /var/lib/myapp/databases
    pattern: "*.db"
    recursive: true
    replica:
      type: s3
      bucket: my-bucket
      path: backups
```

Implementation:
- Extended DBConfig with Directory, Pattern, and Recursive fields
- Added NewDBsFromDirectoryConfig() to create multiple DB instances
- Added FindSQLiteDatabases() for directory scanning
- Added IsSQLiteDatabase() for file validation
- Updated ReplicateCommand to handle directory configurations
- Added comprehensive test coverage

Documentation Note:
A follow-up issue should be created in the litestream.io repository to document:
- New configuration options (directory, pattern, recursive)
- Use cases for multi-tenant applications
- Examples of mixed single/directory configurations
- Current limitations (no dynamic discovery without restart)

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <[email protected]>
@corylanou corylanou force-pushed the feature/directory-replication branch from bdc882a to 689debe Compare September 13, 2025 18:52
@corylanou corylanou marked this pull request as ready for review September 13, 2025 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Replicate whole directories
1 participant