Skip to content

Commit ed16697

Browse files
edmundmillerclaude
andcommitted
feat: Add Wave integration for unified package management
Integrates the unified package management system with Wave for container building: - Add PackageSpec to Wave PackagesSpec conversion in WaveClient - Support provider mapping (conda/mamba/pixi -> CONDA type in Wave) - Enable package directive processing through Wave container builds - Use HxClient API for HTTP requests (aligned with lib-httpx refactor) - Add comprehensive package management documentation - Remove deprecated Pixi implementation in favor of unified system - Add integration tests for package management workflows This enables the `package` directive to work seamlessly with Wave for building containers with the specified package dependencies. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>
1 parent 146deee commit ed16697

34 files changed

+1195
-1660
lines changed

PACKAGE_MANAGEMENT.md

Lines changed: 219 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,219 @@
1+
# Unified Package Management System
2+
3+
This document describes the new unified package management system in Nextflow, introduced as a preview feature behind the `nextflow.preview.package` flag.
4+
5+
## Overview
6+
7+
The unified package management system provides a consistent interface for managing packages across different package managers (conda, pixi, mamba, etc.) through a single `package` directive.
8+
9+
## Enabling the Feature
10+
11+
Add this to your `nextflow.config`:
12+
13+
```groovy
14+
nextflow.preview.package = true
15+
```
16+
17+
## Basic Usage
18+
19+
### Single Package
20+
21+
```groovy
22+
process example {
23+
package "samtools=1.17", provider: "conda"
24+
25+
script:
26+
"""
27+
samtools --version
28+
"""
29+
}
30+
```
31+
32+
### Multiple Packages
33+
34+
```groovy
35+
process example {
36+
package ["samtools=1.17", "bcftools=1.18"], provider: "conda"
37+
38+
script:
39+
"""
40+
samtools --version
41+
bcftools --version
42+
"""
43+
}
44+
```
45+
46+
### Using Default Provider
47+
48+
Configure a default provider in your config:
49+
50+
```groovy
51+
packages {
52+
provider = 'conda'
53+
}
54+
```
55+
56+
Then use:
57+
58+
```groovy
59+
process example {
60+
package "samtools=1.17" // uses default provider
61+
62+
script:
63+
"""
64+
samtools --version
65+
"""
66+
}
67+
```
68+
69+
### Advanced Configuration
70+
71+
```groovy
72+
process example {
73+
package {
74+
provider = "conda"
75+
packages = ["samtools=1.17", "bcftools=1.18"]
76+
channels = ["conda-forge", "bioconda"]
77+
options = [
78+
createTimeout: "30 min"
79+
]
80+
}
81+
82+
script:
83+
"""
84+
samtools --version
85+
bcftools --version
86+
"""
87+
}
88+
```
89+
90+
### Environment Files
91+
92+
```groovy
93+
process example {
94+
package {
95+
provider = "conda"
96+
environment = file("environment.yml")
97+
}
98+
99+
script:
100+
"""
101+
python script.py
102+
"""
103+
}
104+
```
105+
106+
## Supported Providers
107+
108+
- `conda` - Anaconda/Miniconda package manager
109+
- `pixi` - Fast conda alternative with lockfiles
110+
- `mamba` - Fast conda alternative
111+
- `micromamba` - Minimal conda implementation
112+
113+
## Configuration
114+
115+
### Global Configuration
116+
117+
```groovy
118+
// nextflow.config
119+
nextflow.preview.package = true
120+
121+
packages {
122+
provider = 'conda' // default provider
123+
}
124+
125+
// Provider-specific configurations
126+
conda {
127+
channels = ['conda-forge', 'bioconda']
128+
createTimeout = '20 min'
129+
}
130+
131+
pixi {
132+
cacheDir = '/tmp/pixi-cache'
133+
}
134+
```
135+
136+
## Wave Integration
137+
138+
The unified package system integrates with Wave for containerization:
139+
140+
```groovy
141+
process example {
142+
package "samtools=1.17", provider: "conda"
143+
144+
script:
145+
"""
146+
samtools --version
147+
"""
148+
}
149+
```
150+
151+
Wave will automatically create a container with the specified packages.
152+
153+
## Backward Compatibility
154+
155+
Old `conda` and `pixi` directives continue to work but show deprecation warnings when the preview feature is enabled:
156+
157+
```groovy
158+
process oldStyle {
159+
conda 'samtools=1.17' // Shows deprecation warning
160+
161+
script:
162+
"""
163+
samtools --version
164+
"""
165+
}
166+
```
167+
168+
## Migration Guide
169+
170+
### From conda directive
171+
172+
**Before:**
173+
```groovy
174+
process example {
175+
conda 'samtools=1.17 bcftools=1.18'
176+
script: "samtools --version"
177+
}
178+
```
179+
180+
**After:**
181+
```groovy
182+
process example {
183+
package ["samtools=1.17", "bcftools=1.18"], provider: "conda"
184+
script: "samtools --version"
185+
}
186+
```
187+
188+
### From pixi directive
189+
190+
**Before:**
191+
```groovy
192+
process example {
193+
pixi 'samtools bcftools'
194+
script: "samtools --version"
195+
}
196+
```
197+
198+
**After:**
199+
```groovy
200+
process example {
201+
package ["samtools", "bcftools"], provider: "pixi"
202+
script: "samtools --version"
203+
}
204+
```
205+
206+
## Plugin Architecture
207+
208+
The system is extensible through plugins. Package managers are implemented as plugins that extend the `PackageProviderExtension` interface:
209+
210+
- `nf-conda` - Conda support
211+
- `nf-pixi` - Pixi support
212+
213+
Custom package managers can be added by implementing the `PackageProvider` interface and registering as a plugin.
214+
215+
## Examples
216+
217+
See the test files for complete examples:
218+
- `tests/package-test.nf` - Basic usage examples
219+
- `tests/integration-test.nf` - Integration and backward compatibility tests

docs/conda.md

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,10 @@ The Conda environment feature is not supported by executors that use remote obje
3535

3636
The use of Conda recipes specified using the {ref}`process-conda` directive needs to be enabled explicitly in the pipeline configuration file (i.e. `nextflow.config`):
3737

38+
:::{note}
39+
Nextflow also provides a unified {ref}`package-page` system that supports conda and other package managers through a single interface. This newer system is enabled with the `preview.package` feature flag and provides a more consistent experience across different package managers.
40+
:::
41+
3842
```groovy
3943
conda.enabled = true
4044
```
@@ -191,6 +195,49 @@ process hello {
191195

192196
It is also possible to use [mamba](https://github.com/mamba-org/mamba) to speed up the creation of conda environments. For more information on how to enable this feature please refer to {ref}`Conda <config-conda>`.
193197

198+
## Migration to Unified Package Management
199+
200+
The unified {ref}`package-page` system provides a modern alternative to the conda directive. When the `preview.package` feature is enabled, you can use the new syntax:
201+
202+
### Before (conda directive):
203+
```nextflow
204+
process example {
205+
conda 'samtools=1.15 bcftools=1.15'
206+
207+
script:
208+
"""
209+
samtools --version
210+
bcftools --version
211+
"""
212+
}
213+
```
214+
215+
### After (package directive):
216+
```nextflow
217+
process example {
218+
package 'samtools=1.15 bcftools=1.15', provider: 'conda'
219+
220+
script:
221+
"""
222+
samtools --version
223+
bcftools --version
224+
"""
225+
}
226+
```
227+
228+
To enable the unified package system:
229+
230+
```groovy
231+
// nextflow.config
232+
nextflow.preview.package = true
233+
```
234+
235+
The unified system provides:
236+
- Consistent interface across different package managers
237+
- Plugin-based architecture for extensibility
238+
- Better integration with containerization platforms
239+
- Support for multiple package managers (conda, pixi, etc.)
240+
194241
## Best practices
195242

196243
When a `conda` directive is used in any `process` definition within the workflow script, Conda tool is required for the workflow execution.

0 commit comments

Comments
 (0)