|
| 1 | +# Unified Package Management System |
| 2 | + |
| 3 | +This document describes the new unified package management system in Nextflow, introduced as a preview feature behind the `nextflow.preview.package` flag. |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +The unified package management system provides a consistent interface for managing packages across different package managers (conda, pixi, mamba, etc.) through a single `package` directive. |
| 8 | + |
| 9 | +## Enabling the Feature |
| 10 | + |
| 11 | +Add this to your `nextflow.config`: |
| 12 | + |
| 13 | +```groovy |
| 14 | +nextflow.preview.package = true |
| 15 | +``` |
| 16 | + |
| 17 | +## Basic Usage |
| 18 | + |
| 19 | +### Single Package |
| 20 | + |
| 21 | +```groovy |
| 22 | +process example { |
| 23 | + package "samtools=1.17", provider: "conda" |
| 24 | + |
| 25 | + script: |
| 26 | + """ |
| 27 | + samtools --version |
| 28 | + """ |
| 29 | +} |
| 30 | +``` |
| 31 | + |
| 32 | +### Multiple Packages |
| 33 | + |
| 34 | +```groovy |
| 35 | +process example { |
| 36 | + package ["samtools=1.17", "bcftools=1.18"], provider: "conda" |
| 37 | + |
| 38 | + script: |
| 39 | + """ |
| 40 | + samtools --version |
| 41 | + bcftools --version |
| 42 | + """ |
| 43 | +} |
| 44 | +``` |
| 45 | + |
| 46 | +### Using Default Provider |
| 47 | + |
| 48 | +Configure a default provider in your config: |
| 49 | + |
| 50 | +```groovy |
| 51 | +packages { |
| 52 | + provider = 'conda' |
| 53 | +} |
| 54 | +``` |
| 55 | + |
| 56 | +Then use: |
| 57 | + |
| 58 | +```groovy |
| 59 | +process example { |
| 60 | + package "samtools=1.17" // uses default provider |
| 61 | + |
| 62 | + script: |
| 63 | + """ |
| 64 | + samtools --version |
| 65 | + """ |
| 66 | +} |
| 67 | +``` |
| 68 | + |
| 69 | +### Advanced Configuration |
| 70 | + |
| 71 | +```groovy |
| 72 | +process example { |
| 73 | + package { |
| 74 | + provider = "conda" |
| 75 | + packages = ["samtools=1.17", "bcftools=1.18"] |
| 76 | + channels = ["conda-forge", "bioconda"] |
| 77 | + options = [ |
| 78 | + createTimeout: "30 min" |
| 79 | + ] |
| 80 | + } |
| 81 | + |
| 82 | + script: |
| 83 | + """ |
| 84 | + samtools --version |
| 85 | + bcftools --version |
| 86 | + """ |
| 87 | +} |
| 88 | +``` |
| 89 | + |
| 90 | +### Environment Files |
| 91 | + |
| 92 | +```groovy |
| 93 | +process example { |
| 94 | + package { |
| 95 | + provider = "conda" |
| 96 | + environment = file("environment.yml") |
| 97 | + } |
| 98 | + |
| 99 | + script: |
| 100 | + """ |
| 101 | + python script.py |
| 102 | + """ |
| 103 | +} |
| 104 | +``` |
| 105 | + |
| 106 | +## Supported Providers |
| 107 | + |
| 108 | +- `conda` - Anaconda/Miniconda package manager |
| 109 | +- `pixi` - Fast conda alternative with lockfiles |
| 110 | +- `mamba` - Fast conda alternative |
| 111 | +- `micromamba` - Minimal conda implementation |
| 112 | + |
| 113 | +## Configuration |
| 114 | + |
| 115 | +### Global Configuration |
| 116 | + |
| 117 | +```groovy |
| 118 | +// nextflow.config |
| 119 | +nextflow.preview.package = true |
| 120 | +
|
| 121 | +packages { |
| 122 | + provider = 'conda' // default provider |
| 123 | +} |
| 124 | +
|
| 125 | +// Provider-specific configurations |
| 126 | +conda { |
| 127 | + channels = ['conda-forge', 'bioconda'] |
| 128 | + createTimeout = '20 min' |
| 129 | +} |
| 130 | +
|
| 131 | +pixi { |
| 132 | + cacheDir = '/tmp/pixi-cache' |
| 133 | +} |
| 134 | +``` |
| 135 | + |
| 136 | +## Wave Integration |
| 137 | + |
| 138 | +The unified package system integrates with Wave for containerization: |
| 139 | + |
| 140 | +```groovy |
| 141 | +process example { |
| 142 | + package "samtools=1.17", provider: "conda" |
| 143 | + |
| 144 | + script: |
| 145 | + """ |
| 146 | + samtools --version |
| 147 | + """ |
| 148 | +} |
| 149 | +``` |
| 150 | + |
| 151 | +Wave will automatically create a container with the specified packages. |
| 152 | + |
| 153 | +## Backward Compatibility |
| 154 | + |
| 155 | +Old `conda` and `pixi` directives continue to work but show deprecation warnings when the preview feature is enabled: |
| 156 | + |
| 157 | +```groovy |
| 158 | +process oldStyle { |
| 159 | + conda 'samtools=1.17' // Shows deprecation warning |
| 160 | + |
| 161 | + script: |
| 162 | + """ |
| 163 | + samtools --version |
| 164 | + """ |
| 165 | +} |
| 166 | +``` |
| 167 | + |
| 168 | +## Migration Guide |
| 169 | + |
| 170 | +### From conda directive |
| 171 | + |
| 172 | +**Before:** |
| 173 | +```groovy |
| 174 | +process example { |
| 175 | + conda 'samtools=1.17 bcftools=1.18' |
| 176 | + script: "samtools --version" |
| 177 | +} |
| 178 | +``` |
| 179 | + |
| 180 | +**After:** |
| 181 | +```groovy |
| 182 | +process example { |
| 183 | + package ["samtools=1.17", "bcftools=1.18"], provider: "conda" |
| 184 | + script: "samtools --version" |
| 185 | +} |
| 186 | +``` |
| 187 | + |
| 188 | +### From pixi directive |
| 189 | + |
| 190 | +**Before:** |
| 191 | +```groovy |
| 192 | +process example { |
| 193 | + pixi 'samtools bcftools' |
| 194 | + script: "samtools --version" |
| 195 | +} |
| 196 | +``` |
| 197 | + |
| 198 | +**After:** |
| 199 | +```groovy |
| 200 | +process example { |
| 201 | + package ["samtools", "bcftools"], provider: "pixi" |
| 202 | + script: "samtools --version" |
| 203 | +} |
| 204 | +``` |
| 205 | + |
| 206 | +## Plugin Architecture |
| 207 | + |
| 208 | +The system is extensible through plugins. Package managers are implemented as plugins that extend the `PackageProviderExtension` interface: |
| 209 | + |
| 210 | +- `nf-conda` - Conda support |
| 211 | +- `nf-pixi` - Pixi support |
| 212 | + |
| 213 | +Custom package managers can be added by implementing the `PackageProvider` interface and registering as a plugin. |
| 214 | + |
| 215 | +## Examples |
| 216 | + |
| 217 | +See the test files for complete examples: |
| 218 | +- `tests/package-test.nf` - Basic usage examples |
| 219 | +- `tests/integration-test.nf` - Integration and backward compatibility tests |
0 commit comments