Error: starting container ... setting up Pasta: pasta failed with exit code 1

[paulrenn67]

Issue Description

Hi,

I'm new to Podman, and trying to setup a new server with only a basic Debian 12 install. I spent a while configuring the server, trying to load all the dependencies needed for Podman, upgraded Go, and Podman should be the latest:

Client:       Podman Engine
Version:      5.0.0-dev
API Version:  5.0.0-dev
Go Version:   go1.22.1
Git Commit:   38e22d443cc4ba0300712fa6532c79730b829f21
Built:        Mon Mar 11 19:39:45 2024
OS/Arch:      linux/amd64

I'm attempting to install a rootless user, and I'm finally at a place where I can actually create and attempt to start a pod.

podman network create test-network
podman pod create --name psp-pod --network test-network 

However, Pasta networking fails at startup, here is output from podman pod start psp-pod --log-level=debug

INFO[0000] podman filtering at log level debug
DEBU[0000] Called start.PersistentPreRunE(podman pod start psp-pod --log-level=debug)
DEBU[0000] Using conmon: "/usr/bin/conmon"
INFO[0000] Using sqlite as database backend
DEBU[0000] systemd-logind: Unknown object '/'.
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/psp/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1000/containers
DEBU[0000] Using static dir /home/psp/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/psp/.local/share/containers/storage/volumes
DEBU[0000] Using transient store: false
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument
DEBU[0000] Configured OCI runtime crun-vm initialization failed: no valid executable found for OCI runtime crun-vm: invalid argument
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 49
DEBU[0000] Strongconnecting node 2ba3568f71cfd3779ebfbc53ccfeb084e539605c2f3299ef4630f9ba8a743964
DEBU[0000] Pushed 2ba3568f71cfd3779ebfbc53ccfeb084e539605c2f3299ef4630f9ba8a743964 onto stack
DEBU[0000] Finishing node 2ba3568f71cfd3779ebfbc53ccfeb084e539605c2f3299ef4630f9ba8a743964. Popped 2ba3568f71cfd3779ebfbc53ccfeb084e539605c2f3299ef4630f9ba8a743964 off stack
DEBU[0000] Cached value indicated that idmapped mounts for overlay are not supported
DEBU[0000] Made network namespace at /run/user/1000/netns/netns-8238d692-8b38-1c9b-7cae-83eb84bc3852 for container 2ba3568f71cfd3779ebfbc53ccfeb084e539605c2f3299ef4630f9ba8a743964
DEBU[0000] Check for idmapped mounts support
DEBU[0000] overlay: mount_data=lowerdir=/home/psp/.local/share/containers/storage/overlay/l/BK6K7MHETGSAMGGN5DHK5PRYGQ,upperdir=/home/psp/.local/share/containers/storage/overlay/f4267117530de68e42c6edef8bb7a81a5662f3f0cb708f39ca2baa1b61f95efb/diff,workdir=/home/psp/.local/share/containers/storage/overlay/f4267117530de68e42c6edef8bb7a81a5662f3f0cb708f39ca2baa1b61f95efb/work,userxattr
DEBU[0000] Mounted container "2ba3568f71cfd3779ebfbc53ccfeb084e539605c2f3299ef4630f9ba8a743964" at "/home/psp/.local/share/containers/storage/overlay/f4267117530de68e42c6edef8bb7a81a5662f3f0cb708f39ca2baa1b61f95efb/merged"
DEBU[0000] Successfully loaded network test-network: &{test-network 0932e8bed08d20d078792b39e9dbad47d1e1ba5de0482b12a5eabf71ccd371e0 bridge podman1 2024-03-11 20:17:16.380692008 +0100 CET [{{{10.89.0.0 ffffff00}} 10.89.0.1 <nil>}] [] false false true [] map[] map[] map[driver:host-local]}
DEBU[0000] Successfully loaded 2 networks
DEBU[0000] Created root filesystem for container 2ba3568f71cfd3779ebfbc53ccfeb084e539605c2f3299ef4630f9ba8a743964 at /home/psp/.local/share/containers/storage/overlay/f4267117530de68e42c6edef8bb7a81a5662f3f0cb708f39ca2baa1b61f95efb/merged
DEBU[0000] Creating rootless network namespace at "/run/user/1000/containers/networks/rootless-netns/rootless-netns"
DEBU[0000] pasta arguments: --config-net --pid /run/user/1000/containers/networks/rootless-netns/rootless-netns-conn.pid -t none -u none -T none -U none --no-map-gw --dns none --quiet --netns /run/user/1000/containers/networks/rootless-netns/rootless-netns
DEBU[0000] Unmounted container "2ba3568f71cfd3779ebfbc53ccfeb084e539605c2f3299ef4630f9ba8a743964"
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Cleaning up container 2ba3568f71cfd3779ebfbc53ccfeb084e539605c2f3299ef4630f9ba8a743964
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Container 2ba3568f71cfd3779ebfbc53ccfeb084e539605c2f3299ef4630f9ba8a743964 storage is already unmounted, skipping...
Error: starting container 2ba3568f71cfd3779ebfbc53ccfeb084e539605c2f3299ef4630f9ba8a743964: setting up Pasta: pasta failed with exit code 1:
Couldn't open network namespace /run/user/1000/containers/networks/rootless-netns/rootless-netns

Checking the rootless-netns file, it is empty:

psp@primary:~$ ls -l /run/user/1000/containers/networks/rootless-netns/rootless-netns
-rw------- 1 psp psp 0 Mar 11 21:08 /run/user/1000/containers/networks/rootless-netns/rootless-netns

Not sure how to continue further, does anybody have any clues? Happy to assist and provide any further information if required!

Thanks in advance,
Paul

Steps to reproduce the issue

Steps to reproduce the issue

1. Install clean version of Debian 12
2. Install a ton of packages required by Podman (mostly
3. Upgrade to latest version of Go
4. Compiled Podman from source (mostly followed instructions here https://podman.io/docs/installation)
5. Attempt to start a pod
6. Tear out last remaining hair

Describe the results you received

Error: starting container 2ba3568f71cfd3779ebfbc53ccfeb084e539605c2f3299ef4630f9ba8a743964: setting up Pasta: pasta failed with exit code 1:
Couldn't open network namespace /run/user/1000/containers/networks/rootless-netns/rootless-netns

Describe the results you expected

Pod should start cleanly

podman info output

host:
  arch: amd64
  buildahVersion: 1.35.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.6+ds1-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: unknown'
  cpuUtilization:
    idlePercent: 99.95
    systemPercent: 0.03
    userPercent: 0.02
  cpus: 16
  databaseBackend: sqlite
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: journald
  freeLocks: 2046
  hostname: primary.my-hostname.io
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.1.0-18-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 133924184064
  memTotal: 134750760960
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.4.0-3_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.4.0
    package: netavark_1.4.0-3_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: crun
    package: crun_1.8.1-1+deb12u1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20230309.7c7625d-1_amd64
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU Affero GPL version 3 or later <https://www.gnu.org/licenses/agpl-3.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 4289720320
  swapTotal: 4289720320
  uptime: 1h 41m 1.00s (Approximately 0.04 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/psp/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 0
    stopped: 1
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/psp/.local/share/containers/storage
  graphRootAllocated: 229732732928
  graphRootUsed: 5696118784
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/psp/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.0-dev
  Built: 1710182385
  BuiltTime: Mon Mar 11 19:39:45 2024
  GitCommit: 38e22d443cc4ba0300712fa6532c79730b829f21
  GoVersion: go1.22.1
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.0-dev

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[sbrivio-rh]

@paulrenn67, thanks for reporting this. Could you please try installing the passt package from Debian testing or unstable?

You're running pretty much the latest upstream Podman version, but the package providing pasta(1) is the one from Debian Bookworm, and I'm fairly sure Podman can't use it to set up custom networks. For example, this commit is not included in passt_0.0~git20230309.7c7625d-1_amd64, so pasta(1) can't associate to the existing network namespace matching your test-network.

[Luap99]

Yes I think the old pasta version is the problem, we test with debian sid in CI so this should work fine with the lastest passt version.

[afbjorklund]

Pasta doesn't seem to be documented anywhere under https://podman.io/docs/installation#building-from-source

  pasta:
    executable: ""
    package: ""
    version: ""

Only slirp4netns is mentioned, but not the version.

EDIT: Actually slirp4netns was only mentioned for dnf, but it was missing from apt for the runtime packages

[paulrenn67]

Sorry guys, wasn't able to continue with this. I downgraded podman first to 4.9.3 and then to 4.1.0 (which matches another server we use) finally I was able to get the server running. Unfortunately my woes continue, not able to connect to the machine from my Mac, I have configured networks thus:

Name                         URI                                                         Identity                                    Default
my_podman_server             ssh://psp@XX.XX.XX.XX:22/run/podman/podman.sock             /Users/me/.ssh/id_ed25519     false
podman-machine-default       ssh://core@127.0.0.1:49764/run/user/503/podman/podman.sock  /Users/canapay/.ssh/podman-machine-default  true
podman-machine-default-root  ssh://root@127.0.0.1:49764/run/podman/podman.sock           /Users/canapay/.ssh/podman-machine-default  false

I am able to ssh into my_podman_server directly without password having added the key to the remote machine and to my Mac's ssh agent. However, any attempt to upload an image to the server always fails as follows (and adding my_podman_server to my etc/hosts did not help):

podman --ssh=native image scp localhost/whatever/psp:prod my_podman_server:22:: --log-level=debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Called scp.PersistentPreRunE(podman --ssh=native image scp localhost/whatever/psp:prod my_podman_server:22:: --log-level=debug)
DEBU[0000] SSH Ident Key "/Users/me/.ssh/podman-machine-default" SHA256:my-key ssh-ed25519
DEBU[0000] DoRequest Method: GET URI: http://d/v4.9.3/libpod/_ping
DEBU[0000] DoRequest Method: POST URI: http://d/v4.9.3/libpod/images/scp/localhost/whatever/psp:prod
Error: failed to connect: dial tcp: lookup my_podman_server: no such host
DEBU[0004] Shutting down engines

Incidentally, if I don't supply the :22 port number on the image scp command, I get the following, which I thought I read was fixed a while back (I have podman 4.9.3 on my Mac):

Error: strconv.Atoi: parsing "": invalid syntax

If I can't get this to work today then going to resort to old fashioned scp'ing and bash scripts :/

[Luap99]

@afbjorklund Well updates to the docs are always welcome as you know. Feel free to report that on podman.io repo, the install instructions haven't been updated in a while.

[Luap99]

@paulrenn67 I am going to close this one then assuming it will work with the lastest pasta version.

If you have other problems please make sure to file separate issues. The port thing was fixed in #21583 AFAIK and should be fixed in 5.0 and was backported to 4.9, it will in a future 4.9.4.

[sbrivio-rh]

I downgraded podman first to 4.9.3 and then to 4.1.0 (which matches another server we use) finally I was able to get the server running

By the way, there isn't really a need to downgrade anything. To install a single Debian package from testing or unstable there are a number of established ways, but given that passt has no dependencies, you could keep it quick and just dpkg -i a package you would download from https://packages.debian.org/trixie/amd64/passt/download.

[afbjorklund]

As usual, the package versions vary a bit between the distribution and releases.

• https://packages.ubuntu.com/passt (not in 20.04, not in 22.04, but in 24.04)

• https://packages.debian.org/passt

Well, once you find out that 1) you need pasta and 2) it is in the passtpackage

[sbrivio-rh]

As usual, the package versions vary a bit between the distribution and releases.

• https://packages.ubuntu.com/passt (not in 20.04, not in 22.04, but in 24.04)

Yes, I just maintain the Debian package, and rely on automatic Ubuntu imports which happen before every import freeze (https://wiki.ubuntu.com/DebianImportFreeze). The first Ubuntu version where this happened was 23.04.

• https://packages.debian.org/passt

Well, once you find out that 1) you need pasta and 2) it is in the passtpackage

Point 1) is a distribution matter I think -- I wouldn't know how to improve on it. On Debian, for example, Podman recommends the passt package.

About 2):

$ apt-cache search pasta | grep passt
passt - user-mode networking daemons for virtual machines and namespaces

...how bad can it be?

[afbjorklund]

It could still be worth a line in the install documentation. Even if not github (git clone https://passt.top/passt)

Otherwise it will fail at runtime, since you would need one of them in order to create a rootless network.

Error: could not find slirp4netns, the network namespace can't be configured: exec: "slirp4netns": executable file not found in $PATH

Error: could not find pasta, the network namespace can't be configured: exec: "pasta": executable file not found in $PATH