[RFC] Add Debian packaging for release_1.6, release_2.0 and master

[tttech-industrial-buchsbaum]

Motivation

To be able to install ACRN and friends easily on a Debian system the Debian packaging data have to provided. These are located in the debian directory which contains all the packaging control files. This proposal only adds this directory.

The presented solution has been applied to the the release_1.6 branch of ACRN and uses git-buildpackage via a Docker container to build the ACRN Debian packages.

These packages are meant to be installed on a vanilla Debian Buster installation and have been verified on an Intel NUC7i7DNH and a Kontron COMe-mAL10 based industrial PC.

Prerequisites

• Appropriate Grub Package
  A grub bootloader with "multiboot2: Set min address for mbi allocation to 0x1000" patch applied (http://git.savannah.gnu.org/cgit/grub.git/commit/?id=0f3f5b7c13fa9b677a64cf11f20eca0f850a2b20) is required. I am using a the grub-2.04-7 package build of Debian unstable (https://salsa.debian.org/grub-team/grub) with the above patch added and rebuilt resulting in a grub-2.04-7.1 package.

• Multiboot2 boot protocol
  Since deprivileged boot mode is deprecated according to Like Yan, and furthermore needs adjusting the EFI boot variables at install time, I switched to direct boot mode, and since I need an EFI enable Linux guest system multiboot2 boot protocol is the only choice.

• ACRN SOS Enabled Debian Linux Image Package
  This is required, since this also guarantees the creation of an appropriate initramfs image file needed for booting. It can be easily created by using the ACRN kernel at https://github.com/projectacrn/acrn-kernel with the kernel_config_uefi_sos as config to start with and creating your own Debian linux-image-*.deb (use make bindeb-pkg) required to be installed as an SOS.

Board and Scenario Configurations

Since all configurations contained within the ACRN source tree do not specify an initramfs tag entry for the SOS they cannot be used without change. Instead, I provide the configurations I tested with my Kontron-based IPC and an Intel NUC7i7DNH:
Boards/Scenarios:

• kontron-COMe-mAL10: industry, industry2 (no KATA, but two RT-VMs)
• nuc7i7dnb2: industry

The nuc7i7dnb2 board configuration is derived from the original nuc7i7dnb board configuration basically with the stuff removed a more generic configuration for different variants of target board does not really need. I compared what my NUC7i7DNH provided using board_parser.py with the
original nuc7i7dnb configuration and just left what they have in common. BTW, I also did this for different variants of my kontron-COMe-mAL10!

This also makes it easy to add custom configurations without patching the upstream sources!

Remark: If you switch to the source built-in configs (see debian/rules), they all build, but at least the nuc7i7dnb does not run!

Patches

As you can see in debian/patches there are several patches contained in that build. They are subject to be discussed and solved in separate issues (to be posted ASAP) and touch the following topics:

• Makefile related: fixups & reproducibility
• kontron-COMe-mAL10 related issues
• Provide arbitary initramfs sizes
• Make mandatory config items optional
• Multiboot2 related issues

Usage

Checkout my fork at https://github.com/tttech-industrial-buchsbaum/acrn-hypervisor/tree/debian/1.6 (debian/1.6 branch) and simply run debian/run-docker-build.sh. This will create a Docker image and run the package build in a respective container. The resulting package files are then available in build/debs.

Packages

acrnd_1.6.1-1_amd64.deb: ACRN control daemon
acrnd-dbgsym_1.6.1-1_amd64.deb: Debug symbols of ACRN control daemon
acrn-dev_1.6.1-1_amd64.deb: ACRN header files and libacrn-mngr.a
acrn-devicemodel_1.6.1-1_amd64.deb: ACRN devicemodel
acrn-devicemodel-dbgsym_1.6.1-1_amd64.deb: Debug symbols of ACRN devicemodel
acrn-doc_1.6.1-1_all.deb: ACRN documentation
acrn-hypervisor_1.6.1-1_amd64.deb: ACRN hypervisors
acrn-lifemngr_1.6.1-1_amd64.deb: ACRN lifemngr for UOS
acrn-lifemngr-dbgsym_1.6.1-1_amd64.deb: Debug symbols of ACRN lifemngr
acrn-system_1.6.1-1_amd64.deb: Metapackage to install an ACRN system
acrn-tools_1.6.1-1_amd64.deb: ACRN helper/debugging tools
acrn-tools-dbgsym_1.6.1-1_amd64.deb: Debug symbols of ACRN tools
grub-acrn_1.6.1-1_amd64.deb: Grub setup scripts for ACRN

Other files make up the Debian source package of ACRN.

Build Remarks

ACRN is built as DEBUG version (RELEASE=0), so the ACRN console on the respective serial interface is available.

The acrn-hypervisor package contains ALL possible board/scenario configs which you choose from at install time. This is done using debconf.

The build of the acrn-doc package requires some ACRN kernel files. These are downloaded file by file during build instead of cloning the entire ACRN kernel repository.

Unfortunately there is neither a python3-kconfig nor a python3-sphinx-tab package available for Debian (required for ACRN build). To circumvent these deficiencies you have either to install them via 'pip3 install' on a native build system or use the Dockerfile provided for the build. Whenever there are such packages available, they must be added to the build dependencies for a clean Debian package build!

Installation

On your NUC7i7DN(B/H) or Kontron-COMe-mAL10 based board do:

• install Debian Buster
• install an appropriately updated grub package
• install the ACRN SOS enabled kernel package
• install acrnd, acrn-devicemodel, grub-acrn and acrn-hypervisor packages: beware to choose the appropriate board/scenario configuration!

Rebooting should now automatically boot ACRN and the Linux ACRN SOS.

• recommended: switch networking configuration to systemd-networkd
• optional: install ACRN enabled libvirt

For convenience I provide the respective packages I used during testing
via Dropbox, use them on your own risk:
https://www.dropbox.com/sh/54niwn5rubuuale/AACrdXvpcfXSbdwdudqmzRi6a?dl=0

TODOs

• Important: Add & resolve issues to get rid of the patches!
• Rebase/Cherry-Pick the packaging from debian/1.6 to debian/master (and eventually to the upcoming release branch debian/2.0) to be able to build the 'bleeding edge' of ACRN. This work has already been started.
• Extend this to be able to use it with Ubuntu or any other Debian based distribution.

[tttech-industrial-buchsbaum]

I just updated Debian packaging to debian/1.6.1-2, which contains the following fixes and improvements after a discussion with @NanlinXie

• debian: add lifemngr-split-linux-and-windows-build patch
• debian/rules: only build linux part for life-mngr
• debian: acrn-doc: refer to ACRN web page only
• debian/control: remove acrn-doc related build dependencies
• debian: Dockerfile: remove acrn-doc related build dependencies
• debian: fix acrn-hypervisor.postinst
• debian: add missing postinst/postrm for grub-acrn

@NanlinXie: the changes remove the make error when building life_mngr, documentation page now just redirects to https://projectacrn.github.io/ and the fixes in the postinst/postrm script should fix your problems when installing grub-acrn. Please rebuild the packages using debian/1.6.1-2!

[tttech-industrial-buchsbaum]

Just pushed debian/master to my repository, which provides Debian packages at the current master branch (2.0-unstable). Smoke-tested on NUC7i7DNH only.

[NanlinXie]

I tried the build debian package script successfully, just one comment for proxy user, need add proxy setting into script "acrn-hypervisor/debian/run-docker-build.sh" as below, maybe we could add one optional parameter input for run-docker-build.sh script:

In run-docker-build.sh:
${DOCKER} build -q --build-arg HTTP_PROXY=$HTTP_PROXY
--build-arg HTTPS_PROXY=$HTTP_PROXY --build-arg http_proxy=$HTTP_PROXY --build-arg https_proxy=$HTTP_PROXY -f debian/docker/Dockerfile -t acrn-hypervisor-build debian/docker

${DOCKER} run --rm -u $(id -u):$(id -g) --env http_proxy=$HTTP_PROXY --env https_proxy=$HTTP_PROXY --env HTTP_PROXY=$HTTP_PROXY --env HTTPS_PROXY=$HTTP_PROXY -v $(pwd):/acrn-hypervisor acrn-hypervisor-build $@

[NanlinXie]

@tttech-industrial-buchsbaum , I met the issue that grub.conf is not updated automatically via the *.deb package installation. And when I run "update-grub", the grub.conf is still not generated by 25_linux_acrn.

Here is what I saw under /boot in my environment:
root@debian:/boot# ls -lat
total 72872
drwxr-xr-x 5 root root 4096 Jun 2 18:54 .
drwxr-xr-x 5 root root 4096 May 30 16:11 grub
-rw-r--r-- 1 root root 6797915 May 29 00:58 initrd.img-4.19.119-acrn-sos
-rw-r--r-- 1 root root 5246197 May 10 09:08 System.map-4.19.119-acrn-sos
-rw-r--r-- 1 root root 173892 May 10 09:08 config-4.19.119-acrn-sos
-rw-r--r-- 1 root root 13805440 May 10 09:08 vmlinuz-4.19.119-acrn-sos
-rw-r--r-- 1 root root 13863072 Apr 16 03:48 bzImage
drwxr-xr-x 2 root root 4096 Apr 16 03:02 acrn
-rw-r--r-- 1 root root 25809430 Apr 14 23:25 initrd.img-4.19.0-8-amd64
drwxr-xr-x 18 root root 4096 Apr 14 23:25 ..
-rw-r--r-- 1 root root 3408461 Jan 26 12:01 System.map-4.19.0-8-amd64
-rw-r--r-- 1 root root 206194 Jan 26 12:01 config-4.19.0-8-amd64
-rw-r--r-- 1 root root 5270768 Jan 26 12:01 vmlinuz-4.19.0-8-amd64
drwx------ 3 root root 4096 Dec 31 1969 efi

As you saw, I have the necessary files:
vmlinuz-, config- , System.map-<version and initrd.img- that grub helper scripts needed.

And when I already have ACRN configuration in kernel configuration, see below output:
root@debian:/boot# grep 'CONFIG_ACRN_VHM=y' /boot/config-*
/boot/config-4.19.119-acrn-sos:CONFIG_ACRN_VHM=y

Any more ideas for that issue?

[tttech-industrial-buchsbaum]

@NanlinXie, the acrn-* files are missing!

root@nuc7i7dnh:~# ls -la /boot
total 81788
drwxr-xr-x  4 root root     4096 Jun  2 15:06 .
drwxr-xr-x 19 root root     4096 May 29 09:05 ..
-rw-r--r--  1 root root   342696 Jun  2 15:06 acrn-1.6.bin
-rw-r--r--  1 root root     1181 Jun  2 15:06 acrn-1.6.config
-rw-r--r--  1 root root   688099 Jun  2 15:06 acrn-1.6.map
-rw-r--r--  1 root root   348144 Jun  2 15:06 acrn-1.6.out
-rw-r--r--  1 root root   206157 Apr 29 11:38 config-4.19.0-9-amd64
-rw-r--r--  1 root root   173185 May 29 09:09 config-4.19.106-acrn-sos
drwx------  3 root root     4096 Jan  1  1970 efi
drwxr-xr-x  5 root root     4096 Jun  2 15:06 grub
-rw-r--r--  1 root root 38888011 May 28 12:31 initrd.img-4.19.0-9-amd64
-rw-r--r--  1 root root 15622895 May 29 10:15 initrd.img-4.19.106-acrn-sos
-rw-r--r--  1 root root  3411358 Apr 29 11:38 System.map-4.19.0-9-amd64
-rw-r--r--  1 root root  5132490 May 29 09:09 System.map-4.19.106-acrn-sos
-rw-r--r--  1 root root  5278960 Apr 29 11:38 vmlinuz-4.19.0-9-amd64
-rw-r--r--  1 root root 13616800 May 29 09:09 vmlinuz-4.19.106-acrn-sos

They are contained in the acrn-hypervisor package

root@nuc7i7dnh:~/work/acrn-packages/1.6.1-3# dpkg -c ./acrn-hypervisor_1.6.1-3_amd64.deb 
drwxr-xr-x root/root         0 2020-06-02 14:26 ./
drwxr-xr-x root/root         0 2020-06-02 14:26 ./usr/
drwxr-xr-x root/root         0 2020-06-02 14:26 ./usr/lib/
drwxr-xr-x root/root         0 2020-06-02 14:26 ./usr/lib/acrn/
-rw-r--r-- root/root    343976 2020-06-02 14:26 ./usr/lib/acrn/acrn.kontron-COMe-mAL10.industry.bin
-rw-r--r-- root/root      1178 2020-06-02 14:26 ./usr/lib/acrn/acrn.kontron-COMe-mAL10.industry.config
-rw-r--r-- root/root    716633 2020-06-02 14:26 ./usr/lib/acrn/acrn.kontron-COMe-mAL10.industry.map
-rw-r--r-- root/root    343976 2020-06-02 14:26 ./usr/lib/acrn/acrn.kontron-COMe-mAL10.industry2.bin
-rw-r--r-- root/root      1178 2020-06-02 14:26 ./usr/lib/acrn/acrn.kontron-COMe-mAL10.industry2.config
-rw-r--r-- root/root    719936 2020-06-02 14:26 ./usr/lib/acrn/acrn.kontron-COMe-mAL10.industry2.map
-rw-r--r-- root/root    352368 2020-06-02 14:26 ./usr/lib/acrn/acrn.kontron-COMe-mAL10.uefi.industry.32.out
-rw-r--r-- root/root    342696 2020-06-02 14:26 ./usr/lib/acrn/acrn.nuc7i7dnb2.industry.bin
-rw-r--r-- root/root      1181 2020-06-02 14:26 ./usr/lib/acrn/acrn.nuc7i7dnb2.industry.config
-rw-r--r-- root/root    688099 2020-06-02 14:26 ./usr/lib/acrn/acrn.nuc7i7dnb2.industry.map
-rw-r--r-- root/root    348144 2020-06-02 14:26 ./usr/lib/acrn/acrn.nuc7i7dnb2.uefi.industry.32.out
drwxr-xr-x root/root         0 2020-06-02 14:26 ./usr/share/
drwxr-xr-x root/root         0 2020-06-02 14:26 ./usr/share/doc/
drwxr-xr-x root/root         0 2020-06-02 14:26 ./usr/share/doc/acrn-hypervisor/
-rw-r--r-- root/root       431 2020-06-02 14:26 ./usr/share/doc/acrn-hypervisor/changelog.Debian.gz
-rw-r--r-- root/root      1886 2020-06-02 14:26 ./usr/share/doc/acrn-hypervisor/copyright
drwxr-xr-x root/root         0 2020-06-02 14:26 ./usr/share/lintian/
drwxr-xr-x root/root         0 2020-06-02 14:26 ./usr/share/lintian/overrides/
-rw-r--r-- root/root       117 2020-06-02 14:26 ./usr/share/lintian/overrides/acrn-hypervisor
lrwxrwxrwx root/root         0 2020-06-02 14:26 ./usr/share/doc/acrn-hypervisor/html -> ../acrn/html

and get installed to /boot during the debconf phase of install (maybe there is something missing for debconf ?) But you always can reconfigure manually:

root@nuc7i7dnh:~# dpkg-reconfigure acrn-hypervisor

BTW, please use the latest versions on debian/master or debian/1.6, respectively, I update them regularly first to keep pace with master and second to fix issues I discover during my work.

Here is what I did to completely reinstall ACRN. I am using 1.6.1 here but the same procedure also works with 2.0-unstable. BTW, if there are any leftovers after purging, please remove these files manually. This has already been fixed in the latest versions for Debian packaging!
So I just purged my ACRN installation, rebooted and reinstalled.

root@nuc7i7dnh:~# dpkg -l '*acrn*' | grep ii
ii  acrn-devicemodel              1.6.1-3             amd64        Devicemodel for ACRN Hypervisor
ii  acrn-hypervisor               1.6.1-3             amd64        ACRN Hypervisor for IoT
ii  acrnd                         1.6.1-3             amd64        ACRN Hypervisor control daemon
ii  grub-acrn                     1.6.1-3             amd64        Grub setup scripts for ACRN Hypervisor
ii  linux-image-4.19.106-acrn-sos 4.19.106-acrn-sos-1 amd64        Linux kernel, version 4.19.106-acrn-sos

root@nuc7i7dnh:~# apt purge acrn-devicemodel acrn-hypervisor acrnd grub-acrn

root@nuc7i7dnh:~# dpkg -l '*acrn*' | grep ii
ii  linux-image-4.19.106-acrn-sos 4.19.106-acrn-sos-1 amd64        Linux kernel, version 4.19.106-acrn-sos

root@nuc7i7dnh:~# ls -al /boot/
total 80436
drwxr-xr-x  4 root root     4096 Jun  3 09:24 .
drwxr-xr-x 19 root root     4096 May 29 09:05 ..
-rw-r--r--  1 root root   206157 Apr 29 11:38 config-4.19.0-9-amd64
-rw-r--r--  1 root root   173185 May 29 09:09 config-4.19.106-acrn-sos
drwx------  3 root root     4096 Jan  1  1970 efi
drwxr-xr-x  5 root root     4096 Jun  3 09:24 grub
-rw-r--r--  1 root root 38888011 May 28 12:31 initrd.img-4.19.0-9-amd64
-rw-r--r--  1 root root 15622895 May 29 10:15 initrd.img-4.19.106-acrn-sos
-rw-r--r--  1 root root  3411358 Apr 29 11:38 System.map-4.19.0-9-amd64
-rw-r--r--  1 root root  5132490 May 29 09:09 System.map-4.19.106-acrn-sos
-rw-r--r--  1 root root  5278960 Apr 29 11:38 vmlinuz-4.19.0-9-amd64
-rw-r--r--  1 root root 13616800 May 29 09:09 vmlinuz-4.19.106-acrn-sos

root@nuc7i7dnh:~# reboot

root@nuc7i7dnh:~# cd work/acrn-packages/1.6.1-3/
root@nuc7i7dnh:~/work/acrn-packages/1.6.1-3# apt install ./acrn-devicemodel_*.deb  ./acrnd_*.deb ./acrn-hypervisor_*.deb ./grub-acrn_*.deb

root@nuc7i7dnh:~# reboot

I then reinstalled ACRN, chose the appropriate board/scenario (here: nuc7i7dnb2/industry), and rebooted again:

root@nuc7i7dnh:~# cd work/acrn-packages/1.6.1-3/
root@nuc7i7dnh:~/work/acrn-packages/1.6.1-3# apt install ./acrn-devicemodel_*.deb  ./acrnd_*.deb ./acrn-hypervisor_*.deb ./grub-acrn_*.deb

root@nuc7i7dnh:~# reboot

ACRN installed and came up without any problems:

root@nuc7i7dnh:~# dmesg | grep Hypervisor
[    0.000000] Hypervisor detected: ACRN
[    1.607621] vhm: Virtio & Hypervisor service module initialized
root@nuc7i7dnh:~# dmesg | grep 'Hypervisor detected'
[    0.000000] Hypervisor detected: ACRN
root@nuc7i7dnh:~# ls -al /dev/acrn_vhm 
crw------- 1 root root 243, 0 Jun  3 09:39 /dev/acrn_vhm

root@nuc7i7dnh:~# ls -al /boot
total 81788
drwxr-xr-x  4 root root     4096 Jun  3 09:38 .
drwxr-xr-x 19 root root     4096 May 29 09:05 ..
-rw-r--r--  1 root root   342696 Jun  3 09:29 acrn-1.6.bin
-rw-r--r--  1 root root     1181 Jun  3 09:29 acrn-1.6.config
-rw-r--r--  1 root root   688099 Jun  3 09:29 acrn-1.6.map
-rw-r--r--  1 root root   348144 Jun  3 09:29 acrn-1.6.out
-rw-r--r--  1 root root   206157 Apr 29 11:38 config-4.19.0-9-amd64
-rw-r--r--  1 root root   173185 May 29 09:09 config-4.19.106-acrn-sos
drwx------  3 root root     4096 Jan  1  1970 efi
drwxr-xr-x  5 root root     4096 Jun  3 09:29 grub
-rw-r--r--  1 root root 38888011 May 28 12:31 initrd.img-4.19.0-9-amd64
-rw-r--r--  1 root root 15622895 May 29 10:15 initrd.img-4.19.106-acrn-sos
-rw-r--r--  1 root root  3411358 Apr 29 11:38 System.map-4.19.0-9-amd64
-rw-r--r--  1 root root  5132490 May 29 09:09 System.map-4.19.106-acrn-sos
-rw-r--r--  1 root root  5278960 Apr 29 11:38 vmlinuz-4.19.0-9-amd64
-rw-r--r--  1 root root 13616800 May 29 09:09 vmlinuz-4.19.106-acrn-sos

root@nuc7i7dnh:~# grep multiboot /boot/grub/grub.cfg 
	multiboot2	/boot/acrn-1.6.bin   root=UUID=ad0a614e-d1b7-4ca1-9165-ca96a31687df ro  quiet
			multiboot2	/boot/acrn-1.6.bin   root=UUID=ad0a614e-d1b7-4ca1-9165-ca96a31687df ro  quiet
			multiboot2	/boot/acrn-1.6.bin  root=UUID=ad0a614e-d1b7-4ca1-9165-ca96a31687df ro single 

[NanlinXie]

@tttech-industrial-buchsbaum , good news! I have successfully boot up on KBL NUC i7 using your Debian packaging script.

1. Build all the Debian packages by adding proxy
2. Install below Debian packages
   root@debian:~/tttech/debs# cat apt_install.sh
   apt install ./grub-common_2.04-7.1_amd64.deb
   apt install ./grub-efi-amd64_2.04-7.1_amd64.deb
   apt install ./grub-efi-amd64-bin_2.04-7.1_amd64.deb
   apt install ./grub2-common_2.04-7.1_amd64.deb

apt install ./linux-image-4.19.119-acrn-sos_4.19.119-acrn-sos-1_amd64.deb

apt install ./acpica-tools_20200326-1_amd64.deb
apt install ./acrnd_2.0-unstable-1~2.gbpbeeb50_amd64.deb

apt install ./acrn-devicemodel_2.0-unstable-1~2.gbpbeeb50_amd64.deb

apt install ./acrn-hypervisor_2.0-unstable-1~2.gbpbeeb50_amd64.deb

apt install ./grub-acrn_2.0-unstable-1~2.gbpbeeb50_amd64.deb

3. Boot up with Grub option "Debian GNU/Linux, with ACRN hypervisor"

[tttech-industrial-buchsbaum]

I just added a debian/2.0 branch that contains the preparation for Debian packaging for the upcoming 2.0 release (using release_2.0 branch as base). Additionally I updated debian/master accordingly now containing the results of the 2.0 patch rework.

The patches could be drastically reduced and are now containing some Makefile improvements, minor compile warning fixes, my 3 RTVM patch, the tap-name patch and most important the fix to provide booting bzimge/initrd without overwriting each other (this could be dramatically simplified). Furthermore there is a patch compensating for a grub problem providing a bogus E820 mmap entry (for Kontron COMe-mAL10). For the debian/master branch I had to add changes by @shuox from the release_2.0 branch, otherwise my Kontron COMe-mAL10 base board freezes as soon as I start a VM. Furthermore there is no need for special grub packages any more. My systems now all use the stock Debian Buster grub-2.01+dfsg1-20 package version.

Still to do: backport the simplifications to 1.6.x.

[tttech-industrial-buchsbaum]

An update for debian/1.6 just has been pushed. I now uses the results of patch rework from debian/2.0 plus the changes by @jsun26intel for multiboot/multiboot2 stuff.