Skip to content

Add an example of golang:1.21-alpine3.18 #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Aug 24, 2023
Merged

Add an example of golang:1.21-alpine3.18 #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
7303caa
Show more info in the table output
AkihiroSuda Aug 23, 2023
627a4fa
Add IgnoreFileModeRedundantBits
AkihiroSuda Aug 23, 2023
77de54b
Fix comparing nil map vs empty map
AkihiroSuda Aug 24, 2023
74d487c
load: silence `failed to read stream source error="read |0: file alre…
AkihiroSuda Aug 24, 2023
f8ab41e
Experimental support for comparing multi-platform vs single-platform
AkihiroSuda Aug 23, 2023
866f5e2
Experimental support for comparing layers[N] vs layers[M]
AkihiroSuda Aug 23, 2023
ef3e663
diffLayer: fix comparison on length mismatch
AkihiroSuda Aug 24, 2023
53428c8
README.md: add an example of golang:1.21-alpine3.18
AkihiroSuda Aug 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 126 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,3 +110,129 @@ See the `docker://` example above, and read `docker` as `podman`.

### Accessing private images
To access private images, create a credential file as `~/.docker/config.json` using `docker login`.


- - -
# Examples
## Non-reproducible Docker Hub images

### `golang:1.21-alpine3.18`
The sources of the official Docker Hub images are available at <https://github.com/docker-library>.

For example, the source of [`golang:1.21-alpine3.18`](https://hub.docker.com/layers/library/golang/1.21-alpine3.18/images/sha256-dd8888bb7f1b0b05e1e625aa29483f50f38a9b64073a4db00b04076cec52b71c?context=explore)
can be found at <https://github.com/docker-library/golang/blob/d1ff31b86b23fe721dc65806cd2bd79a4c71b039/1.21/alpine3.18/Dockerfile>.

The source can be built as follows:

```console
$ DOCKER_BUILDKIT=0 docker build -t my-golang-1.21-alpine3.18 'https://github.com/docker-library/golang.git#d1ff31b86b23fe721dc65806cd2bd79a4c71b039:1.21/alpine3.18'
...
Successfully tagged my-golang-1.21-alpine3.18:latest
```

> **Note**
>
> `DOCKER_BUILDKIT=0` is specified here because the official `golang:1.21-alpine3.18` image is currently built with the legacy builder.
> A future revision of the official image may be built with BuildKit, and in such a case, `DOCKER_BUILDKIT=1` will rather need to be specified here.

The resulting image binary (`my-golang-1.21-alpine3.18`) can be compared with the official image binary (`golang:1.21-alpine3.18`) as follows:

```console
$ diffoci diff docker://golang:1.21-alpine3.18 docker://my-golang-1.21-alpine3.18 --semantic --report-dir=~/diff
INFO[0000] Loading image "docker.io/library/golang:1.21-alpine3.18" from "docker"
docker.io/library/golang:1.21 alpine3.18 saved
Importing elapsed: 2.6 s total: 0.0 B (0.0 B/s)
INFO[0004] Loading image "docker.io/library/my-golang-1.21-alpine3.18:latest" from "docker"
docker.io/library/my golang 1.21 alpine3 saved
Importing elapsed: 2.6 s total: 0.0 B (0.0 B/s)
TYPE NAME INPUT-0 INPUT-1
Layer ctx:/layers-1/layer length mismatch (457 vs 454)
File lib/apk/db/scripts.tar eef110e559acb7aa00ea23ee7b8bddb52c4526cd394749261aa244ef9c6024a4 342eaa013375398497bfc21dff7dd017a647032ec5c486011142c576b7ccc989
Layer ctx:/layers-1/layer name "usr/local/share/ca-certificates/.wh..wh..opq" only appears in input 0
Layer ctx:/layers-1/layer name "usr/share/ca-certificates/.wh..wh..opq" only appears in input 0
Layer ctx:/layers-1/layer name "etc/ca-certificates/.wh..wh..opq" only appears in input 0
Layer ctx:/layers-2/layer length mismatch (13927 vs 13926)
Layer ctx:/layers-2/layer name "usr/local/go/.wh..wh..opq" only appears in input 0
File lib/apk/db/scripts.tar 073bb5094fc5bba800f06661dc7f1325c5cb4250b13209fb9e3eaf4e60e4bfc4 1369581b62bd60304c59556ea85f585bd498040c8fa223243622bb7990833063
Layer ctx:/layers-3/layer length mismatch (4 vs 3)
Layer ctx:/layers-3/layer name "go/.wh..wh..opq" only appears in input 0
```

> **Note**
> The `--semantic` flag is specified to ignore differences of timestamps, image names, and other "boring" attributes.
> Without this flag, the `diffoci` command may print an enourmous amount of output.

In the `my-golang-1.21-alpine3.18` image, special files called ["Opaque whiteouts"](https://github.com/opencontainers/image-spec/blob/v1.0.2/layer.md#whiteouts) (`.wh..wh..opq`)
are missing due to filesystem difference between Docker Hub's build machine and the local machine.

Also, the `lib/apk/db/scripts.tar` file in the layer 1 is not reproducible due to the timestamps of the tar entries inside it.
The differences can be inspected by running the [`diffoscope`](https://diffoscope.org/) command for `~/diff/input-{0,1}/layers-1/lib/apk/db/scripts.tar`:
```console
$ sudo apt-get install -y diffoscope

$ diffoscope ~/diff/input-0/layers-1/lib/apk/db/scripts.tar ~/diff/input-1/layers-1/lib/apk/db/scripts.tar
--- /home/suda/diff/input-0/layers-1/lib/apk/db/scripts.tar
+++ /home/suda/diff/input-1/layers-1/lib/apk/db/scripts.tar
├── file list
│ @@ -1,9 +1,9 @@
│ --rwxr-xr-x 0 root (0) root (0) 56 2023-08-09 03:36:47.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-install
│ --rwxr-xr-x 0 root (0) root (0) 983 2023-08-09 03:36:47.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-install
│ --rwxr-xr-x 0 root (0) root (0) 755 2023-08-09 03:36:47.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-upgrade
│ --rwxr-xr-x 0 root (0) root (0) 983 2023-08-09 03:36:47.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-upgrade
│ --rwxr-xr-x 0 root (0) root (0) 139 2023-08-09 03:36:47.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-install
│ --rwxr-xr-x 0 root (0) root (0) 1239 2023-08-09 03:36:47.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-upgrade
│ --rwxr-xr-x 0 root (0) root (0) 546 2023-08-09 03:36:47.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.trigger
│ --rwxr-xr-x 0 root (0) root (0) 137 2023-08-09 03:36:47.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.post-deinstall
│ --rwxr-xr-x 0 root (0) root (0) 63 2023-08-09 03:36:47.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.trigger
│ +-rwxr-xr-x 0 root (0) root (0) 56 2023-08-24 07:50:41.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-install
│ +-rwxr-xr-x 0 root (0) root (0) 983 2023-08-24 07:50:41.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-install
│ +-rwxr-xr-x 0 root (0) root (0) 755 2023-08-24 07:50:41.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-upgrade
│ +-rwxr-xr-x 0 root (0) root (0) 983 2023-08-24 07:50:41.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-upgrade
│ +-rwxr-xr-x 0 root (0) root (0) 139 2023-08-24 07:50:41.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-install
│ +-rwxr-xr-x 0 root (0) root (0) 1239 2023-08-24 07:50:41.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-upgrade
│ +-rwxr-xr-x 0 root (0) root (0) 546 2023-08-24 07:50:41.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.trigger
│ +-rwxr-xr-x 0 root (0) root (0) 137 2023-08-24 07:50:41.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.post-deinstall
│ +-rwxr-xr-x 0 root (0) root (0) 63 2023-08-24 07:50:41.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.trigger
```

These differences are boring, but not filtered out by the `--semantic` flag of the `diffoci` command, because `diffoci` is not aware of the formats of the files inside the image layers.

The `lib/apk/db/scripts.tar` file in the layer 2 has the same issue:
<details>
<p>

```console
$ diffoscope ~/diff/input-0/layers-2/lib/apk/db/scripts.tar ~/diff/input-1/layers-2/lib/apk/db/scripts.tar
--- /home/suda/diff/input-0/layers-2/lib/apk/db/scripts.tar
+++ /home/suda/diff/input-1/layers-2/lib/apk/db/scripts.tar
├── file list
│ @@ -1,9 +1,9 @@
│ --rwxr-xr-x 0 root (0) root (0) 56 2023-08-09 04:41:27.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-install
│ --rwxr-xr-x 0 root (0) root (0) 983 2023-08-09 04:41:27.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-install
│ --rwxr-xr-x 0 root (0) root (0) 755 2023-08-09 04:41:27.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-upgrade
│ --rwxr-xr-x 0 root (0) root (0) 983 2023-08-09 04:41:27.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-upgrade
│ --rwxr-xr-x 0 root (0) root (0) 139 2023-08-09 04:41:27.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-install
│ --rwxr-xr-x 0 root (0) root (0) 1239 2023-08-09 04:41:27.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-upgrade
│ --rwxr-xr-x 0 root (0) root (0) 546 2023-08-09 04:41:27.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.trigger
│ --rwxr-xr-x 0 root (0) root (0) 137 2023-08-09 04:41:27.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.post-deinstall
│ --rwxr-xr-x 0 root (0) root (0) 63 2023-08-09 04:41:27.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.trigger
│ +-rwxr-xr-x 0 root (0) root (0) 56 2023-08-24 07:50:52.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-install
│ +-rwxr-xr-x 0 root (0) root (0) 983 2023-08-24 07:50:52.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-install
│ +-rwxr-xr-x 0 root (0) root (0) 755 2023-08-24 07:50:52.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.pre-upgrade
│ +-rwxr-xr-x 0 root (0) root (0) 983 2023-08-24 07:50:52.000000 alpine-baselayout-3.4.3-r1.Q1zwvKMnYs1b6ZdPTBJ0Z7D5P3jyA=.post-upgrade
│ +-rwxr-xr-x 0 root (0) root (0) 139 2023-08-24 07:50:52.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-install
│ +-rwxr-xr-x 0 root (0) root (0) 1239 2023-08-24 07:50:52.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.post-upgrade
│ +-rwxr-xr-x 0 root (0) root (0) 546 2023-08-24 07:50:52.000000 busybox-1.36.1-r2.Q1gQ/L3UBnSjgkFWEHQaUkUDubqdI=.trigger
│ +-rwxr-xr-x 0 root (0) root (0) 137 2023-08-24 07:50:52.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.post-deinstall
│ +-rwxr-xr-x 0 root (0) root (0) 63 2023-08-24 07:50:52.000000 ca-certificates-20230506-r0.Q1FG8M+7w+dkjV9Vy0mGFWW2t4+Do=.trigger
```

</p>
</details>

Depending on the time to build the image, more differences may happen, especially when the Alpine packages on the internet are bumped up.

#### Conclusion
This example indicates that although the official `golang:1.21-alpine3.18` image binary is not fully reproducible, its non-reproducibility is practically negligible, and
this image binary can be assured to be certainly built from with the [published source](https://github.com/docker-library/golang/blob/d1ff31b86b23fe721dc65806cd2bd79a4c71b039/1.21/alpine3.18/Dockerfile).
**If the published source is trustable**, this image binary can be trusted too.
6 changes: 6 additions & 0 deletions cmd/diffoci/commands/diff/diff.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ func NewCommand() *cobra.Command {
"ignore-timestamps",
"ignore-history",
"ignore-file-order",
"ignore-file-mode-redundant-bits",
"ignore-image-name",
}
for _, f := range flagNames {
Expand All @@ -60,6 +61,7 @@ func NewCommand() *cobra.Command {
flags.Bool("ignore-timestamps", false, "Ignore timestamps")
flags.Bool("ignore-history", false, "Ignore history")
flags.Bool("ignore-file-order", false, "Ignore file order in tar layers")
flags.Bool("ignore-file-mode-redundant-bits", false, "Ignore redundant bits of file mode")
flags.Bool("ignore-image-name", false, "Ignore image name annotation")
flags.Bool("semantic", false, "[Recommended] Alias for --ignore-*=true")

Expand Down Expand Up @@ -96,6 +98,10 @@ func action(cmd *cobra.Command, args []string) error {
if err != nil {
return err
}
options.IgnoreFileModeRedundantBits, err = flags.GetBool("ignore-file-mode-redundant-bits")
if err != nil {
return err
}
options.IgnoreImageName, err = flags.GetBool("ignore-image-name")
if err != nil {
return err
Expand Down
14 changes: 13 additions & 1 deletion cmd/diffoci/imagegetter/imagegetter.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,18 @@ func (g *ImageGetter) getPodman(ctx context.Context, rawRef string, plats []ocis
return g.loadDocker(ctx, podman, name, plats)
}

type readerWithEOF struct {
io.Reader
}

func (r *readerWithEOF) Read(p []byte) (int, error) {
n, err := r.Reader.Read(p)
if errors.Is(err, os.ErrClosed) {
err = io.EOF
}
return n, err
}

// loadDocker runs `docker save` and loads the result
func (g *ImageGetter) loadDocker(ctx context.Context, docker, name string, plats []ocispec.Platform) (*images.Image, error) {
log.G(ctx).Infof("Loading image %q from %q", name, docker)
Expand All @@ -151,7 +163,7 @@ func (g *ImageGetter) loadDocker(ctx context.Context, docker, name string, plats
if err = dockerCmd.Start(); err != nil {
return nil, fmt.Errorf("failed to run %v: %w", dockerCmd.Args, err)
}
if err = Load(ctx, g.progressWriter, g.transferrer, r, plats, name); err != nil {
if err = Load(ctx, g.progressWriter, g.transferrer, &readerWithEOF{r}, plats, name); err != nil {
return nil, fmt.Errorf("failed to load an archive (from %v): %w", dockerCmd.Args, err)
}
if err = r.Close(); err != nil {
Expand Down
Loading