Delete fails and HTTP 401 on every other request

[matutter]

I'm running into an issue where DELETE is failing dispite what appears to be a correct registry config. It also looks like every request is tried without basic-auth resulting in an HTTP 401 from the registry but browsing works otherwise.

docker-compose.yml for registry on server

I'm using Ansible to deploy this so these are jinja2 templates.

version: '3'
services:
  registry:
    image: "{{ image }}"
    ports:
    - "5000:5000"
    volumes:
      - "{{ volume_root }}/auth:/auth"
      - "{{ volume_root }}/certs:/certs"
      - "{{ volume_root }}/config.yml:/etc/docker/registry/config.yml"
      - "{{ registry_storage }}:/var/docker"
    restart: always

docker-compose for docker-registry-ui on laptop

version: "3"
services:
  browser:
    image: joxit/docker-registry-ui
    ports:
    - "8004:80"
    environment:
    - REGISTRY_TITLE=Lumarch Docker Registry
    - REGISTRY_URL="{{ registry_http_url }}" # This is "https://domain:5000"
    - DELETE_IMAGES=true

config.yml

I can login and push/pull from this registry just fine. Again, the docker-registry-ui mostly works, I can browse my images and when I disable all auth on the registry deleting tags does work.

version: 0.1
log:
  fields:
    service: registry
storage:
  delete:
    enabled: true
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/docker
auth:
  htpasswd:
    realm: Registry Realm
    path: /auth/htpasswd
http:
  addr: :5000
  host:  "{{ registry_http_url }}" # This is "https://domain:5000"
  tls:
    certificate: /certs/domain.crt
    key: /certs/domain.key
  headers:
    X-Content-Type-Options: [nosniff]
    Access-Control-Allow-Origin: ["http://localhost:8004"]
    Access-Control-Allow-Methods: ['HEAD', 'GET', 'OPTIONS', 'DELETE']
    Access-Control-Expose-Headers: ['Docker-Content-Digest']
    Access-Control-Allow-Credentials: [true]
    Access-Control-Allow-Headers: ['Accept', 'Authorization', 'Content-Type', 'Access-Control-Allow-Headers', 'X-Requested-With']
    Access-Control-Max-Age: [1728000]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3

[matutter]

Guess it looks just like #75 but I'm using local storage.

[isigmund]

I do experience the same issue

[Joxit]

Hum... Maybe that's because you are using proxy pass (REGISTRY_URL) on a registry with https?...

Try with URL variable instead instead of REGISTRY_URL and tell me what happens.

[isigmund]

I am actually using the URL parameter. Here my command how I start the container:

docker run --detach \   
           --restart always \
           --name docker-registry-frontend \
           --env VIRTUAL_HOST=registry-frontend.xxxxxxx.de \
           --env LETSENCRYPT_HOST=registry-frontend.xxxxxxx.de \
           --env URL=https://registry.xxxxxxx.de \
           --env DELETE_IMAGES=true \
           --env REGISTRY_TITLE="SCS Docker Registry" \
           --publish 8080:80 \
           joxit/docker-registry-ui:static 

The one thing that might be worth mentioning is that I use a jwilder/nginx-proxy container in front of this container as well as in front if the registry container itself to handle SSL. I have the following HTTP headers set for the registry container:

  headers:
    X-Content-Type-Options: [nosniff]
    # additional http headers to to allow access from docker-registry-frontend
    Access-Control-Allow-Origin: ['https://registry-frontend.xxxxxxx.de']
    Access-Control-Allow-Methods: ['HEAD', 'GET', 'OPTIONS', 'DELETE']
    Access-Control-Allow-Headers: ['Authorization']
    Access-Control-Max-Age: [1728000]
    Access-Control-Allow-Credentials: [true]
    Access-Control-Expose-Headers: ['Docker-Content-Digest']

[matutter]

@Joxit I tried replacing REGISTRY_URL with just URL in the compose file - yet no change. I also tried this on a variety of browsers to see if it was just a Chrome CORS issue but there was also no change there.

This is from the registry containers docker service logs when trying to delete a container.

infrax_registry.1.43myp6pv3bcq@l3    | 10.255.0.2 - - [24/Sep/2019:21:07:43 +0000] "HEAD /v2/lmch/manifests/test HTTP/2.0" 401 147 "http://localhost:8004/?page=1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36"
infrax_registry.1.43myp6pv3bcq@l3    | time="2019-09-24T21:07:43.46419455Z" level=info msg="authorized request" go.version=go1.11.2 http.request.host="l3.lumarch.dev:5000" http.request.id=1d1ebff7-ffe9-474e-8b7d-dafcbfa1a737 http.request.method=HEAD http.request.referer="http://localhost:8004/?page=1" http.request.remoteaddr="10.255.0.2:39628" http.request.uri="/v2/lmch/manifests/test" http.request.useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" vars.name=lmch vars.reference=test
infrax_registry.1.43myp6pv3bcq@l3    | time="2019-09-24T21:07:43.464632129Z" level=info msg="response completed" go.version=go1.11.2 http.request.host="l3.lumarch.dev:5000" http.request.id=1d1ebff7-ffe9-474e-8b7d-dafcbfa1a737 http.request.method=HEAD http.request.referer="http://localhost:8004/?page=1" http.request.remoteaddr="10.255.0.2:39628" http.request.uri="/v2/lmch/manifests/test" http.request.useragent="Mozilla/5.0 (X11;
Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" http.response.duration=320.428326ms http.response.status=304 http.response.written=0
infrax_registry.1.43myp6pv3bcq@l3    | 10.255.0.2 - - [24/Sep/2019:21:07:43 +0000] "HEAD /v2/lmch/manifests/test HTTP/2.0" 304 0 "http://localhost:8004/?page=1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36"
infrax_registry.1.43myp6pv3bcq@l3    | time="2019-09-24T21:07:43.598917522Z" level=warning msg="error authorizing context: basic authentication
challenge for realm "Registry Realm": invalid authorization credential" go.version=go1.11.2 http.request.host="l3.lumarch.dev:5000" http.request.id=af18ea1a-2e9b-4a9a-a255-804046dd1f04 http.request.method=OPTIONS http.request.referer="http://localhost:8004/?page=1" http.request.remoteaddr="10.255.0.2:39626" http.request.uri="/v2/lmch/manifests/sha256:9539a9b3f7eaef0e77c9f616c7d18843dd896f97b1051dd61124af4a8e0668ec" http.request.useragent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" vars.name=lmch vars.reference="sha256:9539a9b3f7eaef0e77c9f616c7d18843dd896f97b1051dd61124af4a8e0668ec"
infrax_registry.1.43myp6pv3bcq@l3    | 10.255.0.2 - - [24/Sep/2019:21:07:43 +0000] "OPTIONS /v2/lmch/manifests/sha256:9539a9b3f7eaef0e77c9f616c7d18843dd896f97b1051dd61124af4a8e0668ec HTTP/2.0" 401 87 "http://localhost:8004/?page=1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36"

[Joxit]

Hum... Okay, can I see the the header of your failing request (the one who cause the CORS error) ? In the network tab.

The 401 status code is totally normal, the UI do 2 requests when the registry is protected. When the UI get a 401 error, it will do another requests with your credentials and it's OK.

[matutter]

Ok. That explains the requests without credentials.

As I understand it, deleting a tag through the UI should do 3 requests.

1. HEAD - To get the manifest for the tag.
2. OPTIONS - To check if the registry has delete enabled.
3. DELETE - To actually delete some tag(s).

But what I'm seeing is that.

1. HEAD - Sent without credentials, then with credentials as expected.
2. OPTIONS - Is sent without credentials and gets a HTTP 401, then never sends a request with credentials.
3. DELETE - Is never sent.

XHR from the browsers for reference.

[jesseVDwolf]

Having the same issue as @matutter although I can access my tags history. I get the same 401 errors when i try to delete anything. Looking at my chrome network console i can see a couple of things:

1. A HEAD request being made, which fails (probably because no basic auth creds with the request)
2. Another HEAD request being made which succeeds
3. An OPTION request being made, which fails (also because no basic auth creds)

Nothing happens after that. the DELETE request is never send.

[Joxit]

Hi,

I did some research, and I found that is a docker registry miss-configuration of OPTIONS requests.
Our browsers do what they call preflight requests, it's the OPTIONS requests before the real one (the DELETE in our case).
In the issue here, they said that the server respond with an non ok/200 status code, that's "normal" because we need credentials....
I tried a OPTIONS request with curl and that's really the case, the server respond with a 401 status code.

But if I'm understanding well the W3C spec about preflight requests, on OPTION requests, the server must respond a 200 status code with all correct headers event if it needs credentials.

[Joxit]

Can you tell me if this example is working ?

When you use the UI as proxy, you should not have CORS errors...

The other solution will be to override OPTIONS responses on the proxy where your docker registry is hosted. You will need to return 200 status code with all correct headers (those of your docker registry)

[cr1st1p]

@Joxit You got it right, the problem is the answer to the OPTIONS call, from Docker registry. The UI code has nothing to do. Server should return a 2xx error code instead of 401. Because of that 401, nothing further will work.

I just had to change the Cors setup today, for the same reason, by moving the handling from the docker registry's responsibilities (because it does it wrong!), to the upper level, that is to the nginx ingress controller (I'm running docker registry inside a kubernetes cluster).
Nginx is correctly set: on an OPTIONS call, it returns the appropriate Cors header, empty content, and http 204 (If I remember well). It doesn't even pass the request further down to Docker registry.

[cr1st1p]

For the others: also, if Cors headers issue is solved, remember that you also have to enable, inside the Docker registry, the DELETE operation. See https://docs.docker.com/registry/configuration/#delete
Some docker registry images seem to work by setting an environment variable to enable it.

[cr1st1p]

@Joxit -maybe it would be good to document the part about cors header and docker registry being buggy ?