Unable to verify Docker Registry endpoint

[pr]

I tried to create an External Feed with:

Feed Type: Docker Container Registry
Name: xxx Docker Registry feedxxx-docker-registry-feed
URL:https://docker.xxxxxx.net/v2
Registry Path: Please enter the path of the registry
Credentials: Credentials have been entered; username is yyyyyyy

But when testing I have error:

Octopus.Core.Packages.Docker.DockerRegistryException: Unable to verify Docker Registry endpoint https://docker.xxxxxx.net/v2 against the v2 version of the API. Please verify the feed uri or check that your registry exposes a valid v2 Docker Registry API
at Octopus.Core.Packages.Docker.DockerRegistryResolver.TryDetectFromUri(DockerFeed feed, IDockerRegistryImageFeed& imageFeed) in ./source/Octopus.Core/Packages/Docker/DockerRegistryResolver.cs:line 166
at Octopus.Core.Packages.Docker.DockerRegistryResolver.ProbeFeedAddressForBestCompatibleEndpoint(DockerFeed feed) in ./source/Octopus.Core/Packages/Docker/DockerRegistryResolver.cs:line 75
at Octopus.Core.Packages.Docker.DockerRegistryResolver.Get(DockerFeed feed) in ./source/Octopus.Core/Packages/Docker/DockerRegistryResolver.cs:line 65
at Octopus.Core.Packages.PackageFeedFactory.CreatePackageFeed(IFeed feed) in ./source/Octopus.Core/Packages/PackageFeedFactory.cs:line 72

But when I try a GET URL:https://docker.xxxxxx.net/v2 from curl and browser I have correct exptected response:

Response body:
{}
Response header:

1. Connection: Keep-Alive
2. Content-Length:2
3. Content-Type:application/json; charset=utf-8
4. Date:Wed, 11 Jan 2023 18:24:42 GMT
5. Docker-Distribution-Api-Version:registry/2.0
6. Keep-Alive:timeout=5, max=98
7. Server:nginx/1.21.6
8. X-Content-Type-Options:nosniff

I tried also url GET https://docker.xxxxxx.net but it does not work.

Any suggestion ?

Pierpaolo

[Justin_Walsh]

Hi @pr

Thanks for reaching out, and sorry to hear that you’re having issues with setting up your docker registry as a feed in Octopus.

Digging through the code here, it looks like this should be working - basically what we’re doing is determining the API version for the URL, which in this case it does successfully, then it does a HTTP GET call to the specified endpoint to get the Docker-Distribution-Api-Version header value, and ensure that it’s registry/2.0

Judging by what you’ve posted up it looks like everything is correct though - are you able to share some information about what you’re using for your docker registry?

[pr]

Thanks @Justin_Walsh

sure
We run docker registry in Centos 8 Linux VM

Here is the docker-compose.yml file of our docker registry container:

version: "3.9"
services:
  registry:
    image: registry:2
    container_name: 'registry'
    restart: always
    environment:
      - REGISTRY_STORAGE_DELETE_ENABLED=True
    volumes:
      - $PWD/data:/var/lib/registry
    networks:
      - registry-net

  ui:
    image: joxit/docker-registry-ui:latest
    container_name: 'docker-registry-ui'
    restart: always
    ports:
      - 127.0.0.1:5000:80
    environment:
      - REGISTRY_TITLE=xxxxxx Docker Registry
      - NGINX_PROXY_PASS_URL=http://registry:5000
      - SINGLE_REGISTRY=true
      - DELETE_IMAGES=true
    depends_on:
      - registry
    networks:
      - registry-net

networks:
  registry-net:

Here is the the .conf file for httpd virtual host

<VirtualHost *:443>
ServerName docker.xxxxxxxx.net

RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}

ProxyPreserveHost On
ProxyPass / http://127.0.0.1:5000/
ProxyPassReverse / http://127.0.0.1:5000/


LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" vhost_combined
LogFormat "%v %h %l %u %t \"%r\" %>s %b" vhost_common
ErrorLog logs/docker.xxxxxxx.net.error_log
TransferLog logs/docker.xxxxxxx.net.access_log
LogLevel warn

SSLEngine on
SSLCertificateFile /etc/httpd/_.xxxxxxx.net.crt
SSLCertificateKeyFile /etc/httpd/_.xxxxxxx.net.key

<Location />
  Order deny,allow
  Allow from all

  AuthName "xxxxxxxx Docker Registry Authentication"
  AuthType basic
  AuthUserFile "/etc/httpd/conf/auth/registry.password"
  Require valid-user
</Location>

</VirtualHost>

And here are the docker’s packages which are installed on the VM

[root]# yum list installed | grep docker
containerd.io.x86_64                  1.6.14-3.1.el8                                         @docker-ce-stable
docker-ce.x86_64                      3:20.10.22-3.el8                                       @docker-ce-stable
docker-ce-cli.x86_64                  1:20.10.22-3.el8                                       @docker-ce-stable
docker-ce-rootless-extras.x86_64      20.10.22-3.el8                                         @docker-ce-stable
docker-compose-plugin.x86_64          2.14.1-3.el8                                           @docker-ce-stable
docker-scan-plugin.x86_64             0.23.0-3.el8                                           @docker-ce-stable

Do you need other details ?

Pierpaolo

[adam.hollow]

Hi @pr,

Thanks for all the details you’ve sent over.

It seems like a long shot and it might be a silly suggestion and I’m aware you tried a GET request to the endpoint but have you tried setting the feed url within octopus to simply https://docker.xxxxxx.net/, the same URL just without the v2?

Octopus should have its own internal logic to dictate which version of the API is being used by the feed endpoint. Please be aware that searching within a v2 docker feed is currently not supported (on docker’s side). There’s some discussion here: propose registry search functionality · Issue #206 · distribution/distribution · GitHub.

That being said, you should still be able to use a v2 feed within Octopus, providing you enter the image name exactly.

Please let me know the results of the above.

Kind Regards,

[pr]

Hello @adam.hollow

Indeed I already tried that and it does not work.

External Feed with:

Feed Type: Docker Container Registry
Name: xxx Docker Registry feedxxx-docker-registry-feed
URL:https://docker.xxxxxx.net
Registry Path: Please enter the path of the registry
Credentials: Credentials have been entered; username is yyyyyyy

I have the following error

Feed endpoint https://docker.xxxxxx.net does not appear to expose a valid Docker API. The URL was tested against the v1 (https://docker.xxxxxx.net/v1) and v2 (https://docker.xxxxxx.net/v2) endpoints and both failed. Please check the URI and the feed provider, and try again. Read https://oc.to/DockerRegistries for more details.

N.B.: on https://docker.xxxxxx.net is binded to docker registry UI

Pierpaolo

[adam.hollow]

Hey @pr,

Thanks for getting back to me! It’s a strange error that I haven’t seen before, at least not recently.

I wonder if you would be able to send over some of your server logs after triggering the error (the original error with the /v2 URL) a few times, please?

Hopefully there may be something in there that can help us get closer to the issue at hand.
If you’re able to, please upload them to our secure uploads platform: Support - Octopus Deploy.

Kind Regards,
Adam

[Justin_Walsh]

Thanks for the further information @pr

I attempted a reproduction of this, and was able to get it working without Apache in front of it, and appears to have worked as intended (Octopus was able to see the container image):

registry            | time="2023-01-12T16:30:58.04218543Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="<snip>:5000" http.request.id=6c30e639-3d7e-4e80-a3c4-adca6383457d http.request.method=GET http.request.remoteaddr="172.22.0.3:35176" http.request.uri="/v2/_catalog?n=100" http.request.useragent="OctopusDeploy-Server/2023.1.4363-hotfix.5944" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.4015ms http.response.status=200 http.response.written=29 

I didn’t have apache reverse proxying, so I wonder if that could be the issue? Maybe it’s not correctly forwarding all of the headers? This feels like a good place to start your investigation.

[pr]

Thanks @Justin_Walsh for your time in testing in your environment.

In the meanwhile we did a different test.
We disabled authentication in Apache virtual host and then tested again External feed in Octopus using https://docker.xxxxxx.net and it worked. Octopus server was able to browse the containers in the Docker registry.
Then we reactivated authentication in Apache, and the External feed continued to work.

So problem solved without changing our setup, but root cause was not found. I will keep you posted if we will have more infos.

[daniel.doyle]

I can confirm the same behaviour as OP with basic authentication on a private registry, and the Traefik Proxy as well. The docker registry won’t be picked up unless you first disable Basic Authentication, add the docker registry, then enable Basic Authentication again. After that, the secured private registry works with Octopus. It’s the inital add that fails until Basic Authentication is temporarily turned off.

[finnian.dempsey]

Hi @daniel.doyle,

Welcome to the Octopus Community, thanks for adding your details to this issue!

I’ll dig into this and see if I can reproduce this on my end, and raise it with the devs once I’ve confirmed what’s going on with Docker feed registration not using Basic Auth properly.

I’ll reach out if I have any questions about your configuration, feel free to reach out with any of your own!

Best Regards,

[finnian.dempsey]

Hi @daniel.doyle & @pr,

Just an update, I’ve been able to reproduce this and have figured out what’s going on.

As Justin pointed out, when Octopus is discovering the registry it will look for a specific header to determine that it’s using v2 of the registry, otherwise it will fall back to the v1 method where it checks the /_ping endpoint for a header. This error appears when Octopus isn’t able to use either method to determine the version of the registry.

From our Docker Registry documentation about API version discovery:

According to the Docker API documentation, the version 1 API should have a /_ping endpoint which will respond with a X-Docker-Registry-Version HTTP header in the response. Similarly, the version 2 API expects a Docker-Distribution-API-Version HTTP header with a value of registry/2.0. Both of these endpoints are expected to be located at an absolute path of either /v1 or /v2 from the host.

To resolve the issue for without having to disable Basic Auth, add a header like the following to your httpd Virtual Host config (for adding traefik custom headers check out their docs):

Header always set "Docker-Distribution-Api-Version" "registry/2.0"

The Docker docs offer a recipe for this exact scenario which includes the headers they suggest to use, I’d definitely recommend checking it out.

Let me know if that doesn’t help or if you have any questions at all!

Happy Deployments!