Connecting an external Docker Registry feed to Artifactory

[eric.shamay]

My Artifactory server is set up to work as a docker registry exposing an endpoint so that I can pull docker images like this:

docker pull my.server.com:6555/my-repo/subpath:0.0.1-image-tag

I tried in multiple ways to set up an external feed in Octopus to pull images. So far I can only get the feed to connect and list repositories, but when it comes time to create a release, the package list doesn’t work. For example:

URL: https://my.server.com/artifactory/api/docker/my-docker-repo
Registry Path: my.server.com:6555

I can save and test the external feed and I see that the repos all appear as expected when searching.

When I then create a new process step and add the external feed as a package, I configure it like this:

package feed: name of my feed
package ID: my-repo/subpath
name: my-feed

When creating a new release, I get this error:
Request tohttps://my.server.com/v2/my-repo/subpath/tags/list?last=0.0.1-image-tag&n=1000>failed with NotFound:Not Found.

What am I doing wrong? Is it in the external feed configuration, or is it with how I’m configuring my package in the process?

[Kenneth_Bates]

Hi Eric,

Thanks for getting in touch! It looks like it’s probably due to the package ID you’ve configured in your package step, where specifying a subfolder (my-repo/subpath) there isn’t supported. You would need to specify only a package ID in that field, and the feed would need to point specifically to that subfolder. This means if you have multiple folders storing images/packages that you need to deploy, you would need to define a separate feed per directory.

Let me know if that helps get you going, and don’t hesitate to reach out if you have any other questions or concerns moving forward.

Best regards,

Kenny

[eric.shamay]

I’m trying to migrate from using an ECR external feed to using a docker registry external feed, and Artifactory is acting as my docker registry.

Docker images are published with names like my-repo/subpath:my-version-tag in both ECR and in Artifactory.

When connecting the ECR feeds everything works great. Doing the identical work with the docker registry feed backed by artifactory does not work.

Why can I set a path/subpath in my package ID with a single ECR external feed, but not with the Docker registry external feed?

Also, I tried the suggestion and I still don’t have any luck. I set the docker registry feed url to include the subpath but when I try to ‘save and test’ it gives me the same error:

Does the external feed for docker registry not support the same as a typical docker pull?

[eric.shamay]

Here’s some more info.

I can curl this endpoint and I get a full listing of all the repos in my docker registry:
curl -u user:pass https://my.artifactoryserver.com/artifactory/api/docker/my-artifactory-repo/v2/_catalog

If I set that url as my Feed url then I can ‘save and test’ and then see a full listing of all the “packages”. One of them is foo/bar/baz. If I start typing into the search foo/bar I see foo/bar/baz as one of the listed packages just as expected!

Now, going over to the process step where I set the Package id. I start typing foo/ an I immediately see the list populated with all the options of docker image repos just like I expect! I even see foo/bar/baz, so I click on it and set my package id to foo/bar/baz. Great, my feed looks correct and populates things I know exist in artifactory, my process step package definition populates the package names as I type and I select the one I need to use. Everything looks perfect and ready to go!

Now I go to ‘create release’ and I see this error:

Request to https://my.artifactoryserver.com/v2/foo/bar/baz/tags/list?last=0.0.1.my-tag&n=1000> failed with NotFound:Not Found.

Notice how the url in the error doesn’t match the url I set in my Feed url? Why is that? Is octopus dropping part of my feed url when doing a lookup to create a release?

With regards to the suggestion to create a single feed per package, can you be much more detailed about what I should set (given the urls and values above) for my feed url, registry path, and package id?

I’d much prefer if the docker registry feed worked just like the ECR feed where I can use package ids with forward-slash path-like names instead of creating a new feed per package.

[eric.shamay]

Also, here’s the info of my octopus setup:

self-hosted install, 2019.12.6 LTS

[eric.shamay]

To try and work around this I’ve set up an nginx server (my.proxy-server.com) to act as a proxy that rewrites the urls. I’ve now got the following working:

The feed url: http://my.proxy-server.com/artifactory/api/docker/my-artifactory-docker-repo/v2/
When I hit ‘Save and Test’ I enter my package ID (my/package/path) and I see the package populating as expected. I tried other package IDs and they all work just like I want them to.

I can run this curl command from the octopus server host:
curl -v -u username:password http://my.proxy-server.com/v2/my/package/path/tags/list?n=20
This returns a list of 20 image tags as a json reponse.

However, the problem starts when I try to use the feed in a project. When I add a package reference and set the package ID to my/package/path and then click to create a release I get this error:

Request to http://my.proxy-server.com/v2/my/package/path/tags/list?last=0.0.1-some-image-tag&n=1000> failed with NotFound:Not Found.

(side note: why is there a > appended to the url in the error?)

However, if I take the exact url from the error and run curl I get a proper json response!

Why is Octopus failing with the feed when a connection from the host works both in testing the feed url, and when running curl on the same endpoint?

[eric.shamay]

For future viewers, I’ve resolved this by running an nginx proxy server between Octopus and Artifactory. Since Artifactory doesn’t abide by the docker api spec, I’ve created some translation rules to convert from the url and parameter/values that octopus is sending into the ones expected by Artifactory:

    location /artifactory/api/docker/my-docker-repo/v2/ {
        if ($args ~* (.*)n=([^>]*)>(.*)){
          set $args $1n=$2$3;
        }
        if ($args ~* (.*)n=([0-9]*)%3E(.*)){
          set $args $1n=$2$3;
        }
        proxy_pass         https://my.artifactory.server.com/artifactory/api/docker/my-docker-repo/v2/;
    }

    location ^~ /v2/ {
        if ($args ~* (.*)n=([^>]*)>(.*)){
          set $args $1n=$2$3;
        }
        if ($args ~* (.*)n=([0-9]*)%3E(.*)){
          set $args $1n=$2$3;
        }
        proxy_pass         https://my.artifactory.server.com/artifactory/api/docker/my-docker-repo/v2/;
    }

Using this proxy translation I can now connect the feed, and creating releases no longer returns the error above in the thread. The trick was the missing value rewrite to get rid of the > and %3E that Octopus was appending onto the requests.

[Kenneth_Bates]

Hi Eric,

Thanks for keeping in touch and letting us know the resolution to this! My apologies about not getting back to you sooner. I imagine your solution will be very helpful in case others hit this same issue in the future.

Don’t hesitate to reach out if you have any further questions or concerns moving forward!

Best regards,

Kenny