Project Variable not populated in Time

[kira_resari]

Hello everyone,

I have a slightly convoluted pipeline that starts with a number of TeamCity builds using Octopus to deploy Docker containers.

The premise is that I want to use Team City templates to easily deploy “clones” of basically the same Docker containers with slightly different setting passed down via TeamCity parameters.

I have set up the pipeline and it works to the point where I am trying to get Octopus to deploy the right package.

In TeamCity, I distinguish the different versions using the %env.TEAMCITY_BUILDCONF_NAME% variable. I pass this Variable along to Octopus into a custom variable named _#{TEAMCITY_BUILDCONF_NAME} using the -v "TEAMCITY_BUILDCONF_NAME:%env.TEAMCITY_BUILDCONF_NAME%" parameter. This works just fine so long as I do not try to include _#{TEAMCITY_BUILDCONF_NAME} in the package source.

However, in my latest attempt, I was trying to do just that, and include _#{TEAMCITY_BUILDCONF_NAME} in the package source as such: Docker-Program_#{TEAMCITY_BUILDCONF_NAME}

I have already configured TeamCity to push the package to the right repository (e.g. Docker-Program_Docker-Program_Test for the Project Docker-Program_Test ) and confirmed that the package is in the Octopus library.

The problem happens when Octopus tries to resolve the name of #{TEAMCITY_BUILDCONF_NAME} when creating the release: Instead of correctly taking the name I passed on via -v "TEAMCITY_BUILDCONF_NAME:%env.TEAMCITY_BUILDCONF_NAME%", it instead uses the default value (Undefined_Build_Configuration) and then fails with the error Could not find any packages with ID 'Docker-Program-Database_Undefined_Build_Configuration ' in the feed 'octopus://'.

This confuses me since in previous configurations, I was able to confirm that the parameter gets succesfully passed on (I deployed the files into custom paths based on #{TEAMCITY_BUILDCONF_NAME}).

My current assumption as for why this behaviour occurs is that Octopus first needs to create the release before it can prompt for variables, then resolves the yet undefined Variable for the Package Source and does subsequently not update it when it is populated.

Is there any way I can get around this?

Any insight and help will be much appreciated.
Thanks in advance,
Cheers,
Kira
=^,^=

[Matthew_Casperson]

Hi Kira, thanks for reaching out.

I’ll need to get some additional information to try and resolve the issue:

• A screenshot of the variables page in the Octopus project that you are trying to deploy.
• A screenshot of the TC build steps that trigger deployments in Octopus.
• The team city deployment logs.
• The Octopus deployment logs with variable debugging enabled. You can find more details on debugging variables with the documentation at https://octopus.com/docs/support/debug-problems-with-octopus-variables#DebugproblemswithOctopusvariables-Writethevariablestothedeploymentlog. Essentially you add the variable OctopusPrintEvaluatedVariables set to True in your Octopus project and all the variables will be printed to the logs during deployment.

Regards

Matt C

[kira_resari]

Hi Matthew,

thank you for getting back to me on this. I have gathered the data you requested, and based on what I see my initial assumption is correct: For some reason (probably technical) the deployment seems to try to acquire the packages before prompting for the variable names which are required to build the correct package names. Is there something I can do about this, or some sort of workaround?

Anyway, here’s the requested files:

TeamCity Configuration Screenshot for Step OctopusDeploy: Push packages:

TeamCity Configuration Screenshot for Step OctopusDeploy: Create release:

Additional command line arguments: -v "TEAMCITY_BUILDCONF_NAME:%env.TEAMCITY_BUILDCONF_NAME%"

Octopus Deploy Project Variables Page:

I am also including these following ones since I feel they might be illustrative of what happens:

Octopus Deploy Create Release Page (where the packages fail to be acquired correctly because they miss the Variable which I can only set in the next step):

Octopus Deploy Create Deployment Page (where I can set the variable that was missing on the previous page):

Team City build Log (relevant steps only):

[08:37:59]
Step 8/9: Push Image to OctopusDeploy (OctopusDeploy: Push packages) (54s)
[08:37:59]
[Step 8/9] Creating archive Docker-Project_Docker-Project_Test.0.0.2.zip (40s)
[08:37:59]
[Creating archive Docker-Project_Docker-Project_Test.0.0.2.zip] Creating C:\TeamCity\buildAgent\temp\buildTmp\ZipPreprocessor3322282743156780590\Docker-Project_Docker-Project_Test.0.0.2.zip
[08:38:40]
[Creating archive Docker-Project_Docker-Project_Test.0.0.2.zip] Archive was created, file size 92.13 MB (96605475 bytes)
[08:38:40]
[Step 8/9] Creating archive Docker-Project-Database_Docker-Project_Test.0.0.2.zip
[08:38:40]
[Creating archive Docker-Project-Database_Docker-Project_Test.0.0.2.zip] Creating C:\TeamCity\buildAgent\temp\buildTmp\ZipPreprocessor5055288874630007184\Docker-Project-Database_Docker-Project_Test.0.0.2.zip
[08:38:40]
[Creating archive Docker-Project-Database_Docker-Project_Test.0.0.2.zip] Archive was created, file size 3.82 KB (3917 bytes)
[08:38:48]
[Step 8/9] Octopus Deploy (5s)
[08:38:48]
[Octopus Deploy] Running command:   octo.exe push --server http://localhost:8888/octopus --apikey SECRET --package C:\TeamCity\buildAgent\temp\buildTmp\ZipPreprocessor3322282743156780590\Docker-Project_Docker-Project_Test.0.0.2.zip --package C:\TeamCity\buildAgent\temp\buildTmp\ZipPreprocessor5055288874630007184\Docker-Project-Database_Docker-Project_Test.0.0.2.zip
[08:38:48]
[Octopus Deploy] Pushing packages to Octopus server
[08:38:48]
[Octopus Deploy] Octopus Deploy Command Line Tool, version 4.21.0
[08:38:48]
[Octopus Deploy] 
[08:38:49]
[Octopus Deploy] Handshaking with Octopus server: http://localhost:8888/octopus
[08:38:49]
[Octopus Deploy] Handshake successful. Octopus version: 2018.9.16; API version: 3.0.0
[08:38:49]
[Octopus Deploy] Authenticated as: OctoAdmin <kira.resari@company.com> 
[08:38:49]
[Octopus Deploy] Pushing package: C:\TeamCity\buildAgent\temp\buildTmp\ZipPreprocessor3322282743156780590\Docker-Project_Docker-Project_Test.0.0.2.zip...
[08:38:54]
[Octopus Deploy] Pushing package: C:\TeamCity\buildAgent\temp\buildTmp\ZipPreprocessor5055288874630007184\Docker-Project-Database_Docker-Project_Test.0.0.2.zip...
[08:38:54]
[Octopus Deploy] Push successful
[08:38:54]
[Octopus Deploy] Octo.exe exit code: 0

[08:38:54]
Step 9/9: Create & Deploy Release via Octopus (OctopusDeploy: Create release) (3s)
[08:38:56]
[Step 9/9] Octopus Deploy
[08:38:56]
[Octopus Deploy] Running command:   octo.exe create-release --server http://localhost:8888/octopus --apikey SECRET --project Docker-Project --enableservicemessages --deployto Docker-Project Environment -v TEAMCITY_BUILDCONF_NAME:Docker-Project_Test
[08:38:56]
[Octopus Deploy] Creating Octopus Deploy release
[08:38:56]
[Octopus Deploy] Octopus Deploy Command Line Tool, version 4.21.0
[08:38:56]
[Octopus Deploy] 
[08:38:57]
[Octopus Deploy] Build environment is Handshaking with Octopus server: http://localhost:8888/octopus
[08:38:57]
[Octopus Deploy] Handshake successful. Octopus version: 2018.9.16; API version: 3.0.0
[08:38:57]
[Octopus Deploy] Authenticated as: OctoAdmin <kira.resari@company.com> 
[08:38:57]
[Octopus Deploy] This Octopus Server supports channels
[08:38:57]
[Octopus Deploy] Finding project: Docker-Project
[08:38:57]
[Octopus Deploy] Automatically selecting the best channel for this release...
[08:38:57]
[Octopus Deploy] Building a release plan for Channel 'Default'...
[08:38:57]
[Octopus Deploy] Finding deployment process...
[08:38:57]
[Octopus Deploy] Finding release template...
[08:38:57]
[Octopus Deploy] The package version for some steps was not specified. Going to try and resolve those automatically...
[08:38:57]
[Octopus Deploy] Finding latest package for step: Deploy Docker-Project Package
[08:38:57]
[Octopus Deploy] Could not find any packages with ID 'Docker-Project_Undefined_Build_Configuration' in the feed 'octopus://'
[08:38:57]
[Octopus Deploy] Finding latest package for step: Deploy Docker-Project-Database Package
[08:38:57]
[Octopus Deploy] Could not find any packages with ID 'Docker-Project-Database_Undefined_Build_Configuration ' in the feed 'octopus://'
[08:38:57]
[Octopus Deploy] There are no viable release plans in any channels using the provided arguments. The following release plans were considered:
[08:38:57]
[Octopus Deploy] Channel: 'Default' (this is the default channel)
[08:38:57]
[Octopus Deploy]   #   Name                                   Version   Source           Version rules      
[08:38:57]
[Octopus Deploy]   --- -------------------------------------- --------- ---------------- -------------------
[08:38:57]
[Octopus Deploy]   1   Deploy Docker-Project Package            ERROR     Cannot resolve   Allow any version  
[08:38:57]
[Octopus Deploy]   2   Deploy Docker-Project-Database Package   ERROR     Cannot resolve   Allow any version  
[08:38:57]
[Octopus Deploy] 
[08:38:57]
[Octopus Deploy] Exit code: -1
[08:38:57]
[Octopus Deploy] Octo.exe exit code: -1
[08:38:57]
[Step 9/9] Unable to create or deploy release. Please check the build log for details on the error.
[08:38:57]
[Step 9/9] Unable to create or deploy release. Please check the build log for details on the error.
[08:38:57]
[Step 9/9] Step Create & Deploy Release via Octopus (OctopusDeploy: Create release) failed

Octopus Deployment Log:

Task ID:        ServerTasks-24888
Task status:    Failed
Task queued:    Wednesday, December 19, 2018 12:04:42 AM
Task started:   Wednesday, December 19, 2018 12:04:42 AM
Task completed: Wednesday, December 19, 2018 12:04:42 AM
Task duration:  less than a second
Server version: 2018.9.17+Branch.master.Sha.9e332cbb3a61d59f7f14c94196e6c4ed481b10a3
Server node:    EC2AMAZ-52G5JJ8

                    | == Failed: Deploy Docker-Project release 0.0.36 to Docker-Project Environment ==
00:04:42   Verbose  |   Guided failure is not enabled for this task
00:04:42   Fatal    |   The deployment failed because one or more steps failed. Please see the deployment log for details.
                    | 
                    |   == Failed: Acquire packages ==
00:04:42   Info     |     Acquiring packages
00:04:42   Info     |     Making a list of packages to acquire
00:04:42   Verbose  |     No packages are required on the Octopus Server
00:04:42   Verbose  |     Delta compression is enabled for package transfers from the Octopus Server to deployment targets
00:04:42   Verbose  |     Upload Docker-Project-Database_Docker-Project_Test .0.0.79 to Moby Dock
                    |     Upload Docker-Project_Docker-Project_Test.0.0.79 to Moby Dock
00:04:42   Verbose  |     Checking package cache for package Docker-Project-Database_Docker-Project_Test  v0.0.79
00:04:42   Info     |     Downloading Docker-Project-Database_Docker-Project_Test  v0.0.79 to package cache...
00:04:42   Verbose  |     Downloading Docker-Project-Database_Docker-Project_Test  v0.0.79 from Octopus Server (built-in) using cache policy UseCache...
00:04:42   Verbose  |     Downloading package (attempt 1 of 5)
00:04:42   Fatal    |     The step failed: Activity failed with error 'The package Docker-Project-Database_Docker-Project_Test  v0.0.79 could not be downloaded to the package cache from Octopus Server (built-in) after making 5 attempts over a total of 0s. Make sure the package is pushed to the feed and try the deployment again. For a detailed troubleshooting guide go to http://g.octopushq.com/TroubleshootMissingPackages'.
00:04:42   Verbose  |     Acquire Packages completed
                    |   
                    |     Failed: Moby Dock
00:04:42   Error    |       The package Docker-Project-Database_Docker-Project_Test  v0.0.79 could not be downloaded to the package cache from Octopus Server (built-in) after making 5 attempts over a total of 0s. Make sure the package is pushed to the feed and try the deployment again. For a detailed troubleshooting guide go to http://g.octopushq.com/TroubleshootMissingPackages
                    |     
                    |       Failed: Upload package Docker-Project-Database_Docker-Project_Test  v0.0.79
                    |       
                    |   Canceled: Step 1: Deploy Docker-Project Package
00:04:42   Verbose  |     Step "Deploy Docker-Project Package" runs only when all previous steps succeeded; skipping
                    |   
                    |   Canceled: Step 2: Deploy Docker-Project-Database Package
00:04:42   Verbose  |     Step "Deploy Docker-Project-Database Package" runs only when all previous steps succeeded; skipping
                    |   
                    |   Canceled: Step 3: Create Prerequisite Folders
00:04:42   Verbose  |     Step "Create Prerequisite Folders" runs only when all previous steps succeeded; skipping
                    |   
                    |   Canceled: Step 4: Run Docker-Project Images
00:04:42   Verbose  |     Step "Run Docker-Project Images" runs only when all previous steps succeeded; skipping
                    |   

Thank you very much for your effort,
Cheers,
Kira
=^,^=

[Matthew_Casperson]

Hi @kira_resari, thanks for that additional information.

You are correct in that prompted variables can’t be used to define a package ID. When a prompted variable is used for a package ID, it’s default value is used rather than the supplied value, whether that value is supplied via the web UI or with the -v option.

To have a different package selected at deployment time you will need to have a variable scoped to something like a channel (https://octopus.com/docs/deployment-process/variables/scoping-variables). So in this case you would have two channels to represent build configurations like Test and Prod. Then a variable would be scoped to those channels, and the channel would be specified in the TC create-release steps.

Alternatively it may make sense to scope packages by environment if the TC build configurations conceptually represent environments.

There may be a better solution though than scoping package ids to variables. What is the outcome that this build pipeline is trying to achieve? Are you building different Docker images per environment?

Regards
Matt C

[kira_resari]

Hi Matthew,

I have now “solved” this by instead including the Team City Project name into the Package Version, like this: [0.0.9-Docker-Buyer-Test](https://collaboration.netfira.de/octopus/app#/library/builtinrepository/packages-Docker-Buyer.0.0.9-Docker-Project-Test) and then using the --package to select that “Version” like this: --package=Docker-Buyer:%build.number% --package=Docker-Buyer-Database:%build.number%.

That way I can achieve the desired result, which is to have a configurable template in TeamCity, where you can create a fully functional deployment from scratch in a matter of seconds just by adjusting a few parameters.

The basic idea is that we want to deploy different versions of our application into docker containers on our testing server and then access them over different ports and compare them against one another. With the “Version Hack”-Pipeline I’ve just set up, exactly this becomes possible. No need to set up a whole new build configuration from scratch or make any changes on Octopus, simply create a new build configuration from the template, fill in the fields for the source branches and target IP addresses, run the build, and it automatically builds and deploys the product to a dockerized instance on the target machine using the Octopus Project.

Anyway, from what you told me it seems that Octopus currently doesn’t support any better way than my “Version Hack” for achieving this (from what I know about channels, it sounds like you would need to set up an additional one for each new build configuration). However, I may be wrong, so please correct me if there is a better way.

To sum it up: The premise is that I can create a completely functional new “Pipeline Path” just by creating a new TeamCity build from a template and adjusting the Parameters. I should not have to change anything in Octopus Deploy for every new build configuration - all relevant information should be passed on in the form of variables or something.

Cheers,
Kira
=^,^=