Hi,
We have been trialing TeamCity using Octopus Deploy to manage the deployment side of things. All has been working well until recently where all projects are failing to deploy when using either OctopusDeploy: Create Release or OctopusDeploy: Deploy ReleaseRelease. In the log we always get the following error: -
[10:31:19][Octopus Deploy] Running command: octo.exe deploy-release --server http://cibuild --apikey SECRET --project Transport --enableservicemessages --version 1.054 --deployto Deployment.Symphony
[10:31:19][Octopus Deploy] Deploying Octopus Deploy release
[10:31:19][Octopus Deploy] Octopus Deploy Command Line Tool, version 3.3.8+Branch.master.Sha.f8a34fc6097785d7d382ddfaa9a7f009f29bc5fb
[10:31:19][Octopus Deploy]
[10:31:19][Octopus Deploy] Build environment is NoneOrUnknown
[10:31:19][Octopus Deploy] Handshaking with Octopus server: http://cibuild.symphony.local
[10:31:19][Octopus Deploy] Handshake successful. Octopus version: 3.3.6; API version: 3.0.0
[10:31:19][Octopus Deploy] Authenticated as: Martyn Cork Martyn.cork@symphony-group.co.uk
[10:31:19][Octopus Deploy] Finding project: Transport
[10:31:19][Octopus Deploy] Finding release: 1.054
[10:31:19][Octopus Deploy] Error from Octopus server (HTTP 404): The resource ‘release 1.054 for project Transport’ was not found.
[10:31:19][Octopus Deploy] Exit code: -7
[10:31:19][Octopus Deploy] Octo.exe exit code: -7
Our trial of Octopus Deploy has recently expired (didn’t get an opportunity to evaluate as required) and we are now on the community edition, I was wondering if the API used by the TeamCity plugin stops working on the community edition? if not, could you suggest some settings which i should check to correct the issue.
I forgot to mention, it appears as though TeamCity does create the release in octopus as it appears in the audit log, additionally if a manual release (clicking the release button on the project) is done through Octopus then everything works well, its just the deployment through TeamCity that doesn’t seem to work
Hi Martyn,
Thanks for getting in touch! The community edition does everything the full version does within the Project/Machine and User number limits. However if you run into these during your evaluation, just reach out to support we are usually more than happy to extend your trial license. I will get you to confirm you don’t have more than five projects set up yet? That could cause issues with the community edition.
Now, onto your problem.
First, can you confirm for me that when you run a Create Release from TeamCity it actually does create a release and you can see it in the Releases page for that Project? Even if the deploy part fails, you should be able to see the created release in Octopus (not just in the logs).
Second, can you confirm that in the Octopus UI you can deploy the exact release that was created by TeamCity?
Regards,
Mark
Hi,
Sorry I haven’t replied - my partner and I had a child which came a little sooner than expected so I’ve been off for the last few weeks. I’ve managed to narrow the issue down and it leads me to another issue which I’m hoping you can help resolve…
To perform a NuGet package restore in TeamCity I set the TeamCity build agent to execute under a domain account so it could access the internet and this bit works fine, however it is this setting of the domain account against the build agent which stops the Octopus Deploy step from working, it just seems to hang on the “Creating release…” step, then times out/fails. I would guess that there is a permissions issue going on?
Regards,
Martyn Cork
Hi Martyn,
Firstly congratulations on the addition to your family. As for your problem, this definitely looks like a permission issue as you say. When using the TeamCity plugin we use the API key which you provide to do all Octopus authorization. From your original logs that’s a key you’ve created on your account. Provided you can create a release through the Octopus UI we shouldn’t be doing any further account checking. Also we will generally return immediately with a 403 unauthorized error rather than hanging or timing out.
One thing I’ve seen is that users have their network configured such that particular machines, or particular users are blocked at the network level from connecting to the Octopus server, this may explain the hang as the connection is taking a while to time out. Are you able to ping the Octopus server from the TeamCity machine when logged in as that domain account? Or can you bring up the Octopus web app from that machine as that user?
Failing that can you send me through a bit more of the TeamCity log to show the command it is calling that hangs? You can also take a look at the Octopus Server logs (see here for their location: http://docs.octopusdeploy.com/display/OD/Log+files) to see if anything is happening at the Octopus end.
Mark
Hi,
Please see attached build log for the failing build (this issue affects all of our build configurations though), I’ve also attached the octopus server log as well (set in Trace mode).
TeamCity and octopus are hosted on the same server so there shouldn’t be any communication issues, the domain account I’m using during this testing phase is also my own account - I’m running the TeamCity Build Agent service under my account, which has domain admin permissions. The reason I’m doing this is that the build agent needs a domain account to access the internet for NuGet for the package restore.
It’s when running the build agent under my account that we get the issue, if I revert the build agent service to run under Local System then the Octopus release creation works fine (I just can’t restore NuGet packages)
Regards,
Martyn Cork
Transport_Development_Build_1.0.151_(1).log (364 KB)
OctopusServer.txt (120 KB)
Hi Martyn,
So the logs show that you definitely have connectivity, you can see the authentication step and the lines like [Octopus Deploy] Finding project: Transport show a successful GET call against the server.
The call that is failing is the actual POST that creates the release, but if Octopus was rejecting it due to permissions we’d see the 403 in the Teamcity logs, or if it was throwing an exception we’d see the 500 and we’d also see matching things in the Octopus logs. Instead it looks like after 2 minutes something is killing off the http connection. I can’t think of anything at our end that would be doing that although I will keep investigating.
Some things to try:
-
Run the octo.exe create-release --server http://cibuild.symphony.local --apikey SECRET --project Transport --enableservicemessages --version 1.01.0.151 --deployto Development.symphony --packageversion 1.01.0.151 command directly on the command line and see if it still gets killed after two minutes. Also try it with --server http://localhost instead of the full address. You can get octo.exe from here if you need it.
-
Try with anti-virus disabled temporarily if you are running it on your server.
Mark