Sometimes when triggering a deploy from TeamCity using the Octopus plugin it results in an error before starting the deploy.
@@@
Cannot start build runner: Error from Octo.exe: Cannot run program “C:\BuildAgentsNew\BuildNew1\BuildAgent\temp\buildTmp\octo-temp\2.0\octo.exe” (in directory “C:\BuildAgentsNew\BuildNew1\BuildAgent\work\423adb0a3c62b004”): CreateProcess error=2, The system cannot find the file specified
@@@
We have tried to avoid this by making the builds / deploys run on only 1 TeamCity agent hoping that the problem occurred because one of the other agents were locking the file. But this didn’t seem to fix it, so now I turn to you.
Do you have a virus scanner or something else running on TeamCity? Is there a lot of resource usage going on?
At the start of the build we attempt to extract the Octo.exe assembly to that package path, then we call it. If another process prevents the file being extracted or deletes it, I’m not sure what we can do.
Would it be possible to post a copy of the entire build log from TeamCity?
There is no virus scanner on the server, and it’s not using a lot of resources, about 10-40% CPU and about 45-50% memory out of 24gb…
This is the full log as seen from the web:
@@@
[09:07:34]Skip checking for changes - changes are already collected
[09:07:35]Publishing internal artifacts (1s)
[09:07:36]Sending using ArtifactsCachePublisher
[09:07:36]Sending using WebPublisher
[09:07:35]Clearing temporary directory: C:\BuildAgentsNew\BuildNew3\BuildAgent\temp\buildTmp
[09:07:35]Failed to delete empty directory: C:\BuildAgentsNew\BuildNew3\BuildAgent\temp\buildTmp\octo-temp\2.0
[09:07:35]Checkout directory: C:\BuildAgentsNew\BuildNew3\BuildAgent\work\87b08b0fb315d62
[09:07:36]Step 1/1: OctopusDeploy: Create release
[09:07:36][Step 1/1] Octopus Deploy
[09:07:36][Octopus Deploy] Running command: octo.exe create-release --server http://buildserver:8081 --apikey SECRET --project MyProject Web --enableservicemessages --version 1.1.1.43 --deployto Test
[09:07:36][Octopus Deploy] Creating Octopus Deploy release
[09:07:36][Octopus Deploy] Error from Octo.exe: Cannot run program “C:\BuildAgentsNew\BuildNew3\BuildAgent\temp\buildTmp\octo-temp\2.0\octo.exe” (in directory “C:\BuildAgentsNew\BuildNew3\BuildAgent\work\87b08b0fb315d62”): CreateProcess error=2, The system cannot find the file specified
[09:07:36][Octopus Deploy] Step OctopusDeploy: Create release failed with unexpected error
[09:07:36]Publishing internal artifacts
[09:07:36]Sending using ArtifactsCachePublisher
[09:07:36]Sending using WebPublisher
[09:07:36]Build finished
@@@
[09:07:35]Clearing temporary directory: C:\BuildAgentsNew\BuildNew3\BuildAgent\temp\buildTmp
[09:07:35]Failed to delete empty directory: C:\BuildAgentsNew\BuildNew3\BuildAgent\temp\buildTmp\octo-temp\2.0
On your TeamCity build agent server, can you bring up task manager and see if any instances of Octo.exe are running? It looks like perhaps something is locking this directory, preventing the file from being extracted. Could you try deleting the directory manually?
Hello,
I tried deleting it manually, still the same problem.
There is no Octo.exe running continuously, though when another agent is doing some packaging or deploying it runs ofcourse.
Something I might have forgotten to tell you is that for this specific project we have in teamcity, 1 octopack package step and 4 octopus release steps.
The release steps are set to be run on the same agent as the agent that does the packaging (This goes for all our projects).
And it looks like most of the time it successfully manage to do the octopus deploy step for the first one in the queue and fails the other 3.
Thanks for the update Sebastian. I’ll make a change to the TeamCity plugin today and let you know when a new version is available that may be able to fix this problem.
Thank you for the quick update. Unfortunately it didn’t seem to work.
However I got a new error,
@@@
[08:35:43]Skip checking for changes - changes are already collected
[08:35:44]Publishing internal artifacts
[08:35:45]Sending using ArtifactsCachePublisher
[08:35:45]Sending using WebPublisher
[08:35:44]Clearing temporary directory: C:\BuildAgentsNew\BuildNew1\BuildAgent\temp\buildTmp
[08:35:44]Failed to delete empty directory: C:\BuildAgentsNew\BuildNew1\BuildAgent\temp\buildTmp\octo-temp\2.0
[08:35:44]Checkout directory: C:\BuildAgentsNew\BuildNew1\BuildAgent\work\be43aa03d5c03ad7
[08:35:45]Step 1/1: OctopusDeploy: Create release
[08:35:45][Step 1/1] Unable to create temporary file in C:\BuildAgentsNew\BuildNew1\BuildAgent\temp\buildTmp for Octopus: C:\BuildAgentsNew\BuildNew1\BuildAgent\temp\buildTmp\octo-temp\2.0\Octo.exe (Access is denied)
[08:35:46][Step 1/1] Step OctopusDeploy: Create release failed with unexpected error
[08:35:46]Publishing internal artifacts
[08:35:46]Sending using ArtifactsCachePublisher
[08:35:46]Sending using WebPublisher
[08:35:46]Build finished
@@@
“Access is denied” - But it’s still working occasionally like before.
Furthermore information; The account running the agents has been granted full access to the C:\BuildAgentsNew containing all build agents, so we highly doubt that the error actually is an access problem. We even tried giving “Everyone” full access to the folder.
No, they are all set to be run on the same agent so they are queued one after one. Both the packaging and all the deploy steps. Though we do have 2 more agents on that server, but as far as I understand it the agents and the Octopus for the agents are all separate, right?
In your TeamCity VCS settings, have you configured TeamCity to purge the working directory (clean) each time the build runs? Could you try seeing if that has any effect?
I’m attaching a print screen of the TeamCity setup.
To make this occur I start the package step, the other 4 steps are then triggered auto when the Package step is completed and successful. The steps named “Release” triggers a Deploy in Octopus.
Everything running on the same agent (As mentioned before).
The Log file is taken from the “wrapper.log” as this seems to be the only file it writes any logs to when this occurs. The log file is from when I clicked the “Run” on the package step and forward.
Though it doesn’t really seem to be making any more sense than the other Octopus logs to me.
In the screenshot the “Release to Client1 Test” build was successful, also attaching the web log from teamcity for that one.
Also attaching the octopus log of the successful build (The other TeamCity triggered builds does not reach Octopus so I don’t have any logs for those).
Disclosure: Due to that some of the log files contains sensitive information I have replaced project names, server addresses etc with dummy information.
I have checked the logs but I cannot find anything out of the ordinary.
Regarding your 1-4 statements,
1: I don’t know all programs installed on the server. I will take a closer look on this when I can get a hold of the guy who set up the server. But wouldn’t this probably cause a failed build every time as well?
2: In teamcity they are not run parallel but TeamCity doesn’t wait for the Octopus deploy to be fully completed before starting the next deploy in the queue. It runs for a couple of seconds (5-10) then triggers the next.
Could this be a problem?
I think 3 and 4 is highly unlikely as well as you mentioned.
On the assumption that this is an intermittent/temporary issue, I’m going to make a change to the TeamCity plugin that will just retry/sleep a few times when extracting the file. I’ll let you know when a new build is available (expect it next week).
We just noticed that there is a 2.4.1 pre-release version up for download.
Did you include the retry/sleep thing in the teamcity plugin with the same version?
Hi, sorry for the very late reply… Forgot about this thread.
Unfortunately this didn’t help… We did a fresh install on a new server and we’re still seeing the same problem…
To resolve this we have currently setup a retry-build trigger upon a failed deploy in Teamcity that will retry 10 times. This seems to be the only work-around that can manage this at the moment.
Looks like we need to find a new angle to attack this one on; just curious -
2: In teamcity they are not run parallel but TeamCity doesn’t wait for the Octopus deploy to be fully completed before starting the next deploy in the queue. It runs for a couple of seconds (5-10) then triggers the next.
Is the next build triggering on build completion? This sounds a bit unexpected to me - wondering if we have some kind of wait-for-exit missing…
Long time since this was started…
Anyway, I just noticed that it’s still open.
Just wanted to let you know that this got resolved at some point somehow, can’t really tell how or when as I don’t know. But you may close this thread.
