# TeamCity - Cannot start build runner

Hello!

Sometimes when triggering a deploy from TeamCity using the Octopus plugin it results in an error before starting the deploy.

@@@
Cannot start build runner: Error from Octo.exe: Cannot run program “C:\BuildAgentsNew\BuildNew1\BuildAgent\temp\buildTmp\octo-temp\2.0\octo.exe” (in directory “C:\BuildAgentsNew\BuildNew1\BuildAgent\work\423adb0a3c62b004”): CreateProcess error=2, The system cannot find the file specified
@@@

We have tried to avoid this by making the builds / deploys run on only 1 TeamCity agent hoping that the problem occurred because one of the other agents were locking the file. But this didn’t seem to fix it, so now I turn to you.

Referring to me previously created issue: https://github.com/OctopusDeploy/Issues/issues/740.

Regards,
Sebastian

Hi Sebastian,

Do you have a virus scanner or something else running on TeamCity? Is there a lot of resource usage going on?

At the start of the build we attempt to extract the Octo.exe assembly to that package path, then we call it. If another process prevents the file being extracted or deletes it, I’m not sure what we can do.

Would it be possible to post a copy of the entire build log from TeamCity?

Paul

Hi Paul,

There is no virus scanner on the server, and it’s not using a lot of resources, about 10-40% CPU and about 45-50% memory out of 24gb…

This is the full log as seen from the web:
@@@
[09:07:34]Skip checking for changes - changes are already collected
[09:07:35]Publishing internal artifacts (1s)
[09:07:36]Sending using ArtifactsCachePublisher
[09:07:36]Sending using WebPublisher
[09:07:35]Clearing temporary directory: C:\BuildAgentsNew\BuildNew3\BuildAgent\temp\buildTmp
[09:07:35]Failed to delete empty directory: C:\BuildAgentsNew\BuildNew3\BuildAgent\temp\buildTmp\octo-temp\2.0
[09:07:35]Checkout directory: C:\BuildAgentsNew\BuildNew3\BuildAgent\work\87b08b0fb315d62
[09:07:36]Step 1/1: OctopusDeploy: Create release
[09:07:36][Step 1/1] Octopus Deploy
[09:07:36][Octopus Deploy] Running command: octo.exe create-release --server http://buildserver:8081 --apikey SECRET --project MyProject Web --enableservicemessages --version 1.1.1.43 --deployto Test
[09:07:36][Octopus Deploy] Creating Octopus Deploy release
[09:07:36][Octopus Deploy] Error from Octo.exe: Cannot run program “C:\BuildAgentsNew\BuildNew3\BuildAgent\temp\buildTmp\octo-temp\2.0\octo.exe” (in directory “C:\BuildAgentsNew\BuildNew3\BuildAgent\work\87b08b0fb315d62”): CreateProcess error=2, The system cannot find the file specified
[09:07:36][Octopus Deploy] Step OctopusDeploy: Create release failed with unexpected error
[09:07:36]Publishing internal artifacts
[09:07:36]Sending using ArtifactsCachePublisher
[09:07:36]Sending using WebPublisher
[09:07:36]Build finished
@@@

Regards,
Sebastian

Hi Sebastian,

I think this might be the cause of the error:

[09:07:35]Clearing temporary directory: C:\BuildAgentsNew\BuildNew3\BuildAgent\temp\buildTmp
[09:07:35]Failed to delete empty directory: C:\BuildAgentsNew\BuildNew3\BuildAgent\temp\buildTmp\octo-temp\2.0


On your TeamCity build agent server, can you bring up task manager and see if any instances of Octo.exe are running? It looks like perhaps something is locking this directory, preventing the file from being extracted. Could you try deleting the directory manually?

Paul

Hello,
I tried deleting it manually, still the same problem.
There is no Octo.exe running continuously, though when another agent is doing some packaging or deploying it runs ofcourse.
Something I might have forgotten to tell you is that for this specific project we have in teamcity, 1 octopack package step and 4 octopus release steps.
The release steps are set to be run on the same agent as the agent that does the packaging (This goes for all our projects).
And it looks like most of the time it successfully manage to do the octopus deploy step for the first one in the queue and fails the other 3.

Regards,
Sebastian

Thanks for the update Sebastian. I’ll make a change to the TeamCity plugin today and let you know when a new version is available that may be able to fix this problem.

Paul

Hi Sebastian,

Can you try this version of the TeamCity plugin?

Paul

Hello Paul,

Thank you for the quick update. Unfortunately it didn’t seem to work.
However I got a new error,
@@@
[08:35:43]Skip checking for changes - changes are already collected
[08:35:44]Publishing internal artifacts
[08:35:45]Sending using ArtifactsCachePublisher
[08:35:45]Sending using WebPublisher
[08:35:44]Clearing temporary directory: C:\BuildAgentsNew\BuildNew1\BuildAgent\temp\buildTmp
[08:35:44]Failed to delete empty directory: C:\BuildAgentsNew\BuildNew1\BuildAgent\temp\buildTmp\octo-temp\2.0
[08:35:45]Step 1/1: OctopusDeploy: Create release
[08:35:45][Step 1/1] Unable to create temporary file in C:\BuildAgentsNew\BuildNew1\BuildAgent\temp\buildTmp for Octopus: C:\BuildAgentsNew\BuildNew1\BuildAgent\temp\buildTmp\octo-temp\2.0\Octo.exe (Access is denied)
[08:35:46][Step 1/1] Step OctopusDeploy: Create release failed with unexpected error
[08:35:46]Publishing internal artifacts
[08:35:46]Sending using ArtifactsCachePublisher
[08:35:46]Sending using WebPublisher
[08:35:46]Build finished
@@@

“Access is denied” - But it’s still working occasionally like before.
Furthermore information; The account running the agents has been granted full access to the C:\BuildAgentsNew containing all build agents, so we highly doubt that the error actually is an access problem. We even tried giving “Everyone” full access to the folder.

Regards,
Sebastian

Hi Sebastian,

Do you have multiple builds using this Octopus runner running in parallel?

I wonder if build 1 tries to create the Octo.exe file, but then fails because build 2 creates it first.

Paul

Hello Paul,

No, they are all set to be run on the same agent so they are queued one after one. Both the packaging and all the deploy steps. Though we do have 2 more agents on that server, but as far as I understand it the agents and the Octopus for the agents are all separate, right?

Regards,
Sebastian

Hi Sebastian,

To be honest, I’m quite stumped.

We check if this directory exists:

C:\BuildAgentsNew\BuildNew1\BuildAgent\temp\buildTmp\octo-temp\2.0


Since it does, we then attempt to extract the Octo.exe file to it, and this fails with an Access denied error.

There are only a few ways this could happen:

1. Some other process has locked the file (usually a virus scanner)
2. Octo.exe is already running (but your builds don’t run in parallel so this shouldn’t happen)
3. The process doesn’t have write-access to the folder (unlikely since the error doesn’t always happen)
4. The file is marked read-only or some other issue (again not likely)

The next time this happens, can you check the TeamCity build agent log files to see if there are any more details about the error:

http://confluence.jetbrains.com/display/TCD8/Viewing+Build+Agent+Logs

In your TeamCity VCS settings, have you configured TeamCity to purge the working directory (clean) each time the build runs? Could you try seeing if that has any effect?

Thanks,

Paul

Hello,

I’m attaching a print screen of the TeamCity setup.
To make this occur I start the package step, the other 4 steps are then triggered auto when the Package step is completed and successful. The steps named “Release” triggers a Deploy in Octopus.
Everything running on the same agent (As mentioned before).
The Log file is taken from the “wrapper.log” as this seems to be the only file it writes any logs to when this occurs. The log file is from when I clicked the “Run” on the package step and forward.
Though it doesn’t really seem to be making any more sense than the other Octopus logs to me.

In the screenshot the “Release to Client1 Test” build was successful, also attaching the web log from teamcity for that one.

Also attaching the octopus log of the successful build (The other TeamCity triggered builds does not reach Octopus so I don’t have any logs for those).

Disclosure: Due to that some of the log files contains sensitive information I have replaced project names, server addresses etc with dummy information.

I have checked the logs but I cannot find anything out of the ordinary.

1: I don’t know all programs installed on the server. I will take a closer look on this when I can get a hold of the guy who set up the server. But wouldn’t this probably cause a failed build every time as well?

2: In teamcity they are not run parallel but TeamCity doesn’t wait for the Octopus deploy to be fully completed before starting the next deploy in the queue. It runs for a couple of seconds (5-10) then triggers the next.
Could this be a problem?

I think 3 and 4 is highly unlikely as well as you mentioned.

Regards,
Sebastian

TeamCity_Log.txt (15 KB)

Hi Sebastian,

On the assumption that this is an intermittent/temporary issue, I’m going to make a change to the TeamCity plugin that will just retry/sleep a few times when extracting the file. I’ll let you know when a new build is available (expect it next week).

Paul

Hi Paul,

We just noticed that there is a 2.4.1 pre-release version up for download.
Did you include the retry/sleep thing in the teamcity plugin with the same version?

//Sebastian

Hi Sebastian,

No, I don’t believe the current -pre includes the updated TC plugin yet, sorry. Will ensure we get it in the next drop.

Nick

Hello Paul / Nicholas!

Any news on this?

//Sebastian

Hi Sebastian,

Thanks for reminding us and sorry for the delay - we’ve just released the fix today:

Paul

Unfortunately this didn’t help… We did a fresh install on a new server and we’re still seeing the same problem…
To resolve this we have currently setup a retry-build trigger upon a failed deploy in Teamcity that will retry 10 times. This seems to be the only work-around that can manage this at the moment.

Regards,
Sebastian

Hi Sebastian,

Looks like we need to find a new angle to attack this one on; just curious -

2: In teamcity they are not run parallel but TeamCity doesn’t wait for the Octopus deploy to be fully completed before starting the next deploy in the queue. It runs for a couple of seconds (5-10) then triggers the next.

Is the next build triggering on build completion? This sounds a bit unexpected to me - wondering if we have some kind of wait-for-exit missing…

Cheers,
Nick

Hello Nick,

Long time since this was started…
Anyway, I just noticed that it’s still open.
Just wanted to let you know that this got resolved at some point somehow, can’t really tell how or when as I don’t know. But you may close this thread.

Regards,
Sebastian