I’m using OctoPack with a large VS solution (300+ projects) and we produce about 20 packages out of the build. Our build uses the /m flag to MSBuild to tell it to build multiple projects in parallel.
We’re using TeamCity for builds and we recently reconfigured our 3 build agents to live on a single machine instead of one machine per agent. Soon thereafter, we’ve started seeing strange build failures during simultaneous builds where a build executing on one agent will fail because the OctoPackNuGetExePath somehow gets set to the OctoPack nuget package folder of another build agent.
I’ve dumped the MSBuild diag logs and verified that OctoPackNuGetExePath is blank when the build first starts so I looked through the OctoPack source code and believe the bug is actually in CreateOctoPackPackage.FindNuGet():
MSBuild has a feature (on by default) where it will “re-use” msbuild.exe processes after the build runs and if the same process is used by builds for two different agents, it can already have the OctoPack Tasks DLL loaded from the previous build’s packages folder and the CLR won’t load it again from the current build’s packages folder. Therefore, when OctoPack tries to discover the nuget.exe, the assembly’s FullLocalPath is pointing to the wrong folder.
Sample output from MSBuild:
OctoPack error OCT-1676060969: Error when attempting to execute C:\TC\2\work\96e07916d4720aa8\packages\OctoPack.3.4.2\tools\NuGet.exe: The system cannot find the file specified [C:\TC\1\work\96e07916d4720aa8\Foo\Foo.csproj]
OctoPack error OCT-1676060969: System.Exception: Error when attempting to execute C:\TC\2\work\96e07916d4720aa8\packages\OctoPack.3.4.2\tools\NuGet.exe: The system cannot find the file specified ---> System.ComponentModel.Win32Exception: The system cannot find the file specified [C:\TC\1\work\96e07916d4720aa8\Foo\Foo.csproj]
OctoPack error OCT-1676060969: at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo) [C:\TC\1\work\96e07916d4720aa8\Foo\Foo.csproj]
OctoPack error OCT-1676060969: at System.Diagnostics.Process.Start() [C:\TC\1\work\96e07916d4720aa8\Foo\Foo.csproj]
OctoPack error OCT-1676060969: at OctoPack.Tasks.SilentProcessRunner.ExecuteCommand(String executable, String arguments, String workingDirectory, Action`1 output, Action`1 error) [C:\TC\1\work\96e07916d4720aa8\Foo\Foo.csproj]
OctoPack error OCT-1676060969: --- End of inner exception stack trace --- [C:\TC\1\work\96e07916d4720aa8\Foo\Foo.csproj]
OctoPack error OCT-1676060969: at OctoPack.Tasks.SilentProcessRunner.ExecuteCommand(String executable, String arguments, String workingDirectory, Action`1 output, Action`1 error) [C:\TC\1\work\96e07916d4720aa8\Foo\Foo.csproj]
OctoPack error OCT-1676060969: at OctoPack.Tasks.CreateOctoPackPackage.RunNuGet(String specFilePath, String octopacking, String octopacked, String projectDirectory) [C:\TC\1\work\96e07916d4720aa8\Foo\Foo.csproj]
OctoPack error OCT-1676060969: at OctoPack.Tasks.CreateOctoPackPackage.Execute() [C:\TC\1\work\96e07916d4720aa8\Foo\Foo.csproj]
I think I have a workaround by setting the MSBUILDDISABLENODERESUSE environmental variable to disable the msbuild node reuse behavior but it would be good if OctoPack had a better way of locating the nuget.exe. There is a MSBuild property for the current folder (MSBuildThisFileDirectory) so one idea would be to have the octopack.targets file pass $(MSBuildThisFileDirectory)nuget.exe if OctoPackNuGetExe is not specified instead of doing the discovery from within the task.
To clarify, the output I pasted shows a build running on Agent 1 with home directory C:\TC\1 failing because it’s trying to execute the nuget.exe from the packages folder under the Agent 2 home directory C:\TC\2.
Thanks for the report and the great investigation.
In the log we have the nuget.exe from agent 2 C:\TC\2\work\96e07916d4720aa8\packages\OctoPack.3.4.2\tools\NuGet.exe and the agent 1 csproj file C:\TC\1\work\96e07916d4720aa8\Foo\Foo.csproj.
The log seems to be saying that it’s the csproj file that can’t be found? Is that the case or is it the Nuget.exe that isn’t there? If the csproj file can’t be found do you know why that is?
I would have thought that even though we have the wrong exe it should work since one Nuget.exe since they are both the same. Just to be clear I want to fix this, but want to fully understand why it isn’t working to make sure we fix things properly, since it might be tough for us to reproduce. If the logs contain the full text of the nuget commandline can you send me that too? I believe we write it out.
It can’t find the nuget.exe – every line ends with the csproj that is
currently being processed.
As far as why it doesn’t find it, I believe it’s a timing issue. We purge
the work directory at the beginning of the build (which includes deleting
the nuget “packages” folder) so if that happens while another agent is
trying to octopack (but using the wrong nuget.exe path), it could end up
with the “nuget.exe not found” error I’m seeing.
I will look for that full cmd line in log this week and post it.
Thanks for the clarification, I just released Octopack 3.5.2 with the MSBuildThisFileDirectory you suggested. Let me know if this doesn’t solve your issue.
Here is the full command line that I got from running the build at /v:diag level. This one didn’t fail but it still shows that it ran the nuget.exe from the “wrong” folder.
21:26:30.007
292>OctoPack: (TargetId:688)
OctoPack: A NuSpec file named 'Foo.nuspec' was not found in the project root, so the file will be generated automatically. However, you should consider creating your own NuSpec file so that you can customize the description properly. (TaskId:442)
OctoPack: Packaging a console or Window Service application (no Web.config detected) (TaskId:442)
OctoPack: Add binary files (TaskId:442)
OctoPack: Added file: Foo.exe.config (TaskId:442)
OctoPack: Added file: Foo.exe (TaskId:442)
OctoPack: Added file: Foo.pdb (TaskId:442)
OctoPack: NuGet.exe path: C:\TC\2\work\96e07916d4720aa8\packages\OctoPack.3.4.2\tools\NuGet.exe (TaskId:442)
OctoPack: Running NuGet.exe with command line arguments: pack "C:\TC\1\work\96e07916d4720aa8\Foo\obj\octopacking\Foo.nuspec" -NoPackageAnalysis -BasePath "C:\TC\1\work\96e07916d4720aa8\Foo" -OutputDirectory "C:\TC\1\work\96e07916d4720aa8\Foo\obj\octopacked" -Version 3.0.1656 (TaskId:442)
I saw the changes for 3.5.2 and will try them out and report back.