Show Docker logs not all as Errors

fhanggi · 11 November 2022 01:38

Hello
I am running some docker commands inline in a powershell script in a deployment.

write-host "should check or start container"
$DeploymentFolderRoot = $OctopusParameters["DeploymentFolderRoot"]

write-host "shutting down and removing apigw containers ..."
D:
CD $DeploymentFolderRoot
docker compose up -d

write-host "validating krakend configuration"
docker exec -it -u root apigw1 sh ./scripts/kd-check.sh

Everything docker reports is displayed as errors.

I tried this to redirect the output
{docker compose up -d} | Out-Null
but then i don’t get any docker logs in the octopus logs

What i would like to happen is to see Docker info/warnings show up as such in the octopus logs but when there is an error in the docker command octopus should stop and fail the step.

Can this be done?
Thank you very much.
Fabian

Daniel_Fischer · 11 November 2022 02:04

Hi Fabian,

Thanks for getting in touch! To get started with some troubleshooting, would you be able to let me know what versions of Octopus Server and Docker you are using?

It looks like the logs are being passed through as STDERR, but I don’t think that’s the standard behaviour we see. We’re going to do some testing on our end but the above info will be very helpful.

Looking forward to hearing from you and getting to the bottom of this.

Best regards,
Daniel

fhanggi · 14 November 2022 17:00

Hi Daniel
We are using Octopus v2022.2 (Build 6872)
Docker version 20.10.17, build 100c701
Thank you
Fabian

Justin_Walsh · 14 November 2022 20:56

Hey @fhanggi!

Just jumping in for Daniel here, as he’s not yet online as a member of our Australian-based team.

It looks like docker-compose logging all of its output to stderr is a pretty common problem that people hit - sadly, it doesn’t look like the Docker developers have shown any inkling of fixing at this point in time. There’s a great Github issue on this over at https://github.com/docker/compose/issues/7346 - where in the comments, there’s a bunch of workaround from people who have hit this exact issue.

If none of those work, you could look at Octopus’ built-in output redirection service messages: Logging messages from scripts - Octopus Deploy to redirect that output away from the error stream.

I hope this helps!

fhanggi · 15 November 2022 03:27

Hello I read through that post and it is unfortunate that the docker team doesn’t react.
There seems no good way, to deal with this. I tried for a few hours but i’ll forget about it for now.

Maybe you can look at this a bit more
What is really bad is that even with real errors octopus continues. But i suspect the issue is again on the docker side

#line below shows an error in console but still continues. just shows a warning in octopus
#this is a completely wrong command
docker compose upababa -d

#below seems to hide everything even real errors
{docker compose upababa -d} | Out-Null

Is octopus not checking the exit code? or is the above error not considered a real error by docker and they do not exit with an error?

The only thing that kind of worked was the ps script someone posted but it is not very user friendly…

function Invoke-DockerComposeCommand {
    param(
        [string]
        $arguments
    )
    $cmdarguments = @("/c", "docker-compose") + ($arguments -split " ")
    $processStartInfo = New-Object System.Diagnostics.ProcessStartInfo
    $processStartInfo.WorkingDirectory = (pwd).Path
    $processStartInfo.FileName = "C:\windows\system32\cmd.exe"
    $processStartInfo.RedirectStandardError = $true
    $processStartInfo.RedirectStandardOutput = $true
    $processStartInfo.UseShellExecute = $false     
    $processStartInfo.Arguments = $cmdarguments

    $process = New-Object System.Diagnostics.Process

    $process.StartInfo = $processStartInfo
    Write-Host "docker-compose $arguments..." -NoNewline
    $process.Start() | Out-Null
    $process.WaitForExit()
    if ($process.ExitCode -ne 0) {
        Write-Host "error"
        throw [Exception]::new("Error on docker-compose $($arguments): $($process.StandardError.ReadToEnd())")
    }else{
        Write-Host "done"
    }
}

then to execute

Invoke-DockerComposeCommand “upxxxxx -d”

now this correctly throws and error

if the command is correct it just shows “…done” when things are ok.

Ideally all messages from docker should be displayed as Info, except the ones that cause an exit code other than 0. But as i said i could not figure that out.

Thank you
Fabian

fhanggi · 22 November 2022 23:37

Hello, Could you please let me know if anyone is looking into this further or if i’m on my own.
Thank you
Fabian

Daniel_Fischer · 23 November 2022 01:45

Hi Fabian,

I’m very sorry for the delay in getting back to you here. I did a lot of testing on this issue and I have a bit of a workaround. I have been unable to redirect the output to the info stream, but I was able to handle the warnings as verbose using ##octopus[stderr-progress]

We have a short video on how this works and some documentation. The video is for Linux, but you can use Write-Host ##octopus[stderr-progress] in PowerShell.

I’ve tested this and it does work, but you will need to view the raw logs or change your logging to verbose in the task drop down.

I think this could potentially be an oversight from our side in not providing a method for redirecting to the info stream (like ##octopus[stderr-info] or something). I’ve asked the developers to take a look and let me know if this can be improved or whether we need to update our documentation with an explanation as to why it wont work.

Let me know if this helps as a workaround for now.

Best regards,
Daniel

fhanggi · 23 November 2022 18:55

Hi Daniel

Sorry this didn’t work

tried this
write-host “##octopus[stderr-progress]”

and this
write-host ##octopus[stderr-progress]

still everything shows up as errors. i.e. it doesn’t go to verbose

Daniel_Fischer · 24 November 2022 00:05

Hi Fabian,

Thanks for getting back. The testing I did was with a very simple docker compose project based on this example from our docs. (I used a Windows server target.)

Below is a screenshot of my configuration in Octopus which successfully converts the errors to verbose logging. I only run a single docker-compose up -d command and set stderr-progress right before it executes.

I think the next step would be to enable variable logging in your deployment, create a new release and deployment, then send the new logs. This will provide a lot more context around what your deployment is doing and help us solve this.

The following secure link will allow you to upload support files through your Octopus.com account.

Looking forward to hearing from you.

Best regards,
Daniel

fhanggi · 24 November 2022 01:40

Hi Daniel
I made a mistake and now the logs are going to verbose but things are not working like they should

I ran an invalid command but octopus just continues.
see this screen shot with the verbose logs. it says docker composexxxx is not a docker command. But why doesn’t the script stop?
like everything is green like it was successful but obviously this is an error.
Is this an octopus issue or docker issue not reporting the error properly to octopus?

Daniel_Fischer · 24 November 2022 02:23

Hi Fabian,

I’d be curious to see if this command, without setting the ##octopus[stderr-progress], would cause the deployment to fail. If so, then Octopus may not be handling things correctly. If it does the same thing without ##octopus[stderr-progress], then it’s probably a docker issue.

When Octopus runs PowerShell scripts, it watches the $LastExitCode variable. If a value other than 0 is set, then it fails the deployment and returns any details about the error it can. Docker might not be setting the $LastExitCode when it encounters this error. We have a documentation page with details on handling errors in PowerShell scripts which explains this. To work around this you would need to modify your script to watch for this failed command then either throw an error or set the exit code to a non-zero value.

Let me know how you go.

Best regards,
Daniel

fhanggi · 24 November 2022 19:13

so without ##octopus[stderr-progress] it does NOT fail

i checked $LASTEXITCODE and is NOT 0 but octopus still continues
see the code below where i am logging in with wrong credentials.
i can see “LOGIN FAILED” in the console but octopus doesn’t stop. Do i need to check for exit code myself and throw an error?

write-host "logging in with incorrect credentials"
docker login -p "xxxxx" -u "cgf"
if($LASTEXITCODE -ne 0){
write-host "LOGIN FAILED"
exit
} 
else
{
write-host "Logged in"
}

Daniel_Fischer · 24 November 2022 23:44

Hi Fabian,

Thanks for the update. I did some more tests and I’m not sure why Octopus doesn’t fail the deployment when the exit code is set. This is something I’m going to raise with the developers. In the meantime, I got the sample you sent to work by changing your IF statement from write-host to write-error. See below.

write-host "logging in with incorrect credentials"
docker login -p "xxxxx" -u "cgf"
if($LASTEXITCODE -ne 0){
  write-error "LOGIN FAILED"
  exit
  } 
  else
  {
  write-host "Logged in"
}

This caused the deployment to fail and return the LOGIN FIALED message along with more specific details about the error. This should server as a workaround for now, though I’m interested to hear our developers take on this.

Let me know if this one works.

Best regards,
Daniel

fhanggi · 25 November 2022 00:16

Thank you.
I coded by using throw but your write-error may be better

write-host $UserID
docker login -p $Password -u $UserID

if($LASTEXITCODE -ne 0){
throw "Login failed for $($UserID)"
} 
else
{
write-host "Logged in as  $($UserID)"
}

let me know what the developers say.
in the meantime i have added that if($LASTEXITCODE -ne 0) check after every command i have in my script.

system · 26 December 2022 00:17

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.