Gcloud crashed

jamie.springett · 16 March 2023 12:41

Hi I am trying to deploy a docker image to Google Cloud Run using the gcloud step template.
Please see my script here [1]
However I get the error ERROR: gcloud crashed (CommandLoadFailure): Problem loading gcloud.run.services: module 'collections' has no attribute 'MutableMapping'.
Please see the full failing logs here [2]
Please see the full successful logs from 8 days prior here [3]

The only thing of note I could find when comparing these two sets of logs is the difference in Operating System. Changing from Linux 5.4 to Linux 5.15.
OperatingSystem: Linux pkrvmsezufd6p0l 5.4.0-1103-azure #109~18.04.1-Ubuntu
OperatingSystem: Linux pkrvmuwkk4viy9t 5.15.0-1029-azure #36-Ubuntu

Please could you advise if there is a work around that I can employ or whether this will need to be fixed you the Octopus side.

Kind regards
Jamie

[1]

#!/bin/bash

image=$(get_octopusvariable "Octopus.Release.Notes")
image=$(cut -f 1 -d "~"<<<"$image")
region=$(get_octopusvariable "Region")
gcloud run services update ecommerce --platform=managed --image=$image --region=$region --quiet 2>&1

[2]
== Failed: Step 2: Deploy to Cloud Run ==
11:38:16 Fatal | The step failed: Activity Deploy to Cloud Run on a Worker failed with error ‘The remote script failed with exit code 1’.
11:38:16 Verbose | Deploy to Cloud Run completed
|
| == Failed: Worker ==
11:38:13 Verbose | Octopus Server version: 2023.2.2028
11:38:13 Verbose | Environment Information:
| IsRunningInContainer: True
| OperatingSystem: Linux 5.4.0-1100-azure #106~18.04.1-Ubuntu SMP Mon Dec 12 21:49:35 UTC 2022
| OsBitVersion: x64
| Is64BitProcess: True
| CurrentUser: root
| MachineName: octopus-i055569-55c75dd6d9-5mccr
| ProcessorCount: 2
| CurrentDirectory: /Octopus
| TempDirectory: /tmp/
| HostProcessName: Octopus.Server
| PID: 1
11:38:13 Info | Leasing UbuntuDefault dynamic worker…
11:38:13 Info | Obtained UbuntuDefault worker lease successfully.
11:38:13 Verbose | Leased worker 23-03-16-1038-8vlas from pool Hosted Ubuntu (lease Leases-7801700).
11:38:13 Verbose | Executing Deploy to Cloud Run (type Run gcloud in a Script) on DynamicWorker 23-03-16-1038-8vlas
11:38:14 Verbose | Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-4462
11:38:14 Verbose | Executable directory is /bin
11:38:14 Verbose | Executable name or full path: /bin/bash
11:38:14 Verbose | No user context provided. Running as current user.
11:38:14 Verbose | Starting /bin/bash in working directory ‘/home/Octopus/Work/20230316113813-4462-13’ using ‘Unicode (UTF-8)’ encoding running as ‘tentacle’ with the same environment variables as the launching process
11:38:14 Verbose | Process /bin/bash in /home/Octopus/Work/20230316113813-4462-13 exited with code 0
11:38:14 Verbose | Using Calamari.GoogleCloudScripting.linux-x64 25.3.3
11:38:14 Verbose | Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-4462
11:38:14 Verbose | Executable directory is /bin
11:38:14 Verbose | Executable name or full path: /bin/bash
11:38:14 Verbose | No user context provided. Running as current user.
11:38:14 Verbose | Starting /bin/bash in working directory ‘/home/Octopus/Work/20230316113814-4462-14’ using ‘Unicode (UTF-8)’ encoding running as ‘tentacle’ with the same environment variables as the launching process
11:38:14 Verbose | Started Calamari PID 3701
11:38:14 Verbose | Calamari Version: 1.0.0
11:38:14 Verbose | Environment Information:
11:38:14 Verbose | OperatingSystem: Unix 5.15.0.1029
11:38:14 Verbose | OsBitVersion: x64
11:38:14 Verbose | Is64BitProcess: True
11:38:14 Verbose | Running on Mono: False
11:38:14 Verbose | CurrentUser: tentacle
11:38:14 Verbose | MachineName: pkrvmuwkk4viy9t
11:38:14 Verbose | ProcessorCount: 2
11:38:14 Verbose | CurrentDirectory: /home/Octopus/Work/20230316113814-4462-14
11:38:14 Verbose | TempDirectory: /tmp/
11:38:14 Verbose | HostProcess: Calamari.GoogleCloudScripting (3701)
11:38:14 Verbose | Performing variable substitution on ‘/home/Octopus/Work/20230316113814-4462-14/Script.sh’
11:38:14 Verbose | Executing ‘/home/Octopus/Work/20230316113814-4462-14/Script.sh’
11:38:14 Verbose | Setting Proxy Environment Variables
11:38:14 Verbose | Using gcloud from /usr/bin/gcloud.
11:38:14 Verbose | Authenticating to gcloud with key file
11:38:14 Verbose | “/usr/bin/gcloud” auth activate-service-account --key-file=“/home/Octopus/Work/20230316113814-4462-14/apjp2fr4.zlc”
11:38:15 Verbose | Activated service account credentials for: [dev-ops@dev-ops-376414.iam.gserviceaccount.com]
11:38:15 Verbose | Successfully authenticated with gcloud
11:38:15 Verbose | Setting Proxy Environment Variables
11:38:15 Verbose | Bash Environment Information:
11:38:15 Verbose | OperatingSystem: Linux pkrvmuwkk4viy9t 5.15.0-1029-azure #36-Ubuntu SMP Mon Dec 5 19:31:08 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
11:38:15 Verbose | CurrentUser: tentacle
11:38:15 Verbose | HostName: pkrvmuwkk4viy9t
11:38:15 Verbose | ProcessorCount: 2
11:38:15 Verbose | CurrentDirectory: /home/Octopus/Work/20230316113814-4462-14
11:38:15 Verbose | TempDirectory: /tmp
11:38:15 Verbose | HostProcessID: 3730
11:38:16 Info | ERROR: gcloud crashed (CommandLoadFailure): Problem loading gcloud.run.services: module ‘collections’ has no attribute ‘MutableMapping’.
11:38:16 Info | If you would like to report this issue, please run the following command:
11:38:16 Info | gcloud feedback
11:38:16 Info | To check gcloud for common problems, please run the following command:
11:38:16 Info | gcloud info --run-diagnostics
11:38:16 Error | Running rollback behaviours…
11:38:16 Error | Script returned non-zero exit code: 1.
11:38:16 Verbose | Process /bin/bash in /home/Octopus/Work/20230316113814-4462-14 exited with code 1
11:38:16 Verbose | Updating manifest with output variables
11:38:16 Verbose | Updating manifest with action evaluated variables
11:38:16 Verbose | Successfully called release for dynamic pool lease Leases-7801700 : 23-03-16-1038-8vlas
11:38:16 Fatal | The remote script failed with exit code 1
11:38:16 Fatal | The action Deploy to Cloud Run on a Worker failed
|

[3]
== Warning: Step 2: Deploy to Cloud Run ==
15:03:58 Verbose | Deploy to Cloud Run completed
|
| == Warning: Worker ==
15:03:26 Verbose | Octopus Server version: 2023.2.1191-hotfix.1386
15:03:26 Verbose | Environment Information:
| IsRunningInContainer: True
| OperatingSystem: Linux 5.4.0-1100-azure #106~18.04.1-Ubuntu SMP Mon Dec 12 21:49:35 UTC 2022
| OsBitVersion: x64
| Is64BitProcess: True
| CurrentUser: root
| MachineName: octopus-i055569-5f6f7644b7-p8zrg
| ProcessorCount: 2
| CurrentDirectory: /Octopus
| TempDirectory: /tmp/
| HostProcessName: Octopus.Server
| PID: 1
15:03:26 Info | Leasing UbuntuDefault dynamic worker…
15:03:26 Warning | The UbuntuDefault worker image in the Hosted Ubuntu pool is being deprecated and it will be removed on Monday, 03 April 2023 UTC. Please refer to the Dynamic Worker documentation to ensure your Deployments are not impacted by this change.
15:03:26 Info | Obtained UbuntuDefault worker lease successfully.
15:03:26 Verbose | Leased worker 23-03-08-0820-rj1nm from pool Hosted Ubuntu (lease Leases-7732463).
15:03:26 Verbose | Executing Deploy to Cloud Run (type Run gcloud in a Script) on DynamicWorker 23-03-08-0820-rj1nm
15:03:27 Verbose | Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-4166
15:03:27 Verbose | Executable directory is /bin
15:03:27 Verbose | Executable name or full path: /bin/bash
15:03:27 Verbose | No user context provided. Running as current user.
15:03:27 Verbose | Starting /bin/bash in working directory ‘/home/Octopus/Work/20230308150326-4166-113’ using ‘Unicode (UTF-8)’ encoding running as ‘tentacle’ with the same environment variables as the launching process
15:03:27 Verbose | Process /bin/bash in /home/Octopus/Work/20230308150326-4166-113 exited with code 0
15:03:27 Verbose | Using Calamari.GoogleCloudScripting.linux-x64 25.3.3
15:03:27 Verbose | Acquiring isolation mutex RunningScript with NoIsolation in ServerTasks-4166
15:03:27 Verbose | Executable directory is /bin
15:03:27 Verbose | Executable name or full path: /bin/bash
15:03:27 Verbose | No user context provided. Running as current user.
15:03:27 Verbose | Starting /bin/bash in working directory ‘/home/Octopus/Work/20230308150327-4166-114’ using ‘Unicode (UTF-8)’ encoding running as ‘tentacle’ with the same environment variables as the launching process
15:03:27 Verbose | Started Calamari PID 11770
15:03:27 Verbose | Calamari Version: 1.0.0
15:03:27 Verbose | Environment Information:
15:03:27 Verbose | OperatingSystem: Unix 5.4.0.1103
15:03:27 Verbose | OsBitVersion: x64
15:03:27 Verbose | Is64BitProcess: True
15:03:27 Verbose | Running on Mono: False
15:03:27 Verbose | CurrentUser: tentacle
15:03:27 Verbose | MachineName: pkrvmsezufd6p0l
15:03:27 Verbose | ProcessorCount: 2
15:03:27 Verbose | CurrentDirectory: /home/Octopus/Work/20230308150327-4166-114
15:03:27 Verbose | TempDirectory: /tmp/
15:03:27 Verbose | HostProcess: Calamari.GoogleCloudScripting (11770)
15:03:27 Verbose | Performing variable substitution on ‘/home/Octopus/Work/20230308150327-4166-114/Script.sh’
15:03:27 Verbose | Executing ‘/home/Octopus/Work/20230308150327-4166-114/Script.sh’
15:03:27 Verbose | Setting Proxy Environment Variables
15:03:27 Verbose | Using gcloud from /usr/bin/gcloud.
15:03:27 Verbose | Authenticating to gcloud with key file
15:03:27 Verbose | “/usr/bin/gcloud” auth activate-service-account --key-file=“/home/Octopus/Work/20230308150327-4166-114/mkjajcc2.t13”
15:03:28 Verbose | Activated service account credentials for: [dev-ops@dev-ops-376414.iam.gserviceaccount.com]
15:03:28 Verbose | Successfully authenticated with gcloud
15:03:28 Verbose | Setting Proxy Environment Variables
15:03:28 Verbose | Bash Environment Information:
15:03:28 Verbose | OperatingSystem: Linux pkrvmsezufd6p0l 5.4.0-1103-azure #109~18.04.1-Ubuntu SMP Wed Jan 25 20:53:00 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
15:03:28 Verbose | CurrentUser: tentacle
15:03:28 Verbose | HostName: pkrvmsezufd6p0l
15:03:28 Verbose | ProcessorCount: 2
15:03:28 Verbose | CurrentDirectory: /home/Octopus/Work/20230308150327-4166-114
15:03:28 Verbose | TempDirectory: /tmp
15:03:28 Verbose | HostProcessID: 11803
15:03:58 Info | Deploying…
15:03:58 Info | Creating Revision…done
15:03:58 Info | Routing traffic…done
15:03:58 Info | Done.
15:03:58 Info | Service [ecommerce] revision [ecommerce-00073-wir] has been deployed and is serving 100 percent of traffic.
15:03:58 Info | Service URL: https://ecommerce-6krflri77a-ew.a.run.app
15:03:58 Verbose | Process /bin/bash in /home/Octopus/Work/20230308150327-4166-114 exited with code 0
15:03:58 Verbose | Updating manifest with output variables
15:03:58 Verbose | Updating manifest with action evaluated variables
15:03:58 Verbose | Successfully called release for dynamic pool lease Leases-7732463 : 23-03-08-0820-rj1nm
15:03:58 Verbose | Successfully finished Deploy to Cloud Run on a Worker
|

clare.martin · 16 March 2023 15:42

Good afternoon @jamie.springett,

Thank you for contacting Octopus Support and welcome back to the forums!

I am sorry you are seeing an error when trying to deploy a docker image to Google Cloud run using our gcloud step template. Thank you for sending us the logs and the rest of the information that has helped!

It took some googling and a bit of a deep dive but the error here:

11:38:16 Info | ERROR: gcloud crashed (CommandLoadFailure): Problem loading gcloud.run.services: module ‘collections’ has no attribute ‘MutableMapping’.

Seems to suggest the version of gcloud you are using on our workers is not compatible with whatever script you are running.

There is an actual official Google Cloud GitHub issue in for this here.
I also found this article here.
And this one which also is Google Cloud related - Google Issue Tracker.

Here is our documentation on what we have installed on our dynamic worker images, of note is below:

On Ubuntu 18.4 images we were using GCloud CLI (339.0.0) and Python 3.6.8.
And on Ubuntu 22.04 we are using GCloud CLI (367.0.0) and python 3.10.6.

I will get this in front of our engineers to see if they are able to include the environmental variable it mentions in the official Gcloud GitHub issue but it does state in our documentation if anything is incompatible with our dynamic workers and the tools that are on them you should use an execution container and install the software you need in order to get your applications to work.

We actually have a gcloud docker container you can use here if you wanted to use that and set your own environment variable or python version.

I would recommend setting up your own execution container with our gcloud docker image on and run through the suggested fixes in the websites I have linked above and see if you can get this working, I will speak to our engineers but as mentioned we tend to not update our dynamic worker images for specific issues like this as it could have an effect on all the other tools we need to support (ie the AWS CLI, Azure CLI etc) so I would say the execution container is your best bet.

I am sorry I could not be of further help here but this does look to be a known issue dating back quite far with Gcloud and python and various versions. Our 18.04 Ubuntu worker image must have just had the right combination of Gcloud and Python for it to work.

I will let you know what our engineers say, reach out in the meantime if you need further help,
Kind Regards,
Clare

clare.martin · 17 March 2023 08:49

Hey @jamie.springett,

Just a quick update for you, it might be worth holding out on the execution containers, I spoke too soon it seems regarding my comment that the engineers might not update our dynamic worker tools image for Ubuntu 22.04.

They are currently installing newer versions of gcloud CLI (412) on our Ubuntu 22.04 image we use for our dynamic workers on Cloud, and are running some tests so hopefully they can get this working and you can just carry on using the Ubuntu 22.04 dynamic worker.

I will keep you informed on the testing the engineers are currently doing and when the new worker image is out so you can run your own tests.

Kind Regards,
Clare

system · 17 April 2023 08:51

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.