Calamari's "Failed to create CoreCLR, HRESULT: 0x80070008" error in custom execution container

[jinesh.choksi]

In more than one of our Octopus Projects, we have a process step that will run inside a container on a worker from the “Hosted Ubuntu” dynamic worker pool. The custom execution container image used was based on an Official Ubuntu 20.04 image. In our custom docker image, we did not have any OS dependencies for Calamari (the Octopus deployment utility) installed and yet some how Calamari ran fine inside the container.

Execution containers for workers - Octopus Deploy

Am raising this topic to understand how that was working because after we switched our custom image from Ubuntu:20.04 to Ubuntu:22.04, Calamari stopped working inside the container and it instead reported an error saying: “Failed to create CoreCLR, HRESULT: 0x80070008” (container’s exit code was 137)

15:24:39   Verbose  |       Docker version 20.10.7, build f0df350
15:24:39   Verbose  |       docker run --entrypoint='' --rm  --env TentacleHome=/home/Octopus  -w /home/Octopus/Work/20220629152438-87227-14   -v /home/Octopus/Work/20220629152438-87227-14:/home/Octopus/Work/20220629152438-87227-14  -v /home/Octopus:/home/Octopus  gcr.io/opnf-management/deployer:latest
15:24:42   Error    |       Failed to create CoreCLR, HRESULT: 0x80070008
15:24:42   Verbose  |       Process /bin/bash in /home/Octopus/Work/20220629152438-87227-14 exited with code 137

Since we did not have any of the required OS dependencies I thought that could be the reason but then how was it working with Ubuntu:20.04 ?

This is what our Ubuntu:20.04 Dockerfile looked like:

FROM ubuntu:20.04

ARG HELM_VERSION=3.6.3
ARG TERRAFORM_VERSION=1.0.11
ARG TERRAGRUNT_VERSION=0.35.13

RUN apt-get update && \
    DEBIAN_FRONTEND=noninteractive apt-get install -y \
    apt-transport-https \
    software-properties-common \
    curl \
    gnupg2 \
    jq \
    gettext-base \
    git \
    openjdk-8-jre \
    ssh

RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \
    curl -fsSL -o /usr/share/keyrings/cloud.google.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg && \
    apt-get update && apt-get install -y google-cloud-sdk kubectl

RUN echo "deb [signed-by=/usr/share/keyrings/hashicorp.gpg arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | tee -a /etc/apt/sources.list.d/hashicorp.list && \
    curl -fsSL https://apt.releases.hashicorp.com/gpg | gpg --dearmor | tee /usr/share/keyrings/hashicorp.gpg >/dev/null && \
    apt-get update && apt-get install -y terraform=$TERRAFORM_VERSION

RUN curl -s https://deb.nodesource.com/setup_14.x | bash -s && \
    apt-get update && apt-get install -y nodejs

RUN echo "deb [signed-by=/usr/share/keyrings/yarn.gpg] https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list && \
    curl -fsSL https://dl.yarnpkg.com/debian/pubkey.gpg | gpg --dearmor | tee /usr/share/keyrings/yarn.gpg >/dev/null && \
    apt-get update && apt-get install -y yarn

RUN echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list && \
    curl -fsSL -o /usr/share/keyrings/githubcli-archive-keyring.gpg https://cli.github.com/packages/githubcli-archive-keyring.gpg && \
    apt-get update && apt-get install -y gh

RUN curl -LO https://get.helm.sh/helm-v${HELM_VERSION}-linux-amd64.tar.gz && \
    tar xzf helm-v${HELM_VERSION}-linux-amd64.tar.gz && \
    mv linux-amd64/helm /usr/bin/helm && \
    rm -rf linux-amd64 && \
    rm helm-v${HELM_VERSION}-linux-amd64.tar.gz

RUN curl -L https://github.com/gruntwork-io/terragrunt/releases/download/v$TERRAGRUNT_VERSION/terragrunt_linux_amd64 -o /usr/sbin/terragrunt && \
    chmod +x /usr/sbin/terragrunt

COPY helm_charts /resources/helm_charts

COPY terraform /resources/terraform

COPY scripts /resources/scripts

RUN chmod -R +x /resources/scripts

WORKDIR /resources

Calamari stopped working if we changed FROM ubuntu:20.04 to FROM ubuntu:22.04

I understand that we probably need to add the required OS dependencies to our custom image but I would like to understand how it was working without them before.

Thanks.

[dane.falvo]

Hi Jinesh,

Calamari has a number of dependencies so perhaps the other software you may have used on 20.04 provided the extra dependencies needed to have Calamari running fine.

After your 20.04 docker image is built and your 22.04 docker image is built, what versions of dotnet are installed on them?

We have seen some errors resulting from dotnet 6.x, so my initial guess would be that your 20.04 image is running dotnet 5.x and your 22.04 image is running dotnet 6.x. If we can rule this potential issue out and it doesn’t fix the issue, we can keep digging towards a solution.

Regards,

[jinesh.choksi]

Hi Dane,

so perhaps the other software you may have used on 20.04 provided the extra dependencies needed to have Calamari running fine.

This is what confused me. The official Ubuntu 20.04 image does not have dotnet in it.

After your 20.04 docker image is built and your 22.04 docker image is built, what versions of dotnet are installed on them?

From the Dockerfile above, you can see we did not install anything that would pull in dotnet. I did double check this. Also, we did not install Calamari inside the image and so were wondering how it was being executed.

My guess was that Calamari must exist in /home/Octopus on the worker node and this is mounted into the image.

To be honest, we were not aware about the required OS dependencies for custom images until we encountered the error and looked into the matter. So I think I can resolve our issue but it was peculiar how it managed to work so far.

Regards,