AWS 'S3 Upload' step fails as has not assumed specified Role [2018.9.14]

Legacy Cloud
1

@Shaun_Marx I am getting the common ‘Unknown’ permission error from the S3 Upload step template on 2018.9.14. The reason is clear enough, it doesn’t seem to being assuming the specified role.

When I attach the permissions policy directly to the user it works, but when the permissions are attached to the role specified in the step you get a permissions error when uploading. It doesn’t seem to be assuming the role.

I have a AWS PowerShell step in the sample plan that uses the same AWS account and assumes the same role and writes to the same bucket in the same region, and it works correctly, using the permissions of the role.

And testing the user and assumed role from the command line works fine too.

The ‘S3 Upload’ step just doesn’t see to be assuming the specified role.

There is nothing in the raw log except that same 100 exit code and unknown Forbidden error.

With the Policy attached to the Role specified in the S3 Upload step you get the error:

23:07:46   Verbose  |       Bucket some-great-bucket exists in region Asia Pacific (Sydney) (us-east-2). Skipping creation.
23:07:47   Info     |       Glob pattern '**/*' matched 51 files
23:07:47   Info     |       Attempting to upload C:\Octopus\Work\20190603230744-29651-472\staging\foo.txt to bucket some-great-bucket with key myprefix/foo.txt.
23:07:47   Error    |       Calamari.Aws.Exceptions.UnknownException: An unrecognised Forbidden error was thrown while uploading to bucket some-great-bucket
23:07:47   Error    |       at Calamari.Aws.Deployment.Conventions.UploadAwsS3Convention.Install(RunningDeployment deployment)
23:07:47   Error    |       at Calamari.Deployment.ConventionProcessor.RunInstallConventions()
23:07:47   Error    |       at Calamari.Deployment.ConventionProcessor.RunConventions()
23:07:47   Error    |       Running rollback conventions...
23:07:47   Error    |       An unrecognised Forbidden error was thrown while uploading to bucket some-great-bucket

But with the same step and the role added directly to the AWS account, it works fine, even those the step still has a specified role to assume.

22:35:59   Verbose  |       Bucket some-great-bucket exists in region Asia Pacific (Sydney) (us-east-2). Skipping creation.
22:36:00   Info     |       Glob pattern '**/*' matched 51 files
22:36:00   Info     |       Attempting to upload C:\Octopus\Work\20190603223557-29646-462\staging\foo.txt to bucket some-great-bucket with key myprefix/foo.txt.
22:36:00   Info     |       Saving object version id to variable "Octopus.Action[AWS - Deploy to S3].Output.Files[myprefix/foo.txt]"

There is nothing in the raw logs of the AWS Powershell step nor the S3 Upload step saying when and what role it has assumed. That would be useful to log! Is there a way to enable debug logging from the step so I can see it log when if it assumes the role.

I checked the release logs for all newer versions but didn’t see a fix for this problem, so I assume it is similarly doesn’t work in the latest release?

3

Hi Aaron,

Thanks for reaching out, Sorry to hear you have hit this issue with Octopus.
Are you able to provide me with the raw logs ?
I need to check all the verbose logging, to figure out what is going on.
Are you running the step on an ec2 instance?

Hope to hear from you soon.

Regards
John

4

Hi Aaron,

Me again!
@Shaun_Marx and I have been looking at the code, and we think if there is an issue it seems to have been fixed in later versions. Are you able to try a more recent version of Octopus?

Regards
John

5

Thanks for investigating @John_Simons if there is a fix in newer releases I will organise an upgrade window over the next week and report back here.

Does the newest version also include a verbose or info log message when assuming a role? I think it is a noteworthy step for your and our debugging that should ideally show in the logs. If not, please toss it in the backlog.

6

Hi Aaron,

Yes the new version includes the same logging.
Are you still sending me logs for the current version you are on?

Regards
John

7

Confused now. The ‘same logging’ means it is the same in that it doesn’t log assuming roles?

In the version I tested neither the AWS PowerShell step nor the S3 Upload step log when assuming roles. The AWS PowerShell step template is doing it, it just doesn’t log it even in the verbose logging.

I downloaded the logs and the only useful logging is in the snippets above and that the exit code was 100.

Running locally on Octopus. This instance was launched from your official Azure template. Seems like the PowerShell being on your Azure template is quite old, if that is relevant.

23:07:44   Verbose  |       Starting C:\Windows\system32\WindowsPowershell\v1.0\PowerShell.exe in working directory
8

Hi Aaron,

I am not sure what Calamari version you are running, but a year ago we added the role logging to Calamari, it is possible the version you are running is older and hence does not have it.
The Calamari version is in the logs.

Regards
John

9

Hi Aaron,

Upgrading to the latest version of Octopus will definitely give us more verbose logging, but I cannot guarantee that will solve this issue. However with the new logging we should be able to figure out what is wrong and hopefully fix it.
Does this help you make a decision about updating?

Regards
John

10

Hi @John_Simons, the Calamari version is determines by the Octopus version, since each Octopus release deploys or upgrades to a specific Calarmari version.

In this case 2018.9.14 deploys Calamari version 4.10.0 and that is the version used here.

We have upgraded 2019.x so I’ll test again soon. However 2019 broke all our carefully crafted User Roles and the ‘Default’ namespace is cluttering the UI and confusing users, so now we have a bucket-load of other issues to sort out first! Will update when I can.