[Bug] Missing flag to configure "awsfirelens" Log Driver on the "Deploy Amazon ECS Service" step template

Hi!
I’m actually trying to figure out how to use the log driver configuration so I can forward the logs to our self-hosted FluentD instance, instead of using CloudWatch Logs.

The problem is that it seems “impossible” to configure it using the “Deploy Amazon ECS Service” step template because it’s missing the firelensConfiguration flag inside the Container Definitions options.

The first configuration that I’ve tried was setting only the application container inside the Container Definitions block.

Then, I used the “Customise log options” inside “Log Configuration”, changed the “Log Driver” to “awsfirelens” and set some parameters inside the “Log Options” block (setting the “Name”, “Host”, “Port”, “Retry_Limit” and “Tag” parameters).

The response I get when I run this is an error message from CloudFormation:

Resource handler returned message: "Invalid request provided: Create TaskDefinition: When awsfirelens log driver is specified in log configuration, a firelens configuration object must be configured in one of the containers.

And it makes sense, because I didn’t configure the log_router container inside my Container Definitions block. But that’s when things get weird…

So, I looked into some examples from AWS to define that log_router container: (https://github.com/aws-samples/amazon-ecs-firelens-examples/blob/mainline/examples/fluent-bit/forward-to-aggregator/task-definition.json)

As the example specifies, I need to specify the log_router container that uses the amazon/aws-for-fluent-bit:stable image and has a firelensConfiguration block with the parameter type set to fluentbit.

The problem is: with the current official Step Template from Octopus Deploy, there is no option to set that firelensConfiguration block.

This looks like a bug. Without being able to configure that block, the entire Log Configuration block is useless as you can only use CloudWatch Logs to send logs.

Hi @eduardo.saporski,

Thanks for reaching out and letting us know about this!

I’ve confirmed that this functionality is missing and agree that it definitely should be there. I’ve reached out to the team who agree, so I’ll work on creating a public issue to track it and will post it here as soon as it’s ready.

I’ll also keep investigating to see if there are workarounds in the meantime. It should be possible to use our Deploy CloudFormation Template Step.

I’ll keep you posted as soon as I have any updates or suggestions!

Best Regards,

2 Likes

Hi @finnian.dempsey!
Thanks for taking a look into this :smile:
We were trying to use the Deploy CloudFormation Template step but we need a nice graphical user interface for our users to set environment variables and secrets inside the Container Definition.

Let me know when you create that issue so I can track the progress. Thanks again :smile:

1 Like

Hi @eduardo.saporski,

No problem at all, I’ll make sure to post it here as soon as it’s ready. I’m taking a little longer on this one to fully understand the issue to hopefully speed up the resolution time.

In this case I’d recommend looking into our Custom Step Template feature, which will allow for you to specify parameters that get defined when the Step Template is included in a project, which can also be combined with Variable Substitution to have UI filled parameters directly inside your CF template!

First, create a Custom Step Template from the CloudFormation template by navigating to Library -> Step Templates -> Add -> Choose the CloudFormation Step Template

Next configure the Step as usual along with any Parameters (I found the CF template needed to be sourced from a package):

By specifying the Parameters in the Custom Step Template (default values are optional):

Users can define them when using the Custom Template in a Project:

And by using the Octostache syntax in your CloudFormation Template, Octopus will automatically use the provided values:

Feel free to let me know if you have any questions about the process or anything else, I’ll keep you posted as soon as I have any updates!

Best Regards,

1 Like

Hi @eduardo.saporski,

Just a quick update with that public GH issue for tracking the resolution: ECS Step Template - Unable to specify FirelensConfiguration in Container Definition · Issue #7821 · OctopusDeploy/Issues · GitHub

I’ll let you know if I hear of any further updates, feel free to reach out with any questions!

Best Regard,

1 Like

Hi @finnian.dempsey!
Thanks for the detailed answer :smile:

The problem is when we are trying to define the Environment and Secrets block inside the TaskDefinition resource. The environment variables should be a map with a key and a value, and there is no way of creating a CloudFormation parameter that holds a list of maps (a list of environment variables, in case you have more than one). The other thing is that CloudFormation doesn’t have anything like a for loop like Terraform does for creating resources dynamically.

So even interpolating the values using the Variable Substitution from OctopusDeploy, it would need us to update the YAML template manually every time we wanted to add a new Environment Variable.

Thanks for the help anyway! I’m subscribed to that GitHub issue so I can follow any news regarding this problem.

3 Likes

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.