We use the task using the built in Cloudformation Template task to roll out deployments to our tenants on AWS via workers - As these tasks involve Cloudfront udates can take a bit of time (several minutes if it’s a full update) but we want to know the outcome so have the “Wait for Completion” flag set - which I believe has a wait loop that will poll the Cloudformation API for changes to the stack - this polling appears to be quite frequent.
My issue is that we run concurrent builds via Web workers we are hitting cloudformation rate limit errors due to multiple threads polling. Error:
“Amazon.CloudFormation.AmazonCloudFormationException: Rate exceeded” - Concurrent deployments all polling at the same time I suspect is the cause.
Is there a way we could pass in a flag to adjust the frequency this polling occurs? (or back off if the error occurs)
Currently this causes the task to go to fail although the Cloudformation task carries on OK on AWS.
There isn’t currently an option to adjust the polling frequency. I’ve checked with the team and there are two possible options in the interim:
Increase the rate limits in AWS
This is potentially the simplest option and I can see precedence for it from some searching around online, but there’s no guarantee that the limits can be raised either high enough or for long enough.
Use a custom script step to manually poll for success
This would require you to disable the “Wait for Completion” option and add a new step into your deployment process. You would then need to write a script to poll AWS at suitable rate to check for success before proceeding.
I’d be happy to raise this with our engineers as a possible bug with a view to either fixing it or introducing some form of outgoing rate limiting. We’d need to do some investigations first to confirm that it is due to the polling rather than anything else. Please let me know if you’d be interested in working with us on this.
I hope this is helpful. Please let me know if you have any questions.
Hi,
Thanks for your response - that would be great if the engineers could look into it - I’d be interested in looking into it as a potential bug and whether a rate limit (exponential backoff) could be added. I will check with AWS but I think the limit might be fixed in this case.
Custom step might be the only alternative otherwise I suspect otherwise.
I’ve brought this up with the team. I’ll keep you updated as we go along and let you know if they have any questions or if they need anything else in order to investigate this.