Hi,
We are seeing an issue in Octopus and would like to know if you have seen it before.
NotSpecified: Exception calling “GetAllActive” with “0” argument(s): “You have had too many failed authentication attempts in a short period of time. Please try again later.”
I have a runbook that runs every 5 minutes to monitor our tasks running in Octopus.
I have seen the error above failing some, not all of my runbooks. ServerTasks-5012693.log.txt (22.1 KB)
I have had some users also report the same issue in Azure pipeline when trying to push a package to Octopus.
Thank you for contacting Octopus Support and sorry to see you are running into this issue with some of the tasks you are running for Octopus.
This is not something we have seen before and a quick search does not pickup anything from historical tickets or conversations surrounding the auth attempts error message.
The API key you are using, is it from a user account or a service account and is it the same API being used for the runbooks (specifically the ones that runs every 5 mins) and the ADO pipelines?
Its a weird one this, we have been discussing it amongst the team and the wording seems odd to us ‘too many failed authentication attempts’ suggests a bad API key but if you tried with the wrong API key, how would Octopus know which API key to block for “failed authentications”. It would just throw a connection to the server error because it would not let you handshake with the API key.
Our feeling here is that the error message may be coming from either a network device (proxy perhaps not liking that many connection attempts over a small window) or some form of external auth provider.
I am going to ask our engineers if that is an error we would throw out or if it would come from an external source. It just seems odd that the error would show for some runbooks and not others and also in ADO itself, especially if those runbooks / ADO pipelines are using different API keys. If they are using the same API key that also doesn’t make sense as we have other customers who deploy thousands of releases daily using only a handful of API keys and they deploy fine so we wont be throttling API connections to Octopus.
Whilst I confirm with our engineers are you able to check with your networking team to see if they see a similar error with network traffic to Octopus and if its perhaps getting thrown at the proxy end?
One other thing to mention, do you see that error in your Octopus Server log by any chance? Or anything that would point to why that error would occur, if not no worries.
If you do see that error in your Octopus Server log are you able to send us it so we can take a look and see if we can spot anything, I created you a secure link here to upload the file to. Let us know if you are going to upload that file there as we are not notified.
Our engineers confirmed it is Octopus that is logging that error so not a network device thankfully.
They said they think you would see that error if something was using an invalid/expired API Key. If you had all the builds happening through a single agent, for example, and one ADO project had a bad API Key then it could cause the agent’s IP to get blocked for all builds from that agent, so this may be an IP based ban but from one invalid API key in one project (hopefully that makes sense).
I know you said the API key was not the issue but the engineers seem to think that is the route cause, it may just be one specific API key that is causing this, perhaps it is valid in terms of its the correct key but it has expired? Are you able to check the API keys in ADO and against the runbooks and make sure they are all correct and none of them have expired.
I look forward to hearing from you, the server logs may also shed some light on this, perhaps they will log an IP the request is coming from?
Hi @clare.martin,
No the API Keys are not expired, I even created new ones yesterday and got a few fails overnight.
If the API Key was expired, then all my runbooks should fail?
I got 3 fails in the last 12 hours (out of 200+ runs), the 3 fails were all at the same time so maybe at that time something was happening in the background.
I know you did say you checked the API keys but there’s always the possibility of missing one but if they are all the same API key then yes, if that was expired or incorrect nothing would have worked.
Also if you created new ones and you still got failures then we need to dig deeper into this, if you cant see anything specific in the logs it might be worth sending them over to us anyway as we might be able to spot something that may seem unrelated but we can correlate that with this issue possibly.
Let us know if the logs do show something or if you are going to send them over and we can take a closer look. Since we have not seen this issue come up before it does seem environmental but we need to rule out Octopus first before going down that route.
I look forward to hearing from you,
Kind Regards,
Clare
Hi @clare.martin,
I have created a new runbook specific API Key, which seems to have fixed the issue.
I haven’t seen the issue since last Friday.
I will monitor it but it seems to be resolved.
Interesting! Great news creating a new runbook specific API key fixed the issue, maybe it was a case of one specific API key getting hammered?
Forum posts don’t close for around 27 days of inactivity so you are free to reply to this if it starts happening again but if you have not seen this issue since last Friday I would say the same as you and that has fixed the issue for you.
Thank you for updating us on the fix as it will help other customers if they encounter the same errors you were seeing and its also good for our records in case we get a similar issue via email.
Reach out if there are any more issues and we will look into it for you,
Kind Regards,
Clare