Found invalid data while decoding

Since we moved server to Azure and put behind LoadBalancer with open ports 80,443 and 10943 this error start appearing randomly:
Found invalid data while decoding.

Server exception:
System.IO.InvalidDataException: Found invalid data while decoding.
at System.IO.Compression.InflaterZlib.Inflate(FlushCode flushCode)
at System.IO.Compression.InflaterZlib.ReadInflateOutput(Byte[] outputBuffer, Int32 offset, Int32 length, FlushCode flushCode, Int32& bytesRead)
at System.IO.Compression.InflaterZlib.Inflate(Byte[] bytes, Int32 offset, Int32 length)
at System.IO.Compression.DeflateStream.Read(Byte[] array, Int32 offset, Int32 count)
at System.IO.BinaryReader.FillBuffer(Int32 numBytes)
at System.IO.BinaryReader.ReadInt32()
at Newtonsoft.Json.Bson.BsonDataReader.ReadNormal()
at Newtonsoft.Json.Bson.BsonDataReader.Read()
at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
at Halibut.Transport.Protocol.MessageExchangeStream.ReadBsonMessageT in C:\buildAgent\work\fe2b45bbd4978f75\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 250
at Halibut.Transport.Protocol.MessageExchangeStream.ReceiveT in C:\buildAgent\work\fe2b45bbd4978f75\source\Halibut\Transport\Protocol\MessageExchangeStream.cs:line 207
at Halibut.Transport.Protocol.MessageExchangeProtocol.d__18.MoveNext() in C:\buildAgent\work\fe2b45bbd4978f75\source\Halibut\Transport\Protocol\MessageExchangeProtocol.cs:line 231

We are using agents in the pull mode.

Any ideas?

Hi Tomas,

Thanks for getting in touch! I’m sorry to hear you’re starting to hit this error randomly! My first thought is maybe they’re polling the load balancer itself, where they should be instead polling the servers behind the load balancer. Could you check in your Tentacle.config file if they can poll all of the server nodes?

There’s some additional information in the following doc page, and if applicable it outlines options on adding servers via the command line and directly in the config file.

Let me know how you go, or if you have any further questions going forward. :slight_smile:

Best regards,

Kenny

Hm… Our 2 servers are behind the load balancer and agents are in the poling mode… Is there a way to configure that it work properly? So if I understand correctly, tentacle must be checking with both severs at the same time? If we expose those machines vlwith public IP and open only port 10943 and register them in the config of tentacle. Would it solve the problem?

Hi Tomas,

My sincere apologies that this thread slipped through the cracks! Have you been able to solve this issue? You’re correct that polling Tentacles must be configured to poll both servers. The following section in the documentation outlines how you can add this to all servers, either through the command line or directly in the Tentacle.config file.

I hope this helps! Let me know if you have any questions or concerns moving forward. :slight_smile:

Best regards,

Kenny

Hi Kenneth,

No, the issue is still there. We have replaced load balacer with traffic manager, so now it talks to a single server via his external IP.

However, the issue is there and its really annoying…

Oh common man… I will have to wait another month in order to get some suppot???

Hi Tomas,

Thanks for keeping in touch! You certainly won’t have to wait that long. :slight_smile:

I was investigating another report of this same error at the end of last week, so my apologies about the delay here. Looking through the task logs of the other issue, it looks like it might end up being due to a network-related issue like a proxy that’s causing this intermittent error while running a custom script in their process. Is it possible something like that is interfering in your case?

I’d like to get more context around this error to help continue troubleshooting. Would you be willing to supply two separate task logs, one from a failure showing this error, and one from a successful deployment of this same project? Feel free to send it privately here, or email directly at support@octopus.com and I can grab them there.

I look forward to hearing back and getting to the bottom of this one!

Best regards,

Kenny

I sent logs already several times. Sent it again. Can you pass this case to someone who can deal with it ASAP?

We have to deliver a project and Octopus is crashing… with no clear reason.

Really disappointing.

Hi Tomas,

I appreciate you emailing us with the logs. I’ve pulled in some engineers on this issue and we can continue the investigation and discussion in the email thread.

Kind regards,

Kenny