Octopus Database "Padding is invalid and canot be removed"

I’m setting up octopus server as a container in EKS, and I have an MSSQL Express database setup in RDS. I created an empty database, octodb, in Express, and started up the container with the connectionstring. It blows up when trying to initialize the database. Here is the log output:

======== Initializing DB ========
Setting database connection string to: server=address,1433;database=octodb;user id=user;password=********
========== SQL Database Connection Details ========
SQL Server: address,1433
SQL Database: octodb
Authentication Type: SQL Server Credentials
User: octopus

===================================================
Testing connection to the 'master' database...
Successfully connected to 'master' database!
Database 'octodb' already exists on SQL Server at 'address,1433', no need to create it.
Checking the Octopus Master Key has been configured.
Generating a new Master Key for this Octopus Server...
Master Key saved; IMPORTANT: use the Octopus Manager or 'Octopus.Server.exe show-master-key' to back the key up.
Making sure it's safe to upgrade the database schema...
Ensuring pre-conditions for upgrading the database are satisfied...
Searching for indexes that might upset the database upgrade process...
- PASS: All columns use the default collation.
- PASS: Your Octopus Server will be compliant with your license after upgrading.
- PASS: We've done our best to remove any unexpected database indexes.
- PASS: The version of your SQL Server satisfies Octopus Server installation requirements.
Executing always run pre scripts...
Executing TSQL Database Server script 'Octopus.Core.UpgradeScriptsAlwaysPre.Script0000 - Database options.sql'
Ensuring COMPATIBILITY_LEVEL for octodb is set to 140
COMPATIBILITY_LEVEL for octodb is already 140 or higher
Checking to see if database schema upgrade is required...
Database already has the expected schema. No changes are required.
Executing always run post scripts...
Executing TSQL Database Server script 'Octopus.Core.UpgradeScriptsAlwaysPost.Script0000 - Refresh Views.sql'
Refreshing view dbo.Dashboard
Refreshing view dbo.IdsInUse
Refreshing view dbo.MultiTenancyDashboard
Refreshing view dbo.Release_WithDeploymentProcess
Refreshing view dbo.RunbookSnapshot_WithRunbookProcess
Refreshing view dbo.TenantProject
===============================================================================
Error reading row 1, column 1. Padding is invalid and cannot be removed..
Compiled reader expression:

(DbDataReader reader, DocumentReaderContext context) => 
{
    Type deserializeAsType = Octopus.Core.Model.Configuration.CertificateConfiguration
    context.Column = 0
    String temp0 = IIF(reader.IsDBNull(0), null, reader.GetString(0))
    context.Column = 1
    CertificateConfiguration deserializedFromJson = context.DeserializeText(reader, 1, deserializeAsType)
    CertificateConfiguration result = deserializedFromJson
    if (result != null)
    {
    }

    result
}

Nevermore.ReaderException
   at Nevermore.Advanced.ReaderStrategies.Documents.DocumentReaderStrategy.<>c__DisplayClass4_1`1.<CreateReader>b__1(DbDataReader dbDataReader)
   at Nevermore.Advanced.ReadTransaction.ProcessReader[TRecord](DbDataReader reader, PreparedCommand command)+MoveNext()
   at Nevermore.Advanced.ReadTransaction.Stream[TRecord](PreparedCommand command)+MoveNext()
   at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Boolean& found)
   at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
   at Nevermore.Advanced.ReadTransaction.Load[TDocument](String id)
   at Octopus.Core.RelationalStorage.RawRelationalTransaction.Load[TDocument](String id) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Core\RelationalStorage\RawRelationalTransaction.cs:line 42
   at Octopus.Core.Initialization.BuiltIn.OctopusCertificateInitializer.Initialize(IRawRelationalStore store) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Core\Initialization\BuiltIn\OctopusCertificateInitializer.cs:line 40
   at Octopus.Core.Initialization.StoreInitializer.Initialize() in C:\buildAgent\work\7881e50627116cb\source\Octopus.Core\Initialization\StoreInitializer.cs:line 22
   at Octopus.Server.Commands.DatabaseCommand.Start() in C:\buildAgent\work\7881e50627116cb\source\Octopus.Server\Commands\DatabaseCommand.cs:line 203
   at Octopus.Shared.Startup.AbstractCommand.Start(String[] commandLineArguments, ICommandRuntime commandRuntime, OptionSet commonOptions)
   at Octopus.Shared.Startup.OctopusProgram.Start(ICommandRuntime commandRuntime)
   at Octopus.Shared.Startup.ConsoleHost.Run(Action`1 start, Action shutdown)
   at Octopus.Shared.Startup.OctopusProgram.RunHost(ICommandHost host)
   at Octopus.Shared.Startup.OctopusProgram.Run()

--Inner Exception--
Padding is invalid and cannot be removed.
System.Security.Cryptography.CryptographicException
   at Internal.Cryptography.UniversalCryptoDecryptor.DepadBlock(Byte[] block, Int32 offset, Int32 count)
   at Internal.Cryptography.UniversalCryptoDecryptor.UncheckedTransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount)
   at Internal.Cryptography.UniversalCryptoTransform.TransformFinalBlock(Byte[] inputBuffer, Int32 inputOffset, Int32 inputCount)
   at System.Security.Cryptography.CryptoStream.FlushFinalBlockAsync(Boolean useAsync)
   at System.Security.Cryptography.CryptoStream.FlushFinalBlock()
   at System.Security.Cryptography.CryptoStream.Dispose(Boolean disposing)
   at System.IO.Stream.Close()
   at System.IO.Stream.Dispose()
   at Octopus.Core.Security.MasterKey.MasterKeyEncryption.ToPlaintext(Byte[] masterKey, EncryptedBytes encrypted) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Core\Security\MasterKey\MasterKeyEncryption.cs:line 84
   at Octopus.Core.Security.MasterKey.StoredMasterKeyEncryption.ToPlaintext(EncryptedBytes encrypted) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Core\Security\MasterKey\StoredMasterKeyEncryption.cs:line 72
   at Octopus.Core.Security.MasterKey.MasterKeyEncryptionExtensions.ToPlaintextString(IMasterKeyEncryption encryption, EncryptedBytes encrypted) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Core\Security\MasterKey\MasterKeyEncryptionExtensions.cs:line 21
   at Octopus.Core.RelationalStorage.TypeHandlers.SensitiveStringTypeHandler.ReadJson(JsonReader reader, Type objectType, Object existingValue, JsonSerializer serializer) in C:\buildAgent\work\7881e50627116cb\source\Octopus.Core\RelationalStorage\TypeHandlers\SensitiveStringTypeHandler.cs:line 74
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.DeserializeConvertable(JsonConverter converter, JsonReader reader, Type objectType, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue(JsonProperty property, JsonConverter propertyConverter, JsonContainerContract containerContract, JsonProperty containerProperty, JsonReader reader, Object target)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize(JsonReader reader, Type objectType)
   at Nevermore.Advanced.Serialization.NewtonsoftDocumentSerializer.DeserializeSmallText(String text, Type type)
   at Nevermore.Advanced.ReaderStrategies.Documents.DocumentReaderContext.DeserializeText[TDocument](DbDataReader reader, Int32 index, Type concreteType)
   at lambda_method(Closure , DbDataReader , DocumentReaderContext )
   at Nevermore.Advanced.ReaderStrategies.Documents.DocumentReaderStrategy.<>c__DisplayClass4_1`1.<CreateReader>b__1(DbDataReader dbDataReader)
-------------------------------------------------------------------------------
Terminating process with exit code 100
Full error details are available in the log files at:
/home/octopus/.octopus/OctopusServer/Server/Logs
Octopus/Logs
If you need help, please send these log files to https://octopus.com/support
-------------------------------------------------------------------------------

A little google came up with results regarding a migration and versions of Octopus being off, but this is a fresh, empty database. Any help would be appreciated.

Hi @eandrus,

Thank you for contacting Octopus Support.

Have you tried allowing Octopus to create the db during the install? If you did and it errored as well, did you get the same error message?

I look forward to hearing back from you.

Regards,
Donny

@donny.bell I actually just got done testing that. I told the container to use a completely different database in its connection string, one that didn’t exist yet. The log output was exactly the same, just with the new database name in it. I found that weird, given that one of the lines was

Database ‘octodb’ already exists on SQL Server at ‘address,1433’, no need to create it.

But with the new database name that I had not created. Any help would be appreciated. One of my team suspects a problem with password encryption, after some googling.

Hi @eandrus,

Thank you for getting back to me.

Could you tell me what version of Octopus Server you are using to create this installation?

Regards,
Donny

@donny.bell I just pulled the :latest octopusdeploy/octopusdeploy image

@donny.bell just an update to my last message, this should be the version I’m pulling:

https://hub.docker.com/layers/octopusdeploy/octopusdeploy/latest/images/sha256-c6f19d8e2d3026d46a8ff9cdb8ec8b4b6134ff4bdb85cf555a61f35eef4e8ce6?context=explore

Hi @eandrus,

Thank you for getting back to me.

If you haven’t already, double-check that C:\Octopus and C:\Programs Files\Octopus Deploy do not exist prior to installing.

Also, are you enabling encryption in RDS? If so, try turning that off to see if you get a different result.

Let me know what you think.

Regards,
Donny

Hi @eandrus,

I set up a test environment to see if I could replicate the issue.

The error you are receiving is a result of a new instance attempting to connect to an existing SQL db without the correct master key in a 2020.3+ installation. I recommend double-checking the parameters in your script to see if something is misconfigured.

If you would like to share a scrubbed version of your script, I would be happy to run it on my end as well to test. Just let me know.

Regards,
Donny

@donny.bell I will pursue the disabling encryption on RDS, but in the meantime, here is my deployment yaml for Octopus on EKS. I’m already creating the PVCs in a separate yaml, and they seem to work fine.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: octopus-server
  namespace: octopus-server
spec:
  selector:
    matchLabels:
      app: octopus-server
  serviceName: "octopus-server"
  replicas: 2
  template:
    metadata:
      labels:
        app: octopus-server
    spec:
      serviceAccountName: octopus-server
      terminationGracePeriodSeconds: 10
      volumes:
      - name: repository-vol
        persistentVolumeClaim:
          claimName: repository-claim
      - name: artifacts-vol
        persistentVolumeClaim:
          claimName: artifacts-claim
      - name: task-logs-vol
        persistentVolumeClaim:
          claimName: task-logs-claim
      containers:
      - name: octopus-server
        image: octopusdeploy/octopusdeploy:2020.3.4-linux
        securityContext:
          privileged: false
        env:
          - name: ACCEPT_EULA
            value: "Y"
          - name: OCTOPUS_SERVER_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: DB_CONNECTION_STRING
            value: "Server=databaseaddress,1433; Database=octodb; User Id=user; Password=password;"
          - name: ADMIN_USERNAME
            value: octo
          - name: ADMIN_PASSWORD
            value: adminpassword
          - name: OCTOPUS_SERVER_BASE64_LICENSE
            value: "licensekey"
          # - name: MASTER_KEY
          #   value: I'm telling octopus to use a database that doesn't exist, yet when it creates it (and it does in fact create one), I still get the padding error.  I don't know how to get the master key out of an existing database on an RDS instance in order to provide it to Octopus on an existing database to test that way
          - name: DISABLE_DIND
            value: "Y"
        ports:
        - containerPort: 8080
          name: octo-web
        - containerPort: 10943
          name: tentacle
        volumeMounts:
        - name: repository-vol
          mountPath: /repository
        - name: artifacts-vol                                                  
          mountPath: /artifacts
        - name: task-logs-vol
          mountPath: /taskLogs
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/bash
              - -c
              - '[[ -f /Octopus/Octopus.Server ]] && EXE="/Octopus/Octopus.Server" || EXE="dotnet /Octopus/Octopus.Server.dll"; $EXE node --instance=OctopusServer --drain=true --wait=600 --cancel-tasks;'
          # postStart must finish in 5 minutes or the container will fail to create
          postStart:
            exec:
              command:
              - /bin/bash
              - -c
              - 'URL=http://localhost:8080; x=0; while [ $x -lt 9 ]; do response=$(/usr/bin/curl -k $URL/api/octopusservernodes/ping --write-out %{http_code} --silent --output /dev/null); if [ "$response" -ge 200 ] && [ "$response" -le 299 ]; then break; fi; if [ "$response" -eq 418 ]; then [[ -f /Octopus/Octopus.Server ]] && EXE="/Octopus/Octopus.Server" || EXE="dotnet /Octopus/Octopus.Server.dll"; $EXE node --instance=OctopusServer --drain=false; now=$(date); echo "${now} Server cancelling drain mode." break; fi; now=$(date); echo "${now} Server is not ready, can not disable drain mode."; sleep 30; done;'
        readinessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - URL=http://localhost:8080; response=$(/usr/bin/curl -k $URL/api/serverstatus/hosted/internal --write-out %{http_code} --silent --output /dev/null); /usr/bin/test "$response" -ge 200 && /usr/bin/test "$response" -le 299 || /usr/bin/test
          initialDelaySeconds: 30
          periodSeconds: 30
          timeoutSeconds: 5
          failureThreshold: 60
        livenessProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - URL=http://localhost:8080; response=$(/usr/bin/curl -k $URL/api/octopusservernodes/ping --write-out %{http_code} --silent --output /dev/null); /usr/bin/test "$response" -ge 200 && /usr/bin/test "$response" -le 299 || /usr/bin/test "$response" -eq 418
          periodSeconds: 30
          timeoutSeconds: 5
          failureThreshold: 10
        startupProbe:
          exec:
            command:
            - /bin/bash
            - -c
            - URL=http://localhost:8080; response=$(/usr/bin/curl -k $URL/api/octopusservernodes/ping --write-out %{http_code} --silent --output /dev/null); /usr/bin/test "$response" -ge 200 && /usr/bin/test "$response" -le 299 || /usr/bin/test "$response" -eq 418
          failureThreshold: 30
          periodSeconds: 60

That’s how I’m deploying it.

@donny.bell I attempted to disable encryption on the database as you suggested, and got the following:

image

So not sure why Octopus can’t seem to set the master key.

Hi @eandrus,

I just wanted to update you.

I am still attempting to reproduce this error. I should have more information for you by tomorrow.

Thank you for your patience while we get this figured out.

Regards,
Donny

Hi @eandrus,

Thank you for your patience. I was able to successfully reproduce your issue. I’m currently attempting to see if a fix that is already in the pipe-line for 2020.3.5 corrects this issue.

I will confirm with our engineering team in Australia to see if that is the case.

To get you unblocked, I would recommend trying 2020.2.18 instead, then wait until 2020.3.5 before upgrading:

octopusdeploy/octopusdeploy:2020.2.18

Let me know if 2020.2.18 works for you.

Regards,
Donny