Secure deployment to PCI DSS environment without a separate server in the PCI DSS environment

Legacy Uncategorised
1

I know that the current “best practice” for Octopus Deploy in a PCI DSS environment is to have two separate servers; one in the PCI DSS environment and one in a more typical development environment. There’s also suggestions of having separate feeds in Octopus Deploy, but it all feels unnecessarily clunky and bothersome.

According to my understanding of PCI DSS, it doesn’t require a setup as described above. It requires network segmentation, management approval of implemented changes and that the one deploying the changes is not the same person who authored them.

Network segmentation is solved well enough, I believe, by having the networks physically disjointed and by Octopus Deploy’s “environment” support. There is a severe lack of ACL on a per-environment / per-project basis in Octopus Deploy that I believe needs to be addressed, but not for having a secure and verifiable PCI DSS deployment pipeline, I think.

On to the next point; management approval can be solved through authentication and signing. Here is where I think Octopus Deploy in its current state may be lacking features. It would be great if the act of either creating a release or at least promoting it to an environment, could require a manual step. This step should not only be required to be performed by a specific group in Octopus Deploy, but since the Octopus Deploy server itself is not within PCI DSS, the verification itself needs to be done with PKI towards the PCI DSS environment (SSH springs to mind, but there are many viable options). I’m thinking this could be done as a manual step with an input parameter, where the parameter would be an OTP and perhaps a public key.

The last point; proving the identity of the one who built the code: I believe this can be done through code signing. With Git, everything can be signed all the way up to the individual commits or at least the tag representing the commit being created as a release in Octopus Deploy and with NuGet, packages can be signed as well. If the package contains the signed code and the package itself is signed, we can verify the signatures and at least be sure that there’s two different signatures. If the signatures are coupled with PKI that is verified by the PCI DSS environment, we can be pretty sure the signatures belong to different people as well.

So, I’m basically wondering if the above description is something you agree could be approved according to PCI DSS, and more importantly; whether this is something that could be sketched up with Octopus Deploy? Is NuGet code signing supported, for instance? If not, can it be performed by a custom step in Octopus Deploy? How about creating a custom step asking the user for credentials and then verify those credentials against a network service before continuing the deploy?

Also, is it possible for Octopus Deploy tentacles to verify that the package they receive from Octopus adheres to a set of specified criteria, such as verified code signature, etc.?

2

Hi Asbjørn,

You raise some interesting questions. Hopefully I can address them.

But first I need to mention that the one question I can’t answer is whether a scenario such as you describe would be approved for PCI DSS compliance. Because:

a) What we think should be PCI DSS compliant is of little relevance. The only opinion that has weight is that of your auditor :slight_smile:

b) In our experience there are few absolute rules. It depends on your environment, expertise (e.g. what you can effectively argue), and of course the auditor. For example, from the PCI DSS document:

At a high level, adequate network segmentation isolates systems that store,
process, or transmit cardholder data from those that do not. However, the adequacy of a specific implementation of network segmentation is highly
variable and dependent upon a number of factors, such as a given network’s configuration, the technologies deployed, and other controls that may
be implemented.

This is obviously open to interpretation.

My understanding is that Network Segmentation is not required; it is is used to remove systems from the PCI DSS scope.
I don’t believe that having an Octopus instance outside the DCE, capable of deploying to Tentacles within the DCE, means you can’t achieve compliance. It just potentially brings your Octoups instance into scope. You then need to make the case that your usage of Octopus is compliant.
Which leads to your other questions…

Regarding your comment:

There is a severe lack of ACL on a per-environment / per-project basis in Octopus Deploy

I would be interested to know the details of what you believe is missing. You can restrict Teams’ access to specific Environments and Projects. Do you feel that is not granular enough?

Regarding approvals:
There is currently no ability to request approval upon release creation. This is not a feature that is often requested, as releases generally aren’t specific to an environment.
Approval for promoting a release can be achieved using Manual Intervention steps, as you obviously already understand. And your idea to use a Manual Intervention step to capture a key which is then used to authenticate is certainly plausible.

Regarding package-signing:
We have no built-in solution for this (we do have an open UserVoice issue). I don’t believe the community has settled on a solution for this. I can’t find any reference that the NuGet Package Signing spec has actually been implemented?
It really depends what you are trying to achieve.
You can restrict the ability to push packages to the Octopus package repository to, for example, only the account that builds your solution. If only your build account has access to push packages, then by definition that answers the question of who compiled and pushed the package (and you could potentially use an external feed such as MyGet if you need multiple feeds with different permissions).
Any package signing would have to be implemented by you. I don’t personally know of any instances where people have implemented this, but I will ask the team here. But of course anything you can script can be run as a custom Script Step.
It is not possible to prevent the Tentacle from downloading the package, but you can certainly abort the deployment if your custom step doesn’t like the package.

I wish I could provide a definite answer on what is required for compliance. We have many customers who have achieved compliance, but we don’t often receive feedback on the details of how.

A couple of extra points:

  • Do you use Active Directory? Octopus can integrate with Active Directory for authentication, and having multiple AD domains with a one-way trust would (I believe) potentially assist with your environment permissions and remove the need for signing of approvals.

  • In Octopus 3.4 we released support for using a proxy for Tentacle-Server communications. This may be irrelevant, but it does enable certain network architectures.

We are more than happy to provide whatever assistance we can. Please don’t hesitate to ask. I’d be interested to hear about your progress.
As mentioned, I’ll discuss the package-signing issue with the team, in case anyone has any nice ideas.

I hope that helps,
Michael

3

Hi, Michael. Thanks for your thorough reply.

I would be interested to know the details of what you believe is missing. You can restrict Teams’ access to specific Environments and Projects. Do you feel that is not granular enough?

I do. It’s not. What we need is for someone to be able to be “super administrators” within a team, so they have access to all the infrastructural details in Octopus (such as environments) without having access to other teams and their projects.

Having project group security would also help. Basically, I think you can learn a lot from TeamCity’s ACL implementation and granularity, where you have groups, roles and permissions, with several levels on each dimension making it all very customizable. I don’t think Octopus Deploy needs something as granular and complex as that, but something in between the current model and that of TeamCity would be nice.

There is currently no ability to request approval upon release creation. This is not a feature that is often requested, as releases generally aren’t specific to an environment.

Good point.

Approval for promoting a release can be achieved using Manual Intervention steps, as you obviously already understand. And your idea to use a Manual Intervention step to capture a key which is then used to authenticate is certainly plausible.

Yes, but for that to be a true security measure, you would have to implement required steps, no?

I can’t find any reference that the NuGet Package Signing spec has actually been implemented?

I can’t either, but given that they have specified how it should be done, you can bake your own solution based on that, no?

You can restrict the ability to push packages to the Octopus package repository to, for example, only the account that builds your solution. If only your build account has access to push packages, then by definition that answers the question of who compiled and pushed the package (and you could potentially use an external feed such as MyGet if you need multiple feeds with different permissions).

The ability to push packages is limited to a TeamCity service account. And being able to push code to the Git repository used by TeamCity of course requires credentials, but that potentially puts the entire development environment inside the CDE. We would like to avoid that.

Instead of putting all eggs in one basket and basing the verification on the authentication to the Git repository, TeamCity, Octopus or somewhere inbetween, you can add signing as a measure in every step of the way from source code up to released package, so the credentials of those who have performed the actions along the way are transported along with the package. So even if a user has his username and password compromised, his private key will hopefully be intact, making it impossible for the attacker to spoof the user’s digital signature.

Any package signing would have to be implemented by you. I don’t personally know of any instances where people have implemented this, but I will ask the team here. But of course anything you can script can be run as a custom Script Step.

As long as those steps can’t be circumvented, that could indeed work.

It is not possible to prevent the Tentacle from downloading the package, but you can certainly abort the deployment if your custom step doesn’t like the package.

As above; that would work if the custom step can’t be deleted.

Octopus can integrate with Active Directory for authentication, and having multiple AD domains with a one-way trust would (I believe) potentially assist with your environment permissions and remove the need for signing of approvals.

Good to know.

In Octopus 3.4 we released support for using a proxy for Tentacle-Server communications. This may be irrelevant, but it does enable certain network architectures.

That’s also something we can discuss. Thanks!

4

Your point on an improved security model is well taken. It is now one of our highest ranked UserVoice issues, and we will more than likely dedicate some resources to it in the not-too-distant future.
I realize this doesn’t help you at this moment.

Just for the record, Manual Intervention steps cannot be skipped.
Which as you point out, leaves the obvious problem that they can be removed. You can always check whether the deployment process has been modified in the Octopus Audit Logs. I realize this probably isn’t practical for everyday use, but it may assist in satisfying the auditors.

I spoke to our team, and no one had knowledge of anyone implementing custom package-signing (which certainly doesn’t imply that it hasn’t been done).
I agree this would be nice to ensure package contents had not been altered (i.e. digital-signing of a package hash), but I’m not sure that is solves the authentication issue. You still have to trust that the certificate’s’ private-key hasn’t been compromised, which doesn’t feel any different to trusting that the account with permissions to push hasn’t been compromised. Especially given the private-key would have to be used by the TeamCity server to sign the packages.
I understand your point about bringing the entire development domain into scope; this is where I thought perhaps Active Directory domains might help.
If you had your ‘Secure’ domain, and your ‘Dev’ domain, with a one-way trust from Dev to Secure; and your Build account (the one with permission to create packages and push them to Octopus) was defined in the Secure domain; it would still be able to be used from TeamCity and Octopus servers (because the Dev domain trusts the Secure domain). I wonder if this would restrict the scope?

The Octopus Audit Log will also reveal when packages were pushed, so you can potentially also argue that you can verify that a package with the same version has not been uploaded after the original.

5

Your point on an improved security model is well taken. It is now one of our highest ranked UserVoice issues, and we will more than likely dedicate some resources to it in the not-too-distant future.

Excellent! Just so I can follow the issue on UserVoice, can you please provide a link to it?

I realize this doesn’t help you at this moment.

I’m in no rush to have this problem solved, so incremental improvements in Octopus Deploy’s security model is going to help and be appreciated nonetheless. So please keep them coming! :slight_smile:

I understand your point about bringing the entire development domain into scope; this is where I thought perhaps Active Directory domains might help.

Yea, I’m going to discuss this internally to see if it is something that can be implemented. Thanks for the suggestion!

6

Notice:

This issue has been closed due to inactivity. If you encounter the same or a similar issue and require help, please open a new discussion (if we asked for logs or extra details in this thread, consider including them in the new thread). If you are the creator of this thread and believe it should not be closed let us know via our support email.