Models for working with exported data

Legacy Migration
1

Hi,

Question
Is there a NuGet package I can install to get models for Octopus?
For example I’m looking for model of Project. I would like to deserialize projects exported to JSON by the Octopus.Migrator.exe.

Tried
I have looked at Model.ProjectResource in Octopus.Client, but some properties (for example UsedPackages and LogoAttachmentKey) isn’t included in that?

About my env
C#
Octopus version: 2018.9.3
Octopus.Client version: 4.42.2
(we are doing this as a step in our migration of data from one instance to another and for the moment we have created models manually, but it would be nice to use your models.)

Thanks!

3

Hi Tobias,

Thanks for getting in touch. The recommended way to access the data is via the API, using Octopus.Client. We work very hard to ensure we don’t break compatibility on the API, so anything you build against it should work across different versions of Octopus. The models used by migrator are internal to Octopus, we don’t publish them and we don’t recommend people try to use them, and we don’t ensure backward compatibility on them.

All that being said, you obviously have a need to access information that you can’t currently get through the API. Are you able to provide further detail on your scenario to help us understand your requirements? We are happy to look at extending the capabilities of the API if we understand what customers need from it.

Look forward to hearing back from you.

Regards
Shannon

4

Hi,

Okey, thanks for response!
I write our scenario and perhaps there is a better solution to handle this.

We have two instances of Octopus, one for QA/Stage, and one isolated for Production. Instances are isolated but have a common NuGet feed for packages.

We looked at different options but ended up with testing:
Octopus 1:

  1. Use Migrator.exe partial export of projects. (exclude deployments, task logs, machines and tenants)
  2. Push this data to common NuGet feed

Octopus 2:

  1. Get data from NuGet feed
  2. Use Migrator.exe to import

When we did this we found data we didn’t want to import:

  • Accounts
  • TagSets
  • Values of old variables
    (we would like to import new variables. But we don’t want to override values of old variables that has an other value in Production.)
  • Old release variable snapshots
    Dont want to override Production values.
  • Life cycles, we only want to use default in Production.

How we doing it now (test phase)
Before import with Migrator.exe we also export files from Production. And then we run an application that merge/edit exported data. And after that we run import.

“Merge/edit application” performs following actions:

  • Remove JSON files for Accounts
  • Remove JSON files for TagSets
  • Merge JSON files for variables
    (replace value of variable if it already exists in Production)
  • Merge release variable snapshots also
  • Looking at this for the moment but the idea was to remove life cycles and set Production Default life cycle in all projects before import.

Can also ask how environments should be handled? They are of course different between the instances.

Appreciate feedback regarding this. :slight_smile:
Thanks!

7

Hi Tobias,

Thanks for keeping in touch! That’s some nicely detailed information. It sounds like you are implementing your own version of remote release promotions. I’d appreciate if you could read over that RFC and let me know where you see any gaps. Some of the details about ODCM are out of date, but the main concept is still the same.

We haven’t started work on this feature set yet, so I’d like to work with you to find a short-term solution depending on your timing.

Hope that helps!
Mike

8

Yes, that’s correct. As you probably can tell we are really looking forward to Remote Release Promotion :slight_smile:
But we also need a way to handle this now.

Our environments are segregated for security purposes.

Some comments regarding the RFC:

  1. I like the way you think about spaces

  2. Remote environments. If Octopus knows about a remote space, isn’t it possible to know about the environments in that space also? So you don’t have to add them manually.

  3. I agree that Project triggers, Tenants and Channels not should be imported. For us also Accounts and TagSets. If TagSets not are imported you also have to think about what should happen with variables that has a scope for a specific tag. And that could be the same for channels and targets.

  4. Tenants. We do have separated tenants for our “spaces”. So I agree they should be owned by there space, and for us no need to be connected between spaces.

  5. OCDM would be overkill for us. So it would be nice with this feature in “regular” Octopus.

  6. Overall: Sounds like a really nice solution that would be perfect for our scenario!

/Tobias

9

I think we have a solution that works for us now.
Perhaps it can give you input to Remote Release Promotion but I would also appreciate your feedback if it seems like an ok way to handle it. As you understand we are trying to avoid manual work :slight_smile:

In our MergeApplication:
Inputs:

  • Migrator.exe exported files from “Stage instance”
  • Migrator.exe exported files from “Production instance”

Application edit exported files from “Stage instance”:

  1. Remove Accounts
  2. Remove TagSets
  3. Remove all channels except “Default”
  4. Remove 3 environments that not will be used in Prod
  5. Map environment “QA” to “PreProduction” (change name)
  6. Map environment “Stage” to “Production” (change name)
  7. Map lifecycle “QA Stage” to “PreProduction Production” (change name)
    7.1. Map phases in lifecycle (change names)
  8. Remove all lifecycles except “Default Lifecycle” and the one mapped in step 7
  9. Merge project variables:
    9.1. Remove all TenantTags from scopes
    9.2. Replace values with values from Prod
  10. Merge projects LifecycleId:
    10.1. If new project: Set to Default Lifecycle
    10.2. If existing project: Set Lifecycle from Prod (Default or the one in step 7)
  11. Merge Library VariablesSets:
    11.1. Remove all TenantTags from scopes
    11.2. Replace values with values from Prod
  12. Merge Release Snapshots variables:
    12.1. Remove all TenantTags from scopes
    12.2. Replace values with values from Prod
    12.3. Set version from Prod
  13. Save all new Releases to a file (for use in step 15)

Octopus.Migrator.exe:

  1. Run Import

REST API:

  1. Update variable snapshots for all new releases (from step 12)

To be able to get information of changes in exported models (when new versions of Octopus) we throw an error if JSON files misses/contains properties that we have/haven’t included in our MergeApplication.

Thanks
/Tobias

10

Hi Tobias,

Replying to your comments on the RFC for Remote Release Promotions…

  1. Neat, Spaces is coming in Q1 2019, which should be Octopus Server 2019.1.
  2. It is possible to know about the remote environments, however our initial plan is to give you the choice on which remote environments you care about. Some people care about all the remote environments, others only care about the final production environment. I think we will start somewhere and iterate on that particular feature over time as we work with customers.
  3. Accounts and TagSets are interesting.
    a. If you have an Account scoped to say a “Test” environment, and that environment is known to be a local environment, we would not transfer that account - it only applies for local deployments. On the other hand, if you have an unscoped (global) Account, we probably have to transfer that account because it’s viable for deployments to local and remote environments. Perhaps that gives people the control they need?
    b. TagSets are different in that way since they are always global, and if they are used to decide which variable values apply or actions to run for a particular deployment, we have to make a choice about transferring the TagSet and keeping the variables/process intact, or not-transferring the TagSet and mutating the variables/process.
  4. We agree also. The TagSets being an interesting divergence in our thinking.
  5. We aren’t pursuing ODCM any more. We’ve baked Spaces into Octopus and it will be shipping in Q1 2019.
  6. Nice!

Thanks for your feedback. :slight_smile:
Mike

11

Hi Tobias,

Replying to your description of the current solution. Wow! That’s a whole lot of merging - you’ve gone to a lot of effort to understand the Octopus model. If the solution is working just the way you want right now, that’s great and you can probably keep using it.

I’ll throw a few thoughts into the mix in case it helps. I’d also like to offer to spend some time on a screenshare with you if you think that would help as well. :slight_smile:

Firstly, some questions and thoughts overall:

  1. Why did you choose the migrator export -> files -> merge -> migrator import path instead of using the HTTP API to acheive the same result? To offer some insight: we go to great lengths to maintain compatibility for the HTTP API and provide our C# SDK as another layer of compatibility. We don’t think about the migrator file format in quite the same way - we try not to break it, but we treat it as a private protocol subject to change… :thinking: My best guess is because you are bundling up the files to transmit, and the migrator already handles the file writes etc.
  2. The places where you have to merge sound painful, but necessary based on how the migrator was designed. It won’t import things which were not exported, which can be helpful, but it will fail if the Lifecycle for a Project is missing. Which is probably why you need to do the remapping/merging yourself. I can’t think of anything better for this right now, short of us building RRP for real.

Thanks for sending all that through. You’ve certainly given me a lot to think about!
Mike

12

Two separate discussions in same thread (with a title that doesn’t reflect any of them), not the best way, but because we have started it :slight_smile:

Replying to comments about RFC:
3. For both Accounts and TagSets we use them different between our two instances, so we don’t have any global. But I can understand other scenarios where you have global accounts/tagsets.

And regarding our merge solution:
We started to use the Migrator and from the beginning we only removed Accounts and TagSets, a way that is mentioned here: Redirecting to https://octopus.com/docs/administration/data/data-migration

“Transfer it to a new Octopus Server - You can delete files you don’t want to import (e.g., if you’re transferring one project, just delete everything except the files for that project)”

Also we found recommendations of Migrator to keep instances in sync here Redirecting to https://octopus.com/docs/installation/isolated-octopus-deploy-servers.

“The migration tool could be reused periodically to keep both servers in sync. Because the resulting export is simply a collection of JSON files in folders, a source control system like Git could be used for this purpose. Any imports subsequent to the initial import would result in a merge of any changes.”

And with discussions with you it felt like a good solution. But, like I said, from the beginning we didn’t know all the things that didn’t work out.

So during our setup we have ran in to things we needed to fix to be able to automate a continuously sync between our instances. Because of that our MergeApplication has grown during implementation. During this period we haven’t really looked into/thought about other alternatives, because we have been able to solve things, step by step.

Now when we (think) we know what we have to do I guess its good to look at alternatives before going live with it. Do you think we can get this working with API and C# SDK? That would probably be a more solid solution. One thing to have in mind is that our two instances cannot communicate with each other (but can get packages from the same external NuGet feed).

I appreciate all feedback and are happy to give input regarding RRP. We could have a screenshare to look at it more hands on.

Again, thanks!
/Tobias

13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

14

Hi Tobias,

Oh my, I dropped the ball on this and missed your reply! Would you still like to touch base?

Mike