I am evaluating options of how to copy files to AWS Windows FSX share from on prem Octopus server. Is this possible and are there any examples/documentation around same? Most of the online documents suggest using netshare but I can’t use that from octopus because I am not sure where it would mount that drive to on octopus worker nodes. and is it good practice.
Hi @harsh_tech,
Thank you for reaching out to us with your question about copying files from your on-prem Octopus Server to AWS FSx.
I would be keen to understand further what files you are copying from your Octopus Server and why. For example, are they Octopus Log files? Are they packages? Are they files that sit outside of Octopus?
This will help to provide a bigger picture of what it is you are looking to achieve so I can help to offer some advice.
While I am no expert in AWS there may be a few things for you to explore here (depending on what you need to transfer).
Option 1:
You may want to explore the Built-in step to run the AWS CLI AWS CLI and PowerShell scripts - Octopus Deploy or even look at creating your own Custom Step Template to achieve this if the built-in step is insufficient for your needs.
For a custom step, you could use the Run a Script template to create a step that leverages using the AWS CLI to copy files from on-prem to FSx. You could run this from a Worker where the AWS CLI is installed and has access to AWS services and network permissions so the Worker can access the shared folder where the files reside.
Option 2:
Outside of Octopus, you may find using AWS DatSync can achieve what you might be after but keep in mind any costs associated with Amazon for their services.
Option 3:
If these are Octopus specific files for HA cluster i.e. Task logs, Artifacts, Nuget Repository then we do have a guide to Configuring Octopus Server High Availability using AWS FSx
Above are a few suggestions that come to mind until I know a little bit more detail about your end goal here.
I look forward to hearing back from you.
All the Best
Doug
Thanks @doug.pipe for details. I want to connect to one of the on-prem SFTP server, copy files from there to Artifactory and then from Artifactory want to push these files to AWS FSX. Copying the files from on-prem SFTP to Artifactory is planned to be done from Octopus so I would like rest of the steps to be done via Octopus too. I do not want to use DataSync as its not yet adopted in my organisation.
Please let me know if there are any examples from octopus side to achieve similar scenario.
Hi @harsh_tech,
Thank you for getting back to me and outlining your scenario.
I have had a look around our Samples and REST example scripts and there isn’t really much for an example that I can share with you of what you are looking to achieve.
If you are comfortable sharing, I would like to know more about the history of your process and how you are using this (what is the use case for it?).
I would like to make sure I can offer you my best advice.
Ideally, what we find is having the data stored in a centralized place to be more efficient, but there may be business reasons I am unaware of to move these between 3 places, such as DR, security or availability so I don’t want to make any assumptions here.
While Octopus could help to orchestrate some of this by using some of your own scripts as custom steps to leverage the AWS and Artifactory APIs. However, Octopus isn’t really designed to move files around from one place to another in the way that you have asked.
At this time, if using Octopus is your main goal, it feels like your best option is to script this into Custom Steps within Octopus and use the Artifactory REST API to help move these files to FSx. I can’t really advise you about Artifactory I am afraid, so you may want to reach out to them to see what their thoughts are.
I hope this helps
Doug
Thanks @doug.pipe . May be out of all these, let me ask very specific question. Can I use below command from 'Run Script" step running on internal worker node? would the mounting drive work like from Octopus step template?
net use e: \\fsxshare\test password01 /user:clouddomain\user1 /savecred /persistent:no
Hi @harsh_tech,
Apologies for the delay in response, due to public holidays here in the UK I have been away.
For your question, in theory you should be able to set this up using the “Run A Script” step within Octopus.
For this step you have the ability to run a script using your preferred method.
The current options are:
- PowerShell
- Bash
- C#
- F#
- Python3
If you are wanting to use the Net Use command, you can do so using PowerShell as this will treat this command the same.
If possible you may also want consider using the PowerShell cmdlet New-PSDrive to also help achieve this.
I ran a mock test using New-PSDrive in my test environment which looked like this:
This will create a temporary drive while the PowerShell session is running. You could look to use the -Persist switch if you prefer (you will not be able to run the New-PSDrive if the drive already exists, something you would need to factor in with your code). Using this method I will be able to run my Runbook multiple times.
You will also see that I have taken advantage of using stored variables within Octopus to store my password and username.
For the credentials I am leveraging Get-Credential
If you need to set up the mapped network drive as a 1 time thing, I would suggest maybe doing this as outside Octopus for simplicity and use your Runbook to move the files to their destinations.
Above are just some potential ideas for you, I hope you may find use in this.
Many Thanks
Doug
1 Like