How to download and upload a file in Octopus?

Is it possible to download and upload the files from same “Script” template step in Octopus on the internal worker node?

Here is my script but it gets cancelled on upload -

$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$apiKey = $OctopusParameters["APIKey"]
$headers.Add("X-JFrog-Art-Api", $apiKey)

#download
Invoke-RestMethod 'https://artifactory.internal.xyz/testnuget-dev-local/Test.Web.App01-1.0.2.zip' -Headers $headers -OutFile '.\Test.Web.App01-1.0.3.zip' 

$content = Get-ChildItem .\
Write-Host ($content | Format-Table | Out-String)

#Upload
Invoke-RestMethod 'https://artifactory.internal.xyz/testnuget-dev-local/Test.Web.App01' -Headers $headers -Method Put -InFile '.\Test.Web.App01-1.0.3.zip'

I get below error on upload line -

NotSpecified: The request was aborted: The request was canceled. 
May 23rd 2022 15:09:44Error
At E:\Octopus.Tentacle.v4\XXXXXXXXXXXXXXXXX\Work\XXXXX-10\Script.ps1:10 char:1 
May 23rd 2022 15:09:44Error
+ Invoke-RestMethod 'https://artifactory.internal.xyz/testnuget-dev-local ... 
May 23rd 2022 15:09:44Error
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
May 23rd 2022 15:09:44Error
at <ScriptBlock>, E:\Octopus.Tentacle.v4\xxxxx\Work\xxxxxx-10\Script.ps1: line 10 
May 23rd 2022 15:09:44Error
at <ScriptBlock>, <No file>: line 1 
May 23rd 2022 15:09:44Error
at <ScriptBlock>, E:\Octopus.Tentacle.v4\v4-xxxxxx\Work\xxxxxxx-10\Octopus.FunctionAppenderContext.ps1: line 201 
May 23rd 2022 15:09:44Error
at <ScriptBlock>, E:\Octopus.Tentacle.v4\v4-xxxxx\Work\xxxxxx-10\Bootstrap.Octopus.FunctionAppenderContext.ps1: line 1468 
May 23rd 2022 15:09:44Error
at <ScriptBlock>, <No file>: line 1 
May 23rd 2022 15:09:44Error
at <ScriptBlock>, <No file>: line 1 
May 23rd 2022 15:09:44Fatal
The remote script failed with exit code 1 
May 23rd 2022 15:09:44Fatal
The action Run a Script on a Worker failed

Good morning @harsh_tech,

Thank you for contacting Octopus Support and sorry to hear one of your scripts is giving you some issues.

I am pretty sure you should be able to perform a download and an upload in the same script as long as the download actually executes correctly.

One thing I did notice is, that in your upload command you have two apostrophes before your artifactory link. I am wondering whether changing it to just one then allows you to upload correctly?

Let me know if that works for you, if it doesn’t I will do some testing my end to see if I can get this working.

Kind Regards,

Clare Martin

Thanks @clare.martin , actually my bad, I accidently added it in the post but I dont have it in my actual script. Let me correct it in post but the error is same as I do not have it in my original script in octopus.

Hey @harsh_tech,

Thanks for confirming that was a typo, have you tried to run this script outside of Octopus at all and does it work if you do run it outside of Octopus?

I look forward to hearing back from you, I am going to try and test something similar on my end to see if it does work, to help me out here what version of Octopus are you running, our records show 2021.2.7580 but this might not still be the case?

Kind Regards,

Clare Martin

Yes I tried the script outside octopus and it worked, pasted same in octopus and it fails with error "NotSpecified: The request was aborted: The request was canceled. "

Octopus version is 2021.3.12313

Thanks for trying it out for me.

Hi @harsh_tech,

Thanks for trying that out for us!

I wonder if you’d be able to try a couple of different things, please? If you’re able to try the below, one at a time, to rule out any issues.

  • If APIKey is a project variable, you should be able to use $APIKey without the $OctopusParameters.

    • Are you able to try removing the line:
      $apiKey = $OctopusParameters["APIKey"]
      and for good measure, amend the case on the next line to match:
      $headers.Add("X-JFrog-Art-Api", $APIKey)
  • Run a deployment with variable logging enabled and send over the raw task log.
    This should reveal whether the API Key is being populated correctly (it won’t reveal the key itself).

  • Entering the API Key as a raw string into the body of the script:

    • (I realise this isn’t secure and if the environment you’re working on doesn’t allow for you to delete releases after the fact then this may not be possible.)
      Ex:
      $apiKey = "API-XXXXXXXXXXXXXXXXXX"

Hopefully this helps get us further in troubleshooting the issue, please let us know if you have any questions or concerns.

Kind Regards,
Adam

1 Like

Thanks @adam.hollow but I can assure you that API key is correct because download is working fine with same API key which I verified by printing all files in current working folder.

Hi @harsh_tech,

No worries, thanks for confirming that, are you able to try the first and second options from my previous reply to see what the results are?
As they have more to do with how the API key is being fed into the script that Octopus is running, it may provide us with more information.

Kind Regards,
Adam

Yes @adam.hollow tried both the options and same result. I am sure its not about API key because otherwise it would return unauthorised error from Artifactory. But the error is more towards something is cancelling the operation from Octopus side.

Hi @harsh_tech,

Just stepping in for Adam and Clare while they’re offline!

I’ve managed to get the package download/upload working on my own reproduction, essentially using the same script:

$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$apiKey = $OctopusParameters["APIKey"]
$headers.Add("X-JFrog-Art-Api", $apiKey)

#download
Invoke-RestMethod 'https://finnd.jfrog.io/artifactory/generic-repo/package0.0.0.1.zip' -Headers $headers -OutFile '.\package0.0.0.2.zip' 

$content = Get-ChildItem .\
Write-Host ($content | Format-Table | Out-String)

#Upload
Invoke-RestMethod 'https://finnd.jfrog.io/artifactory/generic-repo/package0.0.0.2.zip' -Headers $headers -Method Put -InFile '.\package0.0.0.2.zip'

However I’d really like to get to the bottom of why it’s not working for you, could you please send through the entire log file so that I can compare it with my own? I’m hoping there will be some further indication why it was cancelled.

I also used a package that was ~2gb to see if the size could be contributing to the issue, is your package much larger than that?

Feel free to reach out with any questions!

Best Regards,

Thanks @finnian.dempsey for trying this out. I am surprised why this is not working for me. Below is full error log. another note, I am running the “Run script” task with internal worker node.

  Directory:  
May 25th 2022 13:13:49Info
    E:\Octopus.Tentacle.v4\v4-test_internal.app1.octopus.deploy.internal.cba\Work\20220525031040-4018317-20 
May 25th 2022 13:13:49Info
Mode                LastWriteTime         Length Name                                                                   
May 25th 2022 13:13:49Info
----                -------------         ------ ----                                                                   
May 25th 2022 13:13:49Info
-a----       25/05/2022   1:10 PM           1481 Bootstrap.ps1                                                          
May 25th 2022 13:13:49Info
-a----       25/05/2022   1:10 PM          14859 Octopus.FunctionAppenderContext.ps1                                    
May 25th 2022 13:13:49Info
-a----       25/05/2022   1:10 PM           6634 Output.log                                                             
May 25th 2022 13:13:49Info
-a----       25/05/2022   1:13 PM       92266609 Saranya01-1.0.4.zip                                                    
May 25th 2022 13:13:49Info
-a----       25/05/2022   1:10 PM           1016 Script.ps1                                                             
May 25th 2022 13:13:49Info
-a----       25/05/2022   1:10 PM             36 Variables.PowerShell.secret                                            
May 25th 2022 13:13:49Info
-a----       25/05/2022   1:10 PM          17700 Variables.secret                                                       
May 25th 2022 13:15:30Error
NotSpecified: The request was aborted: The request was canceled. 
May 25th 2022 13:15:30Error
At E:\Octopus.Tentacle.v4\v4-test_internal.app1.octopus.deploy.internal.cba\Work\20220525031040-4018317-20\Script.ps1:16 char:1 
May 25th 2022 13:15:30Error
+ Invoke-RestMethod 'https://artifactory.internal.cba/test-nuget-dev ... 
May 25th 2022 13:15:30Error
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
May 25th 2022 13:15:30Error
at <ScriptBlock>, E:\Octopus.Tentacle.v4\v4-test_internal.app1.octopus.deploy.internal.cba\Work\20220525031040-4018317-20\Script.ps1: line 16 
May 25th 2022 13:15:30Error
at <ScriptBlock>, <No file>: line 1 
May 25th 2022 13:15:30Error
at <ScriptBlock>, E:\Octopus.Tentacle.v4\v4-test_internal.app1.octopus.deploy.internal.cba\Work\20220525031040-4018317-20\Octopus.FunctionAppenderContext.ps1: line 201 
May 25th 2022 13:15:30Error
at <ScriptBlock>, E:\Octopus.Tentacle.v4\v4-test_internal.app1.octopus.deploy.internal.cba\Work\20220525031040-4018317-20\Bootstrap.Octopus.FunctionAppenderContext.ps1: line 1503 
May 25th 2022 13:15:30Error
at <ScriptBlock>, <No file>: line 1 
May 25th 2022 13:15:30Error
at <ScriptBlock>, <No file>: line 1 
May 25th 2022 13:15:30Fatal
The remote script failed with exit code 1 
May 25th 2022 13:15:30Fatal
The action Run a Script on a Worker failed

Hi @harsh_tech,

Thanks for sending that through and that additional info!

I was testing on a deployment target, however I just tested running this on the built-in worker to see if it would make a difference but it also succeeded ok… I think I might need to ask around for some advice with this one!

Is there proxy instance configured by any chance? I’m thinking there could be some type of timeout being reached as it’s approx. 1min40 for the task to be cancelled, however it’s strange that there isn’t another error message being written and that this doesn’t occur outside of Octopus.

I’ll keep you posted as soon as I have any updates from the internal discussions, feel free to let me know if you have any questions!

Best Regards,

Do you know where can I find it on worker?

Thanks, I am waiting for your response.

Hi @harsh_tech,

For the built-in worker, a proxy would typically be configured at the bottom left of the Octopus Manager:

However I have also been caught out in the past by Environment Variables such as: $Env:HTTPS_PROXY being set so definitely check that they haven’t got any values:

Could you please also expand on how you tested the script outside of Octopus? Was it from the same machine & user account that is being used by the Octopus instance? Could you please confirm the approx. size of the package you are working with in GB? I also wonder if it’s possible that adding the flag -DisableKeepAlive could help here?

I’ll keep you posted once I have any more updates from the internal discussion!

Best Regards,

1 Like

Hi @finnian.dempsey Sorry for late response. I think certainly something to do with timeout, I tested with smaller nuget package and it worked like a charm.
But I am unsure whats the size or time limit to proceed with. The error does not provide much detail and also I do not have access to enterprise Octopus level settings to see the page you screened above. But thanks again for providing all the details and clue.

Hey @harsh_tech,

Thanks for confirming it seems to be package size that’s causing this issue, this might then be related to proxy settings depending on how your Octopus Server is configured and how the machine you are running the PowerShell script on is also configured.

What was the size of the original package you were trying to upload? Did you manage to try the
-DisableKeepAlive flag that Finnian suggested?

I look forward to hearing from you, it’s good to see you and Finninan have gotten somewhere with this, I think he was scratching his head a bit as to why this wasn’t working for you but package size would make sense.

Kind Regards,

Clare

1 Like

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.