Powershell - Compress Files from Multiple paths

[nsantos.br]

Hi guys,

I was wondering if someone can help me. I have to create a PowerShell script to compress log files older than 2 days from multiple paths and move then. I want to use Octopus variables in my script, but I don’t know how to accomplish that. So far I have this:

    #get the list of files in the original folder
$rootFolder = "$Octopusvariable1"
$tempVariable = $rootFolder
$files = Get-ChildItem -Path $rootFolder
$FileType = "log"
$targetPath = "$Octopusvariable2"

#create a temporary folder using today's date
$tempFolderRoot = "C:\Temp_"
$date = Get-Date
$date = $date.ToString("yyyy-MM-dd")
$tempFinalFolder = "$tempFolderRoot$date"
New-Item -ItemType directory -Path $tempFinalFolder -Force

#decide how long back to go
$timespan = new-timespan -days 0

#move the files to a temporary location
foreach($file in $files)
{
	$fileLastModifieddate = $file.LastWriteTime
	if(((Get-Date) - $fileLastModifiedDate) -gt $timespan)
	{
		Move-Item "$rootFolder\$file" -destination $tempFinalFolder
	}
}
#Compress the files to ZIP
Get-ChildItem $tempFinalFolder | ForEach-Object {
    $FileName = $_.Name -replace $FileType, ".zip"
    Compress-Archive -LiteralPath $_.FullName -DestinationPath $targetPath$FileName -Update
    Write-Host $FileName
    $FileName = $null
    }
#Remove Temp Folder
Remove-Item $tempFinalFolder -RECURSE

So… Inside #Octopusvariable1, I have to insert multiple paths, for example:
C:\Logs\app1
C:\Logs\app2
C:\Logs\app3

For #Octopusvariable2, I have to send those files to another server folder root but respecting the same folder structure… So files from C:\Logs\app1 must go to \Server2\RootfolderLogs\app1, files from C:\Logs\app2 must go to \Server2\Rootfolder\Logs\app2… and so on.

Do you guys have any idea, how to build my script?

Thanks!
Ney Santos

[paul.calvert]

Hi @nsantos.br,

Thanks for getting in touch!

My initial thoughts on this would be to use a single octopus variable and apply it to both source and target paths, this sample assumes the source will always be C:, it could be amended slightly if that isn’t the case.

$octopusVariable = "logs\app1"

$sourcepath = "C:\" + $octopusVariable
$targetpath = "\Server2\Rootfolder\" + $octopusVariable

Regarding the multiple paths, it’s hard to suggest anything without a bit more information.

If you mean that the path will change based on the environment, then it would just be a case of adding one variable to Octopus and creating a different value scoped to each environment. Then when you run the script, it will use the path relevant to the environment it runs in.

I believe, however, that you need this to do multiple paths in a single run. In which case, I don’t see any way around adding each of the numerous file paths as a separate variable within Octopus. Once that is done, you can place your script into a function with the Octopus variable as a parameter and run it for each different variable.
My sample below calls the function each time manually; if your number of file paths is large or likely to change it may be better to feed the Octopus variables into an array and use a foreach loop to run the function for each of them.

function cleanup{  
    param ($octopusVariable)

     #get the list of files in the original folder
    $rootFolder = "C:\" + $octopusVariable
    $tempVariable = $rootFolder
    $files = Get-ChildItem -Path $rootFolder
    $FileType = "log"
    $targetPath = "\Server2\Rootfolder\" + $octopusVariable

    #create a temporary folder using today's date
    $tempFolderRoot = "C:\Temp_"
    $date = Get-Date
    $date = $date.ToString("yyyy-MM-dd")
    $tempFinalFolder = "$tempFolderRoot$date"
    New-Item -ItemType directory -Path $tempFinalFolder -Force

    #decide how long back to go
    $timespan = new-timespan -days 0

    #move the files to a temporary location
    foreach($file in $files)
    {
	    $fileLastModifieddate = $file.LastWriteTime
	    if(((Get-Date) - $fileLastModifiedDate) -gt $timespan)
	    {
		    Move-Item "$rootFolder\$file" -destination $tempFinalFolder
	    }
    }
    #Compress the files to ZIP
    Get-ChildItem $tempFinalFolder | ForEach-Object {
        $FileName = $_.Name -replace $FileType, ".zip"
        Compress-Archive -LiteralPath $_.FullName -DestinationPath $targetPath$FileName -Update
        Write-Host $FileName
        $FileName = $null
        }
    #Remove Temp Folder
    Remove-Item $tempFinalFolder -RECURSE

}

$octopusVariable1 = "logs\app1"
$octopusVariable2 = "logs\app2"
cleanup $octopusVariable1
cleanup $octopusVariable2

A final way to possibly do this using a single variable would be to enter your file paths in Octopus as a comma-separated list, e.g. $octopusVariable = “logs\apps1,logs\apps2,logs\apps3” and then use the Powershell Split method to push them into an array.

$octopusVariable="logs\app1,logs\app2,logs\app3"
$sourcePaths = $octopusVariable -split ','

 foreach($sourcePath in $sourcePaths)
    {
    cleanup $sourcePath 
    }

Apologies if I have rambled on a bit here, I kept having new ideas and figured it would be useful to include the previous ones as reference.
I hope this helps point you in the right direction, please let me know if I can provide more detail for you.

Best regards,
Paul

[nsantos.br]

Hello Paul,

Thank you very much for your support. I’m trying your suggestion to use the foreach and the Powershell Split for the multiple paths:

#get the list of files in the original folder
$rootFolder = $RootInstallPath -split ','
$tempVariable = $rootFolder
$files = Get-ChildItem -Path $rootFolder
$FileType = "log"
$targetPath = "C:\work\ZipLogs\"

#create a temporary folder using today's date
$tempFolderRoot = "C:\Temp_"
$date = Get-Date
$date = $date.ToString("yyyy-MM-dd")
$tempFinalFolder = "$tempFolderRoot$date"
New-Item -ItemType directory -Path $tempFinalFolder -Force

 foreach($rootFolder in $rootFolder)
{
#decide how long back to go
$timespan = new-timespan -days 2

#move the files to a temporary location
foreach($file in $files)
   {
	$fileLastModifieddate = $file.LastWriteTime
	if(((Get-Date) - $fileLastModifiedDate) -gt $timespan)
	{
		Move-Item "$rootFolder\$file" -destination $tempFinalFolder
	}
   }
#Compress the files to ZIP
Get-ChildItem $tempFinalFolder | ForEach-Object {
    $FileName = $_.Name -replace $FileType, ".zip"
    Compress-Archive -LiteralPath $_.FullName -DestinationPath $targetPath$FileName -Update
    Write-Host $FileName
    $FileName = $null
    }
}    
#Remove Temp Folder
Remove-Item $tempFinalFolder -RECURSE

The Octoputs variable is:
$RootInstallPath = "C:\work\Logs\,C:\work\Logs2\,C:\work\Logs3\"

However, the result of this script is compressing and moving only the log files from C:\work\Logs\

Maybe I’m not using the foreach properly. Do you have any idea?

Thank you very much!

[paul.calvert]

Hi,

Firstly, I noticed that the zip files being created had 2 ‘.’ in them, e.g. ‘test…zip’ so I would recommend amending $FileName = $_.Name -replace $FileType, ".zip" to $FileName = $_.Name -replace $FileType, "zip"

Other than that, I am noticing some errors when I’m running the script. It seems that if the log files in any of the folders share the same name, it will fail to create the file in the temp folder and then not produce the final zip. Could this be what is happening for you?

When I made sure to use differently named files in each folder it did successfully zip all of them; however, I did note that Powershell was still generating errors as it was searching for each of the files in all three folders.
My set up was:
work\logs\test1.log
work\logs2\test2.log
work\logs3\test3.log

I would see errors such as Move-Item : Cannot find path 'C:\work\Logs2\Test.log' because it does not exist.

This last part probably won’t cause a problem in itself, but will generate a lot of noise.

I’ll aim to spend some further time on this today to see if I can come up with a solution for these.

Regards,
Paul

[paul.calvert]

Hi,

Spent some time on this, and have come up with the following. The main changes are that it will now strip the ‘C:’ part of the file path, which allows it to retain the folder path in the destination folder. It will also create the destination folders if they don’t already exist.

I’ve also placed it all into a function which means that it will run the entire script against each folder one at a time, which means that duplicate file names won’t be a problem and it won’t look for files in the wrong folder.

function cleanup{  
    param ($rootfolder)

    #get the list of files in the original folder
    $tempVariable = $rootfolder
    $files = Get-ChildItem -Path $rootfolder
    $FileType = "log"
    $rootPath = $rootfolder.substring(3) #removes 'C:\' from file path
    $targetPath = "C:\work\ZipLogs\" + $rootPath

    #create a temporary folder using today's date
    $tempFolderRoot = "C:\Temp_"
    $date = Get-Date
    $date = $date.ToString("yyyy-MM-dd")
    $tempFinalFolder = "$tempFolderRoot$date"
    New-Item -ItemType directory -Path $tempFinalFolder -Force

    #decide how long back to go
    $timespan = new-timespan -days 2

    #move the files to a temporary location
    foreach($file in $files)
    {
	    $fileLastModifieddate = $file.LastWriteTime
	    if(((Get-Date) - $fileLastModifiedDate) -gt $timespan)
	    {
		    Move-Item "$rootFolder\$file" -destination $tempFinalFolder
	    }
    }
    #Compress the files to ZIP
    Get-ChildItem $tempFinalFolder | ForEach-Object {
        $FileName = $_.Name -replace $FileType, "zip"
        New-Item -ItemType Directory -Path $targetPath -Force #creates target path folder structure
        Compress-Archive -LiteralPath $_.FullName -DestinationPath $targetPath$FileName -Update
        Write-Host $FileName
        $FileName = $null
        }
    #Remove Temp Folder
    Remove-Item $tempFinalFolder -RECURSE
}

$rootFolder = $RootInstallPath -split ','

foreach($path in $rootFolder)
{
    cleanup $path
}

[nsantos.br]

Hi Paul,

I really want to thank you for your support. Your script works perfectly on my local computer, however, I am having the same issue in Octopus. The script reads only the first path inside the Octopus variable RootInstallPath. This is the result I am getting:

The variable Octopus variable is set like this:

What do you think? I don’t understand, if I run the script manually on the server, it works.
I also tried to change the files name to App1_09_09_2019, App2_09_09_2019 and so on, jut to be sure, but it didn’t work anyway.

Thank you!!
Ney Santons

[nsantos.br]

Hi Paul,

I got it… The variable was wrong. Now the script is working.

Thank you very much for you help!
Ney Santos

[paul.calvert]

Hi Ney,

Glad you got that working.

Let me know if you have any further questions.

Best regards,
Paul

[nsantos.br]

Hi Paul,

Sorry to bother you once again. Can I ask you one more question about this script?

Now this script must delete instead of zip some specific files among all the paths. So I created a new Octopus variable with name of these files:

“$LogCleaner.Sensitive” = “file1.log,file2.log,file3.log”.

I tried to move them to another temp folder and then remove the folder, but it didn’t work.

Can you help me? This is the new version of the script, just the “#Move sensitive files to Tempfolder_Sens” is not working, the files are been zipped just like the others:

#move the files to a temporary location
foreach($file in $files)
{
        #Move sensitive files to Tempfolder_Sens
		if($file | Where-Object { $_.name -match $OctopusParameters["LogCleaner.Sensitive"] -split ","})
        {
        Move-Item "$rootFolder\$file" -destination "D:\Tempfolder_Sens" -force
        }
        Else {																				  
     
     #move non sensitive files to Temp folder   
	    $fileLastModifieddate = $file.LastWriteTime
	    if(((Get-Date) - $fileLastModifiedDate) -gt $timespan)
	    {
		    Move-Item "$rootFolder\$file" -destination $tempFinalFolder #Moves files from original folder to temp folder, ex D:\Temp_2019-11-30\App1_logs\*.log
       }
	    }
}

Thank you very much!
Ney Santos

[paul.calvert]

Hi Ney,

Is this script targeting the same folders as the previous script, and running after that script?
If so, the first script is likely already picking up these sensitive log files before this script executes. Running this one first should resolve that.

Best regards,
Paul

[nsantos.br]

Hi Paul,

Actually everything is the same script, I just printed the part that is not working, the whole script is:

function cleanup{  
    param ($rootfolder)

    #get the list of files in the original folder
    $tempVariable = $rootfolder
    $files = Get-ChildItem -Path $rootfolder
    $FileType = "log"
    $rootPath = $rootfolder.substring(3) #removes 'C:\' from file path
    $targetPath = "C:\work\ZipLogs\" + $rootPath

    #create a temporary folder using today's date
    $tempFolderRoot = "C:\Temp_"
    $date = Get-Date
    $date = $date.ToString("yyyy-MM-dd")
    $tempFinalFolder = "$tempFolderRoot$date"
    New-Item -ItemType directory -Path $tempFinalFolder -Force

    #decide how long back to go
    $timespan = new-timespan -days 2

    #move the files to a temporary location
    foreach($file in $files)
    {
                #Move sensitive files to Tempfolder_Sens
		if($file | Where-Object { $_.name -match $OctopusParameters["LogCleaner.Sensitive"] -split ","})
        {
        Move-Item "$rootFolder\$file" -destination "D:\Tempfolder_Sens" -force
        }
        Else {	
	    $fileLastModifieddate = $file.LastWriteTime
	    if(((Get-Date) - $fileLastModifiedDate) -gt $timespan)
	    {
		    Move-Item "$rootFolder\$file" -destination $tempFinalFolder
	    }
    }
    #Compress the files to ZIP
    Get-ChildItem $tempFinalFolder | ForEach-Object {
        $FileName = $_.Name -replace $FileType, "zip"
        New-Item -ItemType Directory -Path $targetPath -Force #creates target path folder structure
        Compress-Archive -LiteralPath $_.FullName -DestinationPath $targetPath$FileName -Update
        Write-Host $FileName
        $FileName = $null
        }
    #Remove Temp Folder
    Remove-Item $tempFinalFolder -RECURSE
}

But the part

	`if($file | Where-Object { $_.name -match $OctopusParameters["LogCleaner.Sensitive"] -split ","})`

Is not working, the script bypass the file names inside Octopus variable and zip everything.

Thanks!
Ney Santos

[paul.calvert]

Hi Ney,

After a lot of confusion, I think I may have figured it out.
Can you try running with the -match statements the opposite way around?
e.g.
Where-Object {$OctopusParameters["LogCleaner.Sensitive"] -split "," -match $_.name})

Regards,
Paul

[nsantos.br]

It worked Paul!

Thank you very much!

Regards,
Ney Santos