Trying to automate a few manual tasks into a single PowerShell script RRS feed

  • Question

  • Hey Scripting Guys!

    I've been trying to automate a few manual tasks into a single PowerShell script, but think I'm way over my head...


    • Every day I receive about a hundred server log files
    • These files are formatted the same, but come from different systems
    • These files are all dropped into the same folder. To keep it simple let's define the path for that folder as C:\raw
    • These files are all consistently formatted (see attached examples) but the files do not have any defined file extension when they arrive.
    • PowerShell 5.1.15063.296
    • Here's a link to some sample data:

    Here is the process I am trying to automate:

    1.  Concatenate all files located within C:\raw into one contiguous document.
    2.  Save the concatenated data file as [today's date].tsv
    3.  Sort the data in the concatenated data file first by column C (Date) and then by column B (City)
    4.  Delete any duplicate rows in the concatenated document
    5.  Save the finished document in TSV format to C:\done

    Below is as far as I've gotten.  Any suggestions for a noob?

    $root = "T:\20"
    $files = Get-ChildItem -Path $root -Filter *.txt

    $data = ForEach ($file in $files){
    Import-Csv -Path $file.FullName -Delimiter "`t"

    gc $filename | sort | get-unique

    $files | Move-Item -Destination "C:\done"

    $data | Export-Csv -NoTypeInformation -Path (Join-Path $root "out.csv") -Append


    • Moved by Bill_Stewart Friday, July 7, 2017 7:04 PM This is not "fix/debug/rewrite my script for me" forum
    Tuesday, June 6, 2017 3:15 AM

All replies

  • Hmmm. You have some pieces but you are not following your recipe.

    Get-ChildItem -Path t:\20\* -Filter *.tsv |
    	Import-Csv -Delimiter "`t" |
    	Export-Csv t:\20\out-csv -NoTypeInformation
    (Get-Content t:\20\out.csv) | 
    	Sort-Object -Unique | 
    	Out-File t:\20\out.csv 


    Tuesday, June 6, 2017 4:28 AM