Multiple computers writing to a domain hosted file

Ask your PowerShell-related questions, including questions on cmdlet development!
Forum rules
Do not post any licensing information in this forum.

Any code longer than three lines should be added as code using the 'Select Code' dropdown menu or attached as a file.
This topic is 4 years and 11 months old and has exceeded the time allowed for comments. Please begin a new topic or use the search feature to find a similar but newer topic.
Locked
PaschalIT
Posts: 38
Last visit: Fri Jan 10, 2020 7:38 am

Multiple computers writing to a domain hosted file

Post by PaschalIT »

Hi everyone!
New to the forums, so let me know if I do anything wrong.

I'm writing a script for my company which runs on every necessary computer on our domain at once, updating a piece of software if needed, and then writes that software's version to a file on a network drive. The issue I'm encountering is that since they're all running the same task, the computers all finish the process at about the same time and try to write to the file at the same time, effectively overwriting or nulling each others' efforts, since Powershell can't replace text in a file without completely rewriting it (to my knowledge).

So my dilemma is, is there a way to "lock" the file a computer is working on it and prevent other computers from accessing it until that one is done? I don't have a problem with the task continuing to run in the background until they get access to the file.
User avatar
mxtrinidad
Posts: 399
Last visit: Tue May 16, 2023 6:52 am

Re: Multiple computers writing to a domain hosted file

Post by mxtrinidad »

We need more information!
It will help if you tell us what cmdlet you're using to write the file to the share drive.
:)
jvierra
Posts: 15439
Last visit: Tue Nov 21, 2023 6:37 pm
Answers: 30
Has voted: 4 times
Been upvoted: 33 times

Re: Multiple computers writing to a domain hosted file

Post by jvierra »

Simultaneous write to a file cannot be done by normal means. You will need to open the file as shared write so the locking is cooperative. This is normal Windows behavior.

A file in Windows can only be opened as "create" or "Append". When appending the writes always get added to the existing file at the end. TO do an update you would have to open a stream in binary mode and seek to the location to change. If the write buffer is not resized then the edit will not require a complete rewrite. If the edit is an insert or delete of bytes then the file will be rewritten. This is only an issue with very large files.

PowerShell CmdLets only allow new files or Append.
jvierra
Posts: 15439
Last visit: Tue Nov 21, 2023 6:37 pm
Answers: 30
Has voted: 4 times
Been upvoted: 33 times

Re: Multiple computers writing to a domain hosted file

Post by jvierra »

A quick fix to your issue will be to have each computer write to a separate file. If you make the files CSV files then it will be easy to concatenate all files and sort/filter the results.


# catenate
$csv = Import-Csv *.csv

# all records from all system for today
$csv = Import-Csv *.csv | sort Mydate | Where{ $_.Mydate -gt [datetime]::Today }
PaschalIT
Posts: 38
Last visit: Fri Jan 10, 2020 7:38 am

Re: Multiple computers writing to a domain hosted file

Post by PaschalIT »

mxtrinidad wrote: Wed Apr 03, 2019 12:49 pm We need more information!
It will help if you tell us what cmdlet you're using to write the file to the share drive.
:)
Sorry! I'm using get-content and set-content to read and write. My code reads in the file, checks to see if that computer's version is already present, appends it if it's not there, or replaces the version if that computer is already present.

Currently it seems like the reads and writes are stepping on each others' toes, as if I watch the file while the scripts are running the entire text occasionally disappears, or values get lost in the mix.

My current code for finding and replacing is:
  1. if ($filetext -match $hostname) # Find comp name in file
  2.         {
  3.             $filetext -replace $find, $replace | Set-Content $file # Replace current data with new data
  4.         }
  5.         else
  6.         {
  7.             Add-Content $file $replace # Append data if comp name not found
  8.         }
User avatar
mxtrinidad
Posts: 399
Last visit: Tue May 16, 2023 6:52 am

Re: Multiple computers writing to a domain hosted file

Post by mxtrinidad »

I would suggest to use PowerShell existing commands as it was designed to such purpose.

As JVierra shown one way using the Import-Csv cmdlet, you can also try using the Out-File cmdlet which has the parameters '-NoClobber' which will not override the file, and in combination with the '-Append' the output is appended to the existing file.

This is a much simple way to collect information.

Try them both to see which give you the better result!
:)
jvierra
Posts: 15439
Last visit: Tue Nov 21, 2023 6:37 pm
Answers: 30
Has voted: 4 times
Been upvoted: 33 times

Re: Multiple computers writing to a domain hosted file

Post by jvierra »

PaschalIT wrote: Wed Apr 03, 2019 1:18 pm
My current code for finding and replacing is:
  1. if ($filetext -match $hostname) # Find comp name in file
  2.         {
  3.             $filetext -replace $find, $replace | Set-Content $file # Replace current data with new data
  4.         }
  5.         else
  6.         {
  7.             Add-Content $file $replace # Append data if comp name not found
  8.         }
This will not work from multiple computers. If you open the file and then append the item you will not lose data except when the open fails which you will have to detect as an exception and retry until you are successful.

Code: Select all

if(Get-Content $filename |Where {$_ -match $computername }){
       # computer record exists
}else{
      Try{
           $text | Out-File $filename -Append -ErrorAction Stop
      }
      Catch{
            # file in use
      }
}
Trying to rewrite the complete file each time will only make the situation worse. You will also lose all write between the time you test the file and when you rewrite it. PowerShell CmdLets do not support advanced locking. For this you will need to use a file stream.

To have an updateable record store we would use a database file like MSAccess and use the Access data provider (ACE). This provides all locking semantics and allows individual records to be updated with no loss.
jvierra
Posts: 15439
Last visit: Tue Nov 21, 2023 6:37 pm
Answers: 30
Has voted: 4 times
Been upvoted: 33 times

Re: Multiple computers writing to a domain hosted file

Post by jvierra »

"NoClobber" will not avoid an exception if the file is in transition. Another process can intercept and the NoClobber will not preserve the file. It can work under some circumstances but not when you try to change a single line in the file. Updates are not cooperative in PS. Only an Append will work.
jvierra
Posts: 15439
Last visit: Tue Nov 21, 2023 6:37 pm
Answers: 30
Has voted: 4 times
Been upvoted: 33 times

Re: Multiple computers writing to a domain hosted file

Post by jvierra »

Here is a good blurb on what "NoClobber" actually does.
noclobber.png
noclobber.png (36.17 KiB) Viewed 3317 times
PaschalIT
Posts: 38
Last visit: Fri Jan 10, 2020 7:38 am

Re: Multiple computers writing to a domain hosted file

Post by PaschalIT »

I've seen a few posts and also heard from my colleagues that a database is the best option, so I will try to look into that for the long run. For now, it sounds like separate files are my best bet, so I think that's what I'll go with for the moment. Thank you everyone for your help! (I'll be back if I can't get it to work :P )
This topic is 4 years and 11 months old and has exceeded the time allowed for comments. Please begin a new topic or use the search feature to find a similar but newer topic.
Locked