VBScript Limitations?

Anything VBScript-related, including Windows Script Host, WMI, ADSI, and more.
Forum rules
Do not post any licensing information in this forum.

Any code longer than three lines should be added as code using the 'Select Code' dropdown menu or attached as a file.
This topic is 16 years and 10 months old and has exceeded the time allowed for comments. Please begin a new topic or use the search feature to find a similar but newer topic.
Locked
User avatar
black777
Posts: 25
Last visit: Thu Apr 02, 2015 12:50 pm

VBScript Limitations?

Post by black777 »

Has anyone hit any size limitations concerning the Scripting.FileSystemObject. I have a HTTP log of about 390 MB, I have enough memory in my system so, I figured a ReadAll would work (It has worked on files up to 150 MB), but suddenly cscript.exe is crashing with a Runtime Error. The cscript.exe process at the time of termination is using up about 540 MB. Are there limitations to cscript/wscript or limitations to the Scripting.FileSystemObject?

--Black
jvierra
Posts: 15439
Last visit: Tue Nov 21, 2023 6:37 pm
Answers: 30
Has voted: 4 times
Been upvoted: 33 times

VBScript Limitations?

Post by jvierra »

I have seen some similar issues when trying to read large files and large bibnary files. SOme issues seem to happen when reading in Unicode files.

I think it has to do with the large reads supressing some event responses in CScript. I have worked around this in the past by using the fso.Read( nChars ) method in a loop. This helps withe EndOfStream detection and seems t avoid some of the unpleasant side affects of some types of a read. If the file is a Unicode file it will create problems when being read using ReadLine and ReadAll as it doesn't have the expected EndOfFile signature like an ansi text file has. REadin it as a binary file may also help to alleviate this issue.

Maybe jeff will find something dffrent if he test this. YOu should probably zip and post teh log file for teting if it doesn't contain proprietary info as this would be the best test.

YOu won't be a ble to post a file of theat size here but you could post it to a web server or a public file store.

User avatar
jhicks
Posts: 1789
Last visit: Mon Oct 19, 2015 9:21 am

VBScript Limitations?

Post by jhicks »

I created a 209MB ASCII text file. I tried using the ReadAll method and got a an out of memory error message. My computer has 1.5GB of RAM. I had almost 800MB of free physical memory. I watched mem usage do down to about 420MB free before it erred out.

Here's the script I threw together to create a text file of just about any size:

Code: Select all

iSize=1048576 'size in bytes

strFile="c:templarge.txt"
strFiller="lorep ipsum dorem ad hoc nauseum"

Dim objFSO
Const ForReading = 1
Const ForWriting = 2
Const Forappending = 8
Set objFSO=CreateObject("Scripting.FileSystemObject")
Set objTS=objFSO.CreateTextFile(strFile)

objTS.WriteLine strFiller
objTS.Close

do Until objFSO.GetFile(strFile).Size >= iSize
    Set objTS=objFSO.OpenTextFile(strFile,Forappending)
    objTS.WriteLine strFiller
    objTS.Close
Loop

WScript.Echo objFSO.GetFile(strFile).Size

User avatar
black777
Posts: 25
Last visit: Thu Apr 02, 2015 12:50 pm

VBScript Limitations?

Post by black777 »

I would post the log file, but it contains every web page, graphic, and file my company has browsed through in the last day. Some of the executives would be a bit embarrased. I read about the Unicode issue, would there be a way to tell whether the file is Unicode or ASCII?
The reason that I was hesitant about splitting up the files into manageable chunks was that I create a Scripting.Dictionary object which contains the domain name and an incremented size value. The script outputs the domain size pairs and I lose the Dictionary object, so in my head I thought I had to read all of the logs in one go, but during lunch, I thought about how easy it would be to process one log file then read the output file back into a new Scripting.Dictionary object and continue. Why this did not occure to me when I started, I have no idea.
I am liking the .Read(nChars) and binary options as well, I will look into those. It is amazing how outside perspectives find so many ways of accomplishing the same task.

--Black
jvierra
Posts: 15439
Last visit: Tue Nov 21, 2023 6:37 pm
Answers: 30
Has voted: 4 times
Been upvoted: 33 times

VBScript Limitations?

Post by jvierra »

Ok - I'll play.

Running on my system teh script blows up as soon as the page file fills up.

Be sure that your paging file has enoughroom. FOr a 400 MB file you would need at least 600 to 700 Mb of free space.
If you are trying to load teh whole file so you can scan it you will, most likely, have probelms and it will be very slow.

Use teh commandline file scanning utilites to extract information from files, or , better yet, use LogParser 2 as it has facilities for reading log files as if they were a database. You can just query the file for matches on key items and teh contents of fields.
It's fast and reliable and can be run from a script.
User avatar
jhicks
Posts: 1789
Last visit: Mon Oct 19, 2015 9:21 am

VBScript Limitations?

Post by jhicks »

LogParser would definitely be an effective tool. Otherwise, you might be able to open the log file using ADO and treat it as a database in VBScript. But Logparser will be more efficient since this is exactly the job it was designed for.
User avatar
jhicks
Posts: 1789
Last visit: Mon Oct 19, 2015 9:21 am

VBScript Limitations?

Post by jhicks »

I'm curious about your page file issue. My pagefile is around 2GB with plenty of free space in the file as well as physical memory.
User avatar
jhicks
Posts: 1789
Last visit: Mon Oct 19, 2015 9:21 am

VBScript Limitations?

Post by jhicks »

I think we can safely say don't try this at home or work. I tried something similiar in PowerShell$large=gc large3.txtI killed PowerShell after 12 minutes. My free physical RAM dropped from 750MB to around 3MB, and my pagefile maxed out its size and started swapping. Jim's statement about no string larger than physical memory makes sense. Unfortunately, you can't assume the size of the file on disk is the same as what will be read into memory. which is an easy thing to do.
User avatar
black777
Posts: 25
Last visit: Thu Apr 02, 2015 12:50 pm

VBScript Limitations?

Post by black777 »

How interesting.

I can consistently build a string with the following size.

Code: Select all

	
iLen = 5*1024*1024*93.8624964714 '492109805
s = String( iLen ,"X")
WScript.Echo Len(s)
	

If I go to 492109806 it errors out. I varied the applications and files I had open to change my memory size, but the error always occured at 492109806. My Windows 2003 Server with less physical memory can go higher than 492109806, trying to figure out how high.

Now that you guys have pointed me towards LogParser, I won't need my script like I did, but it is interesting that a string can only be so long before crashing cscript/wscript.

--Blackblack7772007-05-23 15:03:11
jvierra
Posts: 15439
Last visit: Tue Nov 21, 2023 6:37 pm
Answers: 30
Has voted: 4 times
Been upvoted: 33 times

VBScript Limitations?

Post by jvierra »

The actual string allocation is surely done by teh string alloc primitives in teh system. I would expect that PowerShell and CScript would see similar limits.

On a server it may be because the system can allocate a larger WOrking set to a process.

Microsft has another technique for large strigs that is not available to scripting languages although it shuld be available to PowerShel through NT. This is a memory mapped file. To use it we open the file as a memory object. It all appears to be in memory at teh same time but the memory manager actualy moves pieces in and out through a window of memory. This allows for very fast searches on very large files. Nearly all functions that access memory can access teh file as if it were realy in memory. The address space should be a full 32 bits and a full 64 bits on a 64 bit processor - the lenght of a pointer.

CSCript, vbscript an djscript hava always been known for poor string handling compared to other languages. I never attempt to use a lot of string or very large strings because of this. Whenever I need to parse text I do it with textparsing utilities. They are always faster and more reliable.

LogParser is a text parsing utility that can parse nearly any kind of record oriented file. It excels at parsng logfiles in all common formats including the event logs and even sql databases.

FOr installer logs the newest verion of WIndows Installer has updated utilites to parse and revirew installer logs.

INF files are the missing element. Only custom tools for subsystems are available and they are all limited. The "Profile" functins in teh API do allow for conrolled reading and updating of INF files as they are just INI files.
This topic is 16 years and 10 months old and has exceeded the time allowed for comments. Please begin a new topic or use the search feature to find a similar but newer topic.
Locked