Question

I’ve got a 2GB text file and a 500MB text file. The 2GB is in a slightly daft format: e.g. sample:

CD 15
IG ABH
NU 1223
**
CD 17
IG RFT
NU 3254
**

Where ** is the marker between records.

I need to extract all the values of NU where CD is a certain value; I then need to go through the 500MB text file and then match all the records in there with the NU values from the 2GB file and then write those to a new file.

I know PHP. This is trivial in PHP, apart from the size of the file. Even using fgets to read a line at a time doesn’t really work as it takes for ever and then crashes my computer in localhost (under XAMPP apache.exe grows to use up all system memory). Plus doing it in PHP would be a pain (it’s for non-technical people to run, so they’d need to download the 2GB and 500MB from the FTP server when they become available each week; upload them to my FTP server which is flaky on such large file sizes; run a script on my server that takes ages etc).

I know a bit of VBScript, no Perl, no .NET, no C# etc. How can I write a Windows-based programme that will run locally, load the files a line at a time, and not crash due to the file size?

Answer 1

The following declares a VBScript function to read a source file 1 line at a time and write the destination file only if the cdfilter string matches the cd in the record:

Option Explicit

Const ForReading = 1
Const ForWriting = 2

Sub Extract(srcpath, dstpath, cdfilter)
  Dim fso, src, dst, txt, cd, nu
  Set fso = CreateObject("Scripting.FileSystemObject")
  Set src = fso.OpenTextFile(srcpath, ForReading)
  Set dst = fso.OpenTextFile(dstpath, ForWriting, True)
  While (not src.AtEndOfStream)
    txt = ""
    While (not src.AtEndOfStream) and (txt <> "**")
      txt = src.ReadLine
      If Left(txt, 3) = "CD " Then
        cd = mid(txt, 4)
      End If
      If Left(txt, 3) = "NU " Then
        nu = mid(txt, 4)
      End If
      If txt = "**" Then
        If cd = cdfilter Then
          dst.WriteLine nu
          cd = ""
          nu = ""
        End If
      End If
    Wend
  Wend
End Sub

Convert "input.txt", "output.txt", "17"