Question

I am trying to load and parse a really large text file. Although the loading is not a problem, but there are particular lines that have 2908778 characters on a single line.

This is causing an error in my script.

On the script below, I removed all logic and just got straight to read line.
I also removed all valid lines and just left the really long line in one text file. When running I get the below error :

$ dowhiledebug.sh dump.txt
dowhiledebug.sh[6]: no space
Script Ended dump.txt

The actual script:

 #!/bin/sh
 filename=$1
 count=1
 if [ -f ${filename} ]; then
    echo "after then"
    while read line;
            do
            echo "$count"
            count=$((count+1))
            done < $filename
 else
    echo "Could not open file $filename"
 fi
 echo "Script Ended $filename"

Updated (2013-01-17)

Follow up question : Is it possible to increase the maximum number of characters that ksh variable accepts?

Answer 1

The limit for any shell is the limit of the C command line maximum. Here’s a little program that pulls the information out of /usr/include/limits.h for you:

cpp <<HERE | tail -1
#include <limits.h>
ARG_MAX
HERE

Mine gives me (256 * 1024) or 262144 characters.

Doesn’t work if the C compiler isn’t installed, but it’s probably a similar limit.