Question

I’m a completely new to shell scripting and bash. My question is how do I get awk to interpret a variable which contains linebreaks the same way as awk interpret the data from stdin?

Example:

fileData=`cat /path/to/file`
echo $fileData | awk '{print $1}'

The code above results in the following error message: awk: program limit exceeded: maximum number of fields size=32767 which obviously means that awk interprets all the lines in $fileData at the same time and not one line at a time.

How to make awk interpret one line at a time from a variable?

Answer 1

If I understand you question correctly, I suspect that your issue is less todo with awk, and more to do with shell white-space splitting. Your problem is that the shell has taken the new-lines inside $fileData as white-space field seperators. To prevent that:

echo "$fileData" | awk '{print $1}'

You can also access environment variables (export fileData) directly from awk using the associative array environ, for example environ["fileData"], however it is probably a bad strategy to store an entire file contents in a variable because of the risk of busting memory.

By the way, I hope that your code is just a simple example,

awk '{print $1}' /path/to/file

is much preferred. As is:

fileData=$(cat /path/to/file)

over using backticks (although cat is rarely necessary).