Question

I’m writing a bash shell script and I’m having trouble splitting stdout csv and then looping over it.

I get data from stdout of a database. It is comma delimited and each row is on its own line. I store that in a variable called csv. I have the same thing for my data variable. I get that data from stdout from a url which returns csv…again it is comma delimited and each row has its own line.

Below \n means it is a new line.

I know how to iterate through and get any of the columns for csv using the read (see below). So when I echo out $col1 it displays two results which is what I expect.

This is what I don’t understand:
I then want to get for each of $col1 I want to see if $col1 equals any of the data of the first column of the $data variable. If it exists (it should always exist unless there was an issue) then prepend $col1 of csv onto all the data of the data variable to add that data to form a stdout csv.

csv=$("csv",123\n"csv2",456)
data=$("data1",123\n"data2",456)

echo "$csv" | while IFS=',' read -r col1 col2;do
echo "$col1"
done

example of what is needed:

if $csv[$col1] == [any of the values of $data[$col1]] then;
echo $csv[$col1],$data[all of it]

Answer 1

I’m going to reformat your data as:

DATA

 Col #1    Col #2
 =====     ====
 data1     123
 data2     456

CSV

 Col #1    Col #2
 ====      ====
 csv       123
 csv2      456

Do I have this setup correctly? I know these will be CSV files, but I want to make sure I understand your data structure.

Now you said:

I […] want to get for each of $col1 I want to see if $col1 equals any of the data of the first column of the $data variable.

You want to match column #1 from DATA with column #1 from CSV. In your set, the two column #1 from both sets don’t match. Did you mean Column #2?

I am assuming that your final results should look like this:

DATA

CVS Value    Col #1     Col #2
=========    ======     ======
csv          data1      123
csv2         data2      456

(but in csv format, of course).

Is this correct?

If you have a fairly modern version of BASH, you can use associative arrays. This allows you to have the concept of a key equaling a value.

Let’s say you create an associate array out of both DATA and CSV where the array is keyed by column #2, you could then go through an array, and determine if there’s a matching value, and outputting the data the way you want.

You can set an associative array value by this:

my_array[key]="value"

You can get the value associated with key like this:

echo "${my_array[key]}"

You can get a list of all values like this:

echo "${my_array[*]}"

You can get all keys like this:

echo "${my_array[@]}"

Here’s a quick and dirty program. You probably want something to verify that you don’t have duplicate keys when you create your array, and that a particular key has a value associate with it when you print your array:

#! /bin/bash
csv="csv,123
csv2,456"

data="data1,123
data2,456"

# Create the Data Array Hash keyed by Col #2
while IFS="," read -r col1 col2
do
    data_array[$col2]=$col1
done <<EOD
$data
EOD


# Create the CSV Array Hash keyed by Col #2
while IFS="," read -r col1 col2
do
    csv_array[$col2]=$col1
done <<EOD
$csv
EOD

#For each key in Data Hash, print out corresponding keyed value in CSV Hash
for key in "${!data_array[@]}"
do
    echo "$key: ${data_array[$key]} ${csv_array[$key]}"
done