Question

I have multiple tab delimited data files that are separated by month in the format jan06.txt, feb06.txt, ..., dec07.txt.

Within each file, it looks something like:

Header1 Header2 Header3 ...
Data1   Data2   Data3   ...
Data4   Data5   Data6   ...
...     ...     ...

What I want to do is to combine all all the data files into a single data file, with just a single header at the top, but also include a new data column that contains the month and year so I don’t lose that information from the file name. So my new, single data file would contain:

Date   Header1 Header2 Header3 ...
200601 Data1   Data2   Data3   ...
200602 Data4   Data5   Data6   ...
...    ...     ...     ...

Where 200601 would refer to Jan 06, 200602 refer to Feb 06, etc.

I know if I do something like cat *.txt > data.txt, I could combine all my files. However, the two issues remain:

1. I have a header in every file that would get concatenated, which I don’t want.
2. I would lose the month information that is stored in the file name.

I think I can do this with some combination of cat and sed, but I am not sure how to start.

Answer 1

Try the following:

function month() {

    case ${1:0:3} in
        "jan") echo "20${1:3:2}01" ;;
        "feb") echo "20${1:3:2}02" ;;
        "mar") echo "20${1:3:2}03" ;;
        "apr") echo "20${1:3:2}04" ;;
        "may") echo "20${1:3:2}05" ;;
        "jun") echo "20${1:3:2}06" ;;
        "jul") echo "20${1:3:2}07" ;;
        "aug") echo "20${1:3:2}08" ;;
        "sep") echo "20${1:3:2}09" ;;
        "oct") echo "20${1:3:2}10" ;;
        "nov") echo "20${1:3:2}11" ;;
        "dec") echo "20${1:3:2}12" ;;
    esac

}

# Header
directory="your_directory/"
echo -en "Date\t" > data.txt
head -1 $(ls "${directory}"/*.txt | head -1) >> data.txt

# Contents
for file in "${directory}"/*.txt; do

    date="${file##*/}"
    date="$(month ${date%*.txt})\t"
    tail -n +2 ${file} | sed 's/^/'${date}'/' >> data.txt

done