Question

Two questions concerning using uniq command, please help.

First question

Say I have two files;

$ cat 1.dat
0.1 1.23
0.2 1.45
0.3 1.67

$ cat 2.dat
0.3 1.67
0.4 1.78
0.5 1.89

Using cat 1.dat 2.dat | sort -n | uniq > 3.dat, I am able to merge two files into one. results is:

0.1 1.23
0.2 1.45
0.3 1.67
0.4 1.78
0.5 1.89

But if I have a scientific notation in 1.dat file,

$ cat 1.dat
1e-1 1.23
0.2 1.45
0.3 1.67

the result would be:

0.2 1.45
0.3 1.67
0.4 1.78
0.5 1.89
1e-1 1.23

which is not what I want, how can I let uniq understand 1e-1 is a number, not a string.

Second question

Same as above, but this time, let the second file 2.dat‘s first row be slightly different (from 0.3 1.67 to 0.3 1.57)

$ cat 2.dat
0.3 1.57
0.4 1.78
0.5 1.89

Then the result would be:

0.1 1.23
0.2 1.45
0.3 1.67
0.3 1.57
0.4 1.78
0.5 1.89

My question is this, how could I use uniq just based on the value from the first file and find repetition only from the first column, so that the results is still:

0.1 1.23
0.2 1.45
0.3 1.67
0.4 1.78
0.5 1.89

Thanks

A more complex test cases

$ cat 1.dat
1e-6 -1.23
0.2 -1.45
110.7 1.55
0.3 1.67e-3

Answer 1

one awk (gnu awk) one-liner solves your two problems

  awk '{a[$1*1];b[$1*1]=$0}END{asorti(a);for(i=1;i<=length(a);i++)print b[a[i]];}' file2 file1

test with data: Note, I made file1 unsorted and 1.57 in file2, as you wanted:

kent$  head *
==> file1 <==
0.3 1.67
0.2 1.45
1e-1 1.23

==> file2 <==
0.3 1.57
0.4 1.78
0.5 1.89

kent$  awk '{a[$1*1];b[$1*1]=$0}END{asorti(a);for(i=1;i<=length(a);i++)print b[a[i]];}' file2 file1
1e-1 1.23
0.2 1.45
0.3 1.67
0.4 1.78
0.5 1.89

edit

display 0.1 instead of 1e-1:

kent$  awk '{a[$1*1];b[$1*1]=$2}END{asorti(a);for(i=1;i<=length(a);i++)print a[i],b[a[i]];}' file2 file1
0.1 1.23
0.2 1.45
0.3 1.67
0.4 1.78
0.5 1.89

edit 2

for the precision, awk default (OFMT) is %.6g you could change it. but if you want to display different precision by lines, we have to a bit trick:

(I added 1e-9 in file1)

kent$  awk '{id=sprintf("%.9f",$1*1);sub(/0*$/,"",id);a[id];b[id]=$2}END{asorti(a);for(i=1;i<=length(a);i++)print a[i],b[a[i]];}'  file2 file1 
0.000000001 1.23
0.2 1.45
0.3 1.67
0.4 1.78
0.5 1.89

if you want to display same number precision for all lines:

kent$  awk '{id=sprintf("%.9f",$1*1);a[id];b[id]=$2}END{asorti(a);for(i=1;i<=length(a);i++)print a[i],b[a[i]];}'  file2 file1 
0.000000001 1.23
0.200000000 1.45
0.300000000 1.67
0.400000000 1.78
0.500000000 1.89