Question

I am reading a csv line by line. The CSV looks like this:

29.781646
29.781646
42.698079
43.346914
44.369203
45.006459
45.006459
39.316758

When two numbers are exactly the same I would like to change one slightly.

For example there are two values that are 29.781646 and I would like to change one to be 29.781645

If the csv contains:

29.781646
29.781646
29.781646

then I would like to change it to:

29.781646
29.781645
29.781644

I would very much appreciate your guidance, on implementing this efficiently.

Please note that I want to do this in multiples of 0.000001

Answer 1

You could just iterate over the file line by line, and keep track of seen values using a set, decreasing the value if it is already in the set.

rough example:

seen = set()
with open('test.csv') as input, open('test_out.csv', 'w') as output:
    for line in input:
        value = float(line)
        while value in seen:
            value -= 0.000001
        seen.add(value)
        output.write(str(value) + '\n')

This is efficent since a set offers a O(1) lookup

---

If you’re going to write the values back to the same file anyway, you can use the fileinput module:

import fileinput

seen = set()
for line in fileinput.FileInput('test.csv', inplace=True):
    value = float(line)
    while value in seen:
        value -= 0.000001
    seen.add(value)
    print str(value).strip()

EDIT

To address eumiro’s comment about the floating point issue:

You can use the decimal modules Decimal or just multiply/divide the value with/by 1000000 to work with int instead of using float. As I wrote, this is just a rough example 🙂