Question

i am reading a csv file line by line here:

def GetDistinctValues(theFile, theColumn):
  lines=theFile.split('\n')
  allValues=[]
  for line in lines:
    if line:
        distinctValue=line.split(',')[theColumn]
        allValues.append(distinctValue)        
  return list(set(allValues))

here is what my csv looks like:

1,hat,dog
2,,cat
3,pants,elephant
4,,,

as you can see, sometimes there are blanks.

in the above code i am trying to get all the unique values in a specific column, but this does not work since the column shifts sometimes because it does not account for blanks.

how can account for all the blanks and get all distinct values from a specific column?

Answer 1

Maybe something like:

import csv

def get_distinct_values(filename, column):
    with open(filename, "rb") as fp:
        reader = csv.reader(fp)
        return set(line[column] for line in reader)

which gives

>>> get_distinct_values("ccol.csv", 0)
set(['1', '3', '2', '4'])
>>> get_distinct_values("ccol.csv", 1)
set(['', 'hat', 'pants'])
>>> get_distinct_values("ccol.csv", 2)
set(['', 'elephant', 'dog', 'cat'])

You could use set(line[column] for line in reader if line[column]) or something if you wanted to get rid of the empty values.