Question

I want to use this to get data from row out of mysql database into text files (one entry under another, 50 entries per file):

$ mysql --user=XXX --password=XXX --batch --skip-column-names \
 -e "SELECT userid, displayname FROM Users" stackoverflowdb | \
split -l 50 -a 5 - "result."

but I also don’t want to copy duplicate entries to these files. Will this code remove duplicates or do i need to add something to it to don’t copy duplicate entries?

Answer 1

Modifying the SQL to use the DISTINCT directive

e.g.

 SELECT DISTINCT userid,displayname FROM Users 

will ensure that only unique combinations of userid and displayname are selected.

However this will not prevent userids that have identical displaynames.