Question

Say I have a file like:

apple
pear
lemon
lemon
pear
orange
lemon

How do I make it so that I only keep the unique lines, so I get:

apple
pear
lemon
orange

I can either modify the original file or create a new one.

I’m thinking there’s a way to scan the original file a line at a time, check whether or not the line exists in the new file, and then append if it doesn’t. I’m not dealing with really large files here.

Answer 1

@echo off
setlocal disabledelayedexpansion
set "prev="
for /f "delims=" %%F in ('sort uniqinput.txt') do (
  set "curr=%%F"
  setlocal enabledelayedexpansion
  if "!prev!" neq "!curr!" echo !curr!
  endlocal
  set "prev=%%F"
)

What it does: sorts the input first, and then goes though it sequentially and outputs only if current line is different to previous one. It could have been even simpler if not for need to handle special characters (that’s why those setlocal/endlocal are for).
It just echoes lines to stdout, if you want to write to file do (assuming you named your batch myUniq.bat) myUniq >>output.txt