Question

I want to remove duplicate lines from a file but only remove duplicate lines that match a specific regular expression, leaving all other duplicates in the file. Here is what I currently have:

unique_lines = File.readlines("Ops.Web.csproj").uniq do |line|    
  line[/^.*\sInclude=\".*\"\s\/\>$/]
end

File.open("Ops.Web.csproj", "w+") do |file|
  unique_lines.each do |line|
    file.puts line
  end
end

This will deduplicate the lines correctly but will only add the lines that meet the regular expression back into the file. I need all the other lines in the file to be added back unchanged. I know I am missing something small here. Ideas?

Answer 1

Try this:

lines = File.readlines("input.txt")
out = File.open("output.txt", "w+")
seen = {}

lines.each do |line|
  # check if we want this de-duplicated
  if line =~ /Include/
    if !seen[line]
      out.puts line
      seen[line] = true
    end
  else
    out.puts line
  end
end

out.close

Demo:

➜  12980122  cat input.txt
a
b
c
Include a
Include b
Include a
Include a
d
e
Include b
f
➜  12980122  ruby exec.rb
➜  12980122  cat output.txt
a
b
c
Include a
Include b
d
e
f