Home/ Questions/Q 8211053
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-07T10:17:46+00:00

Applescript: Deleting Duplicates in a text file I have a .txt file which I’ve

  • 0

Applescript: Deleting Duplicates in a text file

I have a .txt file which I’ve created, and I’m trying to delete certain lines of text which are duplicates.

The name I’ve given the file is today.txt, located on my desktop, and it contains a list of URLs for the NYtimes Today’s Paper. However, through parsing an html file, I receive several urls that are duplicates, like so:

http://www.nytimes.com/2012/07/06/education/no-child-left-behind-whittled-down-under-obama.html
http://www.nytimes.com/2012/07/06/business/global/markets-look-to-europes-central-bank-for-action.html
http://www.nytimes.com/2012/07/06/business/global/markets-look-to-europes-central-bank-for-action.html
http://www.nytimes.com/2012/07/06/nyregion/3-children-killed-in-long-island-boating-accident.html
http://www.nytimes.com/2012/07/06/nyregion/3-children-killed-in-long-island-boating-accident.html
http://www.nytimes.com/2012/07/06/world/americas/earthquake-relief-where-haiti-wasnt-broken.html
http://www.nytimes.com/2012/07/06/world/americas/earthquake-relief-where-haiti-wasnt-broken.html
http://www.nytimes.com/2012/07/06/us/politics/journal-critique-of-romney-shows-murdoch-doubt-on-candidacy.html
http://www.nytimes.com/2012/07/06/us/politics/journal-critique-of-romney-shows-murdoch-doubt-on-candidacy.html
http://www.nytimes.com/2012/07/06/technology/at-hacker-hostels-living-on-the-cheap-and-dreaming-of-digital-glory.html
http://www.nytimes.com/2012/07/06/technology/at-hacker-hostels-living-on-the-cheap-and-dreaming-of-digital-glory.html

I’ve been trying to remove the duplicates through a do shell script in Applescript, but I haven’t been able to make it work. Here is my code:

set delDups to do shell script "sort /Users/paolob/Desktop/today.txt | uniq -u"
return delDups

So my question:
How can I delete the duplicates in my today.txt file,
and then save the result to the same today.txt file

Any help would be very appreciated. Thank you in advance.

Edit
It would be more economical and faster if in fact the shell script, or whatever duplicate deleter you suggest, reads the text directly in Applescript Editor, and then sets the new text to a *new_text* variable, for example.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Try this…

    set filePath to (path to desktop as text) & "today.txt"
set theText to read file filePath
set textList to paragraphs of theText

set uniqueList to {}
repeat with i from 1 to count of textList
    set thisParagraph to item i of textList
    if thisParagraph is not in uniqueList then set end of uniqueList to thisParagraph
end repeat

set {tids, text item delimiters} to {text item delimiters, return}
set uniqueText to uniqueList as text
set text item delimiters to tids

set openFile to open for access file filePath with write permission
set eof of openFile to 0
write uniqueText to openFile starting at eof as text
close access openFile
    • 0