Tip: Remove duplicate lines with uniq

Get to know your textutils

Duplicate lines don't often cause a problem, but sometimes they really do. And when they do, there's little need to spend an afternoon working up a filter for them, when the uniq command is at your very fingertips. Find out how it can save you time and headaches.

Share:

Jacek Artymiak (jacek@artymiak.com), Freelance author and consultant

Jacek Artymiak works as a freelance consultant, developer, and writer. Since 1991 he's been developing software for many commercial and free variants of UNIX and BSD operating systems (AIX, HP-UX, IRIX, Solaris, Linux, FreeBSD, NetBSD, OpenBSD, and others), as well as MS-DOS, Microsoft Windows, Mac OS, and Mac OS X. Jacek specializes in business and financial application development, Web design, network security, computer graphics, animation, and multimedia. He's a prolific writer on technology subjects and the coauthor of "Install, Configure, and Customize Slackware Linux" (Prima Tech, 2000) and "StarOffice for Linux Bible" (IDG Books, 2000). Many of Jacek's software projects can be found at SourceForge. You can learn more about him at his personal Web site and contact him at jacek@artymiak.com.



03 April 2003

Also available in Japanese

After sorting, you may discover that some lines are duplicated. Sometimes this duplicate information is not needed and can be removed to save space on disk. The lines of text do not have to be sorted, but you should remember that uniq compares lines as it reads them and will remove only two or more consecutive lines. The following example illustrates how it works in practice:

Listing 1. Removing duplicate lines with uniq
$ cat happybirthday.txt

Happy Birthday to You!

Happy Birthday to You!

Happy Birthday Dear Tux!

Happy Birthday to You!


$ sort happybirthday.txt 

Happy Birthday Dear Tux!

Happy Birthday to You!

Happy Birthday to You!

Happy Birthday to You!


$ sort happybirthday.txt | uniq

Happy Birthday Dear Tux!

Happy Birthday to You!

Be warned that it is a bad idea to use uniq or any other tool to remove duplicate lines from files containing financial or other important data. In such cases, a duplicate line almost always means another transaction for the same amount, and removing it would cause a lot of trouble for the accounting department. Do not do it!

More on uniq

This series offers an introduction to text utilities, which complements information found in the man and info pages. You'll learn even more if you open a new terminal window and type either man uniq or info uniq -- or you can open a new browser window and view the uniq man page at gnu.org.

What if you wanted to make your job a little easier and display, say, only the unique or duplicate lines? You can do this with the -u (unique) and -d (duplicate) options, like this:

Listing 2. Using the -u and -d options
$ sort happybirthday.txt | uniq -u

Happy Birthday Dear Tux!


$ sort happybirthday.txt | uniq -d

Happy Birthday to You!

You can also get some statistics out of uniq with the -c option:

Listing 3. Using the -c option
$ sort happybirthday.txt | uniq -uc

      1 Happy Birthday Dear Tux!


$ sort happybirthday.txt | uniq -dc

      3 Happy Birthday to You!

If uniq compared just full lines it would still be useful, but that's not the end of this command's functionality. Especially handy is its ability to skip the given number of fields, using the -f option followed by the number of fields to skip. This is very useful when you are looking at system logs. Quite often, some entries are duplicated many times, which makes it difficult to look at logs. Using plain uniq won't work, because every entry begins with a different timestamp. But if you tell it to skip all time fields, suddenly your logs will become more manageable. Try uniq -f 3 /var/log/messages and see for yourself.

There is also another option, -s, which works just like -f but skips the given number of characters. You can use -f and -s together. uniq skips fields first, then characters. And what if you wanted to use only a preset number of characters for comparison? Try the -w option.

Questions or comments? I'd love to hear from you -- send mail to jacek@artymiak.com.

Next time, we'll take a look at nl. See you then!

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Linux on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Linux, Open source
ArticleID=11302
ArticleTitle=Tip: Remove duplicate lines with uniq
publish-date=04032003