uniq - Report or filter out repeated lines in a file

Format

  • uniq [-c|-d|-u] [-B] [-f number1] [-s number2] [-W option[option] ... [input_file [output_file]]
  • uniq [-c|-d|-u] [–-number] [-+number] [-B] [-W option[option] ... [input_file [output_file]]

Description

unique manipulates lines that occur more than once in a file. The file must be sorted, since unique only compares adjacent lines. When you invoke this command with no options, it writes only one copy of each line in input_file to output_file. If you do not specify input_file or you specify , unique reads the standard input.

If you do not specify output_file, unique uses the standard output. The specified output_file cannot be a FIFO.

Options

-B
Disables the automatic conversion of tagged files. This option is ignored if the filecodeset or pgmcodeset options (-W option) are specified.
-c
Precedes each output line with the number of times that line occurred in the input.
-d
Displays only lines that are repeated (one copy of each line).
-f number1
Ignores the first number1 fields when comparing lines. Blanks separate fields in the input.
-s number2
Ignores the first number2 characters when comparing lines. If you specify both -s and -f, unique ignores the first number2 characters after the first number1 fields.
-u
Displays only those lines that are not repeated.
-W option[,option]...
Specifies z/OS-specific options. The option keywords are case-sensitive. Possible options are:
filecodeset=codeset
Performs text conversion from one code set to another when reading from the file. The coded character set of the file is codeset. codeset can be a code set name known to the system or a numeric coded character set identifier (CCSID). Note that the command iconv -l lists existing CCSIDs along with their corresponding code set names. The filecodeset and pgmcodeset options can be used on files with any file tag.

If pgmcodeset is specified but filecodeset is omitted, then the default file code set is ISO8859-1 even if the file is tagged with a different code set. If neither filecodeset nor pgmcodeset is specified, text conversion will not occur unless automatic conversion is enabled or the _TEXT_CONV environment variable indicates text conversion. For more information about text conversion, see Controlling text conversion for z/OS UNIX shell commands.

If filecodeset or pgmcodeset is specified, then automatic conversion is disabled for this command invocation and the -B option is ignored if it is also specified. See z/OS UNIX System Services Planning for more information about automatic conversion.

When specifying values for filecodeset, use the values that Unicode Service supports. For more information about supported code sets, see z/OS Unicode Services User's Guide and Reference.

pgmcodeset=codeset
Performs text conversion from one code set to another when reading from the file. The coded character set of the program (command) is codeset. codeset can be a code set name known to the system or a numeric coded character set identifier (CCSID). Note that the command iconv -l lists existing CCSIDs along with their corresponding code set names. The filecodeset and pgmcodeset options can be used on files with any file tag.

If filecodeset is specified but pgmcodeset is omitted, then the default program code set is IBM-1047. If neither filecodeset nor pgmcodeset is specified, text conversion will not occur unless automatic conversion is enabled or the _TEXT_CONV environment variable indicates text conversion. For more information about text conversion, see Controlling text conversion for z/OS UNIX shell commands.

If filecodeset or pgmcodeset is specified, then automatic conversion is disabled for this command invocation and the -B option is ignored if it is also specified. See z/OS UNIX System Services Planning for more information about automatic conversion.

The only supported values for pgmcodeset are IBM-1047 and 1047.

You can choose only one of the -c, -d, or -u options.
–number
Equivalent to -f number (obsolescent).
+number
Equivalent to -s number (obsolescent).

Examples

  1. The command:
    uniq
    is a filter which prints one copy of each different line in its sorted input.
  2. The command:
    uniq -f 2 -s 1
    compares lines starting with the second character of the third field.
  3. The command:
    uniq -d
    prints one instance of each repeated line in the input (and omits all unique lines).
  4. To print one copy of each different line in a text file containing UTF-8 characters, assuming that
    • The text file is untagged and you do not want to tag it or enable automatic conversion, and
    • You cannot alter the tag (for example, you are displaying an untagged public text file or a read-only text file)
    then issue:
    uniq -W filecodeset=UTF-8,pgmcodeset=IBM-1047 myUtf8File
  5. To print only those lines that are not repeated in a text file containing EBCDIC characters, assuming that automatic conversion has been enabled but the text file is incorrectly tagged as ASCII:
    uniq -u -B myMisTaggedFile

Localization

unique uses the following localization environment variables:
  • LANG
  • LC_ALL
  • LC_CTYPE
  • LC_MESSAGES
  • NLSPATH

See Localization for more information.

Environment variables

unique uses the following environment variable:
_TEXT_CONV
Contains text conversion information for the command. The text conversion information is not used when either the -B option or the filecodeset or pgmcodeset option (-w option) is specified. For more information about text conversion, see Controlling text conversion for z/OS UNIX shell commands.

Exit values

0
Successful completion.
1
Failure due to any of the following reasons:
  • Incorrect command-line option.
  • Missing number after –f.
  • Missing or incorrect number after –s.
  • Inability to open the input or output file.
  • The code set is not valid.
  • Could not turn off automatic conversion.
  • Could not perform requested text conversion.
3
Failure while reading the input file:
  • Error during system call.
  • Input line too long.
  • Incorrect character in input.

Messages

Possible error messages include:
Missing character skip count
You specified –s but did not supply a number after the –s.
Missing number of fields to skip
You specified –f but did not supply a number after the –f.
Field skip not a number in string
In a –number or +number construct, number was not a valid number. This could happen because of a typographical error in entering a option.

Portability

POSIX.2, X/Open Portability Guide, UNIX systems.

The -B and -W options are extensions of the POSIX standard. The -number and +number options are considered obsolete.

Related information

comm, sort