Trailing spaces in data files cause silent headaches — deduplication fails, string comparisons return false, joins don’t match. awk '{$1=$1;print}' is the canonical one-liner for stripping leading and trailing whitespace. The sort | uniq pipeline on top gives you clean deduplicated output.
- create a file with duplicate records
1
2
3
4
5
6
7
8
9
10
11
| $ echo "test" > space.txt
$ echo " test" >> space.txt
$ echo " test " >> space.txt
$ echo "test " >> space.txt
$ echo "test1" >> space.txt
$ echo "test2" >> space.txt
|
1
2
3
4
5
6
7
8
| $ cat space.txt
test
test
test
test
test1
test2
|
- remove trailing white space using awk
1
2
3
4
5
6
7
8
| $ cat space.txt | awk '{$1=$1;print}'
test
test
test
test
test1
test2
|
- sort and then get all unique records
1
2
3
4
5
| $ cat space.txt | awk '{$1=$1;print}' | sort | uniq
test
test1
test2
|