Bdtyktl

I have two files - file1 & file2.

file1 contains (only words):

ABC 

YUI 

GHJ 

I8O

file2 contains many paragraphs:

dfghjo ABC kll 

njjgg bla bla GHJ 

njhjckhv chasjvackvh .. 

ihbjhi hbhibb jh jbiibi 

I am using the command below to get the matching lines which contains word from file1 in file2:

grep -Ff file1 file2 

(Gives output of lines where words of file1 found in file2)

I also need the words from file1 which are not found in file2.

Can anyone help in getting this output:

YUI 

I8O

I am looking for a one liner command (via grep, awk, sed), as I am using pssh command and can't use while or for loops.

Here's one way in awk:

$ awk 'NR==FNR{a[$1]++; next}{for(i in a){if($0 ~ i){found[i]++}}}END{for(i in a){if(!found[i]){print i}}}' file1 file2 

YUI

I8O

Or, a bit more legibly:

$ awk 'NR==FNR{

        a[$1]++; 

        next

       }

       {

        for(i in a){

            if($0 ~ i){

                found[i]++

            }

        }

       }

       END{

        for(i in a){

            if(!found[i]){

                print i

            }

        }

       }' file1 file2 

YUI

I8O

Explanation

• 
  NR==FNR : NR is the current line number and FNR is the current line number of the current file. When processing multiple files, the two will be equal only while reading the first file. So this is an easy way of saying "do this for the 1st file only".


• 
  a[$1]++; next : while reading the first file, save each word (the first and only field) in the array a and then skip to the next line. The next also ensures that the rest of the command is not run for the first file.


• 
  for(i in a){ if($0 ~ i){ found[i]++ } }: For each of the words found in the first file (the keys of array a), check if the current line matches that word. If it does, save the word in the found array. This is run for each line of the second input file.


• 
  END{ }: do this after you've processed all input files.


• 
  for(i in a){ if(!found[i]){ print i } }: for each of the words in a, if the word is not also in the found array, print that word.

Alternatively, you can use some of the core Linux utilities:

$ grep -hoP 'w+' file1 file2 | sort | uniq -u | xargs -I{} grep {} file1

I8O

YUI 

Explanation

$ grep -hoP 'w+' file1 file2

ABC

YUI

GHJ

I8O

dfghjo

ABC

kll

njjgg

bla

bla

GHJ

njhjckhv

chasjvackvh

ihbjhi

hbhibb

jh

jbiibi

This will print all the words found in each file. The -o flag means "only print the matching portion of the line", the -P enables Perl Compatible Regular Expressions (PCRE) which let us use w to mean "any word character" (so letters, numbers, _).

$ grep -hoP 'w+' file1 file2 | sort | uniq -u

chasjvackvh

dfghjo

hbhibb

I8O

ihbjhi

jbiibi

jh

kll

njhjckhv

njjgg

YUI

Now we pass the output of the previous command through sort and uniq -u to keep only unique matches: these are the words that are only present in one of the two files.

$ grep -hoP 'w+' file1 file2 | sort | uniq -u | xargs -I{} grep {} file1

I8O

YUI 

Finally, we feed this list of unique words to xargs and have it grep each of them in file1. Only those unique words that are present in file1 will be returned, and unique words present in file1 are therefore not present in file2.

try this command:

grep -oFf file1 file2 | grep -vFf - file1

where first use file1 as PATTERN and get only the part of a matching line that matches PATTERN in file2, first command give you:

ABC

GHJ

then use this output as input file PATTERN and search line in file1 that doesn't match PATTERN, and you will get:

YUI

I8O

Tested on Red Hat Enterprise Linux ES release 4 (Nahant Update 3)