Question

I have to remove common words like (is,are,am,was etc) from a text file. what is the efficient way of doing it in java ?

Answer 1

You will have to read the file in, skipping the words you want to remove, and then write the file back out again.

Because of this, you may prefer to just skip the words you want to ignore each time you read it – depends on your use case.

To actually remove the words on a line-by-line basis (which may not be the way you want to do it anyway), you could do this (using google guava):

    // the words you want to remove from the file:
    //
    Set<String> wordsToRemove = ImmutableSet.of("a", "for");

    // this code will run in a loop reading one line after another from the file
    //
    String line = "Some words read from a file for example";
    StringBuffer outputLine = new StringBuffer();
    for (String word : Splitter.on(Pattern.compile("\\s+")).trimResults().omitEmptyStrings().split(line)) {
        if (!wordsToRemove.contains(word)) {
            if (outputLine.length() > 0) {
                outputLine.append(' ');
            }
            outputLine.append(word);
        }
    }

    // here I'm just printing, but this line could now be written to the output file.
    //
    System.out.println(outputLine.toString());

Running this code will output:

Some words read from file example

I.e, “a” and “for” are omitted.

Notice that this makes for simple code, but, it will change the whitespace formatting in your file. If you had a line with doubled up spaces, tabs etc, then this all gets changed to a single space in this code. This is just an example of how you might do it, depending on your requirements, there will probably be better ways.