Question

I want to go through all available Java (or any other language) source code in a given project and:

1. set up a statistical distribution of permutations of keywords and their relations
2. pick out repetitive sequences characters, constructions (patterns) out of it.

What kind of toolset would you recommend me?

For example:

I want it to be able to pick:

* ( * ) {
  *
}

out of

public static void main ( String[] args ) {
  System.err.println( "Specific Text" );
}

or

\n;

out of

  System.err.println( "Specific Text" );
  System.err.println( "Specific Text" );
  System.err.println( "Specific Text" );
  System.err.println( "Specific Text" );
  System.err.println( "Specific Text" );
  System.err.println( "Specific Text" );
  System.err.println( "Specific Text" );
  System.err.println( "Specific Text" );

Answer 1

You need a language parser. One I can think of right is http://www.antlr.org/. You end up using grammar that already exists for Java and parse a file and look for tokens. For example, if,for,when,etc… are all keywords that the parser will tell you. But 123 is not a keyword.