Question

There are two style of comments , C-style and C++ style, how to recognize them?

/* comments */

// comments

I am feel free to use any methods and 3rd-libraries.

Answer 1

To reliably find all comments in a Java source file, I wouldn’t use regex, but a real lexer (aka tokenizer).

Two popular choices for Java are:

• JFlex: http://jflex.de
• ANTLR: http://www.antlr.org

Contrary to popular belief, ANTLR can also be used to create only a lexer without the parser.

Here’s a quick ANTLR demo. You need the following files in the same directory:

• antlr-3.2.jar
• JavaCommentLexer.g (the grammar)
• Main.java
• Test.java (a valid (!) java source file with exotic comments)

JavaCommentLexer.g

lexer grammar JavaCommentLexer;

options {
  filter=true;
}

SingleLineComment
  :  FSlash FSlash ~('\r' | '\n')*
  ;

MultiLineComment
  :  FSlash Star .* Star FSlash
  ;

StringLiteral
  :  DQuote
     ( (EscapedDQuote)=> EscapedDQuote
     | (EscapedBSlash)=> EscapedBSlash
     | Octal
     | Unicode
     | ~('\\' | '"' | '\r' | '\n')
     )*
     DQuote {skip();}
  ;

CharLiteral
  :  SQuote
     ( (EscapedSQuote)=> EscapedSQuote
     | (EscapedBSlash)=> EscapedBSlash
     | Octal
     | Unicode
     | ~('\\' | '\'' | '\r' | '\n')
     )
     SQuote {skip();}
  ;

fragment EscapedDQuote
  :  BSlash DQuote
  ;

fragment EscapedSQuote
  :  BSlash SQuote
  ;

fragment EscapedBSlash
  :  BSlash BSlash
  ;

fragment FSlash
  :  '/' | '\\' ('u002f' | 'u002F')
  ;

fragment Star
  :  '*' | '\\' ('u002a' | 'u002A')
  ;

fragment BSlash
  :  '\\' ('u005c' | 'u005C')?
  ;

fragment DQuote
  :  '"' 
  |  '\\u0022'
  ;

fragment SQuote
  :  '\'' 
  |  '\\u0027'
  ;

fragment Unicode
  :  '\\u' Hex Hex Hex Hex
  ;

fragment Octal
  :  '\\' ('0'..'3' Oct Oct | Oct Oct | Oct)
  ;

fragment Hex
  :  '0'..'9' | 'a'..'f' | 'A'..'F'
  ;

fragment Oct
  :  '0'..'7'
  ;

Main.java

import org.antlr.runtime.*;

public class Main {
  public static void main(String[] args) throws Exception {
    JavaCommentLexer lexer = new JavaCommentLexer(new ANTLRFileStream("Test.java"));
    CommonTokenStream tokens = new CommonTokenStream(lexer);
      for(Object o : tokens.getTokens()) {
      CommonToken t = (CommonToken)o;
      if(t.getType() == JavaCommentLexer.SingleLineComment) {
        System.out.println("SingleLineComment :: " + t.getText().replace("\n", "\\n"));
      }
      if(t.getType() == JavaCommentLexer.MultiLineComment) {
        System.out.println("MultiLineComment  :: " + t.getText().replace("\n", "\\n"));
      }
    }
  }
}

Test.java

\u002f\u002a <- multi line comment start
multi
line
comment // not a single line comment
\u002A/
public class Test {

  // single line "not a string"

  String s = "\u005C" \242 not // a comment \\\" \u002f \u005C\u005C \u0022;
  /*
  regular multi line comment
  */
  char c = \u0027"'; // the " is not the start of a string

  char q1 = '\u005c'';                  // == '\''
  char q2 = '\u005c\u0027';             // == '\''
  char q3 = \u0027\u005c\u0027\u0027;   // == '\''
  char c4 = '\047';

  String t = "/*";
  \u002f\u002f another single line comment
  String u = "*/";
}

Now, to run the demo, do:

bart@hades:~/Programming/ANTLR/Demos/JavaComment$ java -cp antlr-3.2.jar org.antlr.Tool JavaCommentLexer.g
bart@hades:~/Programming/ANTLR/Demos/JavaComment$ javac -cp antlr-3.2.jar *.java
bart@hades:~/Programming/ANTLR/Demos/JavaComment$ java -cp .:antlr-3.2.jar Main

and you’ll see the following being printed to the console:

MultiLineComment  :: \u002f\u002a <- multi line comment start\nmulti\nline\ncomment // not a single line comment\n\u002A/
SingleLineComment :: // single line "not a string"
SingleLineComment :: // a comment \\\" \u002f \u005C\u005C \u0022;
MultiLineComment  :: /*\n  regular multi line comment\n  */
SingleLineComment :: // the " is not the start of a string
SingleLineComment :: // == '\''
SingleLineComment :: // == '\''
SingleLineComment :: // == '\''
SingleLineComment :: \u002f\u002f another single line comment

EDIT

You can create a sort of lexer with regex yourself, of course. The following demo does not handle Unicode literals inside source files, however:

Test2.java

/* <- multi line comment start
multi
line
comment // not a single line comment
*/
public class Test2 {

  // single line "not a string"

  String s = "\" \242 not // a comment \\\" ";
  /*
  regular multi line comment
  */
  char c = '"'; // the " is not the start of a string

  char q1 = '\'';                  // == '\''
  char c4 = '\047';

  String t = "/*";
  // another single line comment
  String u = "*/";
}

Main2.java

import java.util.*;
import java.io.*;
import java.util.regex.*;

public class Main2 {

  private static String read(File file) throws IOException {
    StringBuilder b = new StringBuilder();
    Scanner scan = new Scanner(file);
    while(scan.hasNextLine()) {
      String line = scan.nextLine();
      b.append(line).append('\n');
    }
    return b.toString();
  }

  public static void main(String[] args) throws Exception {
    String contents = read(new File("Test2.java"));

    String slComment = "//[^\r\n]*";
    String mlComment = "/\\*[\\s\\S]*?\\*/";
    String strLit = "\"(?:\\\\.|[^\\\\\"\r\n])*\"";
    String chLit = "'(?:\\\\.|[^\\\\'\r\n])+'";
    String any = "[\\s\\S]";

    Pattern p = Pattern.compile(
        String.format("(%s)|(%s)|%s|%s|%s", slComment, mlComment, strLit, chLit, any)
    );

    Matcher m = p.matcher(contents);

    while(m.find()) {
      String hit = m.group();
      if(m.group(1) != null) {
        System.out.println("SingleLine :: " + hit.replace("\n", "\\n"));
      }
      if(m.group(2) != null) {
        System.out.println("MultiLine  :: " + hit.replace("\n", "\\n"));
      }
    }
  }
}

If you run Main2, the following is printed to the console:

MultiLine  :: /* <- multi line comment start\nmulti\nline\ncomment // not a single line comment\n*/
SingleLine :: // single line "not a string"
MultiLine  :: /*\n  regular multi line comment\n  */
SingleLine :: // the " is not the start of a string
SingleLine :: // == '\''
SingleLine :: // another single line comment