Question

i want a regular expression to validate string to have only text,operators and these brackets ([ ])

Answer 1

To match these characters:

if (str.matches("[a-zA-Z\\[\\]+\\-/*()]*")) {
  ...
}

A better version is:

if (str.matches("\\s*\\[[a-zA-Z]+\\](\\s*[/*+-]\\s*\\[[a-zA-Z]+\\])*")) {
  ...
}

Supporting parentheses is difficult because if you can put expressions in parentheses it is no longer a regular language (it is a context-free language). Regexes are a poor fit for matching that kind of expression. For that you’ll need a PDA (pushdown automaton).