Question

Good night,

I am currently working on a very simple lexical analiser for human language in C# based on Regex matching, and I am currently facing the problem of specifing a Regex that can match every possible punctuation symbol in the target language, and another one that can match also every kind of blank spaces. After searching here in Stack Overflow and another site I found that I can respectively use Regexs \p{P} for punctuation and \p{Z} for whitespace, and it works when reading from a text file. The question is that in the final program it will be needed to read records from an SQL database and process them using this lexical analiser, and I don’t know what is the encoding of the records in the DB. Can this be a problem in this situation? Can the mentioned Regexes match every punctuation and whitespace character of an input no matter what their encoding is?

Thank you very much.

Answer 1

The .NET Framework will convert strings to Unicode from the database. Whether it converts them correctly depends on whether something told it what the database text encoding was. But the strings your Regex sees will be in Unicode.

So, assuming that the database access layer correctly converted the text from the database record, you don’t have to worry about encoding because it’s always Unicode.