Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I’m designing architecture of a text parser. Example sentence: Content here, content here.
Whole sentence is a… sentence, that’s obvious. The, quick etc are words; , and . are punctuation marks. But what are words and punctuation marks all together in general? Are they just symbols? I simply don’t know how to name what a single sentence consists of in the most reasonable abstract way (because one may write it consists of letters/vowels etc).
Thanks for any help 🙂
What you’re doing is technically lexical analysis (“lexing”), which takes a sequence of input symbols and generates a series of tokens or lexemes. So word, punctuation and white-space are all tokens.
In (E)BNF terms, lexemes or tokens are synonymous with “terminal symbols”. If you think of the set of parsing rules as a tree the terminal symbols are the leaves of the tree.
So what’s the atom of your input? Is it a word or a sentence? If it’s words (and white-space) then a sentence is more akin to a parsing rule. In fact the term “sentence” can itself be misleading. It’s not uncommon to refer to the entire input sequence as a sentence.
A semi-common term for a sequence of non-white-space characters is a “textrun”.