Question

Let me explain with an example.
We have the following text:

“Comme Il Faut was founded in 1927. The tobacco company is most well known for its reputation of producing customized private label brands for its partners worldwide”.

This is normal text. But the following text:

“CommeIlFautwasfounded in 1927. The tobacco companyi most wellknown foritsreputation of producing customizedprivatelabelbrands foritspartners worldwide”

This is text anomaly: typos, words without a space, maybe something else.

How to search for such anomalies?
What algorithms are there for this (statistical)?

It is desirable that the result was a percentage: for example, 80% of the anomalies.

Thanks.

Answer 1

Construct a Trie tree with all the known words in the dictionary.
Take each word that apears in your text and try to find it in the Trie tree. If you don’t find it then try to match prefix of length-k. If you find a match then you apply the same procedure to the rest k characters. It’s recursive and it could catch more than two concatenated words