Home/ Questions/Q 6733909
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-26T10:49:35+00:00

I’m curious if there’s a library for python OR javascript to tokenize sentences of

  • 0

I’m curious if there’s a library for python OR javascript to tokenize sentences of a string of sentences and put new line at each sentence?

IE:

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum aliquet leo in urna hendrerit placerat. Donec adipiscing dignissim adipiscing. Duis adipiscing mollis cursus. Etiam fringilla elit nec enim sagittis a auctor nisi gravida. Nunc sollicitudin, leo sit amet consequat pharetra, mi orci vestibulum mi, a suscipit odio tellus tincidunt erat. Suspendisse a consequat turpis. Morbi eget ante leo, a dignissim mi.

to

Lorem ipsum dolor sit amet, consectetur adipiscing elit.\n
Vestibulum aliquet leo in urna hendrerit placerat.\n
Donec adipiscing dignissim adipiscing. \n
Duis adipiscing mollis cursus. Etiam fringilla elit nec enim sagittis a auctor nisi gravida. Nunc sollicitudin, leo sit amet consequat pharetra, mi orci vestibulum mi, a suscipit odio tellus tincidunt erat. \n
Suspendisse a consequat turpis. \n
Morbi eget ante leo, a dignissim mi.
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    You are looking for a natural language library.

    For Python there is Natural Language Toolkit (NLTK). For example you could take a look at the PunktSentenceTokenizer.

    The PunktSentenceTokenizer divides a text into a list of sentences, by using an unsupervised algorithm to build a model for abbreviation words, collocations, and words that start sentences. It must be trained on a large collection of plaintext in the taret language before it can be used. The algorithm for this tokenizer is described in Kiss & Strunk (2006):

    Kiss, Tibor and Strunk, Jan (2006): Unsupervised Multilingual Sentence
    Boundary Detection. Computational Linguistics 32: 485-525.

    The NLTK data package includes a pre-trained Punkt tokenizer for English.

    • 0