Home/ Questions/Q 8496341
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-10T23:49:32+00:00

I am currently developing algorithms that work with hundreds of thousands of strings (~4000

  • 0

I am currently developing algorithms that work with hundreds of thousands of strings (~4000 chars each) and perform simple operations based on the results of functions applied to these strings. Currently I use Java and a Mysql database with one table: 

 ID | String | attribute a | attribute b | ....
    |        |             |             | ....

Basically, the algorithm gets one ID to start with, reads the string that is stored, performs functions on it (Attributes are set and read for that currently active column). For example, one function extracts an ID from the String (simple string parsing), stores this ID in the “attribute a” column. Once the entry is parsed, the algorithm reads “attribute a”, jumps to the row with this ID and the process starts all over again.

Maybe I am over-thinking this a little bit; but the current set up has so much overhead, that it is nearly impossible to make some quick changes or to quickly test queries. Is there a better tool or programming language that has been designed for directly operating on large data sets like this and that provides efficient functions for string manipulation?

I definitely wouldn’t mind spending time on learning a completely new language as I believe that using the right tool for the job saves time and prevents frustration in the long term.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    I have a pet project that I’ve been working on, on and off, for years. It stores a large number of strings (although not text). In the past I have implemented it in Java in-memory, Scala with a database, MySQL, C in-memory, Python + Redis… and finally, Go.

    Go has done the best job. I have ~300,000 strings (although shorter than yours) stored in a data structure in memory. They form a searchable, analyzable data structure. I’m sure the use case is similar enough to yours for my experience to be relevant.

    Go has similar efficiency to C for data processing. It has nice syntax like Python for quick coding. It has type safety for … type safety. It has garbage collection.

    My suggestion is, learn Go and do it all in-memory. Rely on virtual memory for accommodating a large data-set. Mine is about 500 MB in RAM once loaded, but I have no dobut it would function just fine at twice that.

    I do not persist to disk because I don’t need to. I can re-create the data structure in 15 minutes from input files. The application is a continually running server. If you’re running large batch operations to do analysis that can be suitable. Otherwise I am sure you can easily perisist to disk.

    (FWIW I’m talking about http://www.folktunefinder.com melody search index)

    • 0