Home/ Questions/Q 9142921
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-17T09:53:47+00:00

I have following problem: I’m searching for similarities. Therefore I have a big source

  • 0

I have following problem:

I’m searching for similarities. Therefore I have a big source table with 200000 entries and second table with 10000 entries. Now I’m retrieving a entry set for each table and compare every row in the source table with every row in the second table in java (I’m using some NeedleMan Gotoh algorithm and similar more complex algorithms). That means 1 billion comparisons and that’s too much and too slow…

The goal is a table with all similarities (id from source table, id from second table and a similarity value) or at least something like the best match (or best x matches) for every entry…

Could anyone give me some advice to do such calculations in a “normal” time?

EDIT

Main Table

---+------+-------------+---------+-------+
id | name | address     | country | plz   | ...
---+------+-------------+---------+-------+
20 | Sony | Main Str. 1 | US      | 10000 |
---+------+-------------+---------+-------+

Second Table

---+------+-------------+---------+-------+
id | name | address     | country | plz   | ...
---+------+-------------+---------+-------+
30 | Soni | MainStr. 1  | US      | 10000 |
---+------+-------------+---------+-------+

Goal (similarity table):

---+---------------+--------------+-----------+
id | id_source_tbl | id_second_tbl| similarity|
---+---------------+--------------+-----------+
1  | 20            | 30           | 0.99      |
---+---------------+--------------+-----------+

simil_value is a value that indicates, how likely the company in the source table is the same as the company in the second table

the result indicates, that the two rows are representing the same company… the two entries just differ because of small typos… (0.99 is the similarity and is very high => companies are the same)
Similarity is calculated with a needleman wunsch gotoh algorithm (comparing char for char and considering position in string and so on… typos should result in a high similarity value)

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Actually, I made the problem myself…

    Solution for me was following:
    1) don’t reuse connections, always close them with the corresponing ResultSet
    2) use transactions
    3) split work to threads
    4) if you, like me, have results for single rows (ALL similarities for one single entry) and want to calc something on this subresult (like in my case, for all similarities I wanted to calc the rank), do this in java and use the subresult!!!! instead of doing it afterwards in mysql

    The result for me is about 1 day of calculation time instead of 3 weeks…

    thanks for the help

    • 0