Home/ Questions/Q 8655623
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-12T15:04:35+00:00

I am currently writing an application that needs to be able to select a

  • 0

I am currently writing an application that needs to be able to select a subset of IDs from Millions of users…

I am currently writing software to select a group of 100.000 IDs from a table that contains the whole list of Brazilian population 200.000.000 (200M), I need to be able to do this in a reasonable amount of time… ID on Table = ID on XML

I am thinking of parsing the xml file and starting a thread that performs a SELECT statement on a database, I would need a connection for each thread, still this way seems like a brute force approach, perhaps there is a more elegant way?

1) what is the best database to do this?
2) what is a reasonable limit to the amount of db connections?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Making 100.000 queries would take a long time, and splitting up the work on separate threads won’t help you much as you are reading from the same table.

    Don’t get a single record at a time, rather divide the 100.000 items up in reasonably small batches, for example 1000 items each, which you can send to the database. Create a temporary table in the database with those id values, and make a join against the database table to get those records.

    Using MS SQL Server for example, you can send a batch of items as an XML to a stored procedure, which can create the temporary table from that and query the database table.

    • 0