Home/ Questions/Q 7969883
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-04T07:18:28+00:00

I’m using Windows Azure Table Storage to store millions of entities, however I’m trying

  • 0

I’m using Windows Azure Table Storage to store millions of entities, however I’m trying to figure out the best solution that easily allows for two things:

1) a search on an entity, will retrieve that entity and at least (pageSize) number of entities either side of that entity

2) if there are more entities beyond (pageSize) number of entities either side of that entity, then page next or page previous links are shown, this will continue until either the start or end is reached.

3) the order is reverse chronological order

I’ve decided that the PartitionKey will be the Title provided by the user as each container is unique in the system. The RowKey is Steve Marx’s lexiographical algorithm:

http://blog.smarx.com/posts/using-numbers-as-keys-in-windows-azure

which when converted to javascript instead of c# looks like this:

pad(new Date(100000000 * 86400000).getTime() – new Date().getTime(), 19) + “_” + uuid()

uuid() is a javascript function that returns a guid and pad adds zeros up to 19 chars in length. So records in the system look something like this:

PK                                   RK
TEST    0008638662595845431_ecf134e4-b10d-47e8-91f2-4de9c4d64388
TEST    0008638662595845432_ae7bb505-8594-43bc-80b7-6bd34bb9541b
TEST    0008638662595845433_d527d215-03a5-4e46-8a54-10027b8e23f8
TEST    0008638662595845434_a2ebc3f4-67fe-43e2-becd-eaa41a4132e2

This pattern allows for every new entity inserted to be at the top of the list which satisfies point number 3 above.

With a nice way of adding new records in the system I thought then I would create a mechanism that looks at the first half of the RowKey i.e. 0008638662595845431_ part and does a greater than or less than comparison depending on which direction of the already found item. In other words to get the row immediately before 0008638662595845431 I would do a query like so:

var tableService = azure.createTableService();
var minPossibleDateTimeNumber = pad(new Date(-100000000*86400000).getTime() - new Date().getTime(), 19);

tableService.getTable('testTable', function (error) {
    if (error === null) {
        var query = azure.TableQuery
            .select()
            .from('testTable')
            .where('PartitionKey eq ?', 'TEST')
            .and('RowKey gt ?', minPossibleDateTimeNumber + '_')
            .and('RowKey lt ?', '0008638662595845431_')
            .and('Deleted eq ?', 'false');

If the results returned are greater than 1000 and azure gives me a continuation token, then I thought I would remember the last items RowKey i.e. the number part 0008638662595845431. So now the next query will have the remembered value as the starting value etc.

I am using Windows Azure Node.Js SDK and language is javascript.

Can anybody see gotcha’s or problems with this approach?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    I do not see how this can work effectively and efficiently, especially to get the rows for a previous page.

    To be efficient, the prefix of your “key” needs to be a serially incrementing or decrementing value, instead of being based on a timestamp. A timestamp generated value would have duplicates as well as holes, making mapping page size to row count at best inefficient and at worst difficult to determine.

    Also, this potential algorithm is dependent on a single partition key, destroying table scalability.

    The challenge here would be to have a method of generating a serially incremented key. One solution is to use a SQL database and performing an atomic update on a single row, such that an incrementing or decrementing value is produced in sequence. Something like UPDATE … SET X = X + 1 and return X. Maybe using a stored procedure.

    So the key could be a zero left padded serially generated number. Split such that say the first N digits of the number is the partition key and remaining M digits are the row key.

    For example 
PKey    RKey
00001   10321
00001   10322
….
00954   98912

    Now, since the rows are in sequence it is possible to write a query with the exact key range for the page size.

    Caveat. There is a small risk of a failure occurring between generating a serial key and writing to table storage. In which case, there may be holes in the table. However, your paging algorithm should be able to detect and work around such instances quite easily by specify a page size slightly larger than necessary or by retrying with an adjusted range.

    • 0