Home/ Questions/Q 8837013
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-14T09:37:20+00:00

Lets ay you are building a analytics program for your site, in which you

  • 0

Lets ay you are building a analytics program for your site, in which you log the address of the page every time you visit it, so your log.txt could be

x.com/a
x.com/b
x.com/a
x.com/c
x.com/a

There is no counter, its just a log file and no sql is used, given that this had thousands of elements your a thousand or so unique domains addresses (x.com/a x.com/b), what’s the most efficient way of going though this list and spitting out the top 10 urls.

My best solution would go through the log file, then if that domain does not exist in hashtable add it in as key, and increment it’s value; then search on hash for largest 10 values.

I’m not convinced this is the best solution, not only because of the space complexity (what happens if the unique domain went from a few thousands to few million), and because I would need to conduct another search on hashtable to find the largest values.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Even for few thousands or few millions entries – your approach is just fine – it has a linear on average (O(n)) run time – so it is not that bad.

    However, you can use a map-reduce approach if you want more scalability.

    map(file):
   for each entry in file:
         emitIntermediate(getDomainName(entry),"1"))
reduce(domain,list):
   emit(domain,size(list))

    The above will efficiently give you the list of (domain,count) tupples, and all you have to do is select the top 10.

    Selecting the top 10 can be done using a map-reduce (distributed) sort for scalability – or using a min heap (iterate while maintaining the top 10 elements encountered in the heap). The second is explained with more details in this thread

    About space complexity: If you are using a 64 bit system, you can use it as RAM, and let the OS do what it cans (by swapping elements to disk when needed), it is very unlikely that you will need more then the amount of Virtual Memory you have on a 64 bits machine. An alternative is to use a hash table (or a B+ tree) optimized for file systems and do the same algorithm on it.

    However, if this is indeed the case – and the data does not fit RAM, and you cannot use map-reduce, I suspect sorting and iterating – though will be O(nlogn) will be more efficient (using external sort) – because the number of DISK ACCESSES will be minimized, and disk access is much slower then RAM access.

    • 0