I’m working on a web crawler (please don’t suggest an existing one, it’s not an option).
I have it working the way it is expected to. My only issue is that currently I’m using a sort of server/client model, where by the server does the crawling and processes the data, it then puts it in a central location.
This location is an object created from a class I wrote. Internally the class maintains a HashMap defined as HashMap<String, HashMap<String, String>>
I store data in the map with URL as a key (I keep these unique) and the HashMap value stores the corresponding data fields for that URL such as title or value.
I occasionally serialize the internal objects used, but the spider is multi threaded and as soon as I have say 5 threads crawling, the memory requirements go up exponentially.
So far the performance has been excellent with the HashMap, crawling 15K URLs in 2.r minutes with about 30 seconds CPU time so I really don’t need to be pointed in the direction of an existing spider like most forum users have suggested.
Can anyone suggest a fast disc based solution that will probably support concurrent reading & writing? The data structure doesn’t have to be the same, just needs to be able to store related meta tag values together etc.
I suggest using EhCache for this, even though what you’re building isn’t really a cache. EhCache allows you to configure the cache instance so that it overflows to disc storage, while keeping the most recent items in memory. It can also be configured to be disc-persistent, i.e. data is flushed to disc on shutdown, and read back into memory at startup. On top of all that, it’s key-value based, so it already fits your model. It supports concurrent access, and since the disk storage is managed as a separate thread, you shouldn’t need to worry about disk access concurrency.
Alternatively, you could consider a proper embedded database such as Hypersonic (or numerous others of a similar style), but that’s probably going to be more work.