Question

I need to design a system which has these basic components:

• A Webserver which will be getting ~100 requests/sec. The webserver only needs to dump data into raw data repository.
• Raw data repository which has a single table which gets 100 rows/s from the webserver.
• A raw data processing unit (Simple processing, not much. Removing invalid raw data, inserting missing components into damaged raw data etc.)
• Processed data repository

Does it make sense in such a system to have a service layer on which all components would be built? All inter-component interaction will go through the service layers. While this would make the system easily upgradeable and maintainable, would it not also have a significant performance impact since I have so much traffic to handle?

Answer 1

What do you see as the costs of having a separate service layer?

How do those costs compare with the costs you must incur? In your case that seems to be at least

1. a network read for the request
2. a database write for raw data
3. a database read of raw data
4. a database write of processed data

Plus some data munging.

What sort of services do you have a mind? Perhaps

• saveRawData()
• getNextRawData()
• writeProcessedData()

why is the overhead any more than a procedure call? Service does not need to imply “separate process” or “web service marshalling”.

I contend that structure is always of value, separation of concerns in your application really matters. In comparison with database activities a few procedure calls will rarely cost much.

In passing: the persisting of Raw data might best be done to a queuing system. You can then get some natural scaling by having many queue readers on separate machines if you need them. In effect the queueing system is naturally introducing some service-like concepts.