Question

The system I am developing potentially has a very large number of clients (lets say one million) that need to periodically update a central server with some information. Clients are written in Java.

The specific use-case is that the server backend needs to have an up to date mapping of IP address to clients. But the client IPs are dynamic and subject to (effectively random) change.

The solution I have in mind requires the clients to ping the server to update their IP. The period ideally should be once every minute, but even 1 ping/10 mins is acceptable.

My questions, in sequence:

1. 1M pings per 1 min is over 10k/sec. So first off I want to know
   the approaches can scale to handle such a load. This is to know the options available.

2. Assuming you have more than one solution in mind, which of these
   would be the most economical? The cost effectiveness is critically important. I don’t have my own data center or
   static and fat end-point on the net, so the server application will
   need to run on some sort of provider or ultimately on the cloud.

Notes:

• I considered running the server from home using my own ISP provided connection, but I am neither sure of the performance issues, nor what my ISP will think about a constant stream of pings.

• I can’t see how the server can auto-discover these IP changes.

Answer 1

Erik, your problem is much simpler than it seems to have been made to sound.

This problem been around for a decade maybe two. No need to re-invent the wheel here.

Why Polling/Pinging is a Bad Idea

The dynamic IPs provided by ISPs can have a variable lease time, but will often be at least 24-72 hours. Pinging your server every 1-10m will be a horrible waist of resources potentially making over a 4,320 useless HTTP requests PER CLIENT in a 72 hour period. Each request will be say around 300 bytes * 4,320 wasted http requests equals 1.3mb wasted bandwidth multiplied by your target client count of 1 million clients, you are talking about a monthly wasted bandwidth of ~1.2 TB! And that’s just the wasted bandwidth, not the other bandwidth you might need to run your app and provide useful info.

The clients need to be smarter than just pinging frequently. Rather they should be able to check if their IP address matches the DNS on startup, then only when the IP changes, send a notification to the server. This will cut down your bandwidth and server processing requirements by thousands of times.

What you are describing is Dynamic DNS

What you are talking about is “Dynamic DNS” (both a descriptive name for the technology and also the name of one company that provides a SaaS solution).

Dynamic DNS is quite simply a DNS server that allows you to very rapidly change the mapping between a name and an IP address. Normally this is useful for devices using an ISP which only provides dynamic IPs. Whenever the IP changes for the router/server on a dynamic IP it will inform the Dynamic DNS server of the change.

• The defacto standard protocol for dynamic DNS is well documented. Start here: DNS Update API, I think the specifics you are looking for are here: DynDNS Perform Update. Most commercial implementations out there are very close to the same protocol due to the fact that router hardware usually has a built in DynDNS client which everyone wants to use.
• Most routers (even cheap ones) already have Dynamic DNS clients built into them. (You can write your own soft client, but the router is likely the most efficient location for this as your clients are likely being NAT’d with a private IP – you can still do it but at a cost of more bandwidth for public IP discovery)
• A quick google search for “dynamic DNS java client” brings up full source projects like this one: Java DynDNS client (untested, just illustrating the power of search)

Other Considerations for your System Architecture

Lets say the IP-client mapping thing gets resolved. You figured it all out and it works perfectly, you always knows the IP for each client. Would you then have a nice reliable system for transferring files to clients from mobile devices? I would say no.

Both mobiles and home computers can have multiple connection types, Wi-Fi, Cellular Data, maybe wired data. Each of these networks may have different security systems in place. So a connection from a cellular data mobile to a wifi laptop behind a home router is going to look very different than a wifi mobile device connecting to laptop on the same wifi network.

You may have physical router firewalls to contend with. Also home computers may have windows firewall enabled, maybe norton internet security, maybe symantec, maybe AVG, maybe zone alarm, etc… Do you know the firewall considerations for all these potential clients?