Question

I have a cronjob that runs every hours and parse 150,000+ records. Each record is summarized individually in a MySQL tables. I use two web services to retrieve the user information.

1. User demographic (ip, country, city etc.)
2. Phone information (if landline or cell phone and if cell phone what is the carrier)

Every time I get 1 record I check if I have information and if not I call these web services. After tracing my code I found out both of these calls takes 2 to 4 seconds and it makes my cronjob very slow and I can’t compile statistics on time.

Is there a way to make these web service faster?

Thanks

Answer 1

simple:

get the data locally and use mellissa data:

1. for ip: http://w10.melissadata.com/dqt/websmart/ip-locator.htm
2. for phone: http://www.melissadata.com/fonedata.html

you can also cache them using memcache or APC which will make it faster since he does not have to request the data from the api or database.