Home/ Questions/Q 8408813
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-09T23:44:05+00:00

I have an application that is fetching several e-commerce websites using Curl, looking for

  • 0

I have an application that is fetching several e-commerce websites using Curl, looking for the best price.
This process returns a table comparing the prices of all searched websites.

But now we have a problem, the number of stores are starting to increase, and the loading time actually is unacceptable at the user experience side. (actually 10s pageload)

So, we decided to create a database, and start to inject all Curl filtered result inside this database, in order to reduce the DNS calls, and increase Pageload.

I want to know, despite of all our efforts, is still an advantage implement a Memcache module?
I mean, will it help even more or it is just a waste of time?

The Memcache idea was inspired by this topic, of a guy that had a similar problem: Memcache to deal with high latency web services APIs – good idea?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Memcache could be helpful, but (in my opinion) it’s kind of a weird way to approach the issue. If it was me, I’d go about it this way:

    Firstly, I would indeed cache everything I could in my database. When the user searches, or whatever interaction triggers this, I’d show them a “searching” page with whatever results the server currently has, and a progress bar that fills up as the asynchronous searches complete.

    I’d use AJAX to add additional results as they become available. I’m imagining that the search takes about ten seconds – it might take longer, and that’s fine. As long as you’ve got a progress bar, your users will appreciate and understand that Stuff Is Going On.

    Obviously, the more searches go through your system, the more up-to-date data you’ll have in your database. I’d use cached results that are under a half-hour old, and I’d also record search terms and make sure I kept the top 100 (or so) searches cached at all times.

    Know your customers and have what they want available. This doesn’t have much to do with any specific technology, but it is all about your ability to predict what they want (or write software that predicts for you!)

    Oh, and there’s absolutely no reason why PHP can’t handle the job. Tying together a bunch of unrelated interfaces is one of the things PHP is best at.

    • 0