I’m working on a project in which xml feed is fetched from a different server through cron job to my server and saved it as a file, and my website uses that xml file to display data on website.
But the problem is that the xml file is two large around (4mb) and every time user hit the site, the server get the file(from same location) and parse it and server the user, so if 1000 hits the server would pull it 1000 times (thats the overhead I want to resolve)
I tried to use memcache but that is also limited to 2mb.
Can anyone tell me which is the best method to save that big xml either in DB, as a xml file, or in memcache, should I break that big xml to categories??
I would create a database, have the cron job fetch and then parse the XML and insert everything into the DB, and only serve users from the database.
To make sure the data is consistent you can implement basic versioning. Store the current version number somewhere; whenever you get a new XML, store the next version into the database, then update the current version number, cleanup periodically.