Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I am designing a dynamic site for some form of weather data, which I need to scrape to get the data, as its always changing. I am interested in knowing if its faster to use a database like sqlite or to save objects ,reloading them whenever needed. Both options will hold the scraped data.
Likely will use Python or Ruby, haven’t decided yet.
This depends on a huge number of factors.
If you need to query the data, and search through it, it’s likely that using a database will be more efficient. They are highly optimized for this type of operation.
If, however, you’re purely just trying to dump and reload a bunch of memory, it could be faster to save it directly to a file.
That being said, I’d look at how you’re going to use the data, and choose the method that makes the most sense for your application. I would not base this on speed, as, unless you’re saving a huge amount of data, the speed of saving the data will likely not be a real performance bottleneck.