Question

I’m extremely new to Python, read about half a beginner book for Python3. I figure doing this will get me going and learning with something I actually want to do instead of going through some “boring” exercises.

I’m wanting to build an application that will scrape Reddit for the top URL’s and then post these onto my own page. It would only check a couple times a day so no hammering at all here.

I want to parse the Reddit json (http://www.reddit.com/.json) and other subreddits json into URL’s that I can organize into my own top list and have my own categories as well on my page so I don’t have to keep visiting Reddit.

The website will be a WordPress template with the DB hosted on it’s own server (mysql). I will be hosting this on AWS using RDS, ELB, Auto-scaling, and EC2 instances for the webservers.

My questions are:

-Would it make sense to keep the Python scraper application running on it’s own server, which then writes the scraped URL’s to the database?

-I heard it may make sense to split the application and one does the reading while the other does the writing, whats this about?

-What would the flow of the Python code look like? I can fumble my way around writing it but I just am not entirely sure on how it should flow.

-What else am I not thinking of here, any tips?

Answer 1

Would it make sense to keep the Python scraper application running on
it’s own server, which then writes the scraped URL’s to the database?

Yes, that is a good idea. I would set up a cron job to run the program every so often. Depending on the load you’re expecting, it doesn’t necessarily need to be on its own server. I would have it as its own application.

I heard it may make sense to split the application and one does the
reading while the other does the writing, whats this about?

I am assuming the person who said this meant that you should have an application to write to your database (your python script) and an application to read URLs from the database (your WordPress wrapper, or perhaps another Python script to write something WordPress can understand).

What would the flow of the Python code look like? I can fumble my way
around writing it but I just am not entirely sure on how it should
flow.

This is a somewhat religious matter among programmers. However I feel that your program should be simple enough. I would simply grab the JSON and have a query that inserts into the database if the entry doesn’t exist yet.

What else am I not thinking of here, any tips?

I personally would use urllib2 and MySQLdb modules for the Python script. Good luck!