Question

I am looking for a way to scrape the data from this website: http://www.dectech.org/football/index.php preferably using Python. The difficulty that I seem to be having is that the data is not hard-coded into the HTML of the website, and appears to be wrapped in something called a mochi-kit ( http://mochi.github.com/mochikit/ ).

I’ve done some research and it seems that something like BeautifulSoup might be useful to me, but I think I may not be using it correctly. I’ve also tried using urllib to parse the website with no joy.

My ultimate goal is to have a program that monitors the dectech website and when new predictions are released, automatically picks out value bets using the Betfair API.

Answer 1

It looks like the data is being loaded by javascript from this url

http://www.dectech.org/cgi-bin/new_site/GetUpcomingGames.pl?divID=0

which returns

{
   "games" : [
      {
         "apct" : 0.377838,
         "dpct" : 0.263445,
         "expGoalDiff" : -0.04086,
         "awayID" : "6",
         "homeID" : "17",
         "date" : "20/10/2012",
         "away" : "Chelsea",
         "home" : "Tottenham",
         "hpct" : 0.358717
      },
      {
         "apct" : 0.237829,
         "dpct" : 0.250146,
         "expGoalDiff" : 0.594234,
         "awayID" : "1",
         "homeID" : "8",
         "date" : "20/10/2012",
         "away" : "Aston Villa",
         "home" : "Fulham",
         "hpct" : 0.512025
      }, /* shortened for brevity */

So you’re incredibly lucky, you don’t need to scrape the data (which is tricky), you just need to retreive it and parse it like they’re doing with mochi.

Python’s simplejson module would be able to parse it…