Question

As you can see from here (https://dev.twitter.com/docs/api/1/get/statuses/home_timeline), one tweet has a lot of informations(fields)so that it isn’t easy to store tweet information in MySQL.

If I make this JSON into array, It wouldn’t be 1-depth array. For instance, like URL entities in this JSON, It might have lots of URLs in one “entities” field.

Should I store this information as a just string like “urls[{“aa”: a, “bb”: b}, {“aa”: c, “bb”: d}]” in one field? or Is there any best way to store?

{
    "coordinates": null,
    "favorited": false,
    "created_at": "Fri Jul 16 16:58:46 +0000 2010",
    "truncated": false,
    "entities": {
      "urls": [
        {
          "expanded_url": null,
          "url": "http://www.flickr.com/photos/cindyli/4799054041/",
          "indices": [
            75,
            123
          ]
        }
      ],
      "hashtags": [

      ],
      "user_mentions": [
        {
          "name": "Stephanie",
          "id": 15473839,
          "indices": [
            27,
            39
          ],
          "screen_name": "craftybeans"
        }
      ]
    },
    "text": "got a lovely surprise from @craftybeans. She sent me the best tshirt ever. http://www.flickr.com/photos/cindyli/4799054041/ ::giggles::",
    "annotations": null,
    "contributors": null,
    "id": 18700887835,
    "geo": null,
    "in_reply_to_user_id": null,
    "place": null,
    "in_reply_to_screen_name": null,
    "user": {
      "name": "cindy li",
      "profile_sidebar_border_color": "AD0066",
      "profile_background_tile": false,
      "profile_sidebar_fill_color": "AD0066",
      "created_at": "Wed Nov 29 06:08:08 +0000 2006",
      "profile_image_url": "http://a1.twimg.com/profile_images/553508996/43082001_N00_normal.jpg",
      "location": "San Francisco, CA",
      "profile_link_color": "FF8500",
      "follow_request_sent": false,
      "url": "http://www.cindyli.com",
      "favourites_count": 465,
      "contributors_enabled": false,
      "utc_offset": -28800,
      "id": 29733,
      "profile_use_background_image": true,
      "profile_text_color": "000000",
      "protected": false,
      "followers_count": 3395,
      "lang": "en",
      "notifications": true,
      "time_zone": "Pacific Time (US & Canada)",
      "verified": false,
      "profile_background_color": "cfe8f6",
      "geo_enabled": true,
      "description": "Just me, Cindy Li.Giving cute substance since 1997.\r\nMarried to @themattharris.\r\nProduct designer for Yahoo! ",
      "friends_count": 542,
      "statuses_count": 4847,
      "profile_background_image_url": "http://a3.twimg.com/profile_background_images/3368753/twitter_flowerbig.gif",
      "following": true,
      "screen_name": "cindyli"
    },
    "source": "web",
    "in_reply_to_status_id": null
},

Answer 1

Store it in columns. Then you can place indexes on things that you want to find quickly – not the mention just being able to search through it all. Searching for something specific in the format you suggest would be a nightmare.

If you want to keep using bits from it, write an object that will handle the database aspect of it, or at the very least a function that will pop them in or get them out. It might seem more work now, but in the long run it will save you much more effort later on.

Edit: Yes, I would keep each bit of data in a column of it’s own. Having said that, you might not need to store every single bit of info. If you don’t want to keep info on “user Mentions” for example, just skip it totally.

Edit 2L to put this into perspective, lets say you want to do a search for “Bob”. If you have a column structure like this:

+------+-----------+-----------+-----+
| user | favorited | truncated | url |
+------+-----------+-----------+-----+
| Bob  | true      | false     | ... |
| Sue  | true      | true      | ... |
| Tom  | true      | false     | ... |
+------+-----------+-----------+-----+

You can just write a dead simple query.

vs something like this:

+--------------------------------------------------------------+
| tweetData                                                    |
+--------------------------------------------------------------+
| user:Bob;favorited:true;truncated:false;url:www.example.com  |
| user:Sue;favorited:true;truncated:true;url:www.example2.com  |
| user:Tom;favorited:true;truncated:false;url:www.example2.com |
+--------------------------------------------------------------+

Imagine trying to find how many times Bob was favorited. You would have to extract the entire row each time, do some manipulation/regex/trickery to get the field, then tally it up manually.