Question

I’m building an application and I’m having trouble making a choice about how is the best way to access multiple times to static data in a django app. My experience in the field is close to zero so I could use some help.

The app basically consists in a drag & drop of foods. When you drag a food to a determined place(breakfast for example) differents values gets updated: total breakfast calories, total day nutrients(Micro/Macro), total day calories, …That’s why I think the way I store and access the data it’s pretty important performance speaking.

This is an excerpt of the json file I’m currently using:

foods.json

{
"112": {
    "type": "Vegetables", 
    "description": "Mushrooms", 
    "nutrients": {
        "Niacin": {
            "unit": "mg", 
            "group": "Vitamins", 
            "value": 3.79
        }, 
        "Lysine": {
            "units": "g", 
            "group": "Amino Acids", 
            "value": 0.123
        },
        ... (+40 nutrients)
    "amount": 1, 
    "unit": "cup whole", 
    "grams": 87.0 }
 } 

I’ve thought about different options:

1) JSON(The one I’m currently using):

Every time I drag a food to a “droppable” place, I call a getJSON function to access the food data and then update the corresponding values. This file has a 2mb size, but it surely will increase as I add more foods to it. I’m using this option because it was the most quickest to begin to build the app but I don’t think it’s a good choice for the live app.

2) RDBMS with normalized fields:

I could create two models: Food and Nutrient, each food has 40+ nutrients related by a FK. The problem I see with this is that every time a food data request is made, the app will hit the db a lot of times to retrieve it.

3) RDBMS with picklefield:

This is the option I’m actually considering. I could create a Food models and put the nutrients in a picklefield.

4) Something with Redis/Django Cache system:

I’ll dive more deeply into this option. I’ve read some things about them but I don’t clearly know if there’s some way to use them to solve the problem I have.

Thanks in advance,
Mariano.

Answer 1

This is a typical use case for a relational database. More or less normalized form is the proper way most of the time.

I wrote this data model up from the top of my head, according to your example:

CREATE TABLE unit(
 unit_id integer PRIMARY KEY
,unit text NOT NULL
,metric_unit text NOT NULL
,atomic_amount numeric NOT NULL
);

CREATE TABLE food_type(
 food_type_id integer PRIMARY KEY
,food_type text NOT NULL
);

CREATE TABLE nutrient_type(
 nutrient_type_id integer PRIMARY KEY
,nutrient_type text NOT NULL
);

CREATE TABLE food(
 food_id serial PRIMARY KEY
,food text NOT NULL
,food_type_id integer REFERENCES food_type(food_type_id) ON UPDATE CASCADE
,unit_id integer REFERENCES unit(unit_id) ON UPDATE CASCADE
,base_amount numeric NOT NULL DEFAULT 1
);

CREATE TABLE nutrient(
 nutrient_id serial PRIMARY KEY
,nutrient text NOT NULL
,metric_unit text NOT NULL
,base_amount numeric NOT NULL
,calories integer NOT NULL DEFAULT 0
);

CREATE TABLE food_nutrient(
 food_id integer references food (food_id) ON UPDATE CASCADE ON DELETE CASCADE
,nutrient_id integer references nutrient (nutrient_id) ON UPDATE CASCADE
,amount numeric NOT NULL DEFAULT 1
,CONSTRAINT food_nutrient_pkey PRIMARY KEY (food_id, nutrient_id)
);

CREATE TABLE meal(
 meal_id serial PRIMARY KEY
,meal text NOT NULL
);

CREATE TABLE meal_food(
 meal_id integer references meal(meal_id) ON UPDATE CASCADE ON DELETE CASCADE
,food_id integer references food (food_id) ON UPDATE CASCADE
,amount numeric NOT NULL DEFAULT 1
,CONSTRAINT meal_food_pkey PRIMARY KEY (meal_id, food_id)
);

This is definitely not, how it should work:

every time a food data request is made, the app will hit the db a lot
of times to retrieve it.

You should calculate / aggregate all values you need in a view or function and hit the database only once per request, not many times.

Simple example to calculate the calories of a meal according to the above model:

SELECT sum(n.calories * fn.amount * f.base_amount * u.atomic_amount * mf.amount)
                                                               AS meal_calories
FROM   meal_food mf
JOIN   food f USING (food_id)
JOIN   unit u USING (unit_id)
JOIN   food_nutrient fn USING (food_id)
JOIN   nutrient n USING (nutrient_id)
WHERE  mf.meal_id = 7;

You can also use materialized views. For instance, store computed values per food in a table and update it automatically if underlying data changes. Most likely, those rarely change (but are still easily updated this way).