Question

I have always used rawurlencode to store user entered data into my mysql databases. The main reason I do this is so that stroing foreign characters is very simple I find. I’d then use rawurldecode to retrieve and display the data.

I read somewhere that rawurlencode was not meant for this purpose. Are there any disadvantages to what I’m doing?

---

So let’s say I have a German address with many characters like umlauts etc. What is the simplest way to store this in a mysql database with no risks of it coming out wrong and being searchable using a search script? So far rawurelencode has been excellent for our system. Perhaps the practise can be improved upon by only encoding foreign letters and not common characters like spaces etc, which is a waste of space I totally agree.

Answer 1

Drawbacks I can think of:

• Waste of disk space.
• Waste of CPU cycles encoding and decoding on every read and every write.
• Additional complexity (you can’t even inspect data with a MySQL client).
• Impossibility to use full text searches.
• URL encoding is not necessarily unique (there’re at least two RFCs). It may not lead to data loss but it can lead to duplicate data (e.g., unique indexes where two rows actually contain the same piece of data).
• You can accidentally encode a non-string piece of data such as a date: 2012-04-20%2013%3A23%3A00

But the main consideration is that such technique is completely arbitrary and unnecessary since MySQL doesn’t have the least problem storing the complete Unicode catalogue. You could also decide to swap e’s and o’s in all strings: Holle, werdl!. Your app would run fine but it would not provide any added value.

Update: As Your Common Sense points out, a SQL clause as basic as ORDER BYis no longer usable. It’s not that international chars will be ignored; you’ll basically get an arbitrary sort order based on the ASCII code of the % and hexadecimal characters. If you can’t SELECT * FROM city ORDER BY city_name reliably, you’ve rendered your DB useless.