Question

I have a webpage made in perl and dojo using a PostgreSQL database. I have to search for availale people in the database and since im from Denmark the letters æ,ø and å has to be available in the search. I thought this was standard when using UTF8 and when I normally program in php over mysql I didn’t think it would be that hard.

I have done properly every trick I know to convert this search_word to the right encoding so i can search in the postgre sql database for correct names with æ,ø and å… but it still fails.

i have my perl code making the fetch but this fetch returns 0 rows and when i insert the same command in the psql terminal i get 46 rows returned (copy from “tail -f log terminal” the STDERR statement and inserts it into another terminal connected to the database through the psql command)… the perl code is:

sub dbSearchPersons {
  my $search_word = escapeSql($_[0]);
  $search_word = Encode::decode_utf8($search_word);

  $statement = "SELECT id,name,initials,email FROM person WHERE name ilike '\%".$search_word."\%' OR email ilike '\%".$search_word."\%' OR initials ilike '\%".$search_word."\%' ORDER BY name ASC";
  $sth = $dbh->prepare($statement);
  $num_rows = $sth->execute();

  print STDERR "Statement: " . $statement;
  if($num_rows > 0){
    $persons = $dbh->selectall_hashref($statement,'id');
  }

  dbFinish($sth);
  webdie($DBI::errstr) if($DBI::errstr);
}

and as you can see i write the SQL statement to STDERR and which outputs the following:

[Fri Apr 27 11:24:26 2012] [error] [client 10.254.0.1] Statement: SELECT id,name,initials,email FROM person WHERE name ilike '%Jørgen%' OR email ilike '%Jørgen%' OR initials ilike '%Jørgen%' ORDER BY name ASC, referer: https://xx.xxx.xxx.xx/cgi-bin/users.cgi

The sql I correctly written (as i can see it through the terminal output above) and if I copy and paste the statement from the terminal and inserts it directly into the psql terminal, i get 46 rows returned as I should… But the perl still wont return any rows.

I don’t get it? When formatting a string to display “ø” and not “ø” (as perl translates the UTF8 encoding to, from “J%C3%B8rgen” which gets send through dojo.xhr.post), should I not be able to use it in a SQL statement? Is it because the psql database can have a certain encoding i have to take that into account somehow? Or could it be some completely different?

Hope someone can help me. I have been struggling with this problem for two days now and since the things looks like they should, but don’t work I get a little sad :/

Regards,
Thor Astrup Pedersen

Answer 1

You probably forgot to pg_enable_utf8. The database interface will return then Perl character data to you.

$ createdb -e -E UTF-8 -l en_US.UTF-8 -T template0 so10349280
CREATE DATABASE so10349280 ENCODING 'UTF-8' TEMPLATE template0 LC_COLLATE 'en_US.UTF-8' LC_CTYPE 'en_US.UTF-8';

$ echo 'create table person (id int, name varchar, initials varchar, email varchar)'|psql so10349280
CREATE TABLE

$ echo "insert into person (id, name) values (1, 'Jørgensen')"|psql so10349280
INSERT 0 1

$ echo 'select * from person'|psql so10349280
 id |   name    | initials | email
----+-----------+----------+-------
  1 | Jørgensen |          |

$ perl -Mutf8 -Mstrictures -MDBI -MDevel::Peek -E'
    my $dbh = DBI->connect(
        "DBI:Pg:dbname=so10349280", $ENV{LOGNAME}, "", { RaiseError => 1, AutoCommit => 1, pg_enable_utf8 => 1}
    );
    my $r = $dbh->selectall_hashref("select * from person where name = ?", "id", undef, "Jørgensen");
    Dump $r->{1}{name};
'
SV = PV(0x836e20) at 0xa58dc8
  REFCNT = 1
  FLAGS = (POK,pPOK,UTF8)
  PV = 0xa5a000 "J\303\270rgensen"\0 [UTF8 "J\x{f8}rgensen"]
  CUR = 10
  LEN = 16

You don’t say quite clear, I think you eventually intend to send out the character data as JSON for use with Dojo. You need to encode them into UTF-8 octets; the various JSON libaries take care of that automatically for you, no need to invoke Encode functions manually.