Question

My environment is Ubuntu 12.04 Linux on i386. I can see that only UTF-8 locales are installed (in /var/lib/locales/supported.d/local):

de_DE.UTF-8 UTF-8
en_US.UTF-8 UTF-8
~

Now the problem:

TIdIOHandler.InitComponent calls TIdTextEncoding.Default which in turn calls TIdMBCSEncoding.Create(‘ASCII’).

And there, the line

FMaxCharSize := GetByteCount(PWideChar(@cValue[0]), 2);

is executed, and GetByteCount returns zero – which should not happen (the expected value is 1 because ASCII is a single-byte encoding).

The place which returns the zero value is line 2288 in IdGlobal:

if iconv(FFromUTF16, @LCharsPtr, @LCharCount, @LBytesPtr, {$IFNDEF KYLIX}@{$ENDIF}LByteCount) = size_t(-1) then
begin
  Result := 0;
  Exit;
end;

Could this be caused by a missing UTF-16 locale? (I did not succeed installing UTF-16 support in my first attempt and maybe this problem has been seen by other Indy users on Linux too)

Answer 1

Indy does not currently do adequate handling of iconv() errors because iconv() relies on platform-specific errno variables that are hard to implement in Indy across multiple platforms (this would not have been a problem if iconv() has been implemented to return its own error codes direcgtly). Not all iconv() errors are real failures, but Indy does not know that yet, so it treats all errors as failures (as for why GetByteCount() returns 0 on failure instead of raising an exception, this is because TIdTextEncoding is modeled after Embarcadero’s SysUtils.TEncoding class and thus has to be API-compatible with it. This will change in Indy 11).

ICONV is supposed to implement its charsets natively, so it shouldn’t matter what locates are installed in the OS. In this case, I suspect that iconv() is failing to convert codepoint $10FFFD from UTF-16 to ASCII in TIdMBCSEncoding.Create() since it is outside the ASCII range, and this is one of those cases where Indy needs to look at errno to diffrentiate between a failure and partial conversion, but currently is not doing so.

As a workaround, you can edit IdGlobal.pas to have the TIdTextEncoding.Default property getter use the TIdASCIIEncoding class instead of the TIdMBCSEncoding class, since Indy has its own ASCII implementation that is not dependant on ICONV.

{$IFDEF HAS_CLASSPROPERTIES}
class function TIdTextEncoding.GetDefault: TIdTextEncoding;
{$ELSE}
class function TIdTextEncoding.Default: TIdTextEncoding;
{$ENDIF}
var
  LEncoding: TIdTextEncoding;
begin
  if GIdDefaultEncoding = nil then
  begin
    {$IFDEF USE_ICONV}
    //LEncoding := TIdMBCSEncoding.Create('ASCII');
    LEncoding := TIdASCIIEncoding.Create;
    {$ELSE}
    ...