Question

If I use Moonshine (https://github.com/brandonc/moonshine) on non-English characters – for example if I run the following text

The following is italic Arabic text: *مرحبا، كيف حالك اليوم؟*

through the console app provided in the github project with parameters

extensions = Sundown.MarkdownExtensions.NoIntraEmphasis | Sundown.MarkdownExtensions.FencedCode | Sundown.MarkdownExtensions.AutoLink
smartypants = false

The output is:

<p>The following is italic Arabic text: <em>?????? ??? ???? ??????</em></p>

How can I get it to pass it back properly?

Thanks
Mustafa

EDIT:

So I noticed that in the Moonshine.cs wrapper, the buffer passed to the Sundown library is defined as such:

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi)]
struct buf
{
    [MarshalAs(UnmanagedType.LPStr)]
    public string data;     /* actual character data */
    public uint size;       /* size of the string */
    public uint asize;      /* allocated size (0 = volatile buffer) */
    public uint unit;       /* reallocation unit size (0 = read-only buffer) */
    public int @ref;        /* reference count */
};

Where the Charset is defined as Ansi and the string data is marshaled as an LPStr. Unfortunately, changing it to

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
struct buf
{
    [MarshalAs(UnmanagedType.LPWStr)]
    public string data;     /* actual character data */
    public uint size;       /* size of the string */
    public uint asize;      /* allocated size (0 = volatile buffer) */
    public uint unit;       /* reallocation unit size (0 = read-only buffer) */
    public int @ref;        /* reference count */
};

Doesn’t help, as now the output received is incorrectly encoded text:

瀼吾栀攀 昀漀氀氀漀眀椀渀最 椀猀 戀漀氀搀 䄀爀愀戀椀挀 琀攀砀琀⼼㹰༊ҧÄȪ

Thanks again.

Answer 1

We never found a clean solution to this issue and ended up converting to MarkdownDeep ( https://github.com/toptensoftware/markdowndeep ) which has performed similarly to Moonshine, works well with UTF8, and doesn’t require us to use 32bit mode on our App Pool or compile our app to target x86, which works better for us.

Thanks
Mustafa