Question

Followed MSDN’s GetOpenFileName example using the FFI of LuaJIT. I’ve been trying to get this working for two days, not only the dialog doesn’t opens but it crashes the entire thing.

When debugged with OllyDdb it throws an ugly access violation somewhere in comdlg32.dll.

Here’s the proof of concept with just the essential code. If I specify that the struct size is zero it exits without problem, with GetLastError() throwing a 87 (ERROR_INVALID_PARAMETER)

So I suppose that the problem is struct or FFI related.

If somebody is able to shed some light over here… thankies.

ffi = require"ffi"
bit = require"bit"

collectgarbage("stop")

ffi.cdef[[
  static const int OFN_FILEMUSTEXIST             = 0x1000;
  static const int OFN_NOCHANGEDIR               = 8;
  static const int OFN_PATHMUSTEXIST             = 0x800;

  typedef bool BOOL;
  typedef char CHAR;

  typedef unsigned short WORD; 
  typedef unsigned long DWORD;

  typedef void *PVOID;
  typedef void *LPVOID;
  typedef void *LPOFNHOOKPROC;

  typedef unsigned long HANDLE;
  typedef HANDLE HWND;
  typedef HANDLE HINSTANCE;

  typedef const char *LPCSTR;
  typedef const char *LPCTSTR;

  typedef char *LPSTR;
  typedef char *LPTSTR;

  typedef unsigned long LPARAM;

  typedef struct {
    DWORD         lStructSize;
    HWND          hwndOwner;
    HINSTANCE     hInstance;
    LPCTSTR       lpstrFilter;
    LPTSTR        lpstrCustomFilter;
    DWORD         nMaxCustFilter;
    DWORD         nFilterIndex;
    LPTSTR        lpstrFile;
    DWORD         nMaxFile;
    LPTSTR        lpstrFileTitle;
    DWORD         nMaxFileTitle;
    LPCTSTR       lpstrInitialDir;
    LPCTSTR       lpstrTitle;
    DWORD         flags;
    WORD          nFileOffset;
    WORD          nFileExtension;
    LPCTSTR       lpstrDefExt;
    LPARAM        lCustData;
    LPOFNHOOKPROC lpfnHook;
    LPCTSTR       lpTemplateName;

    LPVOID        pvReserved;
    DWORD         dwReserved;
    DWORD         flagsEx;

  }OPENFILENAME;

  BOOL GetSaveFileNameA( OPENFILENAME lpofn );
  BOOL GetOpenFileNameA( OPENFILENAME lpofn );
]]
com=ffi.load("comdlg32")

ffi.cdef[[
  DWORD GetLastError(void);
]]
krnl=ffi.load("kernel32")

function OpenDialog()
  Ofn=ffi.new("OPENFILENAME")
  ffi.fill(Ofn,ffi.sizeof(Ofn)) --zero fill the structure

  local szFile        = ffi.new("char[260]","\0")
  local hwnd          = ffi.new("HWND",0)

  Ofn.lStructSize     = ffi.sizeof(Ofn)
  Ofn.hwndOwner       = hwnd

  Ofn.lpstrFile       = szFile
  Ofn.nMaxFile        = ffi.sizeof(szFile)

  Ofn.lpstrFilter     = "All\0*.*\0Text\0*.TXT\0"
  Ofn.nFilterIndex    = 1

  Ofn.lpstrFileTitle  = nil
  Ofn.nMaxFileTitle   = 0

  Ofn.lpstrInitialDir = nil
  Ofn.flags           = bit.bor(com.OFN_PATHMUSTEXIST, com.OFN_FILEMUSTEXIST, com.OFN_NOCHANGEDIR)

  print("displaying...")

  if com.GetOpenFileNameA(Ofn) then --luajit converts bool automatically
    print("file->",ffi.string(Ofn.lpstrFile, Ofn.nMaxFile))
  end

  print("lasterror->",krnl.GetLastError())
end

OpenDialog()

Answer 1

These C functions take a pointer to the struct. The correct declaration is: GetOpenFilenameA(OPENFILENAME *lpofn) etc.