Question

In F#, there’s the NativePtr module, but it seems to only support 32 bit offsets for its’ add/get/set functions, just like System.IntPtr does.

Is there a way to add a 64 bit offset to a native pointer (nativeptr<‘a>) in F#? Of course I could convert all addresses to 64 bit integers, do normal integer operations and then convert the result again to nativeptr<‘a>, but this would cost additional add and imul instructions. I really want the AGUs to perform the address calculations.

For instance, using unsafe in C# you could do something like

void* ptr = Marshal.AllocHGlobal(...).ToPointer();
int64 offset = ...;
T* newAddr = (T*)ptr + offset; // T has to be an unmanaged type

Well actually you can’t, because there is no “unmanaged” constraint for type parameters, but at least you can do general pointer arithmetic in a non-generic way.

In F# we finally got the unmanaged constraint; but how do I do the pointer arithmetic?

Answer 1

I’m not an expert in this field, but I took a look at the F# implementation of the NativePtr module and I think that there is no performance overhead associated with converting nativeptr<'a> to nativeint and back.

The implementation uses inline IL and the inline IL code doesn’t contain any code – it is there just to make the F# compiler think that the value on the stack has a different type:

let inline ofNativeInt (x:nativeint)    = (# "" x : nativeptr<_> #)
let inline toNativeInt (x:nativeptr<_>) = (# "" x : nativeint    #)

In fact, the NativePtr.add method also uses these two methods – it converts the pointer to nativeint and then adds the 32bit integer (multiplied by the size of the 'a type).

So, the following function should be fine:

let inline addNativeInt (x:nativeptr<'a>) (n:nativeint) : nativeptr<'a> = 
   (NativePtr.toNativeInt x) + n |> NativePtr.ofNativeInt

All functions used in the code should be inlined, so you’ll end up with just a single instruction for addition (though, I have not verified that). You don’t even have to worry about using the function multiple times in your code (you can work with nativeptr<'a> all the time and use this function for addition).

However, partitioning data may also be an option – as far as I know, the MSR team that was using F# for processing some large (>2GB) data sets used exactly this approach – they partitioned the data into 2GB blocks (stored in arrays).