Question

Is OSCompareAndSwap is immune to ABA problem like CMPXCHG8B?

Answer 1

It all depends on the implementation. OSCompareAndSwap* is only an interface which guarantee an atomic CAS operator (if the CPU supports it).

For x86 this function for 64-bit is implemented as

_OSCompareAndSwap64:
    pushl       %edi
    pushl       %ebx

    movl         4+8(%esp), %eax    #; low 32-bits of oldValue
    movl         8+8(%esp), %edx    #; high 32-bits of oldValue
    movl        12+8(%esp), %ebx    #; low 32-bits of newValue
    movl        16+8(%esp), %ecx    #; high 32-bits of newValue
    movl        20+8(%esp), %edi    #; ptr
    lock
    cmpxchg8b   0(%edi)     #; CAS (eax:edx, ebx:ecx implicit)
    sete        %al         #; did CAS succeed? (TZ=1)
    movzbl      %al, %eax       #; clear out the high bytes

    popl        %ebx
    popl        %edi
    ret

so the answer for you probably is “yes”.