Question

Consider this

Thread 1 in user program:

buf = malloc(9000);
memset(buf, 0xee, 9000);
read(buf, 9000); //for example gives pages [part of 7, 8, 9, part of 10]

Thread 2 in user program:

buf = malloc(9000); //for example gives pages [part of 4, 6, 5, part of 7]
memset(buf, 0xee, 9000);
read(buf, 9000);

Driver read:

get_user_pages();

//build dma sg list from pages
//...

//the platform demands a cachesync
for(all pages) {
    dma_cache_wback_inv();
}

//start dma and wait for it to be done
//...
wait_event_interruptible_timeout(); //blocks calling thread until dma done

for(all pages) {
    if(read) SetPageDirty();
    page_cache_release();
}

Note that page 7 is used by both transfers, and that was a big problem sometimes resulting in bad data (0xee is found in the end of one buf). Just to make it clear, the two reads runs on different DMA channels, so they can run simultaneously.

My solution was to page align the buffers in the user program so that 2 driver DMA will never share parts of the same page.

I wonder if there is another solution for this?
I also wonder why exactly this was a big problem.

Answer 1

This is a limitation of your embedded processor, and DMA that isn’t cache coherent. On high end PowerPC chips this problem goes away.

Your two buffers share a cache line at the point they meet. At the same time that one thread is in the driver writing the cache to RAM, the second thread is still in memset filling the cache line with 0xee.

DMA 1 writes your data to RAM, but the processor still holds a dirty cache line for that data, containing 0xee. When the second thread writes out the cache, it puts the 0xee over the data that came from DMA1.

The solutions are:

1. Cache-align your buffers (highest performance).
2. Use bounce buffers in the kernel driver (most compatible with existing user space code).

The get_user_pages() isn’t part of the problem here – this is about hardware and timing.