Question

My system memory is plenty (a server with 24GB). In my system, the kernel space is allocated with 320MB and 120MB for crash kernel. The rest of the memory is used for other purposes. However, when I use __get_free_pages() to allocate contiguous pages with order of 11, and the kernel fails to allocate 2^10 pages. Why?

According to makelinux

The maximum allowed value for order is 10 or 11 (corresponding to 1024
or 2048 pages), depending on the architecture. The chances of an
order-10 allocation succeeding on anything other than a freshly booted
system with a lot of memory are small, however.

Why is that so? Each page in my system is 4KB (4096 bytes), 2^10 pages = 1024 pages, and the total size is 1024*4096 = 4 194 304 (bytes) ~ 4MB. It’s only 4MB of contiguous space and the kernel is very small: vmlinuz is only 2.1MB and initrd is 15MB. The total memory consumption of the whole kernel is ~300MB. It must be more than enough for the kernel to allocate 4MB of contiguous pages. Even in normal machine with 1GB/3GB kernel/user, and sure the kernel won’t use up the whole 1 GB. But how can the allocation with only 4MB contiguous pages possibly fail? And I think, in kernel space, memory is not scattered in physical memory (due to virtual memory mapping), but is linear and contiguous.

I tried to load my kernel module first with 2^10 pages allocation, but it fails and dump the stack trace:

[    6.037056]  [<ffffffff810041ec>] dump_trace+0x86/0x2de
[    6.037063]  [<ffffffff8122fe83>] dump_stack+0x69/0x6f
[    6.037070]  [<ffffffff8108704e>] warn_alloc_failed+0x13f/0x151
[    6.037076]  [<ffffffff8108786a>] __alloc_pages_nodemask+0x80a/0x871
[    6.037081]  [<ffffffff81087959>] __get_free_pages+0x12/0x50

Answer 1

If I remember correctly, __get_free_pages uses buddy allocation, which not only does scatter its allocations throughout physical memory, it does so in the worst possible pattern for subsequent attempts to allocate large contiguous blocks. If my calculations are correct, on your system with 24GB of physical RAM, even if no space whatsoever were occupied by anything but buddy allocations, it would take less than 8192 order-0 (4KB) allocations to render allocation of a 4MB chunk with __get_free_pages impossible.

There is a thing called the contiguous memory allocator, which is supposed to address the genuine need for large physically-contiguous allocations by device drivers; as of June 2011 it was not in the official kernel, but that was more than a year ago now. You should look into that.