Dillon's comments on FreeBSD/DFly's VM
Well, this is fun. There are over 460 files in the 5.x source tree (360 in DFly) that make calls to malloc(… M_NOWAIT), and so far about 80% of the calls that I’ve reviewed generate inappropriate side effects when/if a failure occurs. CAM is the biggest violator… it even has a few panic() conditionals if a malloc(… M_NOWAIT) fails. Not Fun!
In particular, Dillon said that most of M_NOWAIT flag was used inappropriately in FreeBSD and DFly. Dillon then wrote this in DFly’s list:
I think I’ve come up with a solution! The reason M_NOWAIT is used is primarily because an interrupt thread may be preempting a normal thread and cannot safely manipulate ‘cache’ pages in the VM page queues. This is because reusing a cache page requires messing around with the VM Object the cache page resides in. So M_NOWAIT causes kmem_malloc() to only pull pages out of the VM ‘free’ page queue. At the same time it is allowed to actually exhaust the free page queue whereas normal allocations are not allowed to completely exhaust the free page queue.
More discussion seemed to be related to junsu’s bug report. So I am interested in this thread.
Archived: 2 Comments
delphij | January 19, 2004 10:17 AM
Forgot to copy Dillon’s Re3, which is very important:
More research… correct me if I am wrong but it appears that the 5.x
kmem_malloc() code may have some issues. If you look down at around line
349 there is a comment:
* Note: if M_NOWAIT specified alone, allocate from
* interrupt-safe queues only (just the free list). If
* M_USE_RESERVE is also specified, we can also
* allocate from the cache. Neither of the latter two
* flags may be specified from an interrupt since interrupts
* are not allowed to mess with the cache queue.
if ((flags & (M_NOWAIT|M_USE_RESERVE)) == M_NOWAIT)
pflags = VM_ALLOC_INTERRUPT | VM_ALLOC_WIRED;
pflags = VM_ALLOC_SYSTEM | VM_ALLOC_WIRED;
Here’s the problem… the problem is that malloc(…M_NOWAIT) is used
by interrupts not only to avoid blocking, but also to avoid messing with
the VM Page ‘cache’ queue.
But in 5.x it is possible for non-interrupt threads to preempt other
non-interrupt threads indirectly (due to an interrupt trying to get a
mutex that a non-interrupt thread currently holds). Am I correct?
But the non-interrupt thread will almost certainly be making
memory allocations with M_WAITOK, which means that a preempting thread
*CAN* wind up pulling pages out of the ‘cache’ queue.
Now, my understanding is that 5.x’s mutexes around the VM system means
that this, in fact, will work just fine.
So, that means that the above comment is no longer correct, right? In
fact, interrupts *should* be able to allocate pages from the VM page
‘cache’ queue in 5.x now.
This leads to the obvious conclusion that ‘critical’ code, such as the CAM
code, which cannot afford to block but which also does terrible things
when an M_NOWAIT allocation fails should be able to use
(M_WAITOK|M_USE_RESERVE|M_USE_INTERRUPT_RESERVE) and this would result
in far safer operation then the current M_NOWAIT use results in.
(M_USE_INTERRUPT_RESERVE would be a new M_* flag that allows the system
to exhaust the entire free page reserve if necessary and has the same
effect as M_NOWAIT had before, but the combination of flags would now
allow interrupt-time allocations to also allocate from the cache queue
making it virtually impossible for such allocations to fail and that,
combined with M_WAITOK, would allow all NULL checks to be removed. It
could actually be considered a critical error for the above flags
combination to deadlock.
junsu | January 19, 2004 5:18 PM
This change will bring us a more stable and robust VM. We can reduce the scenerios that panic due to no memory.