Scheduler

Jeff提交了ULE调度器的3.0版(sched_ule.c,v 1.200，巧合？:) 这个版本对调度器本身的上锁进行了细化，从而带来了性能改善(最上面那根黄线，之前是下面那根青色的线）。

Today, Jeff Roberson has committed his version 2.0 ULE scheduler. This new version has addressed several design issues as well as several bugs.

The new scheduler has adopted a circular queue, instead of the double-queue structure which is also found in the Linux O(1) scheduler. The latter has lead to difficulty implementing nice correctly.

For uniprocessor case, ULE is now faster.

MP algorithm has been simplified a bit.

A lot of bugfixes, etc.

To quote the original commit message:

ULE 2.0:

• Remove the double queue mechanism for timeshare threads. It was slow due to excess cache lines in play, caused suboptimal scheduling behavior with niced and other non-interactive processes, complicated priority lending, etc.
• Use a circular queue with a floating starting index for timeshare threads. Enforces fairness by moving the insertion point closer to threads with worse priorities over time.
• Give interactive timeshare threads real-time user-space priorities and place them on the realtime/ithd queue.
• Select non-interactive timeshare thread priorities based on their cpu utilization over the last 10 seconds combined with the nice value. This gives us more sane priorities and behavior in a loaded system as compared to the old method of using the interactivity score. The interactive score quickly hit a ceiling if threads were non-interactive and penalized new hog threads.
• Use one slice size for all threads. The slice is not currently dynamically set to adjust scheduling behavior of different threads.
• Add some new sysctls for scheduling parameters.

Bug fixes/Clean up:

• Fix zeroing of td_sched after initialization in sched_fork_thread() caused by recent ksegrp removal.
• Fix KSE interactivity issues related to frequent forking and exiting of kse threads. We simply disable the penalty for thread creation and exit for kse threads.
• Cleanup the cpu estimator by using tickincr here as well. Keep ticks and ltick/ftick in the same frequency. Previously ticks were stathz and others were hz.
• Lots of new and updated comments.
• Many many others.

Tested on: up x86/amd64, 8way amd64.

今天 David Xu commit了他的新调度器(SCHED_CORE)，这个调度器基于ULE调度器，但改变了一系列算法。打算抽时间好好看一看代码。

Now I got (partially) the point:

• A newly fork’ed thread grabs Giant
• Subsequently, the mutex procedure will call turnstile_wait
• However, the td->td_sched->ke_runq is NULL (means that some part of the new thread is not initialized completely)
• So boom! panic.

Interestingly this does not happen in !PREEMPTION case.

Now my question are:

1. who forks?
2. is it the child? the parent?
3. why the initialization is not completed with a wrong result?

I have done several experiments/asserts to ensure that the panic is related to ULE+PREEMPTION case only.

ULE: no panic
4BSD: no panic
4BSD+PREEMPTION: no panic

My latest progress is that I found that sched_switch in ULE for the threaded case, can decide whether subsequent call would panic. Investigating.

You can obtain latest list from sys/sys/proc.h:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

 * Below is a key of locks used to protect each member of struct proc.  The
 * lock is indicated by a reference to a specific character in parens in the
 * associated comment.
 *      * - not yet protected
 *      a - only touched by curproc or parent during fork/wait
 *      b - created at fork, never changes
 *              (exception aiods switch vmspaces, but they are also
 *              marked 'P_SYSTEM' so hopefully it will be left alone)
 *      c - locked by proc mtx
 *      d - locked by allproc_lock lock
 *      e - locked by proctree_lock lock
 *      f - session mtx
 *      g - process group mtx       
 *      h - callout_lock mtx
 *      i - by curproc or the master session mtx
 *      j - locked by sched_lock mtx
 *      k - only accessed by curthread
 *      l - the attaching proc or attaching proc parent
 *      m - Giant
 *      n - not locked, lazy
 *      o - ktrace lock
 *      p - select lock (sellock)
 *      q - td_contested lock
 *      r - p_peers lock
 *      x - created at fork, only changes during single threading in exec
 *      z - zombie threads/ksegroup lock

It seems that I have hit a bug, where kseq_nice_rem is being called too much. Have read the code and added some KASSERT/MPASS’es.