phk's benchmarking hints
It’s important to have a scientific view when doing any benchmarks. Today, phk and Robert Watson have posted several hints to the -current list.
Poul-Henning Kamp (phk)
A number of people have started to benchmark things seriously now, and run into the usual problem of noisy data preventing any conclusions. Rather than repeat myself many times, I decided to send this email.
I experimented with micro-benchmarking some years back, here are some bullet points with a lot of the stuff I found out. You will not be able to use them all every single time, but the more you use, the better your ability to test small differences will be.
- Disable
APMand any other kind of clock fiddling (ACPI?) - Run in single user mode.
cron(8)and other daemons only add noise. - If syslog events are generated, run
syslogd(8)with an emptysyslogd.conf, otherwise, do not run it. - Minimize disk I/O, avoid it entirely if you can.
- Don’t mount filesystems you do not need.
- Mount
/and/usrand any other filesystem possible as read-only. This removesatimeupdates to disk (etc.) from your I/O picture. newfsyour R/W test filesystem and populate it from atarordumpfile before every run. Unmount and mount it before starting the test. This results in a consistent filesystem layout. For a worldstone test this would apply to/usr/obj(justnewfsand mount). If you want 100% reproducibility, populate your filesystem from add(1)file (i.e.dd if=myimage of=)./dev /ad0s1h bs=1m - Use
malloc-backed or preloadedmd(4)partitions. - Reboot between individual iterations of your test, this gives a more consistent state.
- Remove all non-essential device drivers from the kernel. For instance if you don’t need USB for the test, don’t put USB in the kernel. Drivers which attach often have timeouts ticking away.
- Unconfigure hardware you don’t use. Detach disks with
atacontrolandcamcontrolif you do not use them for the test. - Do not configure the network unless you are testing it (or after your test to ship the results off to another computer).
- Do not run
ntpd. - Put each filesystem on its own disk. This minimizes jitter from head-seek optimizations.
- Minimize output to serial or VGA consoles. Running output into files gives less jitter. (Serial consoles easily become a bottleneck). Do not touch keyboard while test is running, even
<space><back-space>shows up in your numbers. - Make sure your test is long enough, but not too long. If your test is too short, timestamping is a problem. If it is too long, temperature changes and drift will affect the frequency of the quartz crystals in your computer. Rule of thumb: more than a minute, less than an hour.
- Try to keep the temperature as stable as possible around the machine. This affects both quartz crystals and disk drive algorithms. If you really want to get nasty, consider stabilized clock injection (get an OCXO + PLL, inject output into clock circuits instead of motherboard xtal. Send me an email).
- Run at least 3 but better is >20 for both “before” and “after” code. Try to interleave if possible (i.e. do not run 20xbefore then 20xafter), this makes it possible to spot environmental effects. Do not interleave 1:1, but 3:3, this makes it possible to spot interaction effects.
My preferred pattern:bababa{bbbaaa}*. This gives a hint after the first 1+1 runs (so you can stop it if it goes entirely the wrong way), a stddev after the first 3+3 (gives a good indication if it is going to be worth a long run) and trending and interaction numbers later on. - Use
/usrto see if your numbers are significant. Consider buying “Cartoon Guide to Statistics” (ISBN: 0062731025), highly recommended, if you’ve forgotten or never learned about stddev and Student’s t./src /tools /tools /ministat
Enjoy, and please share any other tricks you might develop!
Poul-Henning
Robert Watson
On Mon, 26 Jan 2004, Poul-Henning Kamp wrote:
Run in single user mode.
cron(8)and other daemons only add noise.
Of particular interest here is sshd – either disable its SSHv1 key regeneration, or kill the parent sshd (or don’t run sshd) during the tests.
A few obvious ones just worth remembering:
- Don’t use
bgfsckunless you want to benchmark withbgfsck. Also, disablebackground_fsckinrc.confunless you plan not to start the benchmark for at least 60+fudge seconds after the boot, asrcwakes up and checks to see if anything needs fscking if it is enabled. Likewise, make sure you don’t have snapshots lying around unless you mean to test with snapshots. - If you must leave the system connected to a public network, watch out for spikes of broadcast traffic. Even though you don’t notice it, it will eat your CPU. Multicast has similar caveats.
- If you see unexpectedly bad performance, check for things like high interrupt volume from an unexpected source. I’ve seen several reports of ACPI “misbehaving” and generating excess interrupts under some of the ACPI code drops. If you’re seeing odd results, take a few snapshots of
vmstat -iand look for anything unusual. - Make sure to be careful about optimization parameters for kernel and userspace, likewise debugging. It’s easy to let something slip through and realize later you weren’t comparing the same thing.
- And finally, do not ever benchmark with
WITNESSandINVARIANTSunless you are really just interested in benchmarking those features. :-) I’ve seen 400%+ drops in performance from running withWITNESS. Likewise, userspacemallocparameters default differently in-CURRENTfrom the way they’d ship in production.
Also, it would be very helpful if, while benchmarking, people would do the same benchmark with SCHED_ULE and SCHED_4BSD, and see what impact it has. Scheduler changes are very influential to performance, and we want to make sure we catch edge cases and glitches that will now be being seen with more broad testing.
–
Robert N M Watson
FreeBSD Core Team, TrustedBSD Projects
Senior Research Scientist, McAfee Research
在线阅读本文并参与讨论 delphij's Chaos blog.delphij.net/posts/2004/01/phks-benchmarki/