0011 CPU time for work, wall time only for waits, in tests¶
Context and Problem Statement¶
Some tests check that work finishes within a time, for example that the code handles hostile input in linear time. They measured wall time. When all workspace suites ran at once on a busy machine, they failed now and then. The code under test worked. The process only waited for a core.
Considered Options¶
- Ratio bounds.
- Interleaved best-of-N rounds.
- Bound work by CPU time, and keep wall time only for what CPU time cannot measure.
436 names ratio bounds and interleaved best-of-N rounds, without loosening what the tests detect.¶
532 built the third option.¶
Decision Outcome¶
Chosen option: CPU time for work, wall time only for waits.
- CPU time bounds the work the test process computes:
cpuClock()fromtests/cpu-time.js. - Event latency stays on wall time: a timer, the network, a child process. CPU time misses waits.
- So do a timestamp the code wrote, a polling deadline and a ratio of two measurements.
- Such a wall-time line ends in the comment
// wall time: <why>. tests/time-bounds.test.jsfails on a wall-clock reading in a comparison without that comment.
The reason recorded in #532: wall time grows when other suites share the machine. CPU time does not. Each converted bound must still fail on its defect.
Consequences¶
- Good: the converted bounds keep their old values.
- Bad: a wall-time bound needs a written reason.