Skip to content

0011 CPU time for work, wall time only for waits, in tests

Sources: #436, #532

Context and Problem Statement

Some tests check that work finishes within a time, for example that the code handles hostile input in linear time. They measured wall time. When all workspace suites ran at once on a busy machine, they failed now and then. The code under test worked. The process only waited for a core.

Considered Options

  • Ratio bounds.
  • Interleaved best-of-N rounds.
  • Bound work by CPU time, and keep wall time only for what CPU time cannot measure.

436 names ratio bounds and interleaved best-of-N rounds, without loosening what the tests detect.

532 built the third option.

Decision Outcome

Chosen option: CPU time for work, wall time only for waits.

  • CPU time bounds the work the test process computes: cpuClock() from tests/cpu-time.js.
  • Event latency stays on wall time: a timer, the network, a child process. CPU time misses waits.
  • So do a timestamp the code wrote, a polling deadline and a ratio of two measurements.
  • Such a wall-time line ends in the comment // wall time: <why>.
  • tests/time-bounds.test.js fails on a wall-clock reading in a comparison without that comment.

The reason recorded in #532: wall time grows when other suites share the machine. CPU time does not. Each converted bound must still fail on its defect.

Consequences

  • Good: the converted bounds keep their old values.
  • Bad: a wall-time bound needs a written reason.