mirror of
https://source.quilibrium.com/quilibrium/ceremonyclient.git
synced 2024-11-10 18:25:17 +00:00
44 lines
1.5 KiB
Markdown
44 lines
1.5 KiB
Markdown
# Debugging Flaky Tests
|
|
|
|
If a test is flaky in CI it's probably because there's some timing issue. The
|
|
test probably depends on some Go routine making progress in the background and
|
|
polling to see if the expected outcome is achieved.
|
|
|
|
This will pretty much always work locally because your local machine is likely
|
|
pretty capable and there isn't too many concurrent processes running. In CI, we
|
|
are susceptible to both slower hardware and noisier neighbors. However we can
|
|
mimic this environment locally with
|
|
[cgroups](https://man7.org/linux/man-pages/man7/cgroups.7.html).
|
|
|
|
# Replicating noisy neighbors
|
|
|
|
We can limit the amount of CPU time relative to real time a process gets with
|
|
cgroups. This lets us replicate the environment where many other neighboring
|
|
processes are vying for CPU time.
|
|
|
|
```bash
|
|
# Compile some test we want to run. We do this outside the cgroup so this is
|
|
# fast
|
|
go test -c ./p2p/host/autorelay
|
|
|
|
# Create the group
|
|
sudo cgcreate -g cpu:/cpulimit
|
|
|
|
# Limit the time to 10,000 microseconds for every 1s
|
|
sudo cgset -r cpu.cfs_quota_us=10000 cpulimit
|
|
sudo cgset -r cpu.cfs_period_us=1000000 cpulimit
|
|
|
|
# Run a shell with in our limited environment
|
|
sudo cgexec -g cpu:cpulimit bash
|
|
|
|
# In the shell, run the test
|
|
./autorelay.test -test.v
|
|
```
|
|
|
|
# Flakiness with coverage profile
|
|
|
|
Sometimes adding the `-coverprofile=module-coverage.txt` introduces flaky
|
|
behavior since it adds another goroutine to the mix. If you're having trouble
|
|
reproducing a flaky test, try enabling this flag.
|
|
|