GMS-2021-88: A failed upgrade may lead to hung goroutines

Impact

Processes using tableflip may encounter hung goroutines in the parent process, after a failed upgrade.

The Go runtime has annoying behaviour around setting and clearing O_NONBLOCK: exec.Cmd.Start() ends up calling os.File.Fd() for any file in exec.Cmd.ExtraFiles. os.File.Fd() disables both the use of the runtime poller for the file and clears O_NONBLOCK from the underlying open file descriptor.

This can lead to goroutines hanging in a parent process, after at least one failed upgrade. The bug manifests in goroutines which rely on either a deadline or interruption via Close() to be unblocked being stuck in read or accept like syscalls. As far as I can tell we’ve not experienced this problem in production, so it’s most likely quite rare.

Patches

The problem has been fixed in v1.2.2.

Workarounds

None.

References

• https://github.com/cloudflare/tableflip/commit/cae714b289e199db5da5f08af861ea65be6232c0