Graceful Shutdown Epoll Kqueue Go: Unblock Wait

Graceful shutdown in Go epoll or kqueue event loops hinges on unblocking epoll_wait or kevent with a wakeup file descriptor like eventfd on Linux or a pipe on macOS—register it with your multiplexer so a simple write wakes mp.Wait() instantly. The Go runtime itself uses this self-pipe pattern: create an eventfd or pipe, add it to epoll/kqueue, and write from a signal goroutine to trigger shutdown without interrupting the poller. Production servers like evio follow suit, draining active connections cleanly before exit.

Contents

• The Blocking Problem in Epoll/Kqueue Loops
• Go Runtime’s Self-Pipe Solution for Epoll
• Kqueue Wakeup with Pipes on macOS
• Eventfd: Linux’s Clean Wakeup Mechanism
• Signalfd for Signal Integration
• Production Examples: Evio and Netty Epoll
• Full Go Implementation for Your Server
• Best Practices for Golang Graceful Shutdown
• Sources
• Conclusion

The Blocking Problem in Epoll/Kqueue Loops

Your code nails the issue spot-on. That mp.Wait() call—epoll_wait on Linux, kevent on macOS—blocks indefinitely until an event fires. Signals? They don’t help. A SIGTERM just sits there, ignored, because the kernel doesn’t interrupt the syscall unless you ask nicely with epoll_pwait. Your select never gets another shot.

Why does this happen? Epoll and kqueue are designed for efficiency in epoll server setups. They multiplex thousands of FDs without waking unless data’s ready. But shutdown needs a nudge. Without it, your single-threaded loop freezes, connections linger half-processed, and you kill -9 your way out. Frustrating, right?

Traditional fixes like select poll epoll fall short here too—poll epoll comparisons show epoll wins on scale, but blocking remains the Achilles’ heel for graceful shutdown.

Go Runtime’s Self-Pipe Solution for Epoll

Peek at how Go itself solves this. The runtime’s netpoller registers a wakeup FD right from the start. On Linux, it’s an eventfd paired with epoll. Create it non-blocking with EFD_CLOEXEC | EFD_NONBLOCK, add to epoll via EPOLLIN, and boom—write a uint64(1) to unblock epoll_wait.

Here’s the flow:

1. netpollinit() sets up epfd and efd.
2. netpollBreak() atomically writes to efd if not already waking.
3. Poller sees the event, reads it clear, processes shutdown.

No EINTR races. No signal handlers fighting the loop. Goroutines signal via channels, one writes the FD. Your readyEvents picks up the wakeup, you drain sockets, done.

And it scales. Go’s single poller thread handles millions of conns this way.

Kqueue Wakeup with Pipes on macOS

Switch to macOS or BSD? Go swaps eventfd for a self-pipe. Same idea: pipe() creates wakeR/wakeW, register wakeR with kqueue on _EVFILT_READ.

When shutdown calls, write a byte to wakeW. Kevent wakes, loop reads the pipe (clearing it), resets flags. If blocking, it consumes the full wakeup.

Why pipes over eventfd? macOS lacks eventfd, but pipes are portable and non-blocking. Evio library does this cross-platform too—keeps your kqueue loop pure.

Pro tip: Use processWakeupEvent() logic to drain the pipe fully, avoiding busy loops on multiple wakes.

Eventfd: Linux’s Clean Wakeup Mechanism

Eventfd shines for epoll linux gracefulness. It’s a counter FD: write increments it (64-bit), read decrements and yields the value. Perfect for wakeups—no partial reads like pipes.

In C/Go syscall land:

efd, err := syscall.Eventfd(0, syscall.EFD_CLOEXEC|syscall.EFD_NONBLOCK)
epfd, _ := syscall.EpollCreate1(0)
syscall.EpollCtl(epfd, syscall.EPOLL_CTL_ADD, efd, &syscall.EpollEvent{Events: syscall.EPOLLIN, Fd: int32(efd)})

Shutdown goroutine: syscall.Write(efd, []byte{1}) (or uint64 via binary). Epoll_wait returns it as readable. Read with syscall.Read(efd, buf) to reset.

Man page confirms: another thread writing unblocks without signals. Zero races if atomic.

Signalfd for Signal Integration

Signals tricky? Ditch handlers for signalfd. Block SIGTERM/INT with sigprocmask, create signalfd(-1, &mask, SFD_CLOEXEC), add to epoll.

Epoll_wait now sees signals as FD events. Read signalfd_siginfo, trigger shutdown. No EINTR, no async handlers. Inherited by kids too.

Stack Overflow patterns match: unblock in epoll_pwait sigmask, handle post-wait. But for Go? Self-pipe edges it out—simpler, no sigmask juggling.

Production Examples: Evio and Netty Epoll

Real-world? Evio nails netty epoll server vibes in Go. Serve() makes stopCh, pipeR/pipeW, adds pipeR to poller. SIGTERM? Close stopCh, write pipeW. Loop detects closed chan post-wait, drains, exits.

Netty’s netty epoll? Native epoll with similar wakeup FDs. Errors like netty epoll server io 1 error? Often misregged wakeups—check EPOLLET flags.

Eli Bendersky’s TCP example adds context to net.Accept, but for pure event loops, wakeup FDs rule. Golang graceful shutdown pros wait on wg for handlers.

Full Go Implementation for Your Server

Tie it together. Abstract your mp with wakeup:

type Multiplexer struct {
	epfd int
	wfd int // eventfd or pipeR
	stop chan struct{}
}

func (m *Multiplexer) Init() error {
	// Linux: eventfd, macOS: pipe
	if runtime.GOOS == "linux" {
		var e syscall.Errno
		m.wfd, e = syscall.Eventfd(1, syscall.EFD_CLOEXEC|syscall.EFD_NONBLOCK)
		if e != 0 { return e }
	} else {
		syscall.Pipe([]int{&m.wfd, &wakeW})
	}
	// Add wfd to epoll/kqueue with EPOLLIN/KEV_READ
	m.stop = make(chan struct{})
	return nil
}

func (m *Multiplexer) Wait() ([]Event, error) {
	events := pollEvents[:]
	n, err := syscall.EpollWait(m.epfd, events, -1) // or Kevent
	if n > 0 && events[0].Fd == int32(m.wfd) {
		// Drain wakeup
		var b [8]byte
		syscall.Read(m.wfd, b[:])
		close(m.stop)
		return nil, io.EOF // Signal shutdown
	}
	return events[:n], err
}

// In main goroutine
go func() {
	<-sigs
	syscall.Write(m.wfd, []byte{1})
}()

Loop checks err == io.EOF, drains conns via select{ case <-m.stop: return }. Single-threaded win.

Best Practices for Golang Graceful Shutdown

• Drain, don’t kill: Post-wakeup, process existing events, reject new accepts.
• Atomic wakes: Go’s netpollWakeSig prevents spam writes.
• Timeout: EpollWait with deadline, fallback to signal.
• Test: docker run --rm yourserver & sleep 1; docker kill $!—watch clean exit.
• Go graceful shutdown? Channels + wg for handlers, but wakeup first.

Epoll vs kqueue? Both self-pipe. Avoid threads—keeps it pure event-driven.

Sources

1. Go runtime netpoll_epoll.go
2. Evio GitHub repository
3. Making signals less painful with signalfd
4. Graceful shutdown of TCP server in Go
5. Go runtime netpoll_kqueue.go
6. epoll_wait(2) man page
7. Handle signals with epoll_wait and signalfd - Stack Overflow

Conclusion

Unblocking epoll_wait or kevent for graceful shutdown boils down to one trick: wakeup FD in your multiplexer. Go runtime, evio—everyone does self-pipe or eventfd. Implement it, drain conns on wake, exit clean. Your single-threaded epoll server stays responsive, scalable, production-ready. Skip signals alone; embrace the FD way.