The problem: one thread, thousands of connections

A network server may hold thousands of open connections, but at any moment only a few of them have data waiting. One thread per connection wastes memory and context switches. A single thread that tries each socket in turn wastes CPU on the ones with nothing to say. What the server really wants to ask the kernel is: which of my sockets can I read from right now?

The old answers, select() and poll(), take the whole list of file descriptors on every call. The kernel copies the list in, checks every fd, and copies the result out, so each call costs O(n) in the number of watched fds, even when only one is ready. epoll (Linux 2.6, 2002) splits the job in two: you register the fds once, and the kernel keeps a list of the ones that became ready as it happens. A wait then costs time proportional to the number of ready fds.

Two grids of watched fds with five that have data: poll and select check every fd on every call, 10 000 checks for 10 000 fds; epoll_wait looks only at the ready list, 5 checks
poll() rescans every watched fd on each call; epoll_wait only reads the list of fds that already became ready.

The three system calls

int epfd = epoll_create1(0);                      // a new epoll instance (itself an fd)

struct epoll_event ev = { .events = EPOLLIN | EPOLLET, .data.fd = sock };
epoll_ctl(epfd, EPOLL_CTL_ADD, sock, &ev);         // watch sock (also MOD, DEL)

struct epoll_event events[64];
for (;;) {
    int n = epoll_wait(epfd, events, 64, -1);     // sleep until something is ready
    for (int i = 0; i < n; i++)
        handle(events[i].data.fd);                // only the ready fds
}

What is inside the kernel

epoll_create1 allocates a struct eventpoll. The animation draws its three parts:

  • The interest list (rbr): a red-black tree holding one epitem per watched fd, with the events and flags you asked for. epoll_ctl finds, inserts and erases epitems in O(log n). The animation draws each epitem above its socket instead of as a tree.
  • The ready list (rdllist): a doubly linked list of the epitems whose fds may be ready. This is the list epoll_wait looks at, and it is the whole reason epoll scales.
  • The wait queue (wq): the threads asleep in epoll_wait.

The link between a socket and epoll is a callback. When epoll_ctl(ADD) inserts an epitem, it also hooks a function, ep_poll_callback, into the socket's own wait queue (the grey arrow in the animation). When a packet arrives, the network stack wakes that wait queue as it always does, and the callback runs: it appends the epitem to the ready list, if it is not there already, and wakes a thread sleeping in wq. No fd is scanned; the socket announces itself in O(1). (How the packet gets the CPU’s attention in the first place, through the IRQ line, the interrupt controller and the handler, is shown in The interrupt-driven I/O cycle.)

Inside struct eventpoll for epfd 3: the interest list holds epitems for fds 5 to 10, the ready list, and the wait queue with thread T1 asleep. A packet arrives on socket fd 6, ep_poll_callback appends fd 6's epitem to the ready list, wakes T1, and epoll_wait returns fd 6
A packet's arrival runs ep_poll_callback, which puts that one epitem on the ready list and wakes the waiting thread: no fd is scanned.

What epoll_wait does

epoll_wait(epfd, events, maxevents, timeout):
    if rdllist is empty:
        if timeout == 0: return 0
        sleep on ep->wq until a callback wakes us (or timeout)
    txlist = rdllist;  rdllist = empty           // callbacks may keep adding meanwhile
    n = 0
    for each epitem in txlist, while n < maxevents:
        remove it from txlist
        revents = poll(epitem's file)             // ask the socket AGAIN: is it still ready?
        if revents == 0: continue                 // not ready any more: dropped, not reported
        events[n++] = revents
        if EPOLLONESHOT: disable the epitem       // until epoll_ctl(MOD) re-arms it
        else if not EPOLLET: append it to rdllist // level-triggered: check again next time
    put what is left of txlist back at the FRONT of rdllist
    if n == 0 and we may block: go back to sleep
    return n

Notice the second poll. Being on the ready list only means "something happened"; epoll_wait asks the socket again before reporting it, so a fd you already drained is dropped quietly. The work is one step per ready-list item, which is why the counter in the animation compares it with the number of fds poll() would have checked.

Level-triggered and edge-triggered

The one flag that changes the most is EPOLLET, and the pseudocode shows exactly what it does: whether a reported epitem is put back on the ready list.

  • Level-triggered (the default) behaves like poll(): as long as there is unread data, every epoll_wait reports the fd. After a report the epitem goes back on the ready list, so the next wait polls it again, reports it if data is left, and drops it once the buffer is empty. Run Demo: level-triggered: 4 sockets are watched and 3 of them get data, so epoll_wait checks only those 3 and reports fds 5, 6 and 7. The program then empties fds 6 and 7 but reads only 1 of fd 5's 3 bytes. The next wait checks all 3 again, drops 6 and 7 and reports fd 5 again.
  • Edge-triggered (EPOLLET) reports the fd once per arrival of new data. After a report the epitem is not put back; only the next callback (new data) puts it on the list again. Run Demo: edge-triggered trap: after reading 1 of 3 bytes, epoll_wait returns 0 while 2 bytes sit in the buffer. If the peer is waiting for a reply before sending more, the connection hangs.
Level-triggered and edge-triggered side by side: 3 bytes arrive on fd 5, epoll_wait reports it, the program reads 1 byte leaving 2; on the next epoll_wait level-triggered reports fd 5 again, edge-triggered returns 0 and the 2 bytes stay stuck until new data arrives
After a partial read, level-triggered reports the fd again; edge-triggered stays silent until new data arrives.

The rule for edge-triggered mode is therefore: use non-blocking sockets, and after each event keep reading until read() fails with EAGAIN (for a listening socket, keep calling accept() until EAGAIN). With a blocking socket the last read() would block the whole event loop. Edge-triggered mode saves the repeated reports and the ready-list churn; level-triggered mode is harder to get wrong. nginx uses edge-triggered mode; Redis and libuv use level-triggered mode.

EPOLLONESHOT and maxevents

  • EPOLLONESHOT disables the epitem after one report: the callback ignores new data until you re-arm it with epoll_ctl(MOD). With several threads calling epoll_wait on one epoll instance, this guarantees that only one thread handles a given connection at a time. See Demo: EPOLLONESHOT.
  • maxevents limits how many events one call returns. Unscanned items go back to the front of the ready list, while reported level-triggered items were appended to the tail, so the next call starts with the fds that were skipped. Busy fds cannot starve the others. See Demo: maxevents fairness.

Cost compared with select and poll

select() / poll()epoll
Register an fdnothing to register: the list is passed on every callepoll_ctl, O(log n), once
One waitO(n) watched fds, copied in and out each callO(r) ready-list items
A packet arriveswakes the caller, which then scans everythingcallback: O(1) append to the ready list
fd limitselect: FD_SETSIZE (usually 1024)only memory

With 10 000 idle connections and 5 active ones, poll() checks 10 000 fds per call and epoll checks about 5. With few fds, or when almost all of them are always ready, the difference disappears, and poll() costs fewer system calls because nothing has to be registered.

Common mistakes and edge cases

  • Edge-triggered with a partial read, as in the demo: the fd is not reported again until new data arrives.
  • Edge-triggered with blocking sockets: the "read until EAGAIN" loop blocks on its last read.
  • Level-triggered EPOLLOUT left on: a socket is writable almost all the time, so epoll_wait returns immediately forever and the loop spins at 100% CPU. Ask for EPOLLOUT only while you have data queued to send.
  • Epoll watches the open file, not the fd number. An epitem is removed automatically only when the last descriptor for the file is closed. After dup() or fork(), closing one fd leaves the file registered and still reporting events. Call epoll_ctl(DEL) before close().
  • Several threads on one epoll instance: one event can wake several threads (the thundering herd). EPOLLEXCLUSIVE (Linux 4.5) wakes only one, and EPOLLONESHOT keeps one connection in one thread.
  • Regular files cannot be added (EPERM): a disk file is always "ready", so epoll does not help with disk I/O. That is one of the gaps io_uring fills.

Where epoll is used

Almost every high-performance network program on Linux sits on an epoll loop: nginx, Redis, HAProxy, Node.js (through libuv), Python's asyncio and selectors, Java NIO's Selector, Netty, Go's runtime network poller, Rust's mio/Tokio. The BSDs and macOS have the same idea as kqueue, Windows has I/O completion ports, and newer Linux programs may use io_uring, which reports completed operations instead of readiness.