Demystifying select() & poll(): Kernel Internals and the C10K Challenge

Welcome again, low-level wizards.

My notes and research for this article, showing the prep before writing this article and then coming ones

Before trying to dive deep with modern I/O multiplexing, starts with understanding what came before. We’ll take a deep dive into select() and poll() internals, examining their syscall mechanics, kernel data structures, and architectural limitations.

Now that you understand why select() and poll() hit the scalability wall, we’re ready for the solution. Next up: epoll internals and building high-performance Linux servers. I’ve deliberately split this series to keep each article focused and digestible rather than creating one overwhelming mega-article.

We will cover these topics :

• The C10K problem
• Multiplexing & Batching Concepts
• A Deep Dive into select()
• A Deep Dive into poll()
• Building a Simple Echo Server

In the next article, we’ll take things further and explore how to build high-performance servers using epoll. Stay tuned!

1. The C10K Problem

Back in 1999, Dan Kegel asked a deceptively simple question: “How can one server keep 10,000 connections open at the same time?”

The hardware of that era wasn’t the limiting factor. A typical server — 500MHz CPU and 1GB RAM — had plenty of headroom:

• CPU: 50,000 cycles per client per second
• Memory: ~100KB per client
• Network: Enough bandwidth for modest traffic

And as you see the math is clear, hardware could handle it. The real bottleneck was software architecture. Traditional models (thread per connection, poll(), select()) wasted resources because every idle connection still consumed memory and forced the kernel to scan thousands of file descriptors, asking, “Anything ready yet?” That inefficiency was catastrophic at scale.

2. Multiplexing & Batching Concepts

Before start & dive deep with poll, select and epoll, it’s important to understand the I/O multiplexing & batching. (As an example: Process 100 DB writes in one batch instead of 100 individual write.)

Batching = Group similar operations and execute them together to reduce overhead.

Multiplexing = Combine multiple data streams or signals into one channel for efficient transmission or handling.

I/O multiplexing = Let one thread monitor many file descriptors and react only to those ready for I/O, avoiding blocking on a single connection.

Without multiplexing, you’d need to either:

• Block on each FD one at a time (inefficient)
• Use one thread per connection (doesn’t scale)
• Constantly poll all FDs in a loop (wastes CPU)

3. A Deep Dive into select()

select() was one of the first I/O multiplexing mechanisms in Unix. It allows a process to monitor multiple file descriptors (FDs) to see if they are ready for reading, writing, or have an exceptional condition.

You should know that select() can handle only 1024 File Descriptor and uses Bitmasks (fd_set), which are bits that represent the FDs as a bit, so the kernel can monitor it. Per example, if you want to monitor FD 3 and FD 6, the bit masks will look like :

...  FD 3   FD 4   FD 5   FD 6 ...
 0    1      0     0      1

Or : 

Binary:  0000000001001000
Hex:     0x00000048

There are 3 masks: readfds, writefds, exceptfds. Example: If fd = 5 is set for read → bit 5 in readfds = 1.

How It Works !

1. Setup in User Space

At the user-space or the application level, you are writing you need to do some steps to interact with select():

• App calls FD_ZERO() to clear masks
• then FD_SET(fd, &mask) for each FD.
• Must pass max FD + 1 (nfds) to tell the kernel how far to scan.

Here is a code example :

// preparing variables that we needs 
int sock1, sock2; // this will containe FDs
fd_set readfds; // this var will carry the masks for read
int nfds;  // this tells select() how far to scan (highest FD + 1)
char buffer[1024];

// create a socket and bind it
// ...

// Setup fd_set
FD_ZERO(&readfds); // here we clear masks
FD_SET(sock1, &readfds);  // set bit for sock1 in the mask
FD_SET(sock2, &readfds);  // Set bit for sock2 in the mask

// here we need to get the max FDs select thread needs to scan
// this avoids scanning unused higher FD slots
nfds = (sock1 > sock2 ? sock1 : sock2) + 1;

// select() blocks until at least one FD is ready
printf("Waiting for activity...\n");
int ready = select(nfds, &readfds, NULL, NULL, NULL);
if (ready < 0) {
   perror("select");
   exit(1);
}

// check which socket is ready after the select returns ...
// the returned bitmask has only bits that represents the ready sockets 
if (FD_ISSET(sock1, &readfds)) {
    printf("Socket 1 ready\n");
}
if (FD_ISSET(sock2, &readfds)) {
    printf("Socket 2 ready\n");
}

2. Kernel Side Flow

When the app calls:

select(nfds, &readfds, &writefds, &exceptfds, &timeout);

Kernel copies these masks from user space into kernel memory, Then scans all Bits (O(n)).

• For every FD bit set in the mask, kernel checks readiness (read/write/error).
• If not ready, kernel puts the calling thread to sleep on all those FDs until an event happens or timeout expires.

The Event occurs, the thread woke up and if at least one FD becomes ready, the kernel modifies the bitmasks:

• Ready FDs stay set (1).
• Not ready FDs are cleared (0).
• Then the kernel copies the updated masks back to user space and returns the count of ready FDs.
• App uses FD_ISSET(fd, &mask) to find which FDs are ready.
• Note: The masks are now modified, which means all cleared except ready ones, app must rebuild the masks before the next select() call.

3. Limitations

• FD_SETSIZE cap (usually 1024 FDs) → hard limit on connections.
• O(n) scaling → kernel and user space both scan all FDs every time.
• No persistence → app rebuilds FD sets for each call.
• Data copy overhead → kernel copies FD sets in and out on every call.

4. A Deep Dive into poll()

poll was introduced to fill the gaps of the select() syscall, and to be clear and honest they didn’t make too many changes, like instead of using bitmasks they used array of FDs which means complexity still O(n), they have removed the limitation of 1024 FDs, added new events & introduced a new struct pollfd (this the element of the array of FDs i mentioned).

• No fixed FD_SETSIZE limit
• Provides more event types (POLLIN, POLLOUT, POLLERR, POLLHUP)
• Uses struct pollfd[] instead of bitmasks

How It Works !

1. Setup in User Space

The application creates an array of struct pollfd, one for each file descriptor, and calls poll().

int poll(struct pollfd * fds, int nfds, int timeout); 

struct pollfd {
    int fd;
    short events;   // what we want (POLLIN, POLLOUT, etc.)
    short revents;  // what happened (kernel fills this)
};

And then pass it to the poll() system call, which will block and wait for the ready FDs as the select(). Example :

// prepare variables
int sock1, sock2;              // file descriptors for sockets
struct pollfd fds[2];          // array of pollfd structs (each describes one FD)
char buffer[1024];

// create sockets and bind them...
// ...

// init pollfd array
fds[0].fd = sock1;             // first FD
fds[0].events = POLLIN;        // we care about read readiness
fds[0].revents = 0;            // will be set by the kernel when ready

fds[1].fd = sock2;             // second FD
fds[1].events = POLLIN;        // also care about read readiness
fds[1].revents = 0;            // initially zero

printf("Waiting for activity...\n");

// poll() blocks until at least one FD is ready
int ready = poll(fds, 2, -1);  // 2 = number of FDs, -1 = wait forever
if (ready < 0) {
    perror("poll");
    exit(1);
}

// after poll(), kernel updates revents for each FD
// check which socket is ready
if (fds[0].revents & POLLIN) {
    printf("Socket 1 ready\n");
}
if (fds[1].revents & POLLIN) {
    printf("Socket 2 ready\n");
} 

Before diving into kernel work, we will take a look about the main poll events :

• **POLLIN** - Data available for reading
• **POLLOUT** - Ready for writing (buffer has space)
• **POLLERR** - Error condition (kernel sets automatically)
• **POLLHUP** - Connection closed/hang up
• **POLLNVAL** - Invalid file descriptor

so you can use it like :

fds[0].events = POLLIN | POLLOUT;  // What you want to monitor
// Kernel fills fds[0].revents with what actually happened

2. Kernel Side Flow

When you call poll():

The kernel copies the user-provided pollfd[] array into kernel space and creates its own internal copy.

Allocate internal structures:

The kernel allocates a poll_wqueues object to manage all file descriptors and callbacks for the duration of the call. It also prepares a poll_table used by device drivers to register callbacks.

Check each file descriptor (O(n)) :

For every pollfd in the array:

• The kernel retrieves the file object with fget(fd).
• It calls the file’s f_op->poll() method. Every file type (sockets, pipes, etc.) implements its own poll() logic.
• The driver checks if the FD is ready. If it is, it returns an event mask (e.g., POLLIN, POLLOUT).
• If not ready, the kernel adds the current thread to the FD’s wait queue using the poll_table. Inside this, it registers a poll_wake() callback, so the thread can be woken up later.

Thread parking After all FDs are processed:

• If none are ready and timeout > 0, the kernel parks the thread in an interruptible sleep state.
• If any FD becomes ready later, its driver calls poll_wake(), which removes the thread from the wait queue and wakes it up.

Wake-up and return Once woken up (or when the timeout expires):

• The kernel unregisters callbacks from all FDs (removes them from wait queues).
• It copies the updated revents values back to user space.
• Finally, poll() returns the number of FDs that are ready.

Still O(n) Problem

Despite poll()’s improvements over select(), the fundamental scalability issue remains unsolved. Poll still cannot efficiently handle 10,000+ concurrent connections due to these critical bottlenecks:

Core Performance Issues:

• O(n) rescanning: Even after wake-up, the kernel must scan all nfds again to check each file descriptor’s readiness.
• Linear memory overhead: The entire struct pollfd[] array gets copied between user-space and kernel-space on every call
• No event persistence: Unlike modern solutions, poll doesn’t “remember” which FDs are interesting, it rediscovers everything each time

The Math Problem:

• 10,000 connections = 10,000 pollfd structs scanned per poll() call
• Each pollfd = ~12 bytes → 120KB copied each time
• High-frequency polling = megabytes of unnecessary memory traffic.

5. Simple echo server

In This section, we will look at code examples for a simple echo server: one using select and another using poll, both with detailed comments.

To avoid overcomplicating the code, the socket creation, binding, and setup are abstracted, we focus only on the event loop.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <arpa/inet.h>
#include <sys/socket.h>
#include <poll.h>

#define PORT 12345
#define MAX_CLIENTS 10
#define BUF_SIZE 1024

int main() {

   // variables we need 
  int server_fd;                  // the listening socket for incoming connections
  int client_fd;                  // temporary socket for a newly accepted client
  int max_fd;                     // the highest-numbered FD, needed for select()
  int activity;                   // result of select() call (number of ready FDs)

  int client_sockets[MAX_CLIENTS] = {0}; // array to track all connected client sockets

  struct sockaddr_in address;     // structure to store server IP and port info

  fd_set readfds;                 // set of FDs to be monitored by select()

  char buffer[BUF_SIZE];          // temporary buffer to read/write data to/from clients

  // create a socket and bind it here 
  // ...

    while (1) {
        FD_ZERO(&readfds); // as we said before we zero the bitmask first 
        FD_SET(server_fd, &readfds); // register a bit for our socket FD
        max_fd = server_fd; // max FDs to scan by the kernel

        // for this for loop we rgister each client FD in the bitmask.
        for (int i = 0; i < MAX_CLIENTS; i++) {
            if (client_sockets[i] > 0)
                FD_SET(client_sockets[i], &readfds);
            if (client_sockets[i] > max_fd)
                max_fd = client_sockets[i];
        }

        // calling select syscall and wait for any events 
        activity = select(max_fd + 1, &readfds, NULL, NULL, NULL);

        // we check for the returned bitmask and check if there is a new connection on the socket FD
        if (FD_ISSET(server_fd, &readfds)) {
            int new_socket = accept(server_fd, NULL, NULL); // accept the connection 
            for (int i = 0; i < MAX_CLIENTS; i++) {
                if (client_sockets[i] == 0) {
                    client_sockets[i] = new_socket; // register the client sock so next we register it's bit in the bitmask
                    break;
                }
            }
            printf("New client connected\n");
        }

        // if there is more events on the other FDs, we need to handle it.

        // iterate over clients
        for (int i = 0; i < MAX_CLIENTS; i++) {  

            // check which client is ready
            if (FD_ISSET(client_sockets[i], &readfds)) {
                int valread = read(client_sockets[i], buffer, BUF_SIZE);
                if (valread == 0) {
                    close(client_sockets[i]);
                    client_sockets[i] = 0;
                    printf("Client disconnected\n");
                } else {
                    buffer[valread] = '\0';
                    send(client_sockets[i], buffer, valread, 0);
                }
            }
        }
    }

  return 0
}

Now the poll echo server

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <arpa/inet.h>
#include <sys/socket.h>
#include <poll.h>

#define PORT 12345
#define MAX_CLIENTS 10
#define BUF_SIZE 1024

int main() {
    int server_fd; // the listening socket for incoming connections
    int client_fd; // temporary socket for a newly accepted client

    struct sockaddr_in address; // structure to store server IP and port info

    struct pollfd fds[MAX_CLIENTS + 1]; // array of pollfd structs:
                                        // fds[0] = server socket
                                        // fds[1..MAX_CLIENTS] = client sockets
                                        // each struct contains:
                                        //  .fd = socket FD
                                        //  .events = what we want to monitor (e.g., POLLIN)
                                        //  .revents = what actually happened (set by poll)

    char buffer[BUF_SIZE];// temporary buffer to read/write data to/from clients

    // with the same way connect and bind socket 
    //....

    // register the socket FD 
    fds[0].fd = server_fd;
    fds[0].events = POLLIN; // we need POLLIN event (which means ready for read)
    for (int i = 1; i <= MAX_CLIENTS; i++) fds[i].fd = -1;


    // the event loop
    while (1) {

         // calling poll and waiting for ready events.
        int activity = poll(fds, MAX_CLIENTS + 1, -1);

        // if the first FD has POLLIN revent then it's a new connection 
        if (fds[0].revents & POLLIN) {
            int new_socket = accept(server_fd, NULL, NULL);

            // register client FD in our struct, so we can poll it next
            for (int i = 1; i <= MAX_CLIENTS; i++) {
                if (fds[i].fd == -1) {
                    fds[i].fd = new_socket;
                    fds[i].events = POLLIN;
                    break;
                }
            }
            printf("New client connected\n");
        }

        // handle other events of clients that are ready.
        for (int i = 1; i <= MAX_CLIENTS; i++) {
            if (fds[i].fd != -1 && (fds[i].revents & POLLIN)) {
                int valread = read(fds[i].fd, buffer, BUF_SIZE);
                if (valread == 0) {
                    close(fds[i].fd);
                    fds[i].fd = -1;
                    printf("Client disconnected\n");
                } else {
                    send(fds[i].fd, buffer, valread, 0);
                }
            }
        }
    }

    return 0;
}

The END

I didn’t expect this article will be that long XD.

We covered the C10K problem, multiplexing basics, and dug into select() and poll(). You even got a hands-on echo server to see it all in action. Now you know how the kernel tracks sockets and handles multiple clients efficiently. Next step? Level up to epoll and async frameworks, and start building servers that truly scale, exactly what we’ll dive into in the next article on epoll & high-performance Linux servers.

References

• Linux man pages (man select, man poll)
• Original C10K problem write-ups by Dan Kegel
• I/O Multiplexing in Linux: Select vs Poll vs Epoll
• Medium article