TCP/IP, Ports, and Sockets¶

By Sameer Alam · Updated 20 October 2026

What You'll Learn¶

• How IPv4 addresses and CIDR blocks work, and how to size subnets
• How a Linux host decides where to send a packet, and what NAT changes
• How TCP connections open, close, and linger — and when UDP is used instead
• How ports, sockets, ephemeral ports, and MTU cause real production problems

The Layers That Matter¶

"Layer 4" and "Layer 7" load balancers take their names from this model.

IP Addresses and CIDR¶

An IPv4 address is 32 bits, written as four octets: 10.0.2.15. A CIDR suffix says how many leading bits are the network; the rest identify hosts.

In a traditional subnet, the first address (network) and last (broadcast) aren't usable by hosts. AWS reserves five addresses in every subnet, so a /24 gives 251 usable IPs.

Private address ranges¶

Plan ranges so they never overlap across VPCs, offices, and VPNs you might connect later. Overlapping CIDRs can't be routed to each other without NAT.

Check a plan before you build it. The planner flags overlaps between your ranges and with the defaults Docker, Kubernetes distributions, and clouds pick silently:

ipcalc 10.0.16.0/20        # network, broadcast, host range (sudo apt install ipcalc)
ip -4 addr show            # addresses on this host

Routing¶

Every host has a routing table. For each packet, the kernel picks the most specific matching route.

$ ip route
default via 10.0.0.1 dev eth0 proto dhcp src 10.0.2.15 metric 100
10.0.0.0/16 dev eth0 proto kernel scope link src 10.0.2.15
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1

$ ip route get 8.8.8.8
8.8.8.8 via 10.0.0.1 dev eth0 src 10.0.2.15

• Destinations inside 10.0.0.0/16 go directly out eth0.
• Everything else goes to the default gateway 10.0.0.1.
• ip route get shows exactly which route and source address a destination uses — the fastest way to answer "why is traffic leaving through that interface?"

On the local segment, the host learns the gateway's MAC address with ARP:

ip neigh show

NAT¶

Network address translation rewrites addresses as packets cross a boundary.

Two consequences you'll meet in practice:

• Servers behind SNAT all appear to the outside world as the NAT's public IP. Allow-listing that IP allows the whole private network.
• A NAT device tracks every connection. Exhausting its connection table or source ports causes intermittent timeouts under load.

NAT idle timeouts¶

A NAT device forgets a connection that's been idle too long. The next packet on it gets a reset, and the application sees Connection reset by peer on a connection it thought was healthy. Linux only sends TCP keepalives after 7,200 seconds idle by default, far longer than any of these:

Long-lived idle connections, such as database pools and message consumers, need keepalives shorter than the smallest timeout on the path: set tcp_keepalive_time below 300 seconds for those sockets (most drivers have an option), or have the pool close idle connections first.

TCP vs UDP¶

The three-way handshake¶

sequenceDiagram
    participant C as Client 10.0.2.15:51844
    participant S as Server 10.0.3.20:5432
    C->>S: SYN
    S-->>C: SYN-ACK
    C->>S: ACK
    Note over C,S: ESTABLISHED — data flows
    C->>S: FIN
    S-->>C: ACK, then FIN
    C->>S: ACK
    Note over C: TIME-WAIT, 60 seconds on Linux (up to 4 minutes elsewhere)

What the client sees when the handshake fails tells you a lot:

Connection states¶

ss -tan state established | head
ss -tan state time-wait | wc -l
ss -s

Ports and Sockets¶

A socket is identified by five values: protocol, source IP, source port, destination IP, and destination port. That's why one server port (:443) can hold thousands of connections at once — each has a different client address or port.

sudo ss -tulpn                       # listening TCP and UDP sockets, with the owning process
sudo ss -tnp dst 10.0.3.20:5432      # connections from this host to the database
nc -vz 10.0.3.20 5432                # can I open a TCP connection?

Listening on 127.0.0.1:8080 accepts connections only from the same host. Listening on 0.0.0.0:8080 (or [::]:8080) accepts them on every interface — check which one you need.

Ephemeral ports¶

The client side of each connection uses a temporary port:

$ sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 32768    60999

That's about 28,000 ports per destination IP and port. A service opening a new connection for every request to the same backend, with each socket lingering in TIME-WAIT, can run out — showing up as Cannot assign requested address errors. The fix is connection pooling and keep-alive, not kernel tuning.

Estimate how close a client is to running out:

MTU¶

The maximum transmission unit is the largest packet an interface sends without fragmentation: usually 1500 bytes on Ethernet, and 9001 inside some cloud networks.

When a path has a smaller MTU than the endpoints assume (VPNs, tunnels, overlay networks) and the ICMP messages that report it are blocked, small requests work but large ones hang. This is an MTU black hole.

ip link show eth0 | grep -o 'mtu [0-9]*'
# Test a full-size packet with "don't fragment": 1472 + 28 bytes of headers = 1500
ping -M do -s 1472 -c 3 10.0.3.20

If -s 1472 fails but -s 1372 works, something on the path has a lower MTU.

A Note on IPv6¶

IPv6 uses 128-bit addresses (2001:db8::10), has no NAT in the usual design, and is increasingly the default in cloud and Kubernetes environments. The same tools work with -6 (ip -6 route, ping -6, ss shows both). When something works over IPv4 but not by hostname, check whether the name resolves to an IPv6 address the network doesn't route.

What Most Engineers Miss¶

• Two CIDR blocks never partly overlap. Either they're disjoint or one contains the other, because both are aligned powers of two. That makes overlap checks simple, and it's why "just shift the range a bit" never works.
• Docker takes far more address space than docker0. New bridge networks, including one per Compose project, come from 172.17.0.0/16 through 172.31.0.0/16 in /16s, then 192.168.0.0/16 in /20s, and Swarm overlays come from 10.0.0.0/8. A Compose project on a laptop can silently shadow the corporate VPN's 172.20.0.0/16. Set default-address-pools in /etc/docker/daemon.json.
• TIME-WAIT belongs to whoever closes first. If the server closes the connection, the TIME-WAIT socket is on the server, costing it memory but no client ports. HTTP clients that close after every request pay the full 60 seconds per connection.
• A source port only has to be unique per destination. connect() can reuse the same local port for different destination addresses or ports, so spreading traffic across backends raises the limit. Calling bind() before connect() (to choose a source IP) reserves the port for all destinations at once, unless the socket sets IP_BIND_ADDRESS_NO_PORT.
• AWS NAT gateways have a per-destination limit. Each IP address on a NAT gateway supports up to 55,000 simultaneous connections to each unique destination; beyond that the ErrorPortAllocation metric climbs and new connections fail. Add secondary IPs or spread destinations.
• NAT gateways don't reassemble fragments. AWS NAT gateways don't support IP fragmentation for TCP or ICMP, so ping -s 10000 fails behind one, and so does any server that answers with fragmented TCP.

Common Mistakes¶

• Choosing overlapping VPC or office CIDRs, then being unable to connect them with peering or VPN.
• Sizing Kubernetes or container subnets too small, and running out of IP addresses as the cluster grows.
• Treating "connection refused" and "connection timed out" as the same problem — one is a missing listener, the other a firewall or routing issue.
• Binding a service to 127.0.0.1 and wondering why other hosts can't reach it.
• Raising the ephemeral port range instead of fixing missing connection reuse.
• Blocking all ICMP, which breaks path MTU discovery and creates hard-to-diagnose hangs.

Check Yourself¶

Interview Questions¶

• How many usable IP addresses are in a /26 subnet in AWS, and why?
• What's the difference between "connection refused" and "connection timed out"?
• Walk through the TCP three-way handshake. What is TIME-WAIT, and why does it exist?
• A service logs thousands of CLOSE-WAIT sockets. What does that suggest?
• What is an MTU black hole, and how would you detect one?

Next¶

Continue to DNS.