Justifying Dally's Heuristics

Every figure in the essay rebuilds from six constants

Dally's figures, next to the reconstruction on this page. Thirteen match to the precision he quotes. The add and the sub-array come within \(25\%\) and \(15\%\). The \(230\times\) needs one inferred step.

Arithmetic is nearly free; distance is not

Dally's price list on one log scale. Switch the unit to read each cost as the distance a \(64\)-bit word could travel on chip for the same energy, or as a count of \(32\)-bit adds.

At \(1.92\,\mathrm{pJ}\) per word-millimetre, every cost becomes a distance. A sub-array read is \(0.33\,\mathrm{mm}\) of wire. Going off chip once is \(167\,\mathrm{mm}\), ten chips' width. A DRAM fetch is \(667\,\mathrm{mm}\).

Six constants generate the whole table

Here \(\alpha\) is the fraction of bits that toggle, \(f_{\text{rep}}\) the extra capacitance of repeaters, \(p\) the pipelining overhead of a real fabric, \(b\) the word width and \(\kappa\) the distance convention. Dally names no process; these values fit a \(5\)-nm-class chip at about \(0.75\,\mathrm{V}\).

Energy

A bit-millimetre costs \(30\,\mathrm{fJ}\) because a wire is a capacitor

Moving a bit one millimetre moves \(2{,}000\) gates' worth of charge

A logic \(1\) on a wire is charge held at the supply voltage, \(Q=CV\). A \(5\)-nm gate holds a few hundred electrons; a millimetre of wire holds nearly a million. That ratio, not transistor speed, is why communication dominates.

Charging \(C\) to \(V\) draws \(CV^2\) from the supply. Half is stored, half is burned in the resistance on the way in, and the stored half is burned on discharge. Each transition costs \(\tfrac12CV^2\) whatever the resistance, so lowering \(R\) makes a wire faster, never cheaper. Random data toggles a wire on half its bits, \(\alpha\approx\tfrac12\):

Capacitance per unit length is a shape factor, so it doesn't scale

Each face of a wire is a small parallel-plate capacitor, and every term is a ratio of lengths. Shrink a wire and its neighbours together and \(c\) stays put. Low-k dielectrics offset taller, closer-packed wires, so \(c\) has sat near \(0.2\,\mathrm{fF}/\mu\mathrm{m}\) since the 1990s.

Dally's \(30\,\mathrm{fJ}\) is that wire at \(\tfrac34\,\mathrm{V}\) carrying random data

The \(0.1\,\mathrm{pJ}\) per bit-millimetre rule of the 2010s is the same wire at \(1\,\mathrm{V}\) with every bit toggling. Dally's \(30\,\mathrm{fJ}\) sits between a bare wire (\(28\)) and one with a third more load from repeaters (\(37\)). A model that quotes energy per bit-millimetre should state its \(V\), \(\alpha\) and repeater load.

Speed

Signals crawl at \(c/120\) because on-chip wires are RC lines

Resistance per unit length grows with every shrink

Both dimensions shrink, so \(r\) grows as the square of the shrink. Below copper's \(39\,\mathrm{nm}\) electron mean free path, its resistivity climbs too. Buses ride the intermediate layers, where \(rc\approx3.6\,\mathrm{ns/mm^2}\).

Unrepeated delay grows as \(L^2\); repeated delay grows as \(L\)

A distributed RC line charges by diffusion, and its resistance and capacitance both grow with length. Its \(50\%\) delay is \(t\approx0.38\,rcL^2\); a lumped RC would take \(0.69\,RC\). Unrepeated, \(15\,\mathrm{mm}\) of intermediate metal takes about a thousand clock cycles.

Repeaters trade wire delay for gate delay, and the optimum is a square root

Choosing both the number and the size of the repeaters (Bakoglu, 1990) gives

Delay becomes linear in length, at a velocity set by the geometric mean of the gate and wire time constants: \(0.34\,\mathrm{ns/mm}\) here. Pipelining and arbitration add about \(20\%\), which gives Dally's \(0.4\,\mathrm{ns/mm}\). Each process node shrinks \(R_0C_0\) and grows \(rc\) by similar factors, so the product barely moves. Repeated wires have stayed within \(2\times\) of \(0.3\,\mathrm{ns/mm}\) since the \(130\)-nm node, while gates got \(10\times\) faster.

On-chip signals move \(70\times\) slower than light in the same glass

Light in the chip's dielectric would cover \(173\,\mathrm{mm/ns}\), but on-chip wires are too resistive to act as transmission lines. Board traces do act as transmission lines, so Dally prices the board in energy and adds only \(6\,\mathrm{ns}\) per metre.

Memory

Memory is two-dimensional, so access cost grows as \(\sqrt{N}\)

Where the requester sits sets the distance

For uniform targets \(\mathbb{E}\lvert x-y\rvert=L/3\) along each axis, and Manhattan distance adds the two axes. A core at the centre of its \(1\,\mathrm{mm}\) tile travels \(L/2\) each way: Dally's \(1\,\mathrm{mm}\) round trip. Two random points on a \(16\,\mathrm{mm}\) chip are \(2L/3\) apart: his \(21.3\,\mathrm{mm}\).

His "\(15\,\mathrm{mm}\) each way" for the \(10\,\mathrm{mm}\) block that holds \(256\,\mathrm{MB}\) is no mean. It is \(1.5L\), the \(87.5\)th percentile of distance from a corner: a deliberately conservative figure. Latency budgets want a high percentile; energy budgets want the mean.

Every \(4\times\) in capacity costs \(2\times\) in energy and time

Small memories pay mostly for the endpoint, large ones for the wire. Energy crosses over near \(160\,\mathrm{KB}\), the size of an L2 cache; latency crosses near \(0.8\,\mathrm{MB}\). That is why cache hierarchies are geometric. A level \(4\times\) larger is \(2\times\) slower and \(2\times\) dearer to reach, so each level belongs where the reuse it captures pays for that factor of two.

A sub-array's \(0.64\,\mathrm{pJ}\) is short wires, not bit cells

An \(8\,\mathrm{KB}\) sub-array is \(256\) wordlines by \(256\) columns of six-transistor cells, about \(60\times45\,\mu\mathrm{m}\) with its periphery. A \(64\)-bit read fires one wordline across all \(256\) columns and senses \(64\) of them through a \(4{:}1\) multiplexer.

"The time and energy needed to read or write a single bit cell is negligible."Dally (2022)

The cell itself contributes under \(1\,\mathrm{fJ}\). The \(300\,\mathrm{ps}\) is about \(25\) gate delays: decode, three short RC lines and one small-signal sense. It is the fixed \(t_0\) that a pure distance model lacks.

\(256\,\mathrm{MB}\) fits in \(100\,\mathrm{mm^2}\) at \(45\%\) array efficiency

Density enters only as \(\sqrt{D}\): doubling it shrinks every distance by \(\sqrt2\). SRAM cells have barely shrunk since N5, from \(0.021\,\mu\mathrm{m^2}\) at N5 and N3E to about \(0.0175\,\mu\mathrm{m^2}\) at N2. The way to bring a large memory closer now is to stack it.

Off chip

Leaving the chip costs \(5\,\mathrm{pJ}\) per bit, each way

Dally's \(320\,\mathrm{pJ}\) to go off chip is \(5\,\mathrm{pJ/bit}\) for a \(64\)-bit word. A remote access crosses twice, request out and data back, hence the \(640\,\mathrm{pJ}\) in his summation example. His DRAM figures, \(1.3\,\mathrm{nJ}\) per \(64\) bits and \(640\,\mathrm{pJ}\) to write and re-read a \(16\)-bit activation, are both \(20\,\mathrm{pJ}\) per bit per direction.

Above a few \(\mathrm{Gb/s}\) a board trace is a transmission line, terminated in about \(50\,\Omega\), so it draws current whether or not bits toggle. Pads, ESD structures and package capacitance add more, as do equalizers, clock recovery and serializers at both ends.

Arithmetic

An add costs \(20\,\mathrm{fJ}\); overhead and distance cost the rest

The \(64{,}000\times\) is a DRAM fetch over an add; a CPU shrinks it to \(15\times\)

Horowitz's \(45\)-nm table puts a \(32\)-bit add at \(0.1\,\mathrm{pJ}\). Going to \(5\,\mathrm{nm}\), the switched capacitance falls about \(4.5\times\) and \(V^2\) falls \(1.44\times\), which leaves \(15\,\mathrm{fJ}\), close to Dally's \(20\,\mathrm{fJ}\). Fetching the two operands from DRAM at \(20\,\mathrm{pJ/bit}\) costs \(1.28\,\mathrm{nJ}\), exactly \(64{,}000\) adds.

A CPU wraps the add in instruction fetch, decode, register reads and pipeline control, \(80\,\mathrm{pJ}\) in all. Against that, a \(1.2\,\mathrm{nJ}\) memory access looks only \(15\times\) dearer. A domain-specific accelerator strips the overhead and exposes the whole gap.

Recompute beats store when a layer has fewer than \(4{,}000\) inputs

Backpropagation needs each layer's activations. Storing one to DRAM and reading it back costs \(640\,\mathrm{pJ}\). Recomputing it is a matrix-vector product, \(n\) multiply-accumulates at \(160\,\mathrm{fJ}\) each. Recompute wins while \(n\cdot0.16\,\mathrm{pJ}\lt640\,\mathrm{pJ}\), that is for \(n\lt4{,}000\). With HBM the threshold falls to \(n\approx600\), the same arithmetic that drives FlashAttention-style recomputation.

Placing data at random makes a sum \(230\times\) dearer

Dally's example sums a table spread over \(4\times4\) chips of \(256\) cores, each core holding \(256\mathrm{K}\) words in its own \(2\,\mathrm{MB}\). Summing locally costs \(1.92\,\mathrm{pJ}\) of wire per word. At random, every word travels \(21.3\,\mathrm{mm}\) on its home chip, and the \(15/16\) that live elsewhere add \(640\,\mathrm{pJ}\) and another \(21.3\,\mathrm{mm}\):

Dally's all-in \(230\times\) follows if both sides also pay the per-word work they share: the \(0.64\,\mathrm{pJ}\) sub-array read and about \(0.4\,\mathrm{pJ}\) for the add, \((680+1.04)/(1.92+1.04)\approx230\). He doesn't show this step; it is the simplest one that reproduces his number.

Calculator

Recompute every figure for another process

"Dally" pins the transport constants to his. Drag any constant to derive them from the process instead; every row updates.

import math

# Process constants: 5-nm class, about 2022
c    = 0.2e-12      # F/mm, wire capacitance (0.2 fF/um)
r    = 18e3         # ohm/mm, intermediate-metal resistance
V    = 0.75         # V, supply
R0C0 = 5e-12        # s, inverter time constant
D    = 21.4e6       # bit/mm^2, effective SRAM density
e_io = 5e-12        # J/bit for each chip crossing
alpha, f_rep, pipe = 0.5, 0.1, 1.2   # activity, repeater load, fabric overhead
E0, t0 = 0.64e-12, 0.3e-9            # 64-bit sub-array access (Dally)

e_w  = alpha * 0.5 * c * (1 + f_rep) * V**2         # J per bit-mm
v    = 1 / (2.5 * math.sqrt(R0C0 * r * c)) / pipe    # mm per s
Lseg = math.sqrt(0.7 * R0C0 / (0.4 * r * c))         # mm between repeaters

def move(d_mm, b=64):
    """Energy (J) and time (s) to move b bits d_mm millimetres."""
    return b * e_w * d_mm, d_mm / v

def sram(n_bits, kappa, b=64):
    """Access to a square SRAM of n_bits; kappa sets the one-way distance."""
    side = math.sqrt(n_bits / D)
    E, t = move(2 * kappa * side, b)
    return E0 + E, t0 + t

E1, t1 = move(1.0)                         # core at the centre of its 1-mm tile
E256, t256 = sram(2**31, kappa=1.5)        # 256 MB, Dally's 15 mm each way
Er, tr = move(4 * 16 / 3)                  # random point to random point, 16-mm chip
avg = Er + 15/16 * (2 * 64 * e_io + Er)    # 4x4 chips: 15/16 of accesses leave the chip

print(f"e_w = {e_w*1e15:.1f} fJ/bit-mm, v = {v*1e-9:.2f} mm/ns, L_seg = {Lseg*1e3:.0f} um")
print(f"1 mm   : {E1*1e12:5.2f} pJ {t1*1e9:5.2f} ns   (Dally 1.9 pJ, 0.4 ns)")
print(f"256 MB : {E256*1e12:5.1f} pJ {t256*1e9:5.1f} ns   (Dally 58 pJ, 12.3 ns)")
print(f"random : {avg*1e12:5.0f} pJ, {avg/E1:.0f}x local   (Dally 680 pJ, 354x)")

e_w = 30.9 fJ/bit-mm, v = 2.48 mm/ns, L_seg = 49 um
1 mm   :  1.98 pJ  0.40 ns   (Dally 1.9 pJ, 0.4 ns)
256 MB :  60.1 pJ  12.4 ns   (Dally 58 pJ, 12.3 ns)
random :   682 pJ, 344x local   (Dally 680 pJ, 354x)

Where the heuristics bend