small encoder training throughput on consumer gpus

benchmarks of large language models do not answer a smaller question: how quickly can consumer hardware train a transformer encoder with 5 million or 21 million parameters? this page reports measurements from one training program on three nvidia gpu models and three identical zen 4 machines. the results apply to this workload and software stack, not transformer training in general.

workload

the test used two encoder configurations from the same training program. both use the ModernBERT backbone, a local attention window of 128 tokens, global attention every 3 layers, and PyTorch’s scaled dot-product attention (SDPA). ModernBERT was designed around alternating local and global attention rather than full attention in every layer.³⁴

the 5m model stores 3,670,240 of its 4,871,585 parameters in the 16,385 by 224 token-embedding table. its tied masked-language-model (mlm) output projection therefore accounts for much of its matrix work, even though the encoder has only two transformer layers.

both models used a physical batch of 16, a sequence length of 256, bfloat16 arithmetic, and the AdamW optimizer. throughput is reported in attended tokens per second: tokens that were not padding.

devices

Nvidia’s published specifications confirm the core counts and memory sizes below. The company also specifies 896 GB/s of memory bandwidth for the RTX 5070 Ti.¹²

measured results

production pipeline, 21m model

the full training pipeline was run on each device with a byte-identical configuration (sha256 0fe3a9ce41e8760338f70414637304a7e7068d60dac71e4f86fe2c671b6b12cc), 20 optimizer steps. every run reported 326,444 attended tokens, 21,308,033 parameters, and 7,014,919,680 bytes of peak CUDA device memory. CUDA is Nvidia’s software platform for GPU computing. these checks show that the measured runs used the same model and batch data.

completed epochs, 5m model

two complete training epochs of 23,216,709 records and 3,029,577,143 attended tokens each:

model-compute only

a second benchmark isolates forward/backward/optimizer on the same model classes with synthetic batches and no padding, removing the dataloader and auxiliary loss stack. torch 2.9.1, transformers 5.15.0, 60 steps after 10 warmup.

what the measurements support

bandwidth alone does not explain the result

the rtx 5070 ti has 33% more memory bandwidth than the rtx 4070 ti super (896 versus 672 gb/s), but it was only 5.6% faster on the 5m model and 3.3% faster on the 21m model. bandwidth alone therefore does not explain these results. proving that the workload is compute-bound would require profiler data, which this benchmark did not collect.

throughput fell by 4.02 times when the parameter count grew by 4.37 times. this close relationship is consistent with arithmetic work being important, but parameter count is not itself a count of floating-point operations.

pipeline overhead is substantial

on the rtx 4060 ti, the full pipeline reached 24,086 tokens/s, compared with 80,716 tokens/s in the synthetic model-only test. that is about 30%. the two tests differ in data loading, auxiliary losses, and padding, so the benchmark cannot assign the remaining 70% to any one cause.

model-only rates should not be used as estimates of full-pipeline throughput without measuring the rest of the training loop.

the rtx 4060 ti is usable

at 0.49x an rtx 5070 ti and identical 7.01 gb peak memory, the rtx 4060 ti is roughly half the throughput for a card with a 128-bit bus. two of them approximate one rtx 5070 ti in aggregate.

for experiment designs that run independent concurrent cells rather than distributed data parallel, slot count can matter more than per-run speed. under those conditions, adding a slower card that is already owned is often better than buying a faster one.

projected epoch times

measured values are marked; the remainder are projected from the ratios above and should be treated as estimates.

the cpu range spans two projection methods: the lower bound applies the model-compute ratio directly, the upper bound also applies the 30% pipeline efficiency measured on gpu.

methodology notes

• both benchmarks instantiate the same model classes used in production, not a reimplementation. parameter counts were verified to match exactly (4,871,585 and 21,308,033).
• torch and transformers versions were pinned identically across all devices (2.9.1, 5.15.0). an unpinned install produced torch 2.14 on some hosts, which would have made the comparison invalid.
• flash-attention is not installed in the reference environment, so all devices use sdpa. this keeps the attention implementation portable across cuda and cpu.
• gpu timings use torch.cuda.synchronize() around the measured region.

considerations for cross-vendor hardware

changing gpu vendor changes the numerical environment. for experiment programs that compare training arms at high precision and pool results across runs, a rocm or mps arm is not directly exchangeable with cuda results without re-establishing the comparison basis. staying within one vendor — and preferably one architecture generation — avoids re-validating that basis.

in the measurements above, the rtx 4070 ti super and rtx 4060 ti are both ada lovelace, and the rtx 5070 ti is blackwell. the production runs already pool results across ada and blackwell devices, so adding another ada card stays inside an already-validated envelope.

references

[1] nvidia. rtx 4060 ti specifications.

[2] nvidia. geforce rtx 50 series.

[3] answer.ai. (2024). modernbert: a modern bert replacement.

[4] pytorch. scaled dot product attention.