Practitioner Field Notes
Unwritten production survival tips, real-world war stories, and high-impact operational insights learned the hard way on multi-terabyte clusters.
Never Manually Delete Files from pg_wal!
Running `rm -rf pg_wal/*` during a disk full panic corrupts the transaction log permanently.
Use `pg_archivecleanup` or drop unused temporary tables to allow a graceful checkpoint.
Use Percona pt-online-schema-change for 100M+ Row Tables
Large ALTER TABLE statements can cause Metadata Lock (MDL) queues that stall web traffic.
Use shadow-table based online schema migration tools to migrate large datasets with negligible lock overhead.
Never Leave SQL Server 'Max Server Memory' at Default Unlimited!
The default 2.1TB memory ceiling causes SQL Server to starve the Windows OS, triggering severe thread paging.
Reserve 4GB-8GB (10-15% total RAM) for the operating system and configure explicit max server memory.
Raise Cost Threshold for Parallelism from Default (5) to 40-50
Default cost threshold of 5 triggers premature query parallelism and excessive CXPACKET synchronization waits on multi-core servers.
Set cost threshold for parallelism to 40-50 and bound MAXDOP to the physical core count of a single NUMA node.