Moderators (requests and notices)

PostgreSQL invoked the out of memory process killer.

Feb 21 22:46:14 sql kernel: postgres invoked oom-killer: gfp_mask=0x27000c0(GFP_KERNEL_ACCOUNT|__GFP_NOTRACK), order=2, oom_score_adj=0
Feb 21 22:46:14 sql kernel: postgres cpuset=/ mems_allowed=0
Feb 21 22:46:14 sql kernel: CPU: 5 PID: 17775 Comm: postgres Not tainted 4.8.6-x86_64-linode78 #1
Feb 21 22:46:14 sql kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-prebuilt.qemu-project.org 04/01/2014
Feb 21 22:46:14 sql kernel: 0000000000000286 0000000000000000 ffffffff816e8915 0000000000000000
Feb 21 22:46:14 sql kernel: ffff8801ee173d40 ffff880330713200 ffffffff8122bda5 0000000000000000
Feb 21 22:46:14 sql kernel: ffff88033fffb420 ffff880329e0d780 0000000000000001 ffff88033fffa000
Feb 21 22:46:14 sql kernel: Call Trace:
Feb 21 22:46:14 sql kernel: [<ffffffff816e8915>] ? dump_stack+0x5d/0x88
Feb 21 22:46:14 sql kernel: [<ffffffff8122bda5>] ? dump_header+0x5d/0x1ff
Feb 21 22:46:14 sql kernel: [<ffffffff811f832c>] ? compact_zone_order+0x6c/0x80
Feb 21 22:46:14 sql kernel: [<ffffffff811d1a98>] ? oom_kill_process+0x278/0x460
Feb 21 22:46:14 sql kernel: [<ffffffff8166aea2>] ? security_capable_noaudit+0x42/0x60
Feb 21 22:46:14 sql kernel: [<ffffffff811d1feb>] ? out_of_memory+0x30b/0x3f0
Feb 21 22:46:14 sql kernel: [<ffffffff811d672b>] ? __alloc_pages_nodemask+0xd4b/0xd80
Feb 21 22:46:14 sql kernel: [<ffffffff810fc8d2>] ? copy_process.part.44+0x162/0x1ae0
Feb 21 22:46:14 sql kernel: [<ffffffff81156e11>] ? lock_timer_base.isra.39+0x71/0x90
Feb 21 22:46:14 sql kernel: [<ffffffff81156fcc>] ? mod_timer+0x19c/0x300
Feb 21 22:46:14 sql kernel: [<ffffffff812016ef>] ? handle_mm_fault+0x51f/0xd80
Feb 21 22:46:14 sql kernel: [<ffffffff810fe445>] ? _do_fork+0xf5/0x390
Feb 21 22:46:14 sql kernel: [<ffffffff8105a6ab>] ? __do_page_fault+0x1bb/0x510
Feb 21 22:46:14 sql kernel: [<ffffffff810038f6>] ? do_syscall_64+0x56/0xb0
Feb 21 22:46:14 sql kernel: [<ffffffff81bf24e1>] ? entry_SYSCALL64_slow_path+0x25/0x25

A slow query locked a lot of resources and that caused more requests to back up (we're busy at this time of night) and in turn that caused more resource exhaustion which led to the process being killed automatically.