One of our applications (Freeswitch) just randomly crashed for no apparent reason and didn’t write anything to it’s log files. The service we’re trialling is currently in Beta so there’s room to muck about and do some diagnostics. I want to make the kernel dump a core file whenever Freeswitch dies, in case it happens again, so that we have some stuff to work with after the fact. It’ll also shut up my QA manager.
Check The Current Linux Core Dump Limits
ulimit is used to specify the maximum size of generated coredumps, this is to stop apps chewing up a million GB of RAM and then blowing your disk up, by default it’s a 0, which means nothing gets written to disk and no dump is created!
hstaging:~ # ulimit -c 0
Change The Linux Core Dump Limits To Something Awesome
To set the size limit of the linux core files to 75000 bytes, you can do something like this
hstaging:~ # ulimit -c 75000 hstaging:~ # ulimit -c 75000
but I’m a maverick, this does exactly what you think it does
hstaging:~ # ulimit -c unlimited hstaging:~ # ulimit -c unlimited
Enable Linux Core Dump For Application Crashes And Segfaults And Things
Ok, so we want this to persist across reboots so that basically means we have to stick the ulimit command in /etc/profile, i’m putting this at the bottom of mine:
#corefile stuff ulimit -c unlimited > /dev/null 2>&1
this will stop anything weird getting spat out to the screen and nicely tells us that it’s core file stuff.
For our next trick we’ll set some sysctl flags so in /etc/sysctl.conf add
#corefile stuff kernel.core_uses_pid = 1 kernel.core_pattern = /tmp/core-%e-%s-%u-%g-%p-%t fs.suid_dumpable = 2
this basically says when the application crashes create a coredump file in /tmp with a useful name pattern
kernel.core_uses_pid = 1 - add the pid of the crashed app to the filename.
fs.suid_dumpable = 2 - enable linux core dumps for setuid processes.
kernel.core_pattern = /tmp/core-%e-%s-%u-%g-%p-%t - crazy naming pattern for a successful core dump, here's roughly what all the bits mean:
%e - executable filename
%s - number of signal causing dump
%u - real UID of dumped process
%g - real GID of dumped process
%p - PID of dumped process
%t - time of dump (seconds since 0:00h, 1 Jan 1970)
super usefuls. Then run sysctl -p so it takes effect yo!
hstaging:~ # sysctl -p kernel.core_uses_pid = 1 kernel.core_pattern = /tmp/core-%e-%s-%u-%g-%p-%t fs.suid_dumpable = 2
Enabling Linux Core Dump For All Apps
Now here’s the last part. When you want an application to core dump you create an environment variable, before you start it, telling the kernel to sort itself out and get ready to dump, if you want all apps on the server to generate core dumps then you’re going to want to specify this variable somewhere near the top of the process chain. The best place for this on a redhat style box is /etc/sysconfig/init, so stick the following in that file
DAEMON_COREFILE_LIMIT='unlimited'
now might be an idea to reboot to force it to be set across all applications and things
Enabling Linux Core Dumps For A Specific Application
This is the slightly less rebooty version of the above. Rather than force the environment variable to be loaded when the box starts, we just stick it in the init script for the deamon, and then restart the daemon.
In /etc/init.d/functions the RedHat guys have already stuck in
corelimit="ulimit -S -c ${DAEMON_COREFILE_LIMIT:-0}"
So we need to make sure we put our DEAMON_COREFILE_LIMIT above that. Simples. In our case it’s in /etc/init.d/freeswitch with
DAEMON_COREFILE_LIMIT='unlimited'
Distros That Aren’t RedHat
DAEMON_COREFILE_LIMIT is a RedHatism. If you’re running something cool, like Ubuntu, you’ll want to use
ulimit -c unlimited >/dev/null 2>&1 echo /tmp/core-%e-%s-%u-%g-%p-%t > /proc/sys/kernel/core_pattern
instead.
Testing Core Dumps
This is EASY, we just start the deamon, send a segfault signal, look in the right place!!
hstaging:tmp # /etc/init.d/freewitch start hstaging:tmp # /etc/init.d/freeswitch status freeswitch (pid 8257) is running... hstaging:tmp # kill -s SIGSEGV 8257 hstaging:tmp # ls /tmp/core* core-freeswitch-11-493-492-8257-1371823178
Now you give this file to your developers and take a bow!