Debuging random segvs

This code is a simple and Focused segv catcher, it is designed for transient and random errors, the ones where you never know when or if they are going to happen. The idea is to use a local varable to scope the monitered area and keep tabs of the last line of code with the jump function. There is a streaming mech to gather input params and other data along the way(ie you can log the local vars in it after the fact). Then when a segv actually happens it wakes up and informs you of where it feel over and what the logged data was.

#include <csignal>
#include <iostream>
#include <stdlib.h>
#include <sstream>
 
namespace SigCatchDebug
{
void (*original)(int);
bool installed = false;
std::string   file_;
int           line_  = 0;

std::stringstream data;

void debug_sig(int type)
{
    if(line_ != 0)
    {
        std::cout << "Signal Caught!!!" << std::endl
                  << "last trace loc:" << file_ << ":" << line_ << std::endl
                  << "----- data was ---- "                     << std::endl
                  << data.str() << std::endl
                  << std::endl;
    }

    if(original != SIG_DFL ||
       original != SIG_IGN)
        (*original)(type);
    
    std::exit(EXIT_FAILURE);  //<<-- this really needs to be here!
}

class AutoScope
{
public:
    AutoScope(const char* file,
              const int   line)
    {
        if(!installed)
        {
            installed = true;

            original = std::signal (SIGSEGV,debug_sig);
            if (original == SIG_ERR)
            {
                std::cout << "signal implosion" << std::endl;
            }            
            //std::cout << "signal installed" << std::endl;
        }
        
        file_        = file;
        line_        = line;

        data.str("");
    }

    template<typename T>
    inline AutoScope& operator<<(const T& t) throw()
    {
        try
        {
            data << t;
        }
        catch (...)
        {}
        return *this;
    }

    ~AutoScope()    
    {
        //std::cout << "signal disable" << std::endl;
        line_ = 0;
    }

    void jump(int line)
    {
        line_ = line;
    }

};

}

void evil_random_crashing_func()
{
    static count = 0;
    char** test = NULL;

    SigCatchDebug::AutoScope sigScope(__FILE__,__LINE__);
    sigScope << "test: " << test << "\n"
             << "count: " << count;

    sigScope.jump(__LINE__);
    if(count == 3)
        test[0] = "ABCDEFG";

    sigScope.jump(__LINE__);
    if(count == 4)
         std::cout << "evil_random_crashing_func: " << test[0] << std::endl;
    
    std::cout << "Evil " << count << std::endl;
    count++;
}

void evil_random_crashing_func2()
{
    char** test = NULL;

    test[0] = "ABCDEFG";
    std::cout << "evil_random_crashing_func2: " << test[0] << std::endl;
}

int main(int argc, char const * const *argv)
{
    try
    {
        evil_random_crashing_func();
        evil_random_crashing_func();
        //evil_random_crashing_func2();
        evil_random_crashing_func();
        evil_random_crashing_func();
        
    }
    catch (std::exception const &e)
    {
        std::cout << "Unexpected exception: " <<  e.what() << std::endl;
    }
    catch (...)
    {
        std::cout << "Unexpected exception" << std::endl;
    }
    return 0;
}

Using dbx to get the contents of the stack on sun machines

Dealing with a coredump created by a transient error. The core is 19GB+. So reproducing it isnt possible. To get the job done i need to get the local vars, param passed in an return value out. To do that I actually have to go to the raw memory dump of the stack and back track it. Here is how

First load your core

dbx exefile corefile

This will get you basic stack demunging.

(dbx) where

This outputs

  [5] __sighndlr(0xb, 0xffffffff6eff8a10, 0xffffffff6eff8730, 0xffffffff7c804c20, 0x0, 0xa), at 0xffffffff752d65b4 
  ---- called from signal handler with signal 11 (SIGSEGV) ------
=>[6] function1(0x0, 0x4, 0x1f, 0xffffffff6eff9518, 0x1029feaf0, 0x2), at 0x100139548 
  [7] function1(0x1029feab0, 0x1f, 0x1003a1, 0x100381000, 0x9000, 0x100000), at 0x1000de2a8

Dbx will guess and print the input params input in the () but keep in mind that the real params maybe passed in the regs so take care.

Jump to the frame you want in my case it was 6, this loads the $sp (stack pointer) and $fp(frame pointer) regs to the correct values. Then dump memory at the $fp

(dbx) frame 6  
(dbx) print -F"%lx" $fp 
  ffffffff6eff8a51
(dbx) examine 0xffffffff6eff8a50  / 100 X   

This produces out like the follow. Now you need to know that the Stack bias is 2047 for 64bit sun machines. Your data starts down the mem dump a bit. For me its down at 0xffffffff6eff8b50-ish:

0xffffffff6eff8a50:      0x00000000 0x00000000 0x00000000 0x00000000
0xffffffff6eff8a60:      0x00000000 0x00000000 0x00000000 0x00000000
0xffffffff6eff8a70:      0x00000000 0x00000000 0x00000000 0x00000000
0xffffffff6eff8a80:      0x00000000 0x00000000 0x00000000 0x00000000
0xffffffff6eff8a90:      0x00000000 0x00000000 0x00000000 0x00000000
0xffffffff6eff8aa0:      0x00000000 0x00000000 0x00000000 0x00000000
0xffffffff6eff8ab0:      0x00000000 0x00000000 0x00000000 0x00000000
0xffffffff6eff8ac0:      0x00000000 0x00000000 0x00000000 0x00000000
0xffffffff6eff8ad0:      0x00000000 0x00000000 0x00000000 0x00000000
0xffffffff6eff8ae0:      0x00000000 0x00000000 0x00000000 0x00000000
0xffffffff6eff8af0:      0x00000000 0x00000000 0x00000000 0x00000000
0xffffffff6eff8b00:      0x00000000 0x00000000 0x00000000 0x00000000
0xffffffff6eff8b10:      0x00000001 0x00263000 0x00000000 0x00000000
0xffffffff6eff8b20:      0x00000000 0x00100263 0x00000000 0x00100000
0xffffffff6eff8b30:      0x00000000 0x0003ea86 0x00000000 0x00000000
0xffffffff6eff8b40:      0x00000000 0x00000000 0x00000000 0x0000001f
0xffffffff6eff8b50:      0x00000000 0x00000000 0x00000000 0x00000004
0xffffffff6eff8b60:      0x00000000 0x0000001f 0xffffffff 0x6eff9518
0xffffffff6eff8b70:      0x00000001 0x029feaf0 0x00000000 0x00000002
0xffffffff6eff8b80:      0xffffffff 0x6eff8a51 0x00000001 0x000de2a8
0xffffffff6eff8b90:      0x00000000 0x00000000 0x00000000 0x00000000
0xffffffff6eff8ba0:      0x00000000 0x07db0301 0x00000000 0x00000000
0xffffffff6eff8bb0:      0x00000000 0x00000000 0x00000000 0x00000000
0xffffffff6eff8bc0:      0x00000000 0x00000000 0x00000000 0x00000000
0xffffffff6eff8bd0:      0x00000000 0x00000000 0x00000000 0x00000000

Then read out the raw mem data and map it to you mem layout the local vars. What a headache right...

For the general stack layout: http://en.wikipedia.org/wiki/Call_stack

For "Stack Bias" Refer to: http://www.shrubbery.net/solaris9ab/SUNWdev/SOL64TRANS/p13.html

strace - Debugging a live misbehaving process.

Often when a server is live you dont have the luxury of installing a debug-able version and stepping through its code when the problem appears. Other times you need to get a hold on whatever is killing a program that you never even wrote.

Enter "strace" this tool watches the system calls of a process and dumps it to your console for analysis. strace is not for the faint at heart. You need to know about how shells operate, how dynamic linked libraries are loaded and based and how kernel system calls pass in and out of system. On the flip side if you have never seen it before its a real eye opener. The average programmer most likely just doesn't realize how involved the kernel is with your little program.

To execute it on a program from scratch

strace -v <filename>

Note the "-v" gives you information about the environment variables that are passed from the shell to the new process.

To attach to a live process (you'll need to match the processes privilege level or better)

strace -p <process_id>

Here are some more resources on it:
http://www.redhat.com/magazine/010aug05/features/strace/
http://linux.die.net/man/1/strace

ok and here is an strace of ls in an empty directory so you can clearly see that simple isnt really that simple! (PS I seem to have a problem with my locales is polluting the result)

~/tmp> strace ls ./

execve("/bin/ls", ["ls", "./"], [/* 42 vars */]) = 0
brk(0)                                  = 0x61a000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7da0f51000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7da0f4f000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/home/ashley/obj-mail-release-NDEBUG/dist/bin/tls/x86_64/librt.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/home/ashley/obj-mail-release-NDEBUG/dist/bin/tls/x86_64", 0x7fff25ef42d0) = -1 ENOENT (No such file or directory)
open("/home/ashley/obj-mail-release-NDEBUG/dist/bin/tls/librt.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/home/ashley/obj-mail-release-NDEBUG/dist/bin/tls", 0x7fff25ef42d0) = -1 ENOENT (No such file or directory)
open("/home/ashley/obj-mail-release-NDEBUG/dist/bin/x86_64/librt.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/home/ashley/obj-mail-release-NDEBUG/dist/bin/x86_64", 0x7fff25ef42d0) = -1 ENOENT (No such file or directory)
open("/home/ashley/obj-mail-release-NDEBUG/dist/bin/librt.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/home/ashley/obj-mail-release-NDEBUG/dist/bin", 0x7fff25ef42d0) = -1 ENOENT (No such file or directory)
open("/home/ashley/obj-mail-debug/dist/lib/tls/x86_64/librt.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/home/ashley/obj-mail-debug/dist/lib/tls/x86_64", 0x7fff25ef42d0) = -1 ENOENT (No such file or directory)
open("/home/ashley/obj-mail-debug/dist/lib/tls/librt.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/home/ashley/obj-mail-debug/dist/lib/tls", 0x7fff25ef42d0) = -1 ENOENT (No such file or directory)
open("/home/ashley/obj-mail-debug/dist/lib/x86_64/librt.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/home/ashley/obj-mail-debug/dist/lib/x86_64", 0x7fff25ef42d0) = -1 ENOENT (No such file or directory)
open("/home/ashley/obj-mail-debug/dist/lib/librt.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/home/ashley/obj-mail-debug/dist/lib", 0x7fff25ef42d0) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=102124, ...}) = 0
mmap(NULL, 102124, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7da0f36000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/librt.so.1", O_RDONLY)       = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240#\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=35784, ...}) = 0
mmap(NULL, 2132968, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7da0b2c000
mprotect(0x7f7da0b34000, 2093056, PROT_NONE) = 0
mmap(0x7f7da0d33000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7f7da0d33000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/libselinux.so.1", O_RDONLY)  = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240Q\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=109368, ...}) = 0
mmap(NULL, 2209176, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7da0910000
mprotect(0x7f7da0929000, 2097152, PROT_NONE) = 0
mmap(0x7f7da0b29000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19000) = 0x7f7da0b29000
mmap(0x7f7da0b2b000, 1432, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7da0b2b000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/libacl.so.1", O_RDONLY)      = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\33\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=27600, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7da0f35000
mmap(NULL, 2122744, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7da0709000
mprotect(0x7f7da070f000, 2097152, PROT_NONE) = 0
mmap(0x7f7da090f000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x7f7da090f000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/libc.so.6", O_RDONLY)        = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\342"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1436976, ...}) = 0
mmap(NULL, 3543672, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7da03a7000
mprotect(0x7f7da04ff000, 2097152, PROT_NONE) = 0
mmap(0x7f7da06ff000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x158000) = 0x7f7da06ff000
mmap(0x7f7da0704000, 17016, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7da0704000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/libpthread.so.0", O_RDONLY)  = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260W\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=130224, ...}) = 0
mmap(NULL, 2208624, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7da018b000
mprotect(0x7f7da01a1000, 2097152, PROT_NONE) = 0
mmap(0x7f7da03a1000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7f7da03a1000
mmap(0x7f7da03a3000, 13168, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7da03a3000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/libdl.so.2", O_RDONLY)       = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \16\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14624, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7da0f34000
mmap(NULL, 2109728, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7d9ff87000
mprotect(0x7f7d9ff89000, 2097152, PROT_NONE) = 0
mmap(0x7f7da0189000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f7da0189000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/libattr.so.1", O_RDONLY)     = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000\21\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=16128, ...}) = 0
mmap(NULL, 2111240, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7d9fd83000
mprotect(0x7f7d9fd87000, 2093056, PROT_NONE) = 0
mmap(0x7f7d9ff86000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f7d9ff86000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7da0f33000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7da0f32000
arch_prctl(ARCH_SET_FS, 0x7f7da0f32780) = 0
mprotect(0x7f7da06ff000, 12288, PROT_READ) = 0
munmap(0x7f7da0f36000, 102124)          = 0
set_tid_address(0x7f7da0f32810)         = 15691
set_robust_list(0x7f7da0f32820, 0x18)   = 0
futex(0x7fff25ef4e0c, 0x81 /* FUTEX_??? */, 1) = 0
rt_sigaction(SIGRTMIN, {0x7f7da01902d0, [], SA_RESTORER|SA_SIGINFO, 0x7f7da01997d0}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x7f7da0190350, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7f7da01997d0}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
brk(0)                                  = 0x61a000
brk(0x63b000)                           = 0x63b000
open("/etc/selinux/config", O_RDONLY)   = -1 ENOENT (No such file or directory)
statfs("/selinux", 0x7fff25ef3d30)      = -1 ENOENT (No such file or directory)
open("/proc/mounts", O_RDONLY)          = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7da0f4e000
read(3, "rootfs / rootfs rw 0 0\nnone /sys"..., 1024) = 1024
read(3, "server1/public /home/ashley/file"..., 1024) = 537
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x7f7da0f4e000, 4096)            = 0
open("/usr/lib/locale/locale-archive", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/locale.alias", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2586, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7da0f4e000
read(3, "# Locale name alias data base.\n#"..., 4096) = 2586
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x7f7da0f4e000, 4096)            = 0
open("/usr/lib/locale/en_US.UTF-8/LC_IDENTIFICATION", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_IDENTIFICATION", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=373, ...}) = 0
mmap(NULL, 373, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7da0f4e000
close(3)                                = 0
open("/usr/lib/gconv/gconv-modules.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=25700, ...}) = 0
mmap(NULL, 25700, PROT_READ, MAP_SHARED, 3, 0) = 0x7f7da0f47000
close(3)                                = 0
futex(0x7f7da0703f40, 0x81 /* FUTEX_??? */, 2147483647) = 0
open("/usr/lib/locale/en_US.UTF-8/LC_MEASUREMENT", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_MEASUREMENT", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=23, ...}) = 0
mmap(NULL, 23, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7da0f46000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_TELEPHONE", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_TELEPHONE", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=59, ...}) = 0
mmap(NULL, 59, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7da0f45000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_ADDRESS", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_ADDRESS", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=155, ...}) = 0
mmap(NULL, 155, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7da0f44000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_NAME", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_NAME", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=77, ...}) = 0
mmap(NULL, 77, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7da0f43000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_PAPER", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_PAPER", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=34, ...}) = 0
mmap(NULL, 34, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7da0f42000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_MESSAGES", O_RDONLY) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
close(3)                                = 0
open("/usr/lib/locale/en_US.utf8/LC_MESSAGES/SYS_LC_MESSAGES", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=52, ...}) = 0
mmap(NULL, 52, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7da0f41000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_MONETARY", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_MONETARY", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=286, ...}) = 0
mmap(NULL, 286, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7da0f40000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_COLLATE", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_COLLATE", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=921214, ...}) = 0
mmap(NULL, 921214, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7da0e51000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_TIME", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_TIME", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2451, ...}) = 0
mmap(NULL, 2451, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7da0f3f000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_NUMERIC", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_NUMERIC", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=54, ...}) = 0
mmap(NULL, 54, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7da0f3e000
close(3)                                = 0
open("/usr/lib/locale/en_US.UTF-8/LC_CTYPE", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/en_US.utf8/LC_CTYPE", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=254076, ...}) = 0
mmap(NULL, 254076, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7da0e12000
close(3)                                = 0
ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, TIOCGWINSZ, {ws_row=46, ws_col=156, ws_xpixel=0, ws_ypixel=0}) = 0
stat("./", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("./", O_RDONLY|O_NONBLOCK|O_DIRECTORY|0x80000) = 3
fstat(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl(3, F_GETFD)                       = 0x1 (flags FD_CLOEXEC)
getdents(3, /* 2 entries */, 4096)      = 48
getdents(3, /* 0 entries */, 4096)      = 0
close(3)                                = 0
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
Process 15691 detached

ldd - tracking used libraries/versions

To figure out what libraries(and its location/version as well) an executable is using hit it with "ldd". When using it know in mind that it needs the exact path to the executable.

For example to track down what "ls" is using try this

~> ldd /bin/ls
       linux-vdso.so.1 =>  (0x00007fffdfad4000)
       librt.so.1 => /lib/librt.so.1 (0x00007f33ad7a1000)
       libselinux.so.1 => /lib/libselinux.so.1 (0x00007f33ad585000)
       libacl.so.1 => /lib/libacl.so.1 (0x00007f33ad37e000)
       libc.so.6 => /lib/libc.so.6 (0x00007f33ad01c000)
       libpthread.so.0 => /lib/libpthread.so.0 (0x00007f33ace00000)
       /lib64/ld-linux-x86-64.so.2 (0x00007f33ad9aa000)
       libdl.so.2 => /lib/libdl.so.2 (0x00007f33acbfc000)
       libattr.so.1 => /lib/libattr.so.1 (0x00007f33ac9f8000)

The causes of transient errors

normal programs
* failure to initialize variables.
* Using a piece of memory after it is free
* Overflowing buffers or memory areas that are allocated for the variable. (without causing a GP fault)
* poor pointer arithmetic or initialization leaving the pointer in nomans land
In mutli-threaded:
* improper protection of non thread safe pieces of code.
* in multi-threaded: thread or pipe disruption.
FileIO and networking
* incorrect fileIO fault handling to use of incorrectly initialized file handles.
* incorrect handling of overflows and underflows of the stream

nothing else comes to mind at the moment will update later

Finding all orphaned records where there are multiple parent tables and columns

Finding all orphaned records where there are multiple parent tables and columns;

select id from addresses where id not in (select * from ((select address_id as 'id' from orders) union (select shipping_address_id as 'id' from orders) union (select address_id as 'id' from customers) union (select shipping_address_id as 'id' from customers) order by id) as ui);

mysql stray query find and kill

show processlist;
show full processlist;
KILL QUERY <id>;

Testing sorts

Basically in my last 2 posts I have been using a macro called SCAMBLE to test my sorts

#define SCRAMBLE(x, y) ((0xa57b & ~y) + ((0x3829 & x) << 1))

The average sort algorithm starts to have issues when there is repetition in the data. This SCAMBLE macro is a broken Psedo-Random number generator core. In short since its broken it gets stuck into a loop of numbers which makes it useless as a random number generator but is exactly the input I want to test the sorts.