DBGMEM memory debugger.. 2

Summary.. 2

Introduction.. 2

Features. 2

OPTIONS COMMON TO ALL TOOLS. 3

TOOL: SIMPLE TOOL. 3

TOOL: CHECKING TOOL. 4

License. 4

Downloading.. 4

Installation.. 4

How to prepare for debugging.. 5

Running the debugger.. 5

DBGMEM help printout , summary of options. 9

Memory leaks in more detail.. 11

Terminology.. 11

Causes for memory leaks. 11

Strategy for solving leak issues. 12

Example session for finding leaks. 13

Modifications of the source code: 19

Extending the tool.. 21

Appendix.. 22

Similar tools. 22

Internals. 22

Attributions / Authors.. 25

DBGMEM memory debugger

Summary

DBGMEM is a feature rich memory debugger for C and C++ programs; it currently works for Linux only.

The tool helps you to find problems such as

  1. memory leaks
  2. heap memory corruption,
  3. Misuse, illegal argument to selected standard library functions.
  4. Use of freed or un-initialized heap memory.
  5. Extend the tool, check additional API’s arguments.

The tool overrides GLIBC memory allocation functions, memory and string manipulation functions in order to add its features.

The tool does not require any changes to your program, though you might wish to change your program in order to use some advanced features.

Introduction

Debugging memory leaks and memory corruption problems is something of a black art; there is probably no one right way and no one right tool to tackle these problems, most often you will have to try out multiple approaches and multiple tools.

 

The DGBMEM toolset is a drop-in memory allocator for Linux that adds some debugging functionality, it overrides the standard library functions like malloc and free that are returning new heap memory and adds some extra debugging functionality/

In order to use this tool you can use your process as is, and you don't have link your process with any libraries. This works well for ELF executables.

 

DBGMEM is a relatively primitive tool if compared with other tools like valgrind; though DBGMEM may sometimes be better suited for testing systems under live condition and under heavy load.

 

The tools also helped me to solve problem with real world systems, in other words it helped to save my but with a real job - which I think is very much for a tool like this ;-)

 

DBGMEM is a very feature-rich tool (some say that this is a euphemism for bloat, but well, Features Are a Good Thing).

 

Features

DBGMEM offers the following tools.

OPTIONS COMMON TO ALL TOOLS

TOOL: SIMPLE TOOL

   

TOOL: CHECKING TOOL

License

This software is licensed as BSD license

You can use this product with proprietary applications as is; if you choose to modify this software then you are required to contribute your changes back.

Downloading

Sources can be downloaded as "zip archive":https://github.com/MoserMichael/cstuff/zipball/master or downloaded by the git program.

 

            git clone https://github.com/MoserMichael/cstuff.git

 

The project can be build on Linux (checked 32 or 64 bit Intel/Amd); it will not work on Cygwin.

 

Build and install into /usr/local/ directory

           

            cd build

            ./build-dbgmem.sh install

 

Build and install into /usr/alt directory                                                                                                 a

 

            cd build

`           ./build-dbgmem.sh install INSTALL_PREFIX=/usr/alt

How to prepare for debugging

 

  First of all check that your program meets the following requirements.

 

  Now I am out of reasons why this stuff would not work for you, please contact me if you run into problems.

Running the debugger

 

The debugger is invoked via the /users/local/dbgmem/etc/run script (you can change the root installation dir though)

 

   /users/local/dbgmem/etc/run -d check -a gdb -s 7 test2

 

 

 

   /users/local/dbgmem/scripts/run -d simple -b -s 7 test2

 

  

Lets see what happens with simple debugger

 

/user/local/dbgmem/scripts/run -d simple -b -s 7 test2

 

debug engine:       simple

output report:      dbgmemlog.log

frames per entry:   7

command:            test2

 

DBGMEM: running command with memory debugger...

DBGMEM: Please stop debuged process by signal other with SIGKILL (with Ctl+C SIGINT or SIGTERM)

DBGMEM: debugee started with process id 6701

 

Entry: 0x8427300 size: 10 generation: 1

status [memory low mark overwritten ptr=0x8427300 size=10]

        frame 0 : 0x804841b

        frame 1 : 0x8048468

        frame 2 : 0x8048449

        frame 3 : 0x8048468

        frame 4 : 0x8048449

        frame 5 : 0x8048468

        frame 6 : 0x8048449

Entry: 0x84272b8 size: 10 generation: 1

status [memory upper mark overwritten ptr=0x84272b8 size=10]

        frame 0 : 0x80483fa

        frame 1 : 0x8048468

        frame 2 : 0x8048449

        frame 3 : 0x8048468

        frame 4 : 0x8048449

        frame 5 : 0x8048468

        frame 6 : 0x8048449

 

 

DBGMEM: debugee exit code is 0

DBGMEM: Running analyser script, please wait...

DBGMEM: created report files: dbgmemlog.log 

 

 

 

When a memory error occurs then the event is traced to standard error file;

A report is generated after running the debugged process (see file dbglog.log)

  

 

The report consists of the following sections

 

 

======================

MALLOC INFO (MALLINFO)

======================

       non-mmaped space allocated from system: 135168

       number of free chunks: 1

       number of fastbin blocks: 0

       number of mmapped regions: 0

       space in mmapped regions: 0

       maximum total allocated space: 0

       space available in freed fastbin blocks: 0

       total allocated space: 328

       total free space: 134840

       top-most, releasable (via malloc_trim) space: 134840

       DBGMEM arena size: 12

MALLINFO EOF

 

 

 Entry: 0x9228120 size: 10 generation: 1 status [memory low mark overwritten ptr=0x9228120 size=10]     frame: 0 : (0x804841b) ~"test_malloc2 + 107 in section .text\n" ~"Line 44 of \"test2.c

       frame: 1 : (0x8048468) ~"test_malloc + 20 in section .text\n" ~"Line 63 of \"test2.c

       frame: 2 : (0x8048449) ~"test_malloc2 + 153 in section .text\n" ~"Line 56 of \"test2.c

       frame: 3 : (0x8048468) ~"test_malloc + 20 in section .text\n" ~"Line 63 of \"test2.c

       frame: 4 : (0x8048449) ~"test_malloc2 + 153 in section .text\n" ~"Line 56 of \"test2.c

       frame: 5 : (0x8048468) ~"test_malloc + 20 in section .text\n" ~"Line 63 of \"test2.c

       frame: 6 : (0x8048449) ~"test_malloc2 + 153 in section .text\n" ~"Line 56 of \"test2.c

 Entry: 0x9228100 size: 10 generation: 1 status [memory upper mark overwritten ptr=0x9228100 size=10]     frame: 0 : (0x80483fa) ~"test_malloc2 + 74 in section .text\n" ~"Line 36 of \"test2.c

       frame: 1 : (0x8048468) ~"test_malloc + 20 in section .text\n" ~"Line 63 of \"test2.c

       frame: 2 : (0x8048449) ~"test_malloc2 + 153 in section .text\n" ~"Line 56 of \"test2.c

       frame: 3 : (0x8048468) ~"test_malloc + 20 in section .text\n" ~"Line 63 of \"test2.c

       frame: 4 : (0x8048449) ~"test_malloc2 + 153 in section .text\n" ~"Line 56 of \"test2.c

       frame: 5 : (0x8048468) ~"test_malloc + 20 in section .text\n" ~"Line 63 of \"test2.c

       frame: 6 : (0x8048449) ~"test_malloc2 + 153 in section .text\n" ~"Line 56 of \"test2.c

 MEMALLOC: EOF

 

============

LEAK SUMMARY

============

 

===============

GENERATION [0]

===============

 

All-size: 80 |  Entry: 0x92280e8 size: 8  Entry: 0x92280d0 size: 8  Entry: 0x92280b8 size: 8  Entry: 0x92280a0 size: 8  Entry: 0x9228088 size: 8  Entry: 0x9228070 size: 8  Entry: 0x9228058 size: 8  Entry: 0x9228040 size: 8  Entry: 0x9228028 size: 8  Entry: 0x9228010 size: 8

       frame: 0 : (0x80483e8) ~"test_malloc2 + 56 in section .text\n" ~"Line 32 of \"test2.c

       frame: 1 : (0x8048468) ~"test_malloc + 20 in section .text\n" ~"Line 63 of \"test2.c

       frame: 2 : (0x8048449) ~"test_malloc2 + 153 in section .text\n" ~"Line 56 of \"test2.c

       frame: 3 : (0x8048468) ~"test_malloc + 20 in section .text\n" ~"Line 63 of \"test2.c

       frame: 4 : (0x8048449) ~"test_malloc2 + 153 in section .text\n" ~"Line 56 of \"test2.c

       frame: 5 : (0x8048468) ~"test_malloc + 20 in section .text\n" ~"Line 63 of \"test2.c

       frame: 6 : (0x8048449) ~"test_malloc2 + 153 in section .text\n" ~"Line 56 of \"test2.c

 

-----------------------------------

Sum of all user memory blocks for generation [0]:  80

Number of blocks this generation: 1

DBGMEM overhead: 12

 

===============

GENERATION [1]

===============

 

All-size: 8 |  Entry: 0x9228140 size: 8

       frame: 0 : (0x80483e8) ~"test_malloc2 + 56 in section .text\n" ~"Line 32 of \"test2.c

       frame: 1 : (0x8048468) ~"test_malloc + 20 in section .text\n" ~"Line 63 of \"test2.c

       frame: 2 : (0x8048449) ~"test_malloc2 + 153 in section .text\n" ~"Line 56 of \"test2.c

       frame: 3 : (0x8048468) ~"test_malloc + 20 in section .text\n" ~"Line 63 of \"test2.c

       frame: 4 : (0x8048449) ~"test_malloc2 + 153 in section .text\n" ~"Line 56 of \"test2.c

       frame: 5 : (0x8048468) ~"test_malloc + 20 in section .text\n" ~"Line 63 of \"test2.c

       frame: 6 : (0x80484e6) ~"main + 121 in section .text\n" ~"Line 82 of \"test2.c

 

-----------------------------------

Sum of all user memory blocks for generation [1]:  88

Number of blocks this generation: 1

DBGMEM overhead: 12

 

 

 

See section “Memory leaks in more detail” for more information on this subject.

DBGMEM help printout , summary of options

run [-d simple|check] [-f log_file] [-a] [-s NUMBER] [-b] [-e] [-snext <signum>] [-sdump <signum>] <cmd-to-debug>

 

DBGMEM Memory debugger;

 

Starts a process with the DBGMEM memory debugger. Please run your test sequence and then terminate

the process with any signal other than SIGKILL, use SIGTERM or SIGINT instead.

Be friendly to the environment ;-)

 

Options:

 

-d simple|check     choice of debug engine. possible values for this options:

                     simple - mainly good for detecting memory leaks

                     check  - memory leaks detection + check for pointers in              common library functions

                  The default value is simple.

                    

-f outputfile     set output logfile. if not set then the name is dbgmemlog.log

 

-s NUMBER         number of stack frames that are kept per memory allocation.

                  we record the stack where allocation did happen.

                  The default value is five.

                    

-a core|gdb|skip  what to do when memory error is detected. value

                     core  generate core

                     gdb   attach the gnu debugger

                     skip  continue as if nothing happened

                  default value is skip.

 

-b                fill byte is set - uninitialised memory is set to 0xDD,

                 freed memory set to 0xEE

                  slows thing down but helps to find errors due to accessing

                 uninitialised or freed memory.

                  default is off.

 

-p <n>            whenever an allocated block is checked for overwrites, then

                 adjacent <n> memory

                    blocks are checked too.

                    (this is called healthy paranoia mode)

                  default value is none, only one block is checked for validity

 

-e               Install signal handler for SIGSEGV and SIGBUS. 

                  the signal handler checks if heap is corrupted so that all bad

                 memory blocks are printed out.

                  and then dumps core.

                  this option is off by default

                 

-leak <n1> <n2>   install signal handlers that will help with leak detection

                  <n1> if number is not null then installs signal handler to

                  signal number <n1> which

                     checks if heap is corrupted

                     advances generation tag

                  <n2> if number is not null then installs signal handler to

                  signal number <n2> which

                     dumps all memory blocks to a raw log file.

                  this option is off by default

                    

-t <seconds>      run top command to print memory statistics of debugee; the   statistics

                  are gathered every <seconds> seconds. The output file is <outputfile>.top 

                  this option is off by default

           

-y symfile        use argument symbol file in order to resolve addresses of functions in stack trace.

 

-v                verbose mode; the debugee process prints options received to stdandard error.

 

-version          print version of this software

 

-h                print this help message

 

[cmd-to-debug]           command line of process that will be started and debugged.

 

Note: this debugged process will set its core limit to unlimited. 

 

 

Memory leaks in more detail

Memory leaks cause the process to consume more and more memory, leading to general performance degradation; swapping, and eventually to the process termination due to the fact that the process has run out of free memory. DBGMEM can help you to find these issues effectively.

Terminology

Let first look at the different stages in the life of a process.

The lifetime of a process can be divided into following logical stages:

 

[--- INITIALIZATION ---]

[--- ACTIVE STAGE ---]

[--- SHUTDOWN ---]

[--- EXIT ---]

 

Initialization

The process is initializing. Usually here the configuration is read, caches are initialized and objects and resources used throughout further stages are created and initialized.

Active state

The process is servicing request and does something useful;

During this stage the process completes one or more logical units of processing,

Each such unit may be the processing of a request or series of requests from a network, the processing of a batch data job, or the processing of an interactive user requests. Typically during each logical unit of processing some resources and memory is allocated and then either released or leaked.

Shutdown

The process has received a signal that will cause it to exit.

The process is cleaning up and is caches and is freeing resources.

This is a common but optional stage.

Exit

The process exits.

Causes for memory leaks

A leak is a resource that is allocated and not freed during ACTIVE STAGE of a process. If you allocated a block during initialization and forgot to clean it up then this has a fixed cost and will not have any affect (unless you have a shared memory space between processes).

.

There are the following types of leaks

 

Strategy for solving leak issues

The DBGMEM tool creates a detailed memory report that lists all memory blocks present at a particular point in time, this report includes the stack where each block was allocated; all heap objects that were allocated from the same stack location are summed up in one report entry.

 

Two reports taken at different points in time contain interesting information; 

·        On completing the SHUTDOWN sequence; right before the process exits normally This report is always generated by DBGMEM unless the process is killed by SIGKILL signal; you don't need to modify you program in order to get this report.  This report lists the origin of most leak issues; the stack trace where the leaking memory blocks were allocated will give you a strong hint as to how to fix these issues, alas for some types of leaks related to reference counting you need more information.

·        On completing the ACTIVE STAGE right before the SHUTDOWN sequence

 

We want to tag each memory allocation with the stage in the program lifetime when it was made. For instance we want to know if an allocation was done during INITIALIZATION stage, so we will not have to deal with it at all.

 

This stage is useful if you want to track reference counted leaks; usually these leaked references are removed during SHUTDOWN stage, when the root objects that are holding these references are deleted. When a lot of blocks occur in the report taken before SHUTDOWN stage, but are not present in the report taken before EXIT then this can indicate a leak. This analysis is not done by the tool, it is done by the avid reader, when he compares the two reports; the avid reader is also armed with knowledge of the workings of his program and will be able to conclude what is happening.

 

There are two ways to indicate the two state transitions

 

1.      INITIALIZATION and ACTIVE STAGE

2.      ACTIVE STAGE and SHUTDOWN stage

You can either

1.      Enable DBGMEM command line option that sets up two signal handler, where each signal handler will do the requested state transition.

2.      Modify the debugged program so that it calls the debug library to indicate the two events; this is the preferred solution if you want to run DBGMEM from within unit tests/system tests.

Example session for finding leaks

Lets run an example session – you will find test3.cpp in the src directory.

 

Option –v verbose option prints out additional output

This time –leak 31 33 option is given; the memory debugger installs two additional signal handler;

 

One signal handler is installed on signal 31 – this one increases the generation tag value and checks all memory blocks for overwrites.

 

The other on signal handler is installed on signal 33 – this one will dump all memory blocks.

 

~/dbgmem/src> ../scripts/run -v -d simple -leak 31 33 -s 10 ./test3

debug engine:       simple

output report:      dbgmemlog.log

frames per entry:   10

command:            ./test3

 

DBGMEM: running command with memory debugger...

DBGMEM: Please stop debuged process by signal other with SIGKILL (with Ctl+C SIGINT or SIGTERM)

DBGMEM: debugee started with process id 1914

 

DBGMEM Options:

        process id: 1914

        ignore this process: off

        allocator: simple

        stack frames: 10

        action on memory errors: continue

        check adjacent nodes: off

        init new & freed memory: off

        SIGSEGV/SIGBUS error handler: off

        Install signal for leak detection: on

                signal to increment generation tag: 31

                signal to dump heap contents: 33

        log directory: /disks/uilstore6/disk602/mmoser/dbgmem/src

 

-\|/...

Now lets assume that the application is now up and running; we can now signal that the system is in ACTIVE stage; In another console from the same user let’s issue a signal.


 

kill -31 1914

 

The debugger will write the following

 

DBGMEM: check heap memory blocks

DBGMEM: increasing generation value

 

From now on all allocations will be tagged as generation 1 and for our purpose the debugged process is now in ACTIVE stage.

 

Lets assume that you have run a test scenario, and now its time to enter SHUTDOWN stage. Again in the second terminal send the signal.

 

kill -33 1914

 

The debugger will write

 

DBGMEM: logging all heap blocks...

 

Now exit the process by running Ctrl+C in the debugger.

 

DBGMEM: debugee exit code is 0

DBGMEM: Running analyser script, please wait...

DBGMEM: created report files: dbgmemlog.log dbgmemlog.log_snapshot1

 

 

Now we have the report files;

 

dbgmem.log_snapshot1 – all memory blocks taken when process enters SHUTDOWN stage

dbgmem.log -  when process is about to terminate.

 

Lets look at the following  dbgmem.log_snapshot1 log file; all leaks are printed, loud and clear; sorted by the number of bytes leaked etc. etc. etc.

 

======================

MALLOC INFO (MALLINFO)

======================

       non-mmaped space allocated from system: 135168

       number of free chunks: 1

       number of fastbin blocks: 0

       number of mmapped regions: 0

       space in mmapped regions: 0

       maximum total allocated space: 0

       space available in freed fastbin blocks: 0

       total allocated space: 6024

       total free space: 129144

       top-most, releasable (via malloc_trim) space: 129144

       DBGMEM arena size: 64

MALLINFO EOF

 

 

 MEMALLOC: EOF

 

============

LEAK SUMMARY

============

 

===============

GENERATION [0]

===============

 

All-size: 1624 |  Entry: 0x903d540 size: 1304  Entry: 0x903d0f8 size: 320

       frame: 0 : (0x19c86e) 0x19c86e - (rva: 571502) /usr/lib/libstdc++.so.5.0.3

       frame: 1 : (0x1890f1) 0x1890f1 - (rva: 491761) /usr/lib/libstdc++.so.5.0.3

       frame: 2 : (0x188ffd) 0x188ffd - (rva: 491517) /usr/lib/libstdc++.so.5.0.3

       frame: 3 : (0x188b6c) 0x188b6c - (rva: 490348) /usr/lib/libstdc++.so.5.0.3

       frame: 4 : (0x80493fa) ~"std::__simple_alloc<char*, std::__default_alloc_template<true, 0> >::allocate(unsigned int) + 34 in section .text\n" ~"Line 238 of \"/usr/include/c++/3.2.3/bits/stl_alloc.h

       frame: 5 : (0x80492e1) ~"std::_Vector_alloc_base<char*, std::allocator<char*>, true>::_M_allocate(unsigned int) + 17 in section .text\n" ~"Line 121 of \"/usr/include/c++/3.2.3/bits/stl_vector.h

       frame: 6 : (0x8048f9e) ~"std::vector<char*, std::allocator<char*> >::_M_insert_aux(__gnu_cxx::__normal_iterator<char**, std::vector<char*, std::allocator<char*> > >, char* const&) + 238 in section .text\n" ~"Line 900 of \"/usr/include/c++/3.2.3/bits/stl_vector.h

       frame: 7 : (0x8048d45) ~"std::vector<char*, std::allocator<char*> >::push_back(char* const&) + 81 in section .text\n" ~"Line 498 of \"/usr/include/c++/3.2.3/bits/stl_vector.h

       frame: 8 : (0x8048c04) ~"foo_bar::foo_bar() + 138 in section .text\n" ~"Line 20 of \"test3.cpp

       frame: 9 : (0x8048b2c) ~"__static_initialization_and_destruction_0(int, int) + 34 in section .text\n" ~"Line 34 of \"test3.cpp

All-size: 495 |  Entry: 0x903d490 size: 99  Entry: 0x903d3e0 size: 99  Entry: 0x903d330 size: 99  Entry: 0x903d280 size: 99  Entry: 0x903d048 size: 99

       frame: 0 : (0x19c86e) 0x19c86e - (rva: 571502) /usr/lib/libstdc++.so.5.0.3

       frame: 1 : (0x19c9bf) 0x19c9bf - (rva: 571839) /usr/lib/libstdc++.so.5.0.3

       frame: 2 : (0x8048bef) ~"foo_bar::foo_bar() + 117 in section .text\n" ~"Line 19 of \"test3.cpp

       frame: 3 : (0x8048b2c) ~"__static_initialization_and_destruction_0(int, int) + 34 in section .text\n" ~"Line 34 of \"test3.cpp

       frame: 4 : (0x8048b75) ~"global constructors keyed to global + 21 in section .text\n" ~"Line 32 of \"test3.cpp

       frame: 5 : (0x804c8b1) ~"__do_global_ctors_aux + 25 in section .text\n"

       frame: 6 : (0x80487b5) ~"_init + 21 in section .init\n"

       frame: 7 : (0x804c826) ~"__libc_csu_init + 26 in section .text\n"

       frame: 8 : (0x21f74b) 0x21f74b - (rva: 87883) /lib/tls/libc-2.3.2.so

       frame: 9 : (0x8048969) ~"_start + 33 in section .text\n"

All-size: 420 |  Entry: 0x903e3d8 size: 140  Entry: 0x903e0b8 size: 140  Entry: 0x903dd98 size: 140

       frame: 0 : (0x8048a31) ~"leak_it(int, unsigned int) + 61 in section .text\n" ~"Line 46 of \"test3.cpp

       frame: 1 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 2 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 3 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 4 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 5 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 6 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 7 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 8 : (0x8048aa8) ~"main + 70 in section .text\n" ~"Line 68 of \"test3.cpp

       frame: 9 : (0x21f79a) 0x21f79a - (rva: 87962) /lib/tls/libc-2.3.2.so

All-size: 390 |  Entry: 0x903e308 size: 130  Entry: 0x903dfe8 size: 130  Entry: 0x903dcc8 size: 130

       frame: 0 : (0x8048a31) ~"leak_it(int, unsigned int) + 61 in section .text\n" ~"Line 46 of \"test3.cpp

       frame: 1 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 2 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 3 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 4 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 5 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 6 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 7 : (0x8048aa8) ~"main + 70 in section .text\n" ~"Line 68 of \"test3.cpp

       frame: 8 : (0x21f79a) 0x21f79a - (rva: 87962) /lib/tls/libc-2.3.2.so

       frame: 9 : (0x8048969) ~"_start + 33 in section .text\n"

All-size: 360 |  Entry: 0x903e248 size: 120  Entry: 0x903df28 size: 120  Entry: 0x903dc08 size: 120

       frame: 0 : (0x8048a31) ~"leak_it(int, unsigned int) + 61 in section .text\n" ~"Line 46 of \"test3.cpp

       frame: 1 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 2 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 3 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 4 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 5 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 6 : (0x8048aa8) ~"main + 70 in section .text\n" ~"Line 68 of \"test3.cpp

       frame: 7 : (0x21f79a) 0x21f79a - (rva: 87962) /lib/tls/libc-2.3.2.so

       frame: 8 : (0x8048969) ~"_start + 33 in section .text\n"

All-size: 330 |  Entry: 0x903e190 size: 110  Entry: 0x903de70 size: 110  Entry: 0x903db50 size: 110

       frame: 0 : (0x8048a31) ~"leak_it(int, unsigned int) + 61 in section .text\n" ~"Line 46 of \"test3.cpp

       frame: 1 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 2 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 3 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 4 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 5 : (0x8048aa8) ~"main + 70 in section .text\n" ~"Line 68 of \"test3.cpp

       frame: 6 : (0x21f79a) 0x21f79a - (rva: 87962) /lib/tls/libc-2.3.2.so

       frame: 7 : (0x8048969) ~"_start + 33 in section .text\n"

All-size: 100 |  Entry: 0x903daa0 size: 100

       frame: 0 : (0x8048a31) ~"leak_it(int, unsigned int) + 61 in section .text\n" ~"Line 46 of \"test3.cpp

       frame: 1 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 2 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 3 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 4 : (0x8048aa8) ~"main + 70 in section .text\n" ~"Line 68 of \"test3.cpp

       frame: 5 : (0x21f79a) 0x21f79a - (rva: 87962) /lib/tls/libc-2.3.2.so

       frame: 6 : (0x8048969) ~"_start + 33 in section .text\n"

 

-----------------------------------

Sum of all user memory blocks for generation [0]:  3719

Number of blocks this generation: 7

DBGMEM overhead: 448

 

===============

GENERATION [1]

===============

 

All-size: 140 |  Entry: 0x903e6f8 size: 140

frame: 0 : (0x8048a31) ~"leak_it(int, unsigned int) + 61 in section .text\n" ~"Line 46 of \"test3.cpp

       frame: 1 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 2 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 3 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 4 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 5 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 6 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 7 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 8 : (0x8048aa8) ~"main + 70 in section .text\n" ~"Line 68 of \"test3.cpp

       frame: 9 : (0x21f79a) 0x21f79a - (rva: 87962) /lib/tls/libc-2.3.2.so

All-size: 130 |  Entry: 0x903e628 size: 130

       frame: 0 : (0x8048a31) ~"leak_it(int, unsigned int) + 61 in section .text\n" ~"Line 46 of \"test3.cpp

       frame: 1 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 2 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 3 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 4 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 5 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 6 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 7 : (0x8048aa8) ~"main + 70 in section .text\n" ~"Line 68 of \"test3.cpp

       frame: 8 : (0x21f79a) 0x21f79a - (rva: 87962) /lib/tls/libc-2.3.2.so

       frame: 9 : (0x8048969) ~"_start + 33 in section .text\n"

All-size: 120 |  Entry: 0x903e568 size: 120

       frame: 0 : (0x8048a31) ~"leak_it(int, unsigned int) + 61 in section .text\n" ~"Line 46 of \"test3.cpp

       frame: 1 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 2 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 3 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 4 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 5 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 6 : (0x8048aa8) ~"main + 70 in section .text\n" ~"Line 68 of \"test3.cpp

       frame: 7 : (0x21f79a) 0x21f79a - (rva: 87962) /lib/tls/libc-2.3.2.so

       frame: 8 : (0x8048969) ~"_start + 33 in section .text\n"

All-size: 110 |  Entry: 0x903e4b0 size: 110

       frame: 0 : (0x8048a31) ~"leak_it(int, unsigned int) + 61 in section .text\n" ~"Line 46 of \"test3.cpp

       frame: 1 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 2 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 3 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 4 : (0x8048a11) ~"leak_it(int, unsigned int) + 29 in section .text\n" ~"Line 44 of \"test3.cpp

       frame: 5 : (0x8048aa8) ~"main + 70 in section .text\n" ~"Line 68 of \"test3.cpp

       frame: 6 : (0x21f79a) 0x21f79a - (rva: 87962) /lib/tls/libc-2.3.2.so

       frame: 7 : (0x8048969) ~"_start + 33 in section .text\n"

 

-----------------------------------

Sum of all user memory blocks for generation [1]:  4219

Number of blocks this generation: 4

DBGMEM overhead: 256

 

 

 

Modifications of the source code:

Alternatively you can change the source code, to mark these events; this way you can use the stuff in automatic testing scripts.

 

1) Mark end of INITIALISATION stage and start of ACTIVE stage. Mark end of ACTIVE stage and start of SHUTDOWN stage.

 

    if ( getenv("DBGMEM_ENGINE") != 0) {

      mallopt(1002,0);

    }

   

 

  

2) Dump all objects that are currently allocated (should do so before process termination).

 

    if ( getenv("DBGMEM_ENGINE") != 0) {

      mallopt(1005,0);

    }

 

This will write the files

 

DBGMEM_report.log.bak  - containing all memory blocks with stack trace

DBGMEM_report_mmap.log.bak - containing all mmaped regions with stack trace

 

  1. You can add this functionality in a signal handler for example.

 

3) Force check of all memory blocks for overwrites

.

    if ( getenv("DBGMEM_ENGINE") != 0) {

      mallopt(1003,0);

    }

 

Extending the tool

There can be scenarios where the debugged process wants to access functionality of memory debugger.

 

In order to make full use of all functionality please consider to link with shared library

/usr/local/dbgmem/lib/libdbgmemutil.so

The functions are declared in  /usr/local/dbgmem/inc/dbgmemutil.h

 

See this header file for detailed function documentation.

Appendix

Similar tools

Other tools that replace/augment LIBC memory allocator with debugging functionality:

 

Tools that use LD_PRELOAD to augment / replace LIBC functionality

 

Tools that come as library that has to be linked too

.

Internals

  The tool is a drop in replacement for part of libc/runtime library functionality; you don't have to link your application against a debug library, instead this tool is implemented as a shared library that is loaded before loading the shared library of the runtime library by means of manipulating the LD_PRELOAD environment variable.

 

The tool consists of the following components

 

Each tool is initialized by _init method of shared library.

 This arena header can be overwritten, although things are not quite as bad, since we can double check the linked list as it is traversed; and we are not advancing in it if an entry has gone wrong; bigger problem is that the stack trace can get overwritten.

Attributions / Authors

The inspiration for this tool comes from the XKCD comics (besides the wish to keep my job;

 

 

The author of this package is Michael Moser

 

Special thanks to Dr. Robert Iakobashvili for comments and testing. 

 

Good luck with using this tool; I hope it will help you