Crate mem_dbg

Expand description

§mem_dbg

license

Traits and associated procedural macros to inspect recursively the memory usage and layout of a value.

The trait MemSize can be used to compute the overall memory usage of a value in bytes; the standard library function std::mem::size_of returns the stack size of a type in bytes, but it does not take into consideration heap memory. We provide implementations for most basic types, a derive macro for structs and enums whose fields implement MemSize, and support for a few other crates via optional features.

The trait MemDbg, which depends on MemSize, can be used to display the recursive layout of a value, together with the size of each part and the associated padding bytes.

§Why `MemSize`

Other traits partially provide the functionality of MemSize, but either they require implementing manually a trait, which is prone to error, or they do not provide the flexibility necessary for MemDbg. Most importantly, MemSize uses the type system to avoid iterating over the content of a container (a vector, etc.) when it is not necessary, making it possible to compute instantly the size of values occupying hundreds of gigabytes of heap memory.

This is the result of the benchmark btree_set_comp contained in the examples directory. It builds a B-tree set with a hundred million usize entries and then measures its heap size:

Allocated:    3428571500
get_size:     1600000024 349387500 ns
deep_size_of: 1800000024 284149583 ns
mem_size:     3416666554 41 ns

The first line is the number of bytes allocated by the program as returned by cap. Then, we display the result of get-size, deepsize, and our own MemSize. Note that the first two crates are just measuring the space used by the items, and not by the data structure. Moreover, all other crates are about seven orders of magnitude slower than our implementation, due to the necessity to iterate over all elements.

In general, while the size estimation of BTreeSet, BTreeMap, HashSet, and HashMap is heuristic in all libraries, mem_dbg is significantly more precise.

The following table compares the MemSize trait from this crate against the crates deepsize and get-size. The true memory usage (0% error) is again calculated using the allocator from the cap crate.

Type	Container	Crate	Error (%)	Type	Container	Crate	Error (%)
usize	BTreeMap	`deep_size_of`	36.51 ± 20.88	String	BTreeMap	`deep_size_of`	21.53 ± 8.88
		`get_size`	42.01 ± 21.73			`get_size`	16.04 ± 7.74
		`mem_size`	1.25 ± 2.59			`mem_size`	3.17 ± 1.45
	BTreeSet	`deep_size_of`	28.18 ± 12.41		BTreeSet	`deep_size_of`	18.58 ± 8.51
		`get_size`	46.91 ± 23.70			`get_size`	17.54 ± 8.38
		`mem_size`	1.57 ± 2.97			`mem_size`	3.73 ± 2.65
	HashMap	`deep_size_of`	15.64 ± 6.34		HashMap	`deep_size_of`	6.12 ± 3.42
		`get_size`	15.64 ± 6.34			`get_size`	6.12 ± 3.42
		`mem_size`	0.00 ± 0.00			`mem_size`	0.62 ± 1.30
	HashSet	`deep_size_of`	19.57 ± 7.91		HashSet	`deep_size_of`	6.91 ± 3.82
		`get_size`	19.57 ± 7.91			`get_size`	6.91 ± 3.82
		`mem_size`	0.00 ± 0.00			`mem_size`	1.13 ± 2.32

§References

Two flags, SizeFlags::FOLLOW_REFS and DbgFlags::FOLLOW_REFS, make it possible to follow references when computing the size or displaying the layout of a value. Analogously, SizeFlags::FOLLOW_RCS and DbgFlags::FOLLOW_RCS make it possible to follow Rc/Arc smart pointers.

In both cases, references will be accounted for (when computing size) and followed (when displaying the layout), only at their first instance. Following instances will just display an arrow followed by a pointer.

§Padding

The trait MemDbg is useful to display the layout of a value and understand how much memory is used by each part. In particular, it exploits the new stable macro std::mem::offset_of to display the padding of each field in square brackets; moreover, the flag DbgFlags::RUST_LAYOUT makes it possible to display structures in the layout used by the Rust compiler, rather than that given by declaration order.

These features are also available for enums using the feature offset_of_enum, which however needs the nightly compiler, as it enables the unstable feature offset_of_enum.

§Features

std: enables the use of the standard library; this is enabled by default.
derive: enables the derive macros MemSize and MemDbg; this is enabled by default.
offset_of_enum: support for padding and for the DbgFlags::RUST_LAYOUT flag for enums. Requires the nightly compiler as it enables the unstable feature offset_of_enum. Calling mem_dbg with the flag DbgFlags::RUST_LAYOUT without this feature enabled will result in a panic.
half: support for the half crate.
maligned: support for the maligned crate.
mmap-rs: support for the mmap-rs crate.
rand: support for the rand crate.

§Examples

This is an example program using MemSize and MemDbg. Note that we cannot visualize the effect of the useful DbgFlags::COLOR flag, which colorizes sizes depending on their magnitude.



use mem_dbg::*;

#[derive(MemSize, MemDbg)]
struct Struct<A, B> {
    a: A,
    b: B,
    test: isize,
}

#[derive(MemSize, MemDbg)]
struct Data<A> {
    a: A,
    b: Vec<i32>,
    c: (u8, String),
}

#[derive(Clone, Copy, MemSize, MemDbg)]
#[mem_size_flat]
enum TestEnum {
    Unit,
    Unit2(),
    Unit3 {},
    Unnamed(usize, u8),
    Named { first: usize, second: u8 },
}

let b = Vec::with_capacity(100);

let s = Struct {
    a: TestEnum::Unnamed(0, 16),
    b: Data {
        a: vec![0x42_u8; 700],
        b,
        c: (1, "foo".to_owned()),
    },
    test: -0xbadf00d,
};

println!("size:     {}", s.mem_size(SizeFlags::default()));
println!("capacity: {}", s.mem_size(SizeFlags::CAPACITY));
println!();

s.mem_dbg(DbgFlags::empty())?;

println!();

println!("size:     {}", s.mem_size(SizeFlags::default()));
println!("capacity: {}", s.mem_size(SizeFlags::CAPACITY));
println!();

s.mem_dbg(DbgFlags::default() | DbgFlags::CAPACITY | DbgFlags::HUMANIZE)?;

#[cfg(feature = "offset_of_enum")]
{
    println!();

    println!("size:     {}", s.mem_size(SizeFlags::default()));
    println!("capacity: {}", s.mem_size(SizeFlags::CAPACITY));
    println!();

    s.mem_dbg(DbgFlags::empty() | DbgFlags::RUST_LAYOUT)?;
}

The previous program prints:

size:     807
capacity: 1207

807 B ⏺
 16 B ├╴a
      │ ├╴Variant: Unnamed
  8 B │ ├╴0
  1 B │ ╰╴1
783 B ├╴b
724 B │ ├╴a
 24 B │ ├╴b
 35 B │ ╰╴c
  1 B │   ├╴0 [7B]
 27 B │   ╰╴1
  8 B ╰╴test

size:     807
capacity: 1207

1.207 kB 100.00% ⏺: readme::main::Struct<readme::main::TestEnum, readme::main::Data<alloc::vec::Vec<u8>>>
   16  B   1.33% ├╴a: readme::main::TestEnum
                 │ ├╴Variant: Unnamed
    8  B   0.66% │ ├╴0: usize
    1  B   0.08% │ ╰╴1: u8
1.183 kB  98.01% ├╴b: readme::main::Data<alloc::vec::Vec<u8>>
  724  B  59.98% │ ├╴a: alloc::vec::Vec<u8>
  424  B  35.13% │ ├╴b: alloc::vec::Vec<i32>
   35  B   2.90% │ ╰╴c: (u8, alloc::string::String)
    1  B   0.08% │   ├╴0: u8 [7B]
   27  B   2.24% │   ╰╴1: alloc::string::String
    8  B   0.66% ╰╴test: isize

If run with the feature offset_of_enum, it prints:

size:     807
capacity: 1207

807 B ⏺
 16 B ├╴a
      │ ├╴Variant: Unnamed
  8 B │ ├╴0
  1 B │ ╰╴1 [6B]
783 B ├╴b
724 B │ ├╴a
 24 B │ ├╴b
 35 B │ ╰╴c
  1 B │   ├╴0 [7B]
 27 B │   ╰╴1
  8 B ╰╴test

size:     807
capacity: 1207

1.207 kB 100.00% ⏺: readme::main::Struct<readme::main::TestEnum, readme::main::Data<alloc::vec::Vec<u8>>>
   16  B   1.33% ├╴a: readme::main::TestEnum
                 │ ├╴Variant: Unnamed
    8  B   0.66% │ ├╴0: usize
    1  B   0.08% │ ╰╴1: u8 [6B]
1.183 kB  98.01% ├╴b: readme::main::Data<alloc::vec::Vec<u8>>
  724  B  59.98% │ ├╴a: alloc::vec::Vec<u8>
  424  B  35.13% │ ├╴b: alloc::vec::Vec<i32>
   35  B   2.90% │ ╰╴c: (u8, alloc::string::String)
    1  B   0.08% │   ├╴0: u8 [7B]
   27  B   2.24% │   ╰╴1: alloc::string::String
    8  B   0.66% ╰╴test: isize

size:     807
capacity: 1207

807 B ⏺
783 B ├╴b
724 B │ ├╴a
 24 B │ ├╴b
 35 B │ ╰╴c
  1 B │   ├╴0 [7B]
 27 B │   ╰╴1
 16 B ├╴a
      │ ├╴Variant: Unnamed
  1 B │ ├╴1 [6B]
  8 B │ ╰╴0
  8 B ╰╴test

§Caveats

We support out-of-the-box most basic types, and tuples up to size ten. The derive macros MemSize/MemDbg will generate implementations for structs and enums whose fields implement the associated interface: if this is not the case (e.g., because of the orphan rule) one can implement the traits manually.
RefCell contents can be followed only if the RefCell is not mutably borrowed; MemDbg will show a <mutably borrowed> message, but MemSize will just silently return the size of the RefCell itself.
If you invoke the methods of this crate on a shared reference, the compiler will automatically dereference it, and the method will be invoked on the referenced type:

use mem_dbg::*;

let mut x: [i32; 4] = [0, 0, 0, 0];

assert_eq!(
    (&x).mem_size(SizeFlags::default()),
    std::mem::size_of::<[i32; 4]>()
);

assert_eq!(
    (&mut x).mem_size(SizeFlags::default()),
    std::mem::size_of::<&mut [i32; 4]>()
);

assert_eq!(
    <&[i32; 4] as MemSize>::mem_size(&&x, SizeFlags::default()),
    std::mem::size_of::<&[i32; 4]>()
);

Computation of the size of arrays, slices, vectors, or container types, will be performed by iterating over their elements unless the type is flat. See FlatType for more details.
When all fields of a struct or enum implement FlatType<Flat=True>, a compile-time error will suggest adding #[mem_size_flat] (if the type is flat) or #[mem_size_rec] (to explicitly opt out of the optimization and silence the error).
The content of vectors and slices is not expanded recursively as the output might be too complex; this might change in the future (e.g., via a flag) should interesting use cases arise.
Unions are not supported. See the section below for a manually written example.

§Unions

Unions have no discriminant tag, so the library cannot know which field is active. The recommended solution is to create #[repr(transparent)] wrappers, one per variant, each encoding which field is active at the type level, and delegate the implementation of the traits to the active field. For example:

use mem_dbg::*;

union IntOrFloat {
    i: i32,
    f: f32,
}

/// Wrapper that tells the library the `i` field is active.
#[repr(transparent)]
struct IntOrFloatI(IntOrFloat);

/// Wrapper that tells the library the `f` field is active.
#[repr(transparent)]
struct IntOrFloatF(IntOrFloat);

impl FlatType for IntOrFloatI {
    type Flat = True;
}

impl MemSize for IntOrFloatI {
    fn mem_size_rec(
        &self,
        _flags: SizeFlags,
        _refs: &mut HashMap<usize, usize>,
    ) -> usize {
        core::mem::size_of::<Self>()
    }
}

impl MemDbgImpl for IntOrFloatI {
    fn _mem_dbg_rec_on(
        &self,
        writer: &mut impl core::fmt::Write,
        total_size: usize,
        max_depth: usize,
        prefix: &mut String,
        _is_last: bool,
        flags: DbgFlags,
        dbg_refs: &mut HashSet<usize>,
    ) -> core::fmt::Result {
        unsafe { self.0.i }._mem_dbg_depth_on(
            writer,
            total_size,
            max_depth,
            prefix,
            Some("i"),
            true,
            core::mem::size_of::<Self>(),
            flags,
            dbg_refs,
        )
    }
}

impl FlatType for IntOrFloatF {
    type Flat = True;
}

impl MemSize for IntOrFloatF {
    fn mem_size_rec(
        &self,
        _flags: SizeFlags,
        _refs: &mut HashMap<usize, usize>,
    ) -> usize {
        core::mem::size_of::<Self>()
    }
}

impl MemDbgImpl for IntOrFloatF {
    fn _mem_dbg_rec_on(
        &self,
        writer: &mut impl core::fmt::Write,
        total_size: usize,
        max_depth: usize,
        prefix: &mut String,
        _is_last: bool,
        flags: DbgFlags,
        dbg_refs: &mut HashSet<usize>,
    ) -> core::fmt::Result {
        unsafe { self.0.f }._mem_dbg_depth_on(
            writer,
            total_size,
            max_depth,
            prefix,
            Some("f"),
            true,
            core::mem::size_of::<Self>(),
            flags,
            dbg_refs,
        )
    }
}

let w = IntOrFloatI(IntOrFloat { i: 42 });
assert_eq!(w.mem_size(SizeFlags::default()), 4);
w.mem_dbg(DbgFlags::empty())?;

Structs§

DbgFlags: Flags for MemDbg.
False: One of the two possible implementations of Boolean.
HashMap: A hash map implemented with quadratic probing and SIMD lookup.
HashSet: A hash set implemented as a HashMap where the value is ().
SizeFlags: Flags for MemSize.
True: One of the two possible implementations of Boolean.

Enums§

RefDisplay: How to display a reference address in MemDbgImpl::_mem_dbg_depth_on_impl.

Traits§

Boolean: Internal trait used within FlatType to implement MemSize depending on whether a type is flat or not.
FlatType: Marker trait for flat types.
MemDbg: A trait providing methods to display recursively the content and size of a structure.
MemDbgImpl: Inner trait used to implement MemDbg.
MemSize: A trait to compute recursively the overall size or capacity of a structure, as opposed to the stack size returned by core::mem::size_of().

Functions§

backref_color: Returns the color used to print back-reference arrows (subsequent encounters).
color: Returns the color code corresponding to the size.
humanize_float: Given a size in bytes, returns it in a human readable format using SI suffixes.
n_of_digits: Returns the number of digits of a number.
ref_color: Returns the color used to print reference addresses (first encounter).
reset_color: Returns the color code to reset the color.
type_color: Returns the color used to print types.

Derive Macros§

MemDbg: Generate a mem_dbg::MemDbg implementation for custom types.
MemSize: Generate a mem_dbg::MemSize implementation for custom types.

Crate mem_dbg

Crate mem_dbg Copy item path

§mem_dbg

§Why MemSize

§References

§Padding

§Features

§Examples

§Caveats

§Unions

Structs§

Enums§

Traits§

Functions§

Derive Macros§

Crate mem_dbg

§Why `MemSize`