Serializing Variable Data Structures with binary_buffer

Some applications of STXXL will require variable data structures. Currently there is not much support for this in STXXL.

For serializing information into in-memory data blocks, the STXXL provides the helper classes binary_buffer and binary_reader. These provide functions binary_buffer::put<>() to append arbitrary integral data types and binary_reader::get<>() to read these again. Serialization and deserialization of variable data structures are then composed of identical sequences of put()/get().

Additionally, the classes already provide methods to serialize variable length strings (together with their lengths), and thereby also sub-block serialization. These functions are called binary_buffer::put_string() and binary_reader::get_string().

Furthermore, to squeeze small integers into fewer bytes, they classes also contain "varint" encoding, where each byte contains 7 data bits and one continuation bit. These functions are called binary_buffer::put_varint() and binary_reader::get_varint().

The following example fills a binary_buffer with some data elements:

    // construct a binary blob
    stxxl::binary_buffer bb;
    {
        bb.put<unsigned int>(1);
        bb.put_string("test");

        bb.put_varint(42);
        bb.put_varint(12345678);

        // add a sub block
        stxxl::binary_buffer sub;
        sub.put_string("sub block");
        sub.put_varint(6 * 9);

        bb.put_string(sub);
    }

And the following binary_reader example deserializes the data elements and check's their content.

    // read binary block using binary_reader

    stxxl::binary_reader br(bb);

    STXXL_CHECK(br.get<unsigned int>() == 1);
    STXXL_CHECK(br.get_string() == "test");
    STXXL_CHECK(br.get_varint() == 42);
    STXXL_CHECK(br.get_varint() == 12345678);

    {
        stxxl::binary_reader sub_br = br.get_binary_buffer_ref();
        STXXL_CHECK(sub_br.get_string() == "sub block");
        STXXL_CHECK(sub_br.get_varint() == 6 * 9);
        STXXL_CHECK(sub_br.empty());
    }

    STXXL_CHECK(br.empty());