Libraraies about cpp object serilization in distrirbuted communication.

Discuss one exmaple library about cpp serilization and unserilization

Unlike the golang or java that provide the official serilization and decirilization libraray, we may need to use the third part library for the object serilization and decirilization. One example library that we want to introduce is this library: cereal (https://github.com/USCiLab/cereal). There are detailed documents about how to use this library. We just put an small example here to show some ideas about this library.

Cereal

Just like many convenient cpp tool libraray, we only need to include the header file in order to use the cereal.

this is an exmaple code

#include <cereal/archives/binary.hpp>
#include <cereal/archives/xml.hpp>
#include <cereal/types/vector.hpp>
#include <iostream>
#include <sstream>
#include <vector>

class Object {
 public:
  Object(){};
  Object(unsigned width, unsigned height, unsigned depth, double z_offset,
         float range = 1.2, unsigned nblocks = 1)
      : m_width(width),
        m_height(height),
        m_depth(depth + 1),
        m_extents(6),
        m_origin(3),
        m_data(width * height * (depth + 1)),
        m_z_offset(z_offset),
        m_range(range),
        m_nblocks(nblocks) {
    // initilization
  }

  Object(const Object&) = default;
  Object(Object&&) = default;
  Object& operator=(const Object&) = default;
  Object& operator=(Object&&) = default;
  ~Object() = default;

  bool compare(Object& other) {
    if (other.m_width != m_width || other.m_height != m_height ||
        other.m_depth != m_depth) {
      std::cout << "unequal part1" << std::endl;
      return false;
    }
    if (other.m_z_offset != m_z_offset || other.m_range != m_range ||
        other.m_nblocks != m_nblocks) {
      std::cout << "unequal part2" << std::endl;
      return false;
    }
    if (other.m_extents.size() != m_extents.size() ||
        other.m_origin.size() != m_origin.size() ||
        other.m_data.size() != m_data.size()) {
      std::cout << "unequal part3" << std::endl;
      return false;
    }
    for (int i = 0; i < m_extents.size(); i++) {
      if (other.m_extents[i] != m_extents[i]) {
        std::cout << "unequal m_extents" << std::endl;
        return false;
      }
    }
    for (int i = 0; i < m_origin.size(); i++) {
      if (other.m_origin[i] != m_origin[i]) {
        std::cout << "unequal m_origin" << std::endl;
        return false;
      }
    }
    for (int i = 0; i < m_data.size(); i++) {
      if (other.m_data[i] != m_data[i]) {
        std::cout << "unequal m_data" << std::endl;
        return false;
      }
    }
    return true;
  }

  void caculateSize() {
    size_t stackVar;
    stackVar = sizeof(m_width) + sizeof(m_height) + sizeof(m_depth) +
               sizeof(m_extents) + sizeof(m_origin) + sizeof(m_data) +
               sizeof(m_z_offset) + sizeof(m_range) + sizeof(m_nblocks);
    std::cout << "major ariables";
    std::cout << " m_width " << sizeof(m_width) << ",";
    std::cout << " m_extents " << sizeof(m_extents) << ",";
    std::cout << " m_origin " << sizeof(m_origin) << ",";
    std::cout << std::endl;

    std::cout << " stackVar " << stackVar << std::endl;

    size_t heapVar = sizeof(int) * m_extents.size() +
                     sizeof(double) * m_origin.size() +
                     sizeof(int) * m_data.size();

    std::cout << " heapVar " << heapVar << std::endl;
  }

  // binary output based on cereal
  // https://uscilab.github.io/cereal/serialization_archives.html
  // the function should be public one!
  template <typename A>
  void serialize(A& ar) {
    ar& m_width;
    ar& m_height;
    ar& m_depth;
    ar& m_extents;
    ar& m_origin;
    ar& m_data;
    ar& m_z_offset;
    ar& m_range;
    ar& m_nblocks;
  }

 private:
  size_t m_width;
  size_t m_height;
  size_t m_depth;
  std::vector<int> m_extents;
  std::vector<double> m_origin;
  std::vector<int> m_data;
  unsigned m_z_offset;
  float m_range;
  unsigned m_nblocks;
};

int main() {
  static unsigned LEN = 10;
  Object ob(LEN, LEN, LEN, 1, 1.2, 5);

  std::stringstream ss;  // any stream can be used

  cereal::BinaryOutputArchive oarchive(ss);  // Create an output archive

  oarchive(ob);  // Write the data to the archive

  // oarchive.saveBinaryValue(mb, sizeof(mb), "mb");

  std::string bufferedstr = ss.str();

  std::cout << "serilized size " << bufferedstr.size() << " original size "
            << sizeof(ob);

  // deserialize
  std::stringstream is(bufferedstr);

  cereal::BinaryInputArchive archive_in(is);
  Object mbnew;
  archive_in(mbnew);

  bool ifequal = ob.compare(mbnew);
  std::cout << " ifequal " << ifequal << std::endl;

  mbnew.caculateSize();

  return 0;
}

Several tips of using cereals

you may need to use stringstream to transfer the marshaled data into a string

you also need to add an public template function in the defination of the object in order to use the cereal

When we check the size of the object

serilized size 4508 original size 112 ifequal 1
major ariables m_width 8, m_extents 24, m_origin 24,
 stackVar 108
 heapVar 4448

You could see that the serilized size contains both the stack variable and the heap variable. The sizeof only caculate the stack varaible, that is why the size of the object is 112

DIY

The diy library is maintained by kitware, that is desinged for executing the visualization algorithm in an distributed manner. Talking about the distributed things, we need to send data object back and forth and also includes different types of data communication patterns.

The memory buffer of the diy is easy to undersdand, just a simple vector<char> which is supposed to store the serialized data. Assuming the buffer is a file, It use the position to store the current location of current offset of the memory buffer.

This is an exmaple of data serilization and deseriliazation, remember to reset the position before deserialize the data set.

The difficult thing is all kinds of communication pattern. Instead of using the MPI direactly, it provides all kinds of high level concepts to facilitate the communition in the context of the scientific visualization.

To use DIY, you have to create a Masterobject, set up block neighbors, enqueue objects to send, exchange the data, and then dequeue the objects.

enqueue and dequeue can be viewed as process to put the data into the send queue and recieve queue for memory managed by the

Here is a simple example in the serialization testing code: https://gitlab.kitware.com/vtk/vtk-m/-/blob/master/vtkm/cont/testing/TestingSerialization.h#L189.

References

https://uscilab.github.io/cereal/quickstart.html

https://github.com/wangzhezhe/5MCST/tree/master/unclassified/ceralexample