Many faces for data format/representation for sci-vis

Some tips regarding the common data format used for visualization.

VTK data

VTK-m data

ADIOS

setting python binding’s path

export PYTHONPATH=/path/to/site-packages:${PYTHONPATH} 

remember to update this env before using python bindings for executing adios

bpls checking all variables

This example is output to check variable “VX” by time step, it print the element from position 0 to position 10 for each time step for this variable -d parameter is used to specify the variable.

$ ~/cworkspace/build_adios/bin/bpls /lustre/orion/csc143/proj-shared/gongq/frontier/dataset/bent_pipe.bp/ -d "VX" -s "0" -c "10" -t
Step 0:
  double   VX            {235008000}
    slice (0:9)
    (0)    0 0 0 0 0 0
    (6)    0 0 0 0 
Step 1:
  double   VX    {235008000}
    slice (0:9)
    (0)    -1.00187 -1.00187 -1.00187 -1.00187 -1.00187 -1.00187
    (6)    -1.00187 -1.00187 -1.00187 -1.00187 
Step 2:
  double   VX    {235008000}
    slice (0:9)
    (0)    -1.00281 -1.00281 -1.00281 -1.00281 -1.00281 -1.00281
    (6)    -1.00281 -1.00281 -1.00281 -1.00281 

this is results for checking all variables’s min and max and all informations including the attribute and variable. For this case, there are 6 attributes and 6 variables. When read/write the file, the attributes and variables are defined in different ways.

zw241@login03:~/cworkspace/install_adios/bin$ ./bpls /lustre/orion/csc143/proj-shared/gongq/frontier/dataset/bent_pipe.bp/ -lav
File info:
  of variables:  6
  of attributes: 6
  statistics:    Min / Max 
  string   Fides_Cell_Type              attr   = "hexahedron"
  string   Fides_Connectivity_Variable  attr   = "connectivity"
  string   Fides_Coordinates_Variable   attr   = "points"
  string   Fides_Data_Model             attr   = "unstructured_single"
  string   Fides_Variable_Associations  attr   = {"points", "points", "points"}
  string   Fides_Variable_List          attr   = {"VX", "VY", "VZ"}
  double   VX                           3*{235008000} = -1.36295 / 0.720586
  double   VY                           3*{235008000} = -0.653091 / 0.733072
  double   VZ                           3*{235008000} = -0.103327 / 1.42956
  int32_t  connectivity                 {1880064000} = 0 / 3268607
  float    points                       {235008000, 3} = -20 / 2
  int32_t  step                         3*scalar = 0 / 20

code examples

here are some test files for readling and writting adios files and converting it to vtk data format:

https://github.com/wangzhezhe/5MCST/blob/master/vtk_example/adiosToVTK/adiosToRawCommon.cpp

This is a python example that uses adios python api to extract associated data.

data layout

Another important tip is the meaning for organizing the variable in adios, if the data layout is {d1,d2,d3}, it means that d3 is the dimention that changes the fastest one so if this data is 3d data, it is possible that d3 is the x dimention, then d2 is the y dimention and d1 is the z dimentions. All data is flattern into 1d array and stored in the disk or in memory.

Tips of compiling

On frontier, using default compiler to build adios, there are some errors to find the MPI after switch to the gcc.

HDF5

openPMD

There are three types of backend, hdf5, adios and json.

Foam data

raw image data