Many faces for data format/representation for sci-vis

Some tips regarding the common data format used for visualization.

VTK data

VTK-m data

ADIOS

setting python binding’s path

export PYTHONPATH=/path/to/site-packages:${PYTHONPATH} 

remember to update this env before using python bindings for executing adios

bpls checking all variables

This example is output to check variable “VX” by time step, it print the element from position 0 to position 10 for each time step for this variable -d parameter is used to specify the variable.

$ ~/cworkspace/build_adios/bin/bpls /lustre/orion/csc143/proj-shared/gongq/frontier/dataset/bent_pipe.bp/ -d "VX" -s "0" -c "10" -t
Step 0:
  double   VX            {235008000}
    slice (0:9)
    (0)    0 0 0 0 0 0
    (6)    0 0 0 0 
Step 1:
  double   VX    {235008000}
    slice (0:9)
    (0)    -1.00187 -1.00187 -1.00187 -1.00187 -1.00187 -1.00187
    (6)    -1.00187 -1.00187 -1.00187 -1.00187 
Step 2:
  double   VX    {235008000}
    slice (0:9)
    (0)    -1.00281 -1.00281 -1.00281 -1.00281 -1.00281 -1.00281
    (6)    -1.00281 -1.00281 -1.00281 -1.00281 

this is results for checking all variables’s min and max and all informations including the attribute and variable. For this case, there are 6 attributes and 6 variables. When read/write the file, the attributes and variables are defined in different ways.

zw241@login03:~/cworkspace/install_adios/bin$ ./bpls /lustre/orion/csc143/proj-shared/gongq/frontier/dataset/bent_pipe.bp/ -lav
File info:
  of variables:  6
  of attributes: 6
  statistics:    Min / Max 
  string   Fides_Cell_Type              attr   = "hexahedron"
  string   Fides_Connectivity_Variable  attr   = "connectivity"
  string   Fides_Coordinates_Variable   attr   = "points"
  string   Fides_Data_Model             attr   = "unstructured_single"
  string   Fides_Variable_Associations  attr   = {"points", "points", "points"}
  string   Fides_Variable_List          attr   = {"VX", "VY", "VZ"}
  double   VX                           3*{235008000} = -1.36295 / 0.720586
  double   VY                           3*{235008000} = -0.653091 / 0.733072
  double   VZ                           3*{235008000} = -0.103327 / 1.42956
  int32_t  connectivity                 {1880064000} = 0 / 3268607
  float    points                       {235008000, 3} = -20 / 2
  int32_t  step                         3*scalar = 0 / 20

code examples

here are some test files for readling and writting adios files and converting it to vtk data format:

https://github.com/wangzhezhe/5MCST/blob/master/vtk_example/adiosToVTK/adiosToRawCommon.cpp

This is a python example that uses adios python api to extract associated data.

data layout

Another important tip is the meaning for organizing the variable in adios, if the data layout is {d1,d2,d3}, it means that d3 is the dimention that changes the fastest one so if this data is 3d data, it is possible that d3 is the x dimention, then d2 is the y dimention and d1 is the z dimentions. All data is flattern into 1d array and stored in the disk or in memory.

Tips of compiling

On frontier, using default compiler to build adios, there are some errors to find the MPI after switch to the gcc.

HDF5 与 nc data

nc data is used a lot in climate science

The popular tool for showing nc data is Panoply

典型的表示就是经度和纬度相结合组成的2d网格

python的常用接口可以参考如下例子：

import netCDF4 as nc
import matplotlib.pyplot as plt
import numpy as np 


file_path = 'xxx.nc'
load the nc data
ds = nc.Dataset(file_path)

# general info of data set
print(ds)

# keys
print(ds.variables.keys())

# show information about specific var
uu_var = ds.variables['ww'][:]

# replace the masked value created by netcdf4 to np.nan
# for original maskConstant, the output value is "--"
uu_var_filled = uu_var.filled(np.nan)

# get results in one layer, in this dataset, the first dim is time, the second dim is depth or layer
uu_var_filled = uu_var_filled[0][0][:]

# print out the image by lat and lon
plt.imshow(uu_var_filled, origin='lower')

# save results
plt.savefig("uu_data.png")

cpp的常用接口有

openPMD

There are three types of backend, hdf5, adios and json.

Foam data

raw image data