SC 99-43 Werner Benger, Hans-Christian Hege, Andre Merzky, Thomas Radke, Edward Seidel: Efficient Distributed File I/O for Visualization in Grid
Environments (in preparation)
Abstract: Large scale simulations running in metacomputing
environments
face the problem of efficient file I/O. For efficiency it
is
desirable to write data locally, distributed across the
computing
environment, and then to minimize data transfer, i.e. reduce
remote file access. Both aspects require I/O approaches
which
differ from existing paradigms.
For the data output of distributed simulations, one wants
to use
fast local parallel I/O for all participating nodes,
producing a
single distributed logical file, while keeping changes to
the
simulation code as small as possible. For reading the
data file
as in postprocessing and file based visualization, one
wants to
have efficient partial access to remote and
distributed
files, using a global naming scheme and efficient data
caching,
and again keeping the changes to the postprocessing code
small.
However, all available software solutions require the
entire
data to be staged locally (involving possible data
recombination
and conversion), or suffer from the performance problems
of
remote or distributed file systems.
In this paper we show how to interface the HDF5 I/O
library via
its flexible Virtual File Driver layer to the Globus Data
Grid.
We show, that combining these two toolkits in a suitable
way
provides us with a new I/O framework, which allows
efficient,
secure, distributed and parallel file I/O in a
metacomputing
environment.
Keywords: Grid,
Data Grid,
Globus,
HDF5,
Cactus,
CCTK,
Amira,
VFD,
Storage Systems
CR: C.2.4