Keywords: parallel I/O, multiprocessor file system, pario-bib

Comment: Describes a parallel IBM/370, where they attach several small 370s to a switch, and several disks to each 370. Not much in the way of striping.

Abstract: As computation and communication hardware performance continue to rapidly increase, I/O represents a growing fraction of application execution time. This gap between the I/O subsystem and others is expected to increase in future since I/O performance is limited by physical motion. Therefore, it is imperative that novel techniques for improveing I/O performance be developed. Parallel I/O is a promising approach to alleviating this bottleneck. However, very little work exist with respect to scheduling parallel I/O operations explicitly. In this paper, we address the problem of effective management of parallel I/O in cluster computing systems by using appropriate I/O scheduling strategies. We propose two new I/O scheduling algorithms and compare them with two existing scheduling Approaches. The preliminary results show that the proposed policies outperform existing policies substantially.

Keywords: parallel I/O, I/O scheduling algorithms, pario-bib

Keywords: parallel I/O, out-of-core algorithm, pario-bib

Comment: See also the component papers vitter:survey, arge:lower, crauser:segment, grossi:crosstrees, toledo:survey. Not clear to what extent these papers are about *parallel* I/O.

Keywords: out-of-core algorithm, graph, pario-bib

Keywords: parallel I/O, I/O, pario-bib

Comment: Derives formulas for the speedup with and without I/O considered and with parallel software and hardware format conversion. Considering I/O gives a more optimistic view of the speedup of a program assuming that the parallel version can use its I/O bandwidth as effectively as the serial processor. Concludes that, for a given number of processors, increasing the I/O bandwidth is the most effective way to speed up the program (over the format conversion improvements).

Abstract: Getting good I/O performance from parallel programs is a critical problem for many application domains. In this paper, we report our experience tuning the I/O performance of four application programs from the areas of satellite-data processing and linear algebra. After tuning, three of the four applications achieve application-level I/O rates of over 100 MB/s on 16 processors. The total volume of I/O required by the programs ranged from about 75 MB to over 200 GB. We report the lessons learned in achieving high I/O performance from these applications, including the need for code restructuring, local disks on every node and knowledge of future I/O requests. We also report our experience on achieving high performance on peer-to-peer configurations. Finally, we comment on the necessity of complex I/O interfaces like collective I/O and strided requests to achieve high performance.

Keywords: parallel I/O, filesystem workload, parallel application, pario-bib

Abstract: We provide tight upper and lower bounds, up to a constant factor, for the number of inputs and outputs (I/Os) between internal memory and secondary storage required for five sorting-related problems: sorting, the fast Fourier transform (FFT), permutation networks, permuting, and matrix transposition. The bounds hold both in the worst case and in the average case, and in several situations the constant factors match. Secondary storage is modeled as a magnetic disk capable of transfering $P$ blocks each containing $B$ records in a single time unit; the records in each block must be input from or output to $B$ contiguous locations on the disk. We give two optimal algorithms for the problems, which are variants of merge sorting and distribution sorting. In particular we show for $P=1$ that the standard merge sorting algorithm is an optimal external sorting method, up to a constant factor in the number of I/Os. Our sorting algorithms use the same number of I/Os as does the permutation phase of key sorting, except when the internal memory size is extremely small, thus affirming the popular adage that key sorting is not faster. We also give a simpler and more direct derivation of Hong and Kung's lower bound for the FFT for the special case $B = P = O(1)$.

Keywords: parallel I/O, sorting, pario-bib

Comment: Good comments on typical external sorts, and how big they are. Focuses on parallelism at the disk. They give tight theoretical bounds on the number of I/O's required to do external sorting and other problems (FFTs, matrix transpose, etc.). If $x$ is the number of blocks in the file and $y$ is the number of blocks that fit in memory, then the number of I/Os needed grows as $Θ (x \log x / \log y)$. If parallel transfers of $p$ blocks are allowed, speedup linear in $p$ is obtained.

Keywords: compiler, I/O, pario-bib

Comment: Not really about parallel applications or parallel I/O, but I think it may be of interest to that community. They propose a compiler framework for a compiler to insert asynchronous I/O operations (start I/O, finish I/O), to satisfy the dependency constraints of the program.

Abstract: The motivation for the research presented here is to develop an approach for scheduling I/O operations in distributed/parallel computer systems. First, a general model for specifying the parallel I/O scheduling problem is developed. The model defines the I/O bandwidth for different parallel/distributed architectures. Then the model is used to establish an algorithm for scheduling I/O operations on these architectures.

Keywords: parallel I/O, scheduling, pario-bib

Abstract: This paper describes the performance improvements achieved by a data-intensive application by controlling the storage policies and algorithms of a distributed storage system. The Network Storage Manager (NSM) is a mass distributed storage framework with a unique architecture that provides applications with the high-performance features they need. It also provides the standard most commonly used implementation for storage policies. Distributed Terrain Viewer (DTViewer) is an application that utilizes NSM architecture and for efficient and reliable data delivery. Moreover, it exploits NSM controllable architecture by plugging-in its application-specific optimized implementations. DTViewer overrides the default NSM policies that do not understand its sophisticated access patterns, partitioning, and storage layout requirements. Experimental results have show significant improvement achieved when the application-tailored implementation are used. Such speedups are not achievable on storage systems with no application control such as the Parallel Virtual File System PVFS. (44 Refs.)

Keywords: application-specific storage policies, pario-app, DTViewer, access patterns, data layout, pario-bib

Abstract: An emerging class of data-intensive applications involve the geographically dispersed extraction of complex scientific information from very large collections of measured or computed data. Such applications arise, for example, in experimental physics, where the data in question is generated by accelerators, and in simulation science, where the data is generated by supercomputers. So-called Data Grids provide essential infrastructure for such applications, much as the Internet provides essential services for applications such as e-mail and the Web. We describe here two services that we believe are fundamental to any Data Grid: reliable, high-speed transport and replica management. Our high-speed transport service, GridFTP, extends the popular FTP protocol with new features required for Data Grid applications, such as striping and partial file access. Our replica management service integrates a replica catalog with GridFTP transfers to provide for the creation, registration, location, and management of dataset replicas. We present the design of both services and also preliminary performance results. Our implementations exploit security and other services provided by the Globus Toolkit.

Keywords: computational grid, data transfer, network, I/O, pario-bib

Abstract: We are developing a system for collaborative research and development for a distributed group of researchers at different institutions around the world. In a new paradigm for collaborative computational science, the computer code and supporting infrastructure itself becomes the collaborating instrument, just as an accelerator becomes the collaborating tool for large numbers of distributed researchers in particle physics. The design of this "Collaboratory" allows many users, with very different areas of expertise, to work coherently together, on distributed computers around the world. Different supercomputers may be used separately, or for problems exceeding the capacity of any single system, multiple supercomputers may be networked together through high speed gigabit networks. Central to this Collaboratory is a new type of community simulation code, called "Cactus". The scientific driving force behind this project is the simulation of Einstein's equations for studying black holes, gravitational waves, and neutron stars, which has brought together researchers in very different fields from many groups around the world to make advances in the study of relativity and astrophysics. But the system is also being developed to provide scientists and engineers, without expert knowledge of parallel or distributed computing, mesh refinement, and so on, with a simple framework for solving any system of partial differential equations on many parallel computer systems, from traditional supercomputers to networks of workstations.

Keywords: scientific application, grid, input/output, parallel-io, pario-bib

Comment: invited talk. They describe a computational toolkit (CACTUS) that allows developers to construct code modules (thorns) to plug into the core system (cactus flesh). The toolkit includes thorns for solving partial differential equations using MPI, parallel elliptic solvers, thorns for I/O using FlexIO or HDF5, and thorns for checkpointing. The talk showed results from a cactus code demo that ran at SC'98. The demo combined two tightly-connected supercomputers (one in Europe and one in America) using Globus to simulate the collision of two neutron stars.

Keywords: fault tolerance, RAID, disk array, parallel I/O, pario-bib

Abstract: Enterprise-scale storage systems, which can contain hundreds of host computers and storage devices and up to tens of thousands of disks and logical volumes, are difficult to design. The volume of choices that need to be made is massive, and many choices have unforeseen interactions. Storage system design is tedious and complicated to do by hand, usually leading to solutions that are grossly over-provisioned, substantially under-performing or, in the worst case, both.To solve the configuration nightmare, we present minerva: a suite of tools for designing storage systems automatically. Minerva uses declarative specifications of application requirements and device capabilities; constraint-based formulations of the various sub-problems; and optimization techniques to explore the search space of possible solutions.This paper also explores and evaluates the design decisions that went into Minerva, using specialized micro- and macro-benchmarks. We show that Minerva can successfully handle a workload with substantial complexity (a decision-support database benchmark). Minerva created a 16-disk design in only a few minutes that achieved the same performance as a 30-disk system manually designed by human experts. Of equal importance, Minerva was able to predict the resulting system's performance before it was built.

Keywords: disk array, storage system, RAID, automatic design, parallel I/O, pario-bib

Keywords: parallel architecture, MIMD, NUMA, pario-bib

Comment: Interesting architecture. 3-d mesh of pipelined packet-switch nodes, e.g., 16x16x16 is 4096 nodes, with 256 procs, 512 memory units, 256 I/O cache units, and 256 I/O processors attached. 2816 remaining nodes are just switching nodes. Each processor is 64-bit custom chip with up to 128 simultaneous threads in execution. It alternates between ready threads, with a deep pipeline. Inter-instruction dependencies explicitly encoded by the compiler, stalling those threads until the appropriate time. Each thread has a complete set of registers! Memory units have 4-bit tags on each word, for full/empty and trap bits. Shared memory across the network: ``The Tera ISP-level architecture is UMA, even though the PMS-level architecture is NUMA. Put another way, the memory looks a single cycle away to the compiler writer.'' - Burton Smith. See also tera:brochure.

Keywords: file caching, distributed file system, pario-bib

Comment: Part of jin:io-book; reformatted version of anderson:serverless.

Abstract: In benchmarking I/O systems, it is important to generate, accurately, the I/O access pattern that one is intending to generate. However, timing accuracy ( issuing I/Os at the desired time) at high I/O rates is difficult to achieve on stock operating systems. We currently lack tools to easily and accurately generate complex I/O workloads on modern storage systems. As a result, we may be introducing substantial errors in observed system metrics when we benchmark I/O systems using inaccurate tools for replaying traces or for producing synthetic workloads with known inter-arrival times.

In this paper, we demonstrate the need for timing accuracy for I/O benchmarking in the context of replaying I/O traces. We also quantitatively characterize the impact of error in issuing I/Os on measured system parameters. For instance, we show that the error in perceived I/O response times can be as much as +350% or -15% by using naive benchmarking tools that have timing inaccuracies. To address this problem, we present Buttress, a portable and flexible toolkit that can generate I/O workloads with microsecond accuracy at the I/O throughputs of high-end enterprise storage arrays. In particular, Buttress can issue I/O requests within 100µs of the desired issue time even at rates of 10000 I/Os per second (IOPS).

Keywords: benchmarking software, performance analysis, I/O access patterns, I/O workloads, pario-bib

Comment: Looks like a really cool piece of software. Generates I/O workloads by replaying I/O traces.

Abstract: Disk arrays have a myriad of configuration parameters that interact in counter-intuitive ways, and those interactions can have significant impacts on cost, performance, and reliability. Even after values for these parameters have been chosen, there are exponentially-many ways to map data onto the disk arrays' logical units. Meanwhile, the importance of correct choices is increasing: storage systems represent an growing fraction of total system cost, they need to respond more rapidly to changing needs, and there is less and less tolerance for mistakes. We believe that automatic design and configuration of storage systems is the only viable solution to these issues. To that end, we present a comparative study of a range of techniques for programmatically choosing the RAID levels to use in a disk array. Our simplest approaches are modeled on existing, manual rules of thumb: they "tag" data with a RAID level before determining the configuration of the array to which it is assigned. Our best approach simultaneously determines the RAID levels for the data, the array configuration, and the layout of data on that array. It operates as an optimization process with the twin goals of minimizing array cost while ensuring that storage workload performance requirements will be met. This approach produces robust solutions with an average cost/performance 14-17{PCT} better than the best results for the tagging schemes, and up to 150-200{PCT} better than their worst solutions. We believe that this is the first presentation and systematic analysis of a variety of novel, fully-automatic RAID-level selection techniques.

Keywords: file systems, pario-bib

Keywords: file caching, distributed file system, pario-bib

Comment: See anderson:serverless-sosp.

Abstract: Many parallel scientific applications need high-performance I/O. Unfortunately, end-to-end parallel-I/O performance has not been able to keep up with substantial improvements in parallel-I/O hardware because of poor parallel file-system software. Many radical changes, both at the interface level and the implementation level, have recently been proposed. One such proposed interface is collective I/O, which allows parallel jobs to request transfer of large contiguous objects in a single request, thereby preserving useful semantic information that would otherwise be lost if the transfer were expressed as per-processor non-contiguous requests. Kotz has proposed disk-directed I/O as an efficient implementation technique for collective-I/O operations, where the compute processors make a single collective data-transfer request, and the I/O processors thereafter take full control of the actual data transfer, exploiting their detailed knowledge of the disk-layout to attain substantially improved performance.

Recent parallel file-system usage studies show that writes to write-only files are a dominant part of the workload. Therefore, optimizing writes could have a significant impact on overall performance. In this paper, we propose ENWRICH, a compute-processor write-caching scheme for write-only files in parallel file systems. ENWRICH combines low-overhead write caching at the compute processors with high performance disk-directed I/O at the I/O processors to achieve both low latency and high bandwidth. This combination facilitates the use of the powerful disk-directed I/O technique independent of any particular choice of interface. By collecting writes over many files and applications, ENWRICH lets the I/O processors optimize disk I/O over a large pool of requests. We evaluate our design via simulated implementation and show that ENWRICH achieves high performance for various configurations and workloads.

Keywords: parallel file system, parallel I/O, caching, pario-bib, dfk

Abstract: Many parallel scientific applications need high-performance I/O. Unfortunately, end-to-end parallel-I/O performance has not been able to keep up with substantial improvements in parallel-I/O hardware because of poor parallel file-system software. Many radical changes, both at the interface level and the implementation level, have recently been proposed. One such proposed interface is collective I/O, which allows parallel jobs to request transfer of large contiguous objects in a single request, thereby preserving useful semantic information that would otherwise be lost if the transfer were expressed as per-processor non-contiguous requests. Kotz has proposed disk-directed I/O as an efficient implementation technique for collective-I/O operations, where the compute processors make a single collective data-transfer request, and the I/O processors thereafter take full control of the actual data transfer, exploiting their detailed knowledge of the disk-layout to attain substantially improved performance.

Recent parallel file-system usage studies show that writes to write-only files are a dominant part of the workload. Therefore, optimizing writes could have a significant impact on overall performance. In this paper, we propose ENWRICH, a compute-processor write-caching scheme for write-only files in parallel file systems. ENWRICH combines low-overhead write caching at the compute processors with high performance disk-directed I/O at the I/O processors to achieve both low latency and high bandwidth. This combination facilitates the use of the powerful disk-directed I/O technique independent of any particular choice of interface. By collecting writes over many files and applications, ENWRICH lets the I/O processors optimize disk I/O over a large pool of requests. We evaluate our design via simulated implementation and show that ENWRICH achieves high performance for various configurations and workloads.

Keywords: parallel file system, parallel I/O, caching, pario-bib, dfk

Abstract: High-performance parallel file systems are needed to satisfy tremendous I/O requirements of parallel scientific applications. The design of such parallel file systems depends on a comprehensive understanding of the expected workload, but so far there have been very few usage studies of multiprocessor file systems. In the first part of this dissertation, we attempt to fill this void by measuring a real file-system workload on a production parallel machine, namely the CM-5 at the National Center for Supercomputing Applications. We collect information about nearly every individual I/O request from the mix of jobs running on the machine. Analysis of the traces leads to various recommendations for design of future parallel file systems. Our usage study showed that writes to write-only files are a dominant part of the workload. Therefore, optimizing writes could have a significant impact on overall performance. In the second part of this dissertation, we propose ENWRICH, a compute-processor write-caching scheme for write-only files in parallel file systems. Within its framework, ENWRICH uses a recently proposed high performance implementation of collective I/O operations called disk-directed I/O, but it eliminates a number of limitations of disk-directed I/O. ENWRICH combines low-overhead write caching at the compute processors with high performance disk-directed I/O at the I/O processors to achieve both low latency and high bandwidth. This combination facilitates the use of the powerful disk-directed I/O technique independent of any particular choice of interface, and without the requirement for mapping libraries at the I/O processors. By collecting writes over many files and applications, ENWRICH lets the I/O processors optimize disk I/O over a large pool of requests. We evaluate our design of ENWRICH using simulated implementation and extensive experimentation. We show that ENWRICH achieves high performance for various configurations and workloads. We pinpoint the reasons for ENWRICH`s failure to perform well for certain workloads, and suggest possible enhancements. Finally, we discuss the nuances of implementing ENWRICH on a real platform and speculate about possible adaptations of ENWRICH for emerging multiprocessing platforms.

Keywords: parallel I/O, multiprocessor file system, file access patterns, workload characterization, file caching, disk-directed I/O, pario-bib

Comment: See also ap:enwrich, ap:workload, and nieuwejaar:workload

Abstract: High-performance parallel file systems are needed to satisfy tremendous I/O requirements of parallel scientific applications. The design of such high-performance parallel file systems depends on a comprehensive understanding of the expected workload, but so far there have been very few usage studies of multiprocessor file systems. This paper is part of the CHARISMA project, which intends to fill this void by measuring real file-system workloads on various production parallel machines. In particular, here we present results from the CM-5 at the National Center for Supercomputing Applications. Our results are unique because we collect information about nearly every individual I/O request from the mix of jobs running on the machine. Analysis of the traces leads to various recommendations for parallel file-system design.

Keywords: parallel I/O, file access pattern, multiprocessor file system, file system workload, dfk, pario-bib

Comment: See also kotz:workload, nieuwejaar:strided.

Abstract: Rapid increases in the computational speeds of multiprocessors have not been matched by corresponding performance enhancements in the I/O subsystem. To satisfy the large and growing I/O requirements of some parallel scientific applications, we need parallel file systems that can provide high-bandwidth and high-volume data transfer between the I/O subsystem and thousands of processors.

Design of such high-performance parallel file systems depends on a thorough grasp of the expected workload. So far there have been no comprehensive usage studies of multiprocessor file systems. Our CHARISMA project intends to fill this void. The first results from our study involve an iPSC/860 at NASA Ames. This paper presents results from a different platform, the CM-5 at the National Center for Supercomputing Applications. The CHARISMA studies are unique because we collect information about every individual read and write request and about the entire mix of applications running on the machines.

The results of our trace analysis lead to recommendations for parallel file system design. First, the file system should support efficient concurrent access to many files, and I/O requests from many jobs under varying load condit ions. Second, it must efficiently manage large files kept open for long periods. Third, it should expect to see small requests, predominantly sequential access patterns, application-wide synchronous access, no concurrent file-sharing between jobs, appreciable byte and block sharing between processes within jobs, and strong interprocess locality. Finally, the trace data suggest that node-level write caches and collective I/O request interfaces may be useful in certain environments.

Keywords: parallel I/O, file access pattern, multiprocessor file system, file system workload, dfk, pario-bib

Comment: See also kotz:workload, nieuwejaar:strided.

Keywords: parallel file system, parallel I/O, Intel iPSC/2, pario-bib

Comment: Studies the performance of Intel CFS. Uses an application that reads in a huge file of records, each a genome sequence, and compares each sequence against a given sequence. Looks at cache performance, message latency, cost of prefetches and directory reads, and throughput. He tries one-disk, one-proc transfer rates. Because of contention with the directory server on one of the two I/O nodes, it was faster to put all of the file on the other I/O node. Striping is good for multiple readers. Best access pattern was interleaved, not segmented or separate files, because it avoided disk seeks. Perhaps the files are stored contiguously? Can get good speedup by reading the sequences in big integral record sizes, from CFS, using a load-balancing scheduled by the host. Contention for directory blocks - through single-node directory server.

Abstract: The paper presents a survey of the basic paradigms for designing efficient external-memory algorithms and especially for designing external-memory algorithms for computational geometry problems with applications in GIS. As the area of external-memory algorithms is relatively young the paper focuses on fundamental external-memory design techniques more than on algorithms for specific GIS problems. The presentation is survey-like with a more detailed discussion of the most important techniques and algorithms.

Keywords: out-of-core algorithm, geographic information system, GIS, pario-bib

Comment: not parallel? but mentions some parallel disk stuff.

Abstract: We present a set of algorithms designed to solve large-scale geometric problems involving collections of line segments in the plane. Geographical information systems (GIS) handle large amounts of spatial data, and at some level the data is often manipulated as collections of line segments. NASA's EOS project is an example of a GIS that is expected to store and manipulate petabytes (thousands of terabytes, or millions of gigabytes) of data! In the design of algorithms for this type of large-scale application, it is essential to consider the problem of minimizing I/O communication, which is the bottleneck.

In this paper we develop efficient new external-memory algorithms for a number of important problems involving line segments in the plane, including trapezoid decomposition, batched planar point location, triangulation, red-blue line segment intersection reporting, and general line segment intersection reporting. In GIS systems, the first three problems are useful for rendering and modeling, and the latter two are frequently used for overlaying maps and extracting information from them. To solve these problems, we combine and modify in novel ways several of the previously known techniques for designing efficient algorithms for external memory. We also develop a powerful new technique that can be regarded as a practical external memory version of fractional cascading. Except for the batched planar point location problem, no algorithms specifically designed for external memory were previously known for these problems. Our algorithms for triangulation and line segment intersection partially answer previously posed open problems, while the batched planar point location algorithm improves on the previously known solution, which applied only to monotone decompositions. Our algorithm for the red-blue line segment intersection problem is provably optimal.

Keywords: verify, out-of-core algorithm, computational geometry, pario-bib

Comment: Special issue on cartography and geographic information systems.

Keywords: out-of-core algorithm, computational geometry, pario-bib

Comment: See also the component papers vitter:survey, arge:lower, crauser:segment, grossi:crosstrees, toledo:survey. Not clear to what extent these papers are about *parallel* I/O.

Abstract: In the design of algorithms for large-scale applications it is essential to consider the problem of minimizing I/O communication. Geographical information systems (GIS) are good examples of such large-scale applications as they frequently handle huge amounts of spatial data. In this paper we develop efficient new external-memory algorithms for a number of important problems involving line segments in the plane, including trapezoid decomposition, batched planar point location, triangulation, red-blue line segment intersection reporting, and general line segment intersection reporting. In GIS systems, the first three problems are useful for rendering and modeling, and the latter two are frequently used for overlaying maps and extracting information from them.

Keywords: out-of-core algorithm, computational geometry, pario-bib

Comment: Does deal with parallel disks, though not in great detail.

Abstract: We investigate the I/O complexity of the problem of sorting sequences (or strings of characters) in external memory, which is a fundamental component of many large-scale text applications. In the standard unit-cost RAM comparison model, the complexity of sorting K strings of total length N is Theta (K log/sub 2/ K+N). By analogy, in the external memory (or I/O) model, where the internal memory has size M and the block transfer size is B, it would be natural to guess that the I/O complexity of sorting sequences is Theta ((K/B)log/sub M/B/(K/B)+(N/B)), but the known algorithms do not come even close to achieving this bound. Our results show, somewhat counterintuitively, that the I/O complexity of string sorting depends upon the length of the strings relative to the block size. We first consider a simple comparison I/O model, where the strings are not allowed to be broken into their individual characters, and we show that the I/O complexity of string sorting in this model is Theta ((N/sub 1//B)log/sub M/B/(N/sub 1//B)+K/sub 2/+(N/B)), where N/sub 1/ is the total length of all strings shorter than B and K/sub 2/ is the number of strings longer than B. We then consider two more general I/O comparison models in which string breaking is allowed. We obtain improved algorithms and in several cases lower bounds that match their I/O bounds. Finally, we develop more practical algorithms outside the comparison model.

Keywords: out-of-core algorithm, sorting algorithm, pario-bib

Comment: This paper is really the same paper as arge:sorting-strings.

Abstract: In this paper we address for the first time the I/O complexity of the problem of sorting strings in external memory, which is a fundamental component of many large-scale text applications. In the standard unit-cost RAM comparison model, the complexity of sorting K strings of total length N is theta(K log K + N). By analogy, in the external memory (or I/O) model, where the internal memory has size M and the block transfer size is B, it would be natural to guess that the I/O complexity of sorting strings is $θ(K/B log_(M/B) (K/B) + N/B)$, but the known algorithms do not come even close to achieving this bound. Our results show, somewhat counterintuitively, that the I/O complexity of string sorting depends upon the length of the strings relative to the block size. We first consider a simple comparison I/O model, where one is not allowed to break the strings into their characters, and we show that the I/O complexity of string sorting in this model is $θ(N_1/B log_(M/B) (N_1/B) + K_2 log_(M/B) K_2 + N/B)$, where $N_1$ is the total length of all strings shorter than B and $K_2$ is the number of strings longer than B. We then consider two more general I/O comparison models in which string breaking is allowed. We obtain improved algorithms and in several cases lower bounds that match their I/O bounds. Finally, we develop more practical algorithms without assuming the comparison model.

Keywords: out-of-core algorithm, sorting, parallel I/O, pario-bib

Comment: Not parallel? But mentions some parallel disk stuff.

Abstract: The single-disk, D-head model of parallel I/O was introduced by Agarwal and Vitter to analyze algorithms for problem instances that are too large to fit in primary memory. Subsequently Vitter and Shriver proposed a more realistic model in which the disk space is partitioned into D disks, with a single head per disk. To date, each problem for which there is a known optimal algorithm for both models has the same asymptotic bounds on both models. Therefore, it has been unknown whether the models are equivalent or whether the single-disk model is strictly more powerful.

In this paper we provide evidence that the single-disk model is strictly more powerful. We prove a lower bound on any general simulation of the single-disk model on the multi-disk model and establish randomized and deterministic upper bounds. Let $N$ be the problem size and let $T$ be the number of parallel I/Os required by a program on the single-disk model. Then any simulation of this program on the multi-disk model will require $Ω\left(T \frac{\log(N/D)}{\log \log(N/D)}\right)$ parallel I/Os. This lower bound holds even if replication is allowed in the multi-disk model. We also show an $O\left(\frac{\log D}{\log \log D}\right)$ randomized upper bound and an $O\left(\log D (\log \log D)^2\right)$ deterministic upper bound. These results exploit an interesting analogy between the disk models and the PRAM and DCM models of parallel computation.

Keywords: parallel I/O, theory, parallel I/O algorithm, pario-bib

Keywords: distributed query processing, dataflow, pario-bib

Comment: River is a dataflow programming environment for database query processing applications. River is specifically designed for clusters of computers with heterogeneous performance characteristics. The goal of the River runtime system is to adapt to "performance faults"-portions of the system that perform poorly by dynamically adjusting the transfer of data through the dataflow graph. River uses two constructs to build applications: a distributed queue that deals with performance faults by consumers, and graduated declustering that deals with performance faults of producers. A distributed queue pushes data through the dataflow graph at a rate proportional to the rate of consumption and adapts to changes in consumption rates. Graduated declustering deals with producer performance faults by reading from replicated producers. Although River is designed specifically for query processing, they briefly discuss how one might adapt scientific applications to work in their framework.

Abstract: We introduce River, a data-flow programming environment and I/O substrate for clusters of computers. River is designed to provide maximum performance in the common case- even in the face of non-uniformities in hardware, software, and workload. River is based on two simple design features: a high-performance distributed queue,and a storage redundancy mechanism called graduated declustering.We have implemented a number of data-intensive applications on River, which validate our design with near-ideal performance in a variety of non-uniform performance scenarios.

Keywords: cluster computing, parallel I/O, pario-bib

Keywords: parallel I/O, prefetching, parallel file system, pario-bib

Comment: A related paper is arunachalam:prefetch2.

Abstract: The significant difference between the speeds of the I/O system (e.g., disks) and compute processors in parallel systems creates a bottleneck that lowers the performance of an application that does a considerable amount of disk accesses. A major portion of the compute processors' time is wasted on waiting for I/O to complete. This problem can be addressed to a certain extent, if the necessary data can be fetched from the disk before the I/O call to the disk is issued. Fetching data ahead of time, known as prefetching in a multiprocessor environment depends a great deal on the application's access pattern. The subject of this paper is implementation and performance evaluation of a prefetching prototype in a production parallel file system on the Intel Paragon. Specifically, this paper presents a) design and implementation of a prefetching strategy in the parallel file system and b) performance measurements and evaluation of the file system with and without prefetching. The prototype is designed at the operating system level for the PFS. It is implemented in the PFS subsystem of the Intel Paragon Operating System. It is observed that in many cases prefetching provides considerable performance improvements. In some other cases no improvements or some performance degradation is observed due to the overheads incurred in prefetching.

Keywords: parallel I/O, prefetching, multiprocessor file system, pario-bib

Comment: See arunachalam:prefetch.

Keywords: parallel I/O, disk array, RAID, pario-bib

Comment: Part of jin:io-book; reformatted version of asami:self.

Abstract: This paper shows the suitability of a self-maintaining approach to Tertiary Disk, a large-scale disk array system built from commodity components. Instead of incurring the cost of custom hardware, we attempt to solve various problems by design and software. We have built a cluster of storage nodes connected by switched Ethernet. Each storage node is a PC hosting a few dozen SCSI disks, running the FreeBSD operating system. The system is used as a web-based image server for the Zoom Project in cooperation with the Fine Arts Museums of San Francisco (http://www.thinker.org/). We are designing self-maintenance extension to the OS to run on this cluster to mitigate the system administrator's burden. There are several components required for building self-maintaining system. One is decoupling the time failure from the time of hardware replacement. This implies the system must have some amount of redundancy, and has no single point of failure. Our system is fully redundant, and everything is constructed to avoid a single point of failure. Another is correctly identifying failures and their dependencies. The paper also outlines several approaches to lower the human cost of system administration of such a system and making the system as autonomous as possible.

Keywords: parallel I/O, disk array, RAID, pario-bib

Keywords: parallel I/O, hypercube, Intel iPSC/2, file access pattern, pario-bib

Keywords: parallel I/O, architecture, pario-bib

Comment: They describe an I/O subsystem based on an ``active memory'' called SWIM (Structured Wafer-based Intelligent Memory). SWIM chips are RAM chips with some built-in processing. The idea is that these tiny processors can manipulate the data in the chip at full speed, without dealing with memory bus or off-chip costs. Further, the chips can work in parallel. They demonstrate how they've used this to build a national phone database server, a high-performance IP router, and a call-screening agent.

Abstract: We describe an I/O subsystem based on an active memory named SWIM (Structured Wafer-based Intelligent Memory) designed for efficient storage and manipulation of data structures. The key architectural idea in SWIM is to associate some processing logic with each memory chip that allows it to perform data manipulation operations locally and to communicate with a disk or a communication line through a backend port. The processing logic is specially designed to perform operations such as pointer dereferencing, memory indirection, searching and bounds checking efficiently. The I/O subsystem is built using an interconnected ensemble of such memory logic pairs. A complex processing task can now be distributed between a large number of small memory processors each doing a sub-task, while still retaining a common locus of control in the host CPU for higher level administrative and provisioning functions. We argue that active memory based processing enables more powerful, scalable and robust designs for storage and communications subsystems, that can support emerging network services, multimedia workstations and wireless PCS systems. A complete parallel hardware and software system constructed using an array of SWIM elements has been operational for over a year. We present results from application of SWIM to three network functions: a national phone database server, a high performance IP router, and a call screening agent.

Keywords: parallel I/O architecture, pario-bib

Comment: Part of a whole book on parallel I/O; see iopads-book.

Keywords: parallel I/O, pario-bib

Comment: They describe using the techniques of delrosario and debenedictis (although without mentioning them) to provide for channels (parallel pipes) between independent data-parallel tasks. The technique really is the same as in debenedictus and delrosario, although they extend it a bit to allow multiple "files" within a channel (why not use multiple channels)? Also, they depend on the program to read and write synchronization variables to control access to the flow of data through the channel. While this may provide good performance in some cases, why not have support for automatic flow control? The system can detect when a portion of the channel is written, and release readers waiting on that portion of the channel (if any). The paper is a bit confusing in its use of the word "file", which seems to be used to mean different things at different points. Also, they seem to use an arbitrary distribution for the "file", which may or may not be the same as one of those used by the two endpoints.

Abstract: Access to shared data is critical to the long term success of grids of distributed systems. As more parallel applications are being used on these grids, the need for some kind of parallel I/O facility across distributed systems increases. However, grid middleware has thus far had only limited support for distributed parallel I/O. In this paper, we present an implementation of the MPI-2 I/O interface using the Globus GridFTP client API. MPI is widely used for parallel computing, and its I/O interface maps onto a large variety of storage systems. The limitations of using GridFTP as an MPI-I/O transport mechanism are described, as well as support for parallel access to scientific data formats such as HDF and NetCDF. We compare the performance of GridFTP to that of NFS on the same network using several parallel I/O benchmarks. Our tests indicate that GridFTP can be a workable transport for parallel I/O, particularly for distributed read-only access to shared data sets. (26 refs.)

Keywords: grid I/O, MPI-I/O, grid middleware, gridFTP, pario-bib

Keywords: parallel I/O, pario-bib, prefetching, caching, multiprocessor file system, file access pattern

Comment: Basically there are two parts to this paper. First, they will instrument applications, Intel PFS, and IBM Vesta, to trace I/O-related activity. Then they'll use Pablo to analyze and characterize. They plan to trace some events in detail, and the rest with histogram counters. Second, they plan to develop caching and prefetching policies and to analyze those with simulation, analysis, and implementation. They note that IBM and Intel are developing parallel I/O architecture simulators. See also poole:sio-survey, choudhary:sio-language, bershad:sio-os.

Abstract: RAID storage arrays often possess gigabytes of RAM for caching disk blocks. Currently, most RAID systems use LRU or LRU-like policies to manage these caches. Since these array caches do not recognize the presence of file system buffer caches, they redundantly retain many of the same blocks as those cached by the file system, thereby wasting precious cache space. In this paper, we introduce X-RAY, an exclusive RAID array caching mechanism. X-RAY achieves a high degree of (but not perfect) exclusivity through gray-box methods: by observing which files have been accessed through updates to file system meta-data, X-RAY constructs an approximate image of the contents of the file system cache and uses that information to determine the exclusive set of blocks that should be cached by the array. We use microbenchmarks to demonstrate that X-RAY's prediction of the file system buffer cache contents is highly accurate, and trace-based simulation to show that X-RAY considerably outperforms LRU and performs as well as other more invasive approaches. The main strength of the X-RAY approach is that it is easy to deploy - all performance gains are achieved without changes to the SCSI protocol or the file system above.

Keywords: RAID, x-ray, caching policies, pario-bib

Keywords: parallel I/O, distributed file system, mass storage, pario-bib

Comment: Architecture for distributed information storage. Integrates file systems, databases, etc. Single system image, lots of support for administration. O-O model, with storage device objects, logical device objects, volume objects, and file objects. Methods for each type of object, including administrative methods.

Abstract: Modern applications such as `video on demand' require fast reading of complete files, which can be supported well by file striping. Many conventional applications, however, are only interested in some part of the available records. In order to avoid reading attributes irrelevant to such applications, each attribute could be stored in a separate (transposed) file; Aiming at I/O parallelism, byte-oriented striping could be applied to transposed files. However, such a fragmentation ignores the semantics of data. This fragmentation cannot be optimized by a database management system (DBMS) because a DBMS has to perform its tasks on the basis of data semantics. For example, queries must be translated into file operations using a scheme that maps a data model to a file system. However, details about files, such as the striping width, are invisible to a DBMS. Therefore, we propose to store each transposed file related to a composite type on a separate, independent disk drive, which means I/O parallelism tuned to a data model. As we also aim at system reliability and data availability, each transposed file must be duplicated on another drive. Consequently, a DBMS also has to guarantee correctness and completeness of the allocation of transposed files within an array of disk drives. As a solution independent of the underlying data model, we propose an abstract framework consisting of a meta model and a set of rules

Keywords: database, parallel I/O, pario-bib

Keywords: multiprocessor file system, file access pattern, parallel I/O, hypercube, pario-bib

Comment: Census-data processing on an nCUBE/10 at USC. Their program uses an interleaved pattern, which is like my lfp or gw with multi-record records (i.e., the application does its own blocking). Shifted to asynchronous I/O to do OBL manually. Better results if they did more computation per I/O (of course).

Abstract: We show two algorithms for computing multidimensional Fast Fourier Transforms (FFTs) on a multiprocessor system with distributed memory when problem sizes are so large that the data do not fit in the memory of the entire system. Instead, data reside on a parallel disk system and are brought into memory in sections. We use the Parallel Disk Model for implementation and analysis.

The first method is a straightforward out-of-core variant of a well-known method for in-core, multidimensional FFTs. It performs 1-dimensional FFT computations on each dimension in turn. This method is easy to generalize to any number of dimensions, and it also readily permits the individual dimensions to be of any sizes that are integer powers of 2. The key step is an out-of-core transpose operation that places the data along each dimension into contiguous positions on the parallel disk system so that the data for the 1-dimensional FFTs are contiguous.

The second method is an adaptation of another well-known method for in-core, multidimensional FFTs. This method computes all dimensions simultaneously. It is more difficult to generalize to arbitrary radices and number of dimensions in this method than in the first method. Our present implementation is therefore limited to two dimensions of equal size, that are again integer powers of 2.

We present I/O complexity analyses for both methods as well as empirical results for a DEC 2100 server and an SGI Origin 2000, each of which has a parallel disk system. Our results indicate that the methods are comparable in speed in two-dimensions.

Keywords: parallel I/O, out of core, FFT, parallel algorithm, scientific computing, pario-bib

Comment: Undergraduate Honors Thesis. Advisor: Tom Cormen.

Keywords: parallel I/O, hashing, reliability, disk mirroring, pario-bib

Comment: Describes a file system for a distributed system that scatters records of each file over many disks using hash functions. The hash function is known by all processors, so no one processor must be up to access the file. Any portion of the file whose disknode is available may be accessed. Shadow nodes are used to take over for nodes that go down, saving the info for later use by the proper node. Intended to easily parallelize read/write accesses and global file operations, and to increase file availability.

Keywords: disk model, I/O bus, device model, I/O model, pario-bib

Keywords: disk model, I/O bus, device model, I/O model, pario-bib

Abstract: We provide a competitive analysis framework for online prefetching and buffer management algorithms in parallel I/O systems, using a read-once model of block references. This has widespread applicability to key I/O-bound applications such as external merging and concurrent playback of multiple video streams. Two realistic lookahead models, global lookahead and local lookahead, are defined. Algorithms NOM and GREED based on these two forms of lookahead are analyzed for shared buffer and distributed buffer configurations, both of which occur frequently in existing systems. An important aspect of our work is that we show how to implement both the models of lookahead in practice using the simple techniques of forecasting and flushing.

Given a D-disk parallel I/O system and a globally shared I/O buffer that can hold upto M disk blocks, we derive a lower bound of $Ω(\sqrt{D}$) on the competitive ratio of any deterministic online prefetching algorithm with O(M) lookahead. NOM is shown to match the lower bound using global M-block lookahead. In contrast, using only local lookahead results in an $Ω(D)$ competitive ratio. When the buffer is distributed into D portions of M/D blocks each, the algorithm GREED based on local lookahead is shown to be optimal, and NOM is within a constant factor of optimal. Thus we provide a theoretical basis for the intuition that global lookahead is more valuable for prefetching in the case of a shared buffer configuration whereas it is enough to provide local lookahead in case of the distributed configuration. Finally, we analyze the performance of these algorithms for reference strings generated by a uniformly-random stochastic process and we show that they achieve the minimal expected number of I/Os. These results also give bounds on the worst-case expected performance of algorithms which employ randomization in the data layout.

Keywords: disk prefetching, file caching, parallel I/O, pario-bib

Comment: See also barve:competitive. They propose two methods for scheduling prefetch operations in the situation where the access pattern is largely known in advance, in such a way as to minimize the total number of parallel I/Os. The two methods are quite straightforward, and yet match the optimum lower bound for an on-line algorithm.

Abstract: We consider the problem of sorting a file of N records on the D-disk model of parallel I/O in which there are two sources of parallelism. Records are transferred to and from disk concurrently in blocks of B contiguous records. In each I/O operation, up to one block can be transferred to or from each of the D disks in parallel. We propose a simple, efficient, randomized mergesort algorithm called SRM that uses a forecast-and-flush approach to overcome the inherent difficulties of simple merging on parallel disks. SRM exhibits a limited use of randomization and also has a useful deterministic version. Generalizing the technique of forecasting, our algorithm is able to read in, at any time, the ``right'' block from any disk, and using the technique of flushing, our algorithm evicts, without any I/O overhead, just the ``right'' blocks from memory to make space for new ones to be read in. The disk layout of SRM is such that it enjoys perfect write parallelism, avoiding fundamental inefficiencies of previous mergesort algorithms. By analysis of generalized maximum occupancy problems we are able to derive an analytical upper bound on SRM's expected overhead valid for arbitrary inputs.

The upper bound derived on expected I/O performance of SRM indicates that SRM is provably better than disk-striped mergesort (DSM) for realistic parameter values D, M, and B. Average-case simulations show further improvement on the analytical upper bound. Unlike previously proposed optimal sorting algorithms, SRM outperforms DSM even when the number D of parallel disks is small.

Keywords: parallel I/O algorithm, sorting, pario-bib

Comment: This paper formerly called barve:mergesort; I discovered that the paper had appeared in SPAA96, so the SPAA96 paper is now called barve:mergesort.

Keywords: parallel I/O algorithm, sorting, pario-bib

Abstract: In modern I/O architectures, multiple disk drives are attached to each I/O bus. Under I/O-intensive workloads, the disk latency for a request can be overlapped with the disk latency and data transfers of requests to other disks, potentially resulting in an aggregate I/O throughput at nearly bus bandwidth. This paper reports on a performance impairment that results from a previously unknown form of convoy behavior in disk I/O, which we call rounds. In rounds, independent requests to distinct disks convoy, so that each disk services one request before any disk services its next request. We analyze log files to describe read performance of multiple Seagate Wren-7 disks that share a SCSI bus under a heavy workload, demonstrating the rounds behavior and quantifying its performance impact.

Keywords: disk, I/O bus, parallel I/O, pario-bib

Keywords: parallel architecture, array processor, parallel I/O, SIMD, pario-bib

Comment: This paper is reproduced in Kuhn and Padua's (1981, IEEE) survey ``Tutorial on Parallel Processing.'' The STARAN is an array processor that uses Multi-Dimensional-Access (MDA) memories and permutation networks to access data in bit slices in a variety of ways, with high-speed I/O capabilities. Its router (called the flip network) could permute data among the array processors, or between the array processors and external devices, including disks, video input, and displays.

Keywords: parallel I/O, parallel architecture, simulation, pario-bib

Keywords: performance evaluation, parallel architecture, parallel I/O, pario-bib

Comment: They use a simulator to evaluate the performance of a parallel I/O system. They simulate the network and disks under a synthetic workload, and measure the time it takes for I/O requests to traverse the network, be processed, and return. They also measure the impact of I/O requests on non-I/O messages. Their results are fairly unsurprising.

Keywords: performance evaluation, parallel architecture, parallel I/O, pario-bib

Keywords: parallel I/O, parallel architecture, performance analysis, pario-bib

Comment: See polished version baylor:vulcan-perf-book. Simulation of the I/O architecture for the Vulcan MPP at IBM TJW. This is a distributed-memory MIMD system with a bidirectional omega-type interconnection network, and separate compute and I/O nodes. They use a stochastic workload to evaluate the average I/O performance under a few different situations, and then use that average performance, along with a stochastic workload, in a detailed simulation of the interconnection network. (What would be the effect of adding variance to the I/O-node performance?) A key point is that the I/O node will not accept any more requests until a current write request is finished being processed (copied into the write-back cache). If there are many writes, this can backup the network (would a different write-request protocol help?) Not clear how concurrency of reads are modeled. Results show that network saturates for high request rates and small number of I/O nodes. As request rate decreases or number of I/O nodes increases, performance levels off to a reasonable value. Placement of I/O nodes didn't make much difference, nor did extra non-I/O traffic. Given their parameters, and for reasonable loads, 1 I/O node per 4 compute nodes was a reasonable balance, and was scalable.

Abstract: Presented are the trace-driven simulation results of a study conducted to evaluate the performance of the internal parallel I/O subsystem of the Vulcan massively parallel processor (MPP) architecture. The system sizes evaluated vary from 16 to 512 nodes. The results show that a compute node to I/O node ratio of four is the most cost effective for all system sizes, suggesting high scalability. Also, processor-to-processor communication effects are negligible for small message sizes and the greater the fraction of I/O reads, the better the I/O performance. Worse case I/O node placement is within 13% of more efficient placement strategies. Introducing parallelism into the internal I/O subsystem improves I/O performance significantly.

Keywords: parallel I/O architecture, performance evaluation, pario-bib

Comment: Part of a whole book on parallel I/O; see iopads-book.

Abstract: In recent years, the design and performance evaluation of parallel processors has focused on the processor, memory and communication subsystems. As a result, these subsystems have better performance potential than the I/O subsystem. In fact, the I/O subsystem is the bottleneck in many machines. However, there are a number of studies currently underway to improve the design of parallel I/O subsystems. To develop optimal parallel I/O subsystem designs, one must have a thorough understanding of the workload characteristics of parallel I/O and its exploitation of the associated parallel file system. Presented are the results of a study conducted to analyze the parallel I/O workloads of several applications on a parallel processor using the Vesta parallel file system. Traces of the applications are obtained to collect system events, communication events, and parallel I/O events. The traces are then analyzed to determine workload characteristics. The results show I/O request rates on the order of hundreds of requests per second, a large majority of requests are for small amount of data (less than 1500 bytes), a few requests are for large amounts of data (on the order of megabytes), significant file sharing among processes within a job, and strong temporal, traditional spatial, and interprocess spatial locality.

Keywords: parallel I/O, workload characterization, pario-bib

Comment: See polished version baylor:workload-book. They characterize four parallel applications: sort, matrix multiply, seismic migration, and video server, in terms of their I/O activity. They found results that are consistent with kotz:workload, in that they also found lots of small data requests, some large data requests, significant file sharing and interprocess locality. This study found less of the non-contiguous access than did kotz:workload, because of the logical views provided by Vesta. Note on-line postscript does not include figures.

Abstract: To develop optimal parallel I/O subsystems, one must have a thorough understanding of the workload characteristics of parallel I/O and its exploitation of the associated parallel file system. Presented are the results of a study conducted to analyze the parallel I/O workloads of several applications on a parallel processor using the Vesta parallel file system. Traces of the applications are obtained to collect system events, communication events, and parallel I/O events. The traces are then analyzed to determine workload characteristics. The results show I/O request rates on the order of hundreds of requests per second, a large majority of requests are for small amounts of data (less than 1500 bytes), a few requests are for large amounts of data (on the order of megabytes), significant file sharing among processes within a job, and strong temporal, traditional spatial, and interprocess spatial locality.

Keywords: parallel I/O, file access pattern, workload characterization, file system workload, pario-bib

Comment: Part of a whole book on parallel I/O; see iopads-book.

Keywords: BBN, parallel I/O, pario-bib

Comment: Administrative manual for the TC2000 I/O system. Can stripe over partitions in a user-specified set of disks. Large requests automatically split and done in parallel. See also garber:tc2000.

Keywords: parallel I/O, scientific computing, linear algebra, pario-bib

Comment: They look at out-of-core block and slab solvers for the Maspar. They overlap reading one block with the computation of the previous block. They solve matrices up to 40k x 40k, and obtain 3.14 GFlops even with I/O considered.

Keywords: file access pattern, disk prefetch, file system, pario-bib

Comment: A specialized database system for particle physics codes. Valuable for its description of access patterns and subsequent file access requirements. Particle-in-cell codes iterate over timesteps, updating the position of each particle, and then the characteristics of each cell in the grid. Particles may move from cell to cell. Particle update needs itself and nearby gridcell data. The whole dataset is too big for memory, and each timestep must be stored on disk for later analysis anyway. Regular file systems are inadequate: specialized DBMS is more appropriate. Characteristics needed by their application class: multidimensional access (by particle type or by location, i.e., multiple views of the data), coordination between grid and particle data, coordination between processors, coordinated access to meta-data, inverted files, horizontal clustering, large blocking of data, asynchronous I/O, array data, complicated joins, and prefetching according to user-prespecified order. Note that many of these things can be provided by a file system, but that most are hard to come by in typical file systems, if not impossible. Many of these features are generalizable to other applications.

Keywords: hypercube, graphics, parallel algorithm, parallel I/O, pario-bib

Comment: About using the nCUBE/10's RT Graphics System. They were frustrated by an unusual mapping from the graphics memory to the display, a shortage of memory on the graphics nodes, and small message buffers on the graphics nodes. They wrote some algorithms for collecting the columns of pixels from the hypercube nodes, and routing them to the appropriate graphics node. They also would have liked a better interconnection network between the graphics nodes, at least for synchronization.

Keywords: parallel I/O, pario-bib

Comment: Jovian is a runtime library for use with SPMD codes, eg, HPF. They restrict IO to collective operations, and provide extra processes to 'coalesce' the many requests from multiple CPs into fewer larger requests to the operating system, perhaps optimized for access order. They mention that there is a standardization process underway for specifying data distributions. Also a compact representation for strided access to n-dimensional data structures. Coalescing basically means combining requests to eliminate duplication and to combine adjacent requests. Requests to coalescers are in full blocks, to lower the processing overhead. Nonetheless, their method involves moving requests around twice, and involve several memory-memory copies of the data, so their overhead is high.

Keywords: parallel I/O, network, supercomputer system, pario-bib

Comment: An update of berdahl:woodenman, close to the final draft.

Keywords: parallel I/O, network, supercomputer system, pario-bib

Comment: They describe a protocol for making parallel data transfers of arbitrary data sets from one set of data servers to another set of data servers. The goal is to be independent of specific architectures or even types of data servers, and to work on top of existing transport protocols. The data set is described using a gather set for the source and a scatter set for the destination, and using a linear address space as an intermediate representation. All the servers are contacted, they figure out who they need to talk, and exchange port information with them. Each pair exchanges votes on who will control the transfer (ie, who will control the order of the transfer), and on their maximum data rates. This information is used to settle on the control and set of ports to be used. This proposal is not final and is under active development, so it may change.

Keywords: parallel computing, performance evaluation, parallel file system, pario-bib

Comment: In German. They summarize typical performance of the Intel Paragon, including the communication performance and the parallel file-system performance.

Keywords: scientific computation, application, parallel I/O, pario-bib

Keywords: parallel I/O, multiprocessor file system, pario-bib

Comment: Four major components: networking, memory servers, file system, and persistent object store. Networking part focuses on low-latency support communication within an application, between applications, and between machines (Bershad and Peterson). Memory servers, shared virtual memory, and checkpointing support (Kai Li). File systems support includes benchmarking, transparent informed prefetching (Gibson), a common interface for PFS and Vesta (Snir), and integrating secondary and tertiary storage systems (including the integration of the National Storage Lab's HPSS (see coyne:hpss) into this project in 1995). OSF/1 (Black) will be extended to support parallel file systems, extent-like behavior, and block coalescing. Persistent object store (DeWitt) is radical change to an object-oriented interface, transparent I/O (though extensible and changable with subclassing, presumably), and heterogeneous support via the Object Definition Language standard. Persistent objects may be integrated with the memory servers and shared virtual memory. See also poole:sio-survey, bagrodia:sio-character, choudhary:sio-language.

Keywords: fault tolerance, multimedia, video on demand, parallel I/O, pario-bib

Keywords: parallel I/O, multiprocessor file system, pario-bib

Comment: Much like Intel CFS, with different I/O modes that determine when the compute nodes synchronize, and the semantics of I/Os written to the file. They found it hard to get good bandwidth for independent I/Os, as opposed to coordinated I/Os; part of this was due to their RAID 3 disk array, but it is more complicated than that. Some performance numbers were given in talk.

Keywords: RAID, disk array, reliability, parallel I/O, pario-bib

Comment: Uses the Information Dispersal Algorithm (IDA) to generate $n+m$ blocks from $n$ blocks, to tolerate $m$ disk failures; all of the data from the $n$ blocks is hidden in the $n+m$ blocks. Not with the RAID project.

Abstract: In wide area computing, programs frequently execute at sites that are distant from their data. Data access mechanisms are required that place limited functionality demands on an application or host system yet permit high-performance implementations. To address these requirements, we propose a data movement and access service called Global Access to Secondary Storage (GASS). This service defines a global name space via Uniform Resource Locators and allows applications to access remote files via standard I/O interfaces. High performance is achieved by incorporating default data movement strategies that are specialized for I/O patterns common in wide area applications and by providing support for programmer management of data movement. GASS forms part of the Globus toolkit, a set of services for high-performance distributed computing. GASS itself makes use of Globus services for security and communication, and other Globus components use GASS services for executable staging and real-time remote monitoring. Application experiences demonstrate that the library has practical utility.

Keywords: wide-area network, parallel I/O, pario-bib

Keywords: data grid, filter, pario-bib

Keywords: parallel I/O, disk shadowing, reliability, disk mirroring, disk optimization, pario-bib

Comment: Goes further than bitton:shadow. Uses simulation to verify results from that paper, which were expressions for the expected seek distance of shadowed disks, using shortest-seek-time arm scheduling. Problem is her assumption that arm positions stay independent, in the face of correlating effects like writes, which move all arms to the same place. Simulations match model only barely, and only in some cases. Anyway, shadowed disks can improve performance for workloads more than 60 or 70% reads.

Keywords: parallel I/O, disk shadowing, reliability, disk mirroring, disk optimization, pario-bib

Comment: Also TR UIC EECS 88-1 from Univ of Illinois at Chicago. Shadowed disks are mirroring with more than 2 disks. Writes to all disks, reads from one with shortest seek time. Acknowledges but ignores problem posed by lo:disks. Also considers that newer disk technology does not have linear seek time $(a+bx)$ but rather $(a+b\sqrt{x})$. Shows that with either seek distribution the average seek time for workloads with at least 60% reads decreases in the number of disks. See also bitton:schedule.

Keywords: parallel I/O, file access pattern, pario-bib

Comment: A substantial part of this structural-analysis application was involved in I/O, moving substructures in and out of RAM. The Maspar IO-RAM helped a lot, nearly halving the time required. On the Cray, the SSD had an even bigger impact, perhaps 7-12 times faster. Their main conclusion is that caching helped. Most likely this was due to its double-buffering, since they structured the code to read/compute/write in large ``superblocks''.

Keywords: disk array, RAID, parallel I/O, pario-bib

Comment: Part of jin:io-book.

Abstract: There is a growing interest in using Java as the language for developing high-performance computing applications. To be successful in the high-performance computing domain, however, Java must not only be able to provide high computational performance, but also high-performance I/O. In this paper, we first examine several approaches that attempt to provide high-performance I/O in Java-many of which are not obvious at first glance-and evaluate their performance on two parallel machines, the IBM SP and the SGI Origin2000. We then propose extensions to the Java I/O library that address the deficiencies in the Java I/O API and improve performance dramatically. The extensions add bulk (array) I/O operations to Java, thereby removing much of the overhead currently associated with array I/O in Java. We have implemented the extensions in two ways: in a standard JVM using the Java Native Interface (JNI) and in a high-performance parallel dialect of Java called Titanium. We describe the two implementations and present performance results that demonstrate the benefits of the proposed extensions.

Keywords: parallel I/O, Java, file system interface, pario-bib

Abstract: There is a growing interest in using Java as the language for developing high-performance computing applications. To be successful in the high-performance computing domain, however, Java must not only be able to provide high computational performance, but also high-performance I/O. In this paper, we first examine several approaches that attempt to provide high-performance I/O in Java-many of which are not obvious at first glance-and evaluate their performance on two parallel machines, the IBM SP and the SGI Origin2000. We then propose extensions to the Java I/O library that address the deficiencies in the Java I/O API and improve performance dramatically. The extensions add bulk (array) I/O operations to Java, thereby removing much of the overhead currently associated with array I/O in Java. We have implemented the extensions in two ways: in a standard JVM using the Java Native Interface (JNI) and in a high-performance parallel dialect of Java called Titanium. We describe the two implementations and present performance results that demonstrate the benefits of the proposed extensions.

Keywords: parallel I/O, java, file system interface, pario-bib

Keywords: parallel I/O, database, disk caching, pario-bib

Comment: More recent than copeland:bubba, and a little more general. This gives few details, and doesn't spend much time on the parallel I/O. Bubba does use parallel independent disks, with a significant effort to place data on the disks, and do the work local to the disks, to balance the load and minimize interprocessor communication. Also they use a single-level store (i.e., memory-mapped files) to improve performance of their I/O system, including page locking that is assisted by the MMU. The OS has hooks for the database manager to give memory-management policy hints.

Keywords: file access pattern, parallel I/O, database machine, pario-bib

Comment: Improvements in I/O bandwidth crucial for supporting database machines, otherwise highly parallel DB machines are useless (I/O bound). Two ways to do it: 1) synchronized interleaving by using custom controller and regular disks to read/write same track on all disks, which speeds individual accesses. 2) use very large cache (100-200M) to keep blocks to re-use and to do prefetching. But see dewitt:pardbs.

Keywords: collective I/O, multiprocessor file system, parallel I/O, pario-bib

Comment: bordawekar:collective was renamed bordawekar:collective-tr, so this could be called bordawekar:collective.

Abstract: A majority of parallel applications obtain parallelism by partitioning data over multiple processors. Accessing distributed data structures like arrays from files often requires each processor to make a large number of small non-contiguous data requests. This problem can be addressed by replacing small non-contiguous requests by large collective requests. This approach, known as Collective I/O, has been found to work extremely well in practice. In this paper, we describe implementation and evaluation of a collective I/O prototype in a production parallel file system on the Intel Paragon. The prototype is implemented in the PFS subsystem of the Intel Paragon Operating System. We evaluate the collective I/O performance using its comparison with the PFS M_RECORD and M_UNIX I/O modes. It is observed that collective I/O provides significant performance improvement over accesses in M_UNIX mode. However, in many cases, various implementation overheads cause collective I/O to provide lower performance than the M_RECORD I/O mode.

Keywords: parallel I/O, mutliprocessor file system, pario-bib

Comment: This tech report was called bordawekar:collective, then renamed bordawekar:collective-tr, on the appearance of the ICS paper bordawekar:collective.

Keywords: parallel I/O, inter-processor communication, pario-bib

Comment: bordawekar:comm-tr is nearly identical in content. Also bordawekar:commstrat is a shorter version.

Abstract: In this paper, we show that communication in the out-of-core distributed memory problems requires both inter-processor communication and file I/O. Given that primary data structures reside in files, even communication requires I/O. Thus, it is important to optimize the I/O costs associated with a communication step. We present three methods for performing communication in out-of-core distributed memory problems. The first method, termed as the "out-of-core" communication method, follows a loosely synchronous model. Computation and Communication phases in this case are clearly separated, and communication requires permutation of data in files. The second method, termed as "demand-driven-in-core communication" considers only communication required of each in-core data slab individually. The third method, termed as "producer-driven-in-core communication" goes even one step further and tries to identify the potential (future) use of data while it is in memory. We describe these methods in detail and provide performance results for out-of-core applications; namely, two-dimensional FFT and two-dimensional elliptic solver. Finally, we discuss how "out-of-core" and "in-core" communication methods could be used in virtual memory environments on distributed memory machines.

Keywords: parallel I/O, inter-processor communication, pario-bib

Comment: They compare different ways to do global communications in out-of-core applications, involving file I/O and communication at different times. They also comment briefly on how it would work if it depended on virtual memory at each node.

Keywords: interprocessor communication, parallel I/O, pario-bib

Comment: Small version of bordawekar:comm.

Abstract: It is widely acknowledged that improving parallel I/O performance is critical for widespread adoption of high performance computing. In this paper, we show that communication in out-of-core distributed memory problems may require both inter-processor communication and file I/O. Thus, in order to improve I/O performance, it is necessary to minimize the I/O costs associated with a communication step. We present three methods for performing communication in out-of-core distributed memory problems. The first method called the generalized collective communication method follows a loosely synchronous model; computation and communication phases are clearly separated, and communication requires permutation of data in files. The second method called the receiver-driven in-core communication considers only communication required of each in-core data slab individually. The third method called the owner-driven in-core communication goes even one step further and tries to identify the potential future use of data (by the recipients) while it is in the sender's memory. We describe these methods in detail and present a simple heuristic to choose a communication method from among the three methods. We then provide performance results for two out-of-core applications, the two-dimensional FFT code and the two-dimensional elliptic Jacobi solver. Finally, we discuss how the out-of-core and in-core communication methods can be used in virtual memory environments on distributed memory machines.

Keywords: compiler, communication, out-of-core, parallel I/O, inter-processor communication, pario-bib

Abstract: It is widely acknowledged that improving parallel I/O performance is critical for widespread adoption of high performance computing. In this paper, we show that communication in out-of-core distributed memory problems may require both inter-processor communication and file I/O. Thus, in order to improve I/O performance, it is necessary to minimize the I/O costs associated with a communication step. We present three methods for performing communication in out-of-core distributed memory problems. The first method called the generalized collective communication method follows a loosely synchronous model; computation and communication phases are clearly separated, and communication requires permutation of data in files. The second method called the receiver-driven in-core communication considers only communication required of each in-core data slab individually. The third method called the owner-driven in-core communication goes even one step further and tries to identify the potential future use of data (by the recipients) while it is in the sender's memory. We describe these methods in detail and present a simple heuristic to choose a communication method from among the three methods. We then provide performance results for two out-of-core applications, the two-dimensional FFT code and the two-dimensional elliptic Jacobi solver. Finally, we discuss how the out-of-core and in-core communication methods can be used in virtual memory environments on distributed memory machines.

Keywords: out-of-core, compiler, communication, distributed memory, parallel I/O, pario-bib

Comment: See also bordawekar:comm, at ICS'95.

Abstract: None.

Keywords: parallel I/O, compiler, out-of-core, pario-bib

Comment: Part of a whole book on parallel I/O; see iopads-book.

Abstract: This article presents a case for compositional file systems (CFSs). The CFS is designed using the end-to-end argument; the basic file system attributes, therefore, are independent of the user requirements. The CFS is designed as a functionally compositional, structurally distributed, and dynamically extendable file system. The article also discusses the advantages and implementation alternatives for these file systems, and outlines possible applications.

Keywords: parallel I/O, multiprocessor file system, pario-bib

Abstract: For a high-performance parallel machine to be a scalable system, it must also have a scalable parallel I/O system. Recently, several commercial machines (e.g. Intel Touchstone Delta, Paragon, CM-5, Ncube-2) have been built that provide features for parallel I/O. However, very little is understood about the performance of these I/O systems. This paper presents an experimental evaluation of the Intel Touchstone Delta's Concurrent File System (CFS). The CFS utilizes the declustering of large files across the disks to improve the I/O performance. Data files can be read or written on the CFS using 4 access modes.

We present performance measurements for the CFS on the Touchstone Delta with 512 compute nodes and 32 I/O nodes. The study focuses on file read/write rates for various configurations of I/O and compute nodes. The study attempts to show the effect of access modes, buffer sizes and volume restrictions on the system performance. The paper also shows that the performance of the CFS can greatly vary for various data distributions commonly employed in scientific and engineering applications.

Keywords: performance evaluation, multiprocessor file system, parallel I/O, pario-bib

Comment: Some new numbers over bordawekar:delta-fs-TR, but basically the same conclusions.

Keywords: performance evaluation, multiprocessor file system, parallel I/O, pario-bib

Comment: Evaluating the Caltech Touchstone Delta (512 nodes, 32 I/O nodes, 64 disks, 8 MB cache per I/O node). Basic measurements of different access patterns and I/O modes. Location in network doesn't seem to matter. Throughput is often limited by the software; at least, the full hardware throughputs are rarely obtained. Sometimes they are compnode-limited, and other times they may be being limited by the cache management. There must be a way to push bottleneck back to the disks .

Abstract: Large scale scientific applications, such as the Grand Challenge applications, deal with very large quantities of data. The amount of main memory in distributed memory machines is usually not large enough to solve problems of realistic size. This limitation results in the need for system and application software support to provide efficient parallel I/O for out-of-core programs. This paper describes techniques for translating out-of-core programs written in a data parallel language like HPF to message passing node programs with explicit parallel I/O. We describe the basic compilation model and various steps involved in the compilation. The compilation process is explained with the help of an out-of-core matrix multiplication program. We first discuss how an out-of-core program can be translated by extending the method used for translating in-core programs. We then describe how the compiler can optimize the code by estimating the I/O costs associated with different array access patterns and selecting the method with the least I/O cost. This optimization can reduce the amount of I/O by as much as an order of magnitude. Performance results on the Intel Touchstone Delta are presented and analyzed.

Keywords: parallel I/O, compiler, pario-bib

Comment: Revised as bordawekar: This is actually fairly different from thakur:runtime. They describe the same basic compiler technique, where arrays are distributed across processors, and each processor has a local array file for holding data from its local partitions. Then the I/O needed for a loop is broken into slabs, where the program proceeds as an alternation of (read slabs, compute, write slabs). The big new thing here is that the compiler tries different ways to form slabs (e.g., by row or by column), estimates the number of I/Os and the amount of data moved for each case, and chooses the case with the smallest amount of I/O. They also mention how the choice of memory size allocated to different arrays affects the amount of IO, but give no algorithm other than "try all the possibilities."

Keywords: multiprocessor file system, performance evaluation, parallel I/O, pario-bib

Comment: Part of a special issue on parallel and distributed I/O.

Abstract: This paper presents a unified evaluation of the I/O behavior of a commercial clustered DSM machine, the HP Exemplar. Our study has the following objectives: (1) To evaluate the impact of different interacting system components, namely, architecture, operating system, and programming model, on the overall I/O behavior and identify possible performance bottlenecks and (2) To provide hints to the users for achieving high out-of-box I/O throughput. We find that for the DSM machines that are built as a cluster of SMP nodes, integrated clustering of computing and I/O resources, both hardware and software, is not advantageous for two reasons. First, within an SMP node, the I/O bandwidth is often restricted by the performance of the peripheral components and cannot match the memory bandwidth. Second, since the I/O resources are shared as a global resource, the file-access costs become non-uniform and the I/O behavior of the entire system, in terms of the scalability and balance, degrades. We observe that the buffered I/O performance is determined not only by the I/O subsystem, but also by the programming model, global-shared memory subsystem, and data-communication mechanism. Moreover, programming-model support can be effectively used to overcome the performance constraints created by the architecture and operating system. For example, on the HP Exemplar, users can achieve high I/O throughput by using features of the programming model that balance the sharing and locality of the user buffers and file systems. Finally, we believe that at present, the I/O subsystems are being designed in isolation and there is a need for mending the traditional memory-oriented design approach to address this problem.

Keywords: parallel I/O, pario-bib, workload characterization, distributed shared memory

Keywords: data parallel, parallel I/O, pario-bib

Comment: Although this is mostly a compilers paper, there is a little bit about parallel I/O here. They comment briefly on how their compiler framework will help them make a compiler that can provide advice to the file system about prefetching and cache replacement, and to decide on the layout of scratch files to optimize locality.

Keywords: parallel I/O, pario-bib

Comment: They propose some extensions to HPF to accomodate parallel I/O.

Abstract: This report presents implementation details of the prototype PASSION compiler. The PASSION compiler provides support for: (1) Accessing multidimensional in-core arrays and (2) Out-of-core computations. The PASSION compiler takes as input an annotated I/O intensive (either an out-of-core program or program accessing distributed arrays from files) High Performance Fortran (HPF) program. Using hints provided by the user, the compiler modifies the computation so as to minimize the I/O cost and restructures the program to incorporate explicit I/O calls. In this report, compilation of out-of-core FORALL constructs is illustrated using representative programs. Compiler support for accessing distributed in-core data is explained using illustrative examples and supplemented by experimental results.

Keywords: parallel I/O, compiler, FORTRAN, HPF, pario-bib

Comment: Currently not available on WWW. Describes implementation details of the PASSION Compiler.

Abstract: This paper presents a unified evaluation of the I/O behavior of a commercial clustered DSM machine, the HP Exemplar. Our study has the following objectives: 1) To evaluate the impact of different interacting system components, namely, architecture, operating system, and programming model, on the overall I/O behavior and identify possible performance bottlenecks, and 2) To provide hints to the users for achieving high out-of-box I/O throughput. We find that for the DSM machines that are built as a cluster of SMP nodes, integrated clustering of computing and I/O resources, both hardware and software, is not advantageous for two reasons. First, within an SMP node, the I/O bandwidth is often restricted by the performance of the peripheral components and cannot match the memory bandwidth. Second, since the I/O resources are shared as a global resource, the file-access costs become nonuniform and the I/O behavior of the entire system, in terms of both scalability and balance, degrades.

We observe that the buffered I/O performance is determined not only by the I/O subsystem, but also by the programming model, global-shared memory subsystem, and data-communication mechanism. Moreover, programming-model support can be used effectively to overcome the performance constraints created by the architecture and operating system. For example, on the HP Exemplar, users can achieve high I/O throughput by using features of the programming model that balance the sharing and locality of the user buffers and file systems. Finally, we believe that at present, the I/O subsystems are being designed in isolation, and there is a need for mending the traditional memory-oriented design approach to address this problem.

Keywords: parallel I/O, pario-bib, workload characterization, distributed shared memory

Abstract: It is widely acknowledged in high-performance computing circles that parallel input/output needs substantial improvement in order to make scalable computers truly usable. We present a data storage model that allows processors independent access to their own data and a corresponding compilation strategy that integrates data-parallel computation with data distribution for out-of-core problems. Our results compare several communication methods and I/O optimizations using two out-of-core problems, Jacobi iteration and LU factorization.

Keywords: parallel I/O, compiler, pario-bib

Keywords: compilers, parallel I/O, out-of-core applications, pario-bib

Comment: Basically a summary of their I/O and compilation model for out-of-core compilation of HPF programs. See also paleczny:support.

Abstract: This thesis looks at various issues in providing application-level software support for parallel I/O. We show that the performance of the parallel I/O system varies greatly as a function of data distributions. We present runtime I/O primitives for parallel languages which allow the user to obtain a consistent performance over a wide range of data distributions.

In order to design these primitives, we study various parameters used in the design of a parallel file system. We evaluate the performance of Touchstone Delta Concurrent File System and study the effect of parameters like number of processors, number of disks, file size on the system performance. We compute the I/O costs for common data distributions. We propose an alternative strategy -two phase data access strategy- to optimize the I/O costs connected with data distributions. We implement runtime primitives using the two-phase access strategy and show that using these primitives not only I/O access rates are improved but also user can obtain complex data distributions like block-block and block-cyclic.

Keywords: parallel I/O, pario-bib

Comment: This is basically a consolidation of the other bordawekar papers, in more detail. So he covers an experimental analysis of the touchstone delta; of the problems arising from the direct-access model for non-conforming distributions; of the two-phase model; and of the run-time library to support two-phase access. See also bordawekar:reorganize, thakur:runtime, bordawekar:efficient, thakur:out-of-core, delrosario:two-phase, bordawekar:primitives, bordawekar:delta-fs.

Abstract: This paper describes a framework for analyzing dataflow within an out-of-core parallel program. Dataflow properties of FORALL statement are analyzed and a unified I/O and communication placement framework is presented. This placement framework can be applied to many problems, which include eliminating redudant I/O incurred in communication. The framework is validated by applying it for optimizing I/O and communication in out-of-core stencil problems. Experimental performance results on an Intel Paragon show significant reduction in I/O and communication overhead.

Keywords: parallel I/O, compiler, pario-bib

Abstract: In this paper, we describe a framework for optimizing communication and I/O costs in out-of-core problems. We focus on communication and I/O optimization within a FORALL construct. We show that existing frameworks do not extend directly to out-of-core problems and can not exploit the FORALL semantics. We present a unified framework for the placement of I/O and communication calls and apply it for optimizing communication for stencil applications. Using the experimental results, we demonstrate that correct placement of I/O and communication calls can completely eliminate extra file I/O from communication and obtain significant performance improvement.

Keywords: parallel I/O, compiler, pario-bib

Abstract: In this paper, we show that the performance of parallel file systems can vary greatly as a function of the selected data distributions, and that some data distributions can not be supported. Also, we describe how the parallel language extensions, though simplifying the programming, do not address the performance problems found in parallel file systems.

We have devised an alternative scheme for conducting parallel I/O - the Two-Phase Access Strategy - which guarantees higher and more consistent performance over a wider spectrum of data distributions. We have designed and implemented runtime primitives that make use of the two-phase access strategy to conduct parallel I/O, and facilitate the programming of parallel I/O operations. We describe these primitives in detail and provide performance results which show that I/O access rates are improved by up to several orders of magnitude. Further, we show that the variation in performance over various data distributions is restricted to within a factor of 2 of the best access rate.

Keywords: parallel I/O, pario-bib

Comment: Much of this is the same as delrosario:two-phase, except for section 4 where they describe their actual run-time library of primitives, with a little bit about how it works. It's not clear, for example, how their meta-data structures are distributed across the machine. They also do not describe their methods for the data redistribution.

Keywords: parallel I/O, compilation, pario-bib

Comment: Basically they give a case study of out-of-core matrix multiplication to emphasize that the compiler's choice of loop ordering and matrix distribution for in-core matmult is not a very good choice for out-of-core matmult, because it causes too much I/O. By reorganizing the data and the loops, they get much better performance. In this particular case there are known algorithms which they should have used. In general they make the point that the compiler should consider several organizations, and estimate their costs, before generating code. They don't propose anything more sophisticated than to try all the possible organizations.

Abstract: In this paper, we describe a technique for optimizing commununication for out-of-core distributed memory stencil problems. In these problems, communication may require both inter-processor communication and file I/O. We show that in certain cases, extra file I/O incurred in communication can be completely eliminated by reordering in-core computations. The in-core computation pattern is decided by: (1) how the out-of-core data distributed into in-core slabs (tiling) and (2) how the slabs are accessed. We show that a compiler using the stencil and processor information can choose the tiling parameters and schedule the tile accesses so that the extra file I/O is eliminated and overall performance is improved.

Keywords: compiler, parallel I/O, pario-bib

Abstract: In this paper, we describe a technique for optimizing commununication for out-of-core distributed memory stencil problems. In these problems, communication may require both inter-processor communication and file I/O. We show that in certain cases, extra file I/O incurred in communication can be completely eliminated by reordering in-core computations. The in-core computation pattern is decided by: (1) how the out-of-core data distributed into in-core slabs (tiling) and (2) how the slabs are accessed. We show that a compiler using the stencil and processor information can choose the tiling parameters and schedule the tile accesses so that the extra file I/O is eliminated and overall performance is improved.

Keywords: compiler, parallel I/O, pario-bib

Keywords: parallel I/O, pario-bib

Comment: Contains much of the material from bordawekar:hpf.

Abstract: This dissertation investigates several issues in providing compiler support for I/O intensive parallel programs. In this dissertation, we focus on satisfying two I/O requirements, namely, support for accessing multidimensional arrays and support for out-of-core computations. We analyze working spaces in I/O intensive programs and propose three execution models to be used by users or compilers for developing efficient I/O intensive parallel programs. Different phases in compiling out-of-core parallel programs are then described. Three different methods for performing communication are presented and validated using representative application templates.We illustrate that communication in out-of-core programs may require both inter-processor communication and file I/O. We show that using the copy-in-copy-out semantics of the HPF FORALL construct, extra file I/O incurred in communication can be completely eliminated by reordering in-core computations. Two different approaches for reordering in-core computations are presented, namely, integrated tiling and scheduling heuristic, and dataflow framework for placing communication and I/O calls. The discussion is supplemented with experimental performance results of representative stencil applications. Finally, an overview of the prototype \textsf{PASSION} (Parallel And Scalable Software for I/O) compiler is presented. This compiler takes an annotated out-of-core High Performance Fortran (HPF) program as input and generates the corresponding node+message-passing program with calls to the parallel I/O runtime library. We illustrate various functionalities of the compiler using example programs and supplement them by experimental results.

Keywords: parallel I/O, compiler, HPF, pario-bib

Keywords: parallel I/O, distributed memory, pario-bib

Comment: In a sense, this is about a two-phase technique for network I/O. They consider the problem of feeding a fast network interface (HIPPI) from a distributed-memory parallel machine (iWARP) in which the individual internal links are slower than the external network. So they get the processors to cooperate to reshuffle the data into a canonical layout that is convenient to send to the gateway node, and from there onto the external network.

Keywords: object-based storage, distributed file system, parallel file system, pario-bib

Comment: Describes an open-source project to develop an object-based file system for clusters. Related to the NASD project at CMU (http://www.pdl.cs.cmu.edu/NASD/).

Keywords: hypercube, parallel I/O, Intel, pario-bib

Comment: Some measurements and simulations of early CFS performance. Looks terrible, but they disclaim that it is a beta version of the first CFS. They determined that the disks are the bottleneck. But this may just imply that they need more disks. Their parallel synthetic applications had each process read a separate file. CFS had ridiculous traffic overhead. Again, this was beta CFS.

Keywords: parallel I/O, disk caching, disk architecture, pario-bib

Comment: Some new DASD products with caches overlap cache hits with prefetch of remainder of track into cache. They use analytical model to evaluate performance of these. They find performance improvements of 5-15 percent under their assumptions.

Keywords: supercomputing, fortran, multiprocessor file system interface, pario-bib

Comment: Describing their way of writing arrays to files so that they are written in a fast, parallel way, and so that (if read in same distribution) they can be read fast and parallel. Normal read and write forces standard ordering, but cread and cwrite uses a compiler and runtime selected ordering, which is stored in the file so it can be used when rereading. Good for temp files.

Abstract: Recently several projects have started to implement large-scale high-performance computing on "computational grids" composed of heterogeneous and geographically distributed systems of computers, networks, and storage devices that collectively act as a single "virtual supercomuter". One of the great challenges for this environment is to provide appropriate high-level programming models. High Performance Fortran (HPF) is a language of choice for development of data parallel components of Grid applications. Another challenge is to provide efficient access to data that is distributed across local and remote Grid resources. In this paper, constructs to specify parallel input and output (I/O) operations on multidimensional arrays on the Grid in the context of HPF are proposed. The paper also presents implementation concepts that are based on the HPF compiler VFC, the parallel I/O runtime system Panda, Internet, and Grid technologies. Preliminary experimental performance results are discussed in the context of a real application example.

Keywords: parallel I/O, Fortran, HPF, data-parallel, computational grid, pario-bib

Abstract: For an increasing number of data intensive scientific applications, parallel I/O concepts are a major performance issue. Tackling this issue, we provide an outline of an input/output system designed for highly efficient, scalable and conveniently usable parallel I/O on distributed memory systems. The main focus of this paper is the parallel I/O runtime system support provided for software-generated programs produced by parallelizing compilers in the context of High Performance FORTRAN efforts. Specifically, our design is presented in the context of the Vienna Fortran Compilation System.

Keywords: compiler transformations, runtime support, parallel I/O, prefetching, pario-bib

Keywords: parallel I/O, high performance mass storage system, high performance languages, compilation techniques, data administration, pario-bib

Keywords: compiler transformations, runtime support, declustering, parallel I/O, pario-bib

Comment: They describe some extensions to Vienna Fortran that support parallel I/O, and how they plan to extend the compiler and run-time system to help. They are somewhat short on details, however. The basic idea is that file declustering is based on hints from the compiler or programmer about how the file will be used, eg, as a matrix distributed in thus-and-so way.

Abstract: Large scale irregular applications involve data arrays and other data structures that are too large to fit in main memory and hence reside on disks; such applications are called out-of-core applications. This paper presents techniques for implementing this kind of applications. In particular we present a design for a runtime system to efficiently support parallel execution of irregular out-of-core codes on distributed-memory systems. Furthermore, we describe the appropriate program transformations required to reduce the I/O overheads for staging data as well as for communication while maintaining load balance. The proposed techniques can be used by a parallelizing compiler or by users writing programs in node + message passing style. We have done a preliminary implementation of the techniques presented here. We introduce experimental results from a template CFD code to demonstrate the efficacy of the presented techniques.

Keywords: parallel I/O, out of core, compiler, library, pario-bib

Comment: The authors present techniques for implementing large scale irregular out-of-core applications. The techniques they describe can be used by a parallel compiler (e.g., HPF and its extensions) or by users using message passing. The objectives of the proposed techniques are to ''to minimize I/O accesses in all steps while maintaining load balance and minimal communication''. They demonstrate the effectivness of their techniques by showing results from a Computational Fluid Dynamics (CFD) code.

Keywords: parallel I/O, out of core, irregular applications, compiler, pario-bib

Abstract: Input and output (I/O) is a major performance bottleneck for large-scale scientific applications running on parallel platforms. For example, it is not uncommon that performance of carefully tuned parallel programs can slow dramatically when they read or write files. This is because many parallel applications need to access large amounts of data, and although great advances have been made in the CPU and communication performance of parallel machines, similar advances have not been made in their I/O performance. The densities and capacities of disks have increased significantly, but improvement in performance of individual disks has not followed the same pace. For parallel computers to be truly usable for solving real, large-scale problems, the I/O performance must be scalable and balanced with respect to the CPU and communication performance of the system. Parallel I/O techniques can help to solve this problem by creating multiple data paths between memory and disks. However, simply adding disk drives to an I/O system without considering the overall software design will improve performance only marginally.

Keywords: pario-bib, parallel I/O

Keywords: distributed file system, pario-bib

Comment: See broom:acacia-tr. See also broom:impl, lautenbach:pfs, mutisya:cache, and broom:cap.

Keywords: distributed file system, pario-bib

Comment: This paper is not specifically about parallel I/O, but the file system will be used in the AP-1000 multiprocessor. Acacia is a file server that is optimized for synchronous writes, like those used in stateless protocols (eg, NFS). It writes inodes in blocks in any free location that is close to the current head position, using indirect inode blocks to track those. Indirect blocks are in turn written anywhere convenient, and their positions are tracked by the superblock. There is one slot in each cylinder reserved for the superblock, which is timestamped. They get good performance but claim to need a better implementation, and a faster allocation algorithm. No indication of effect on read performance.

Keywords: distributed file system, multiprocessor file system, pario-bib

Comment: See also broom:acacia, broom:impl, lautenbach:pfs, and mutisya:cache. This describes the semantic model for their file system. Modelled a lot after Amoeba, they have capabilities that represent immutable files. There are create, destroy, read, and write operations, but the read and write can affect only part of the file, if desired. They also have an atomic ``copy'' operation, which creates a snapshot of the current state of the file. They also have ``spawn'' and ``merge'' operations, which are essentially begin and end a transaction, a set of changes that are atomically merged into the file later. These seem to be addressing issues of concurrency more than of parallelism. They also discuss implementation somewhat, mentioning the use of distributed caches and log-structured disk layout. Prototype in Linda (!).

Keywords: distributed file system, multiprocessor file system, pario-bib

Comment: See also broom:acacia, lautenbach:pfs, mutisya:cache, and broom:cap. This paper is a very sketchy overview of those; it is better to read them.

Abstract: To ameliorate the need to spend significant programmer time modifying parallel programs to achieve high-performance, while maintaining compact, comprehensible source codes, the paper advocates the use of telescoping language technology to automatically apply, during the normal compilation process, high-level performance enhancing transformations to applications using a high-level domain-specific I/O library. We believe that this approach will be more acceptable to application developers than new language extensions, but will be just as amenable to optimization by advanced compilers, effectively making it a domain-specific language extension for I/O. The paper describes a domain-specific I/O library for irregular block-structured applications based on the KeLP library, describes high-level transformations of the library primitives for improving performance, and describes how a high-level domain-specific optimizer for applying these transformations could be constructed rising the telescoping languages framework.

Keywords: parallel I/O, domain-specific I/O library, scientific computing, astronomy, pario-bib

Keywords: parallel I/O, multiprocessor file system, pario-bib

Comment: They evaluate the performance of Acacia with some simple synthetic benchmarks. Performance limited by implementation problems in the sequential file system. Otherwise no real surprises.

Abstract: Benchmarks have historically played a key role in guiding the progress of computer science systems research and development, but have traditionally neglected the areas of availability, maintainability, and evolutionary growth, areas that have recently become critically important in high-end system design. As a first step in addressing this deficiency, we introduce a general methodology for benchmarking the availability of computer systems. Our methodology uses fault injection to provoke situations where availability may be compromised, leverages existing performance benchmarks for workload generation and data collection, and can produce results in both detail-rich graphical presentations or in distilled numerical summaries. We apply the methodology to measure the availability of the software RAID systems shipped with Linux, Solaris 7 Server, and Windows 2000 Server, and find that the methodology is powerful enough not only to quantify the impact of various failure conditions on the availability of these systems, but also to unearth their design philosophies with respect to transient errors and recovery policy.

Keywords: RAID, disk array, parallel I/O, pario-bib

Keywords: parallel I/O, disk caching, database, pario-bib

Comment: A fancy interconnection from procs to I/O processors, intended mostly for DB applications, that uses cache at I/O end and a switch with smarts. Cache is associative. Switch helps out in sort and join operations.

Keywords: parallel programming, parallel I/O, pario-bib

Comment: An overview of the CHIMP message-passing library and the PUL set of libraries. Key design goal is portability; they run on many systems. PUL includes PUL-GF, which supports parallel access to files (see chapple:pulgf, chapple:pulgf-adv, and chapple:pario). Other PUL libraries support grids and meshes, global communications, and task farms. Contact pul@epcc.ed.ac.uk.

Abstract: The concept of block-cyclic order elimination can be applied to out-of-core $LU$ and $QR$ matrix factorizations on distributed memory architectures equipped with a parallel I/O system. This elimination scheme provides load balanced computation in both the factor and solve phases and further optimizes the use of the network bandwidth to perform I/O operations. Stability of LU factorization is enforced by full column pivoting. Performance results are presented for the Connection Machine system CM-5.

Keywords: parallel I/O, linear algebra, out-of-core, pario-bib

Comment: Short, not many details. Performance results shows about 3.5 Gflops for all problem sizes, both in-core on small N and out-of-core on large N.

Keywords: parallel I/O, disk striping, distributed file system, pario-bib

Comment: See cabrera:swift, cabrera:swift2. Describes the performance of a Swift prototype and simulation results. They stripe data over multiple disk servers (here SPARC SLC with local disk), and access it from a SPARC2 client. Their prototype gets nearly linear speedup for reads and asynchronous writes; synchronous writes are slower. They hit the limit of the Ethernet and/or the client processor with three disk servers. Adding another Ethernet allowed them to go higher. Simulation shows good scaling. Seems like a smarter implementation would help, as would special- purpose parity-computation hardware. Good arguments for use of PID instead of RAID, to avoid a centralized controller that is both a bottleneck and a single point of failure.

Keywords: parallel I/O, disk striping, distributed file system, pario-bib

Keywords: striping, parallel I/O, distributed system, pario-bib

Comment: See cabrera:swift2, cabrera:swift, cabrera:pario. Not much new here. Simulates higher-performance architectures. Shows reasonable scalability. Counts 5 inst/byte for parity computation.

Keywords: parallel I/O, disk striping, distributed file system, multimedia, pario-bib

Comment: See cabrera:swift2. A brief outline of a design for a high-performance storage system, designed for storing and retrieving large objects like color video or visualization data at very high speed. They distribute data over several ``storage agents'', which are some form of disk or RAID. They are all connected by a high-speed network. A ``storage manager'' decides where to spread each file, what kind of reliability mechanism is used. User provides preallocation info such as size, reliability level, data rate requirements, and so forth.

Keywords: parallel I/O, disk striping, distributed file system, multimedia, pario-bib

Comment: See also cabrera:swift. More detail than the other paper. Experimental results from a prototype that stripes files across a distributed file system. Gets almost linear speedup in certain cases. Much better than NFS. Simulation to extend it to larger systems.

Abstract: This paper describes an implementation of MPI-IO using aB new parallel file system, called Expand (Expandable Parallel File System)1, that is based on NFS servers. Expand combines multiple NFS servers to create a distributed partition where files are declustered. Expand requires no changes to the NFS server and uses RPC operations to provide parallel access to the same file. Expand is also independent of the clients, because all operations are implemented using RPC and NFS protocol. The paper describes the design, the implementation and the evaluation of Expand with MPI-IO. This evaluation has been made in Linux clusters and compares Expand and PVFS.

Keywords: parallel I/O, multiprocessor file system, NFS, pario-bib

Abstract: Applications that explore, query, analyze, visualize, and, in general, process very large scale data sets are known as Data Intensive Applications. Large scale data intensive computing plays an increasingly important role in many scientific activities and commercial applications, whether it involves data mining of commercial transactions, experimental data analysis and visualization, or intensive simulation such as climate modeling. By combining high performance computation, very large data storage, high bandwidth access, and high- speed local and wide area networking, data intensive computing enhances the technical capabilities and usefulness of most systems. The integration of parallel and distributed computational environments will produce major improvements in performance for both computing intensive and data intensive applications in the future. The purpose of this introductory article is to provide an overview of the main issues in parallel data intensive computing in scientific and commercial applications and to encourage the reader to go into the more in-depth articles later in this special issue.

Keywords: parallel application, parallel I/O, pario-bib

Keywords: parallel I/O, RAID, pario-bib

Comment: See cao:tickertaip-tr2.

Keywords: parallel I/O, RAID, pario-bib

Comment: Superceded by cao:tickertaip-tr2 and cao:jtickertaip.

Keywords: parallel I/O, RAID, pario-bib

Comment: A parallelized RAID architecture that distributes the RAID controller operations across several worker nodes. Multiple hosts can connect to different workers, allowing multiple paths into the array. The workers then communicate on their own fast interconnect to accomplish the requests, distributing parity computations across multiple workers. They get much better performance and reliability than plain RAID. They built a prototype and a performance simulator. Two-phase commit was needed for request atomicity, and a request sequencer was needed for serialization. Also found it was good to give the whole request info to all workers and to let them figure out what to do and when. Superceded by cao:tickertaip-tr2 and cao:tickertaip.

Keywords: parallel I/O, RAID, pario-bib

Comment: Revised version of cao:tickertaip, actually: ``It's the ISCA paper with some text edits plus some new results on what happens if you turn disk request-scheduling on. It's been sent to TOCS.''. Thus it supercedes both cao:tickertaip-tr and cao:tickertaip. Eventually published as cao:jtickertaip.

Abstract: Caching has been intensively used in memory and traditional file systems to improve system performance. However, the use of caching in parallel file systems and I/O libraries has been limited to I/O nodes to avoid cache coherence problems. In this paper, we specify an adaptive cache coherence protocol very suitable for parallel file systems and parallel I/O libraries. This model exploits the use of caching, both at processing and I/O nodes, providing performance increase mechanisms as aggressive prefetching and delayed-write techniques. The cache coherence problem is solved by using a dynamic scheme of cache coherence protocols with different sizes and shapes of granularity. The proposed model is very appropriate for parallel I/O interfaces, as MPI-IO. Performance results, obtained on an IBM SP2, are presented to demonstrate the advantages offered by the cache management methods proposed.

Keywords: parallel file system, caching, cache coherence, adaptive caching, protocol specification, pario-bib

Keywords: persistent systems, database, parallel I/O, object-oriented, pario-bib

Comment: SHORE is a persistent object database system. It is intended for parallel or distributed systems, and attempts to combine both DB and file system features. Everything in the database is a typed object, in that there is a registered interface object that defines this type, including the basic data types of elements of the object, and methods that manipulate the object. Every object has an OID, and objects can refer to other objects with the OID. But they also support unix-like namespace, in which the names refer to objects by giving the OID. They also have a unix-compatibility library that provides access to many objects through the unix file interface. Every node has a SHORE server, and applications talk to their local server for all their needs. The local server talks to other servers as needed. The servers are also responsible for caching pages and managing locks and transactions.

Abstract: As Linux clusters have matured as platforms for low-cost, high-performance parallel computing, software packages to provide many key services have emerged, especially in areas such as message passing and networking. One area devoid of support, however, has been parallel file systems, which are critical for high-performance I/O on such clusters. We have developed a parallel file system for Linux clusters, called the Parallel Virtual File System (PVFS). PVFS is intended both as a high-performance parallel file system that anyone can download and use and as a tool for pursuing further research in parallel I/O and parallel file systems for Linux clusters.

In this paper, we describe the design and implementation of PVFS and present performance results on the Chiba City cluster at Argonne. We provide performance results for a workload of concurrent reads and writes for various numbers of compute nodes, I/O nodes, and I/O request sizes. We also present performance results for MPI-IO on PVFS, both for a concurrent read/write workload and for the BTIO benchmark. We compare the I/O performance when using a Myrinet network versus a fast-ethernet network for I/O-related communication in PVFS. We obtained read and write bandwidths as high as 700 Mbytes/sec with Myrinet and 225 Mbytes/sec with fast ethernet.

Keywords: parallel I/O, parallel file system, cluster file system, Linux, pario-bib

Comment: won the Best Paper Award.

Abstract: This document briefly describes the components of the Cache Coherent File System (CCFS) source code. CCFS has three main components: Client File Server (CLFS), Local File Server (LFS), Concurrent Disk System (CDS). The main modules and functions of each component are described here. Special emphasys has been put on interfaces and data structures.

Keywords: multiprocessor file system, parallel I/O, pario-bib

Comment: See carretero:*, rosales:cds, perez:clfs.

Abstract: We present two designs, namely, "collective I/O" and "pipelined collective I/O", of a runtime library for irregular applications based on the two-phase collective I/O technique. We also present the optimization of both models by using chunking and compression mechanisms. In the first scheme, all processors participate in compressions and I/O at the same time, making scheduling of I/O requests simpler but creating a possibility of contention at the I/O nodes. In the second approach, processors are grouped into several groups, overlapping communication, compression, and I/O to reduce I/O contention dynamically. Finally, evaluation results are shown that demonstrates that we can obtain significantly high-performance for I/O above what has been possible so far

Keywords: PASSION, parallel I/O, compression, collective I/O, two-phase I/O, performance evaluation, pario-bib

Abstract: In this paper we present an experimental evaluation of COMPASSION, a runtime system for irregular applications based on collective I/O techniques. It provides a "Collective I/O" model, enhanced with "Pipelined" operations and compression. All processors participate in the I/O simultaneously, alone or grouped, making scheduling of I/O requests simpler and providing support for contention management. In-memory compression mechanisms reduce the total execution time by diminishing the amount of I/O requested and the I/O contention. Our experiments, executed on an Intel Paragon and on the ASCI/Red teraflops machine, demonstrate that COMPASSION can obtain significantly high-performance for I/O above what has been possible so far.

Keywords: PASSION, parallel I/O, compression, collective I/O, pario-bib

Keywords: multiprocessor file system, parallel I/O, pario-bib

Comment: See carretero:*, rosales:cds, perez:clfs.

Abstract: This paper presents a Parallel Disk System (PDS) for general purpose multiprocessors, which provides support for conventional file systems and databases, as well as direct access for applications requiring high performance mass storage. We present a systematic method to characterize a parallel I/O system, using it to evaluate PDS and to identify an optimal PDS configuration. Several devices (single disk, Raid3 and Raid5), and different configurations of I/O nodes, each one with a different type of device, have been simulated. Throughput and I/O rate of each configuration have been obtained for the former configurations and different types of workloads (database, general purpose and scientific applications).

Keywords: multiprocessor file system, parallel I/O, pario-bib

Comment: See carretero:*, rosales:cds, perez:clfs.

Abstract: This document describes the detailed design of the LFS, one of the components of the Cache Coherent File System (CCFS). CCFS has three main components: Client File Server (CLFS), Local File Server (LFS), Concurrent Disk System (CDS). The Local File Servers are located on each disk node, to develop file server functions in a per node basis. The LFS will interact with the Concurrent Disk System (CDS) to execute real input/output and to manage the disk system, partitions, distributed partitions, etc. The LFS includes general file system services and specialized services, and it will be responsible of maintaining cache consistency, distributing accesses to other servers, controlling partition information, etc.

Keywords: multiprocessor file system, parallel I/O, pario-bib

Comment: See carretero:*, rosales:cds, perez:clfs.

Abstract: This paper gives an overview of the I/O data mapping mechanisms of ParFiSys. Grouped management and parallelization are presented as relevant features. I/O data mapping mechanisms of ParFiSys, including all levels of the hierarchy, are described in this paper.

Keywords: parallel I/O, multiprocessor file system, pario-bib

Keywords: multiprocessor file system, parallel I/O, pario-bib

Comment: See carretero:*, rosales:cds, perez:clfs.

Keywords: parallel I/O, multiprocessor file system, pario-bib

Keywords: multiprocessor file system, parallel I/O, pario-bib

Comment: See carretero:*, rosales:cds, perez:clfs.

Keywords: multiprocessor file system, parallel I/O, pario-bib

Comment: See carretero:*, rosales:cds, perez:clfs.

Keywords: parallel I/O, I/O architecture, pario-bib

Comment: See carretero:*, rosales:cds, perez:clfs.

Keywords: parallel I/O, benchmark, pario-bib

Comment: Specs for three scalable-I/O benchmarks to be used for evaluating I/O for multiprocessors. One measures application I/O by mixing I/O and computation, one measures max disk I/O by reading and writing 80% of the total RAM memory, and the last one is for sending that data from the file system, through the network, and back. See fineberg:nht1.

Abstract: To demonstrate the flexibility of the Galley parallel file system and to analyze the efficiency and flexibility of the Vesta parallel file system interface, we implemented Vesta's application-programming interface on top of Galley. We implemented the Vesta interface using Galley's file-access methods, whose design arose from extensive testing and characterization of the I/O requirements of scientific applications for high-performance multiprocessors. We used a parallel CPU, parallel I/O, out-of-core matrix-multiplication application to test the Vesta interface in both its ability to specify data access patterns and in its run-time efficiency. In spite of its powerful ability to specify the distribution of regular, non-overlapping data access patterns across disks, we found that the Vesta interface has some significant limitations. We discuss these limitations in detail in the paper, along with the performance results.

Keywords: parallel I/O, multiprocessor file system, pario-bib, dfk

Comment: See also nils/galley.html

Abstract: With a view to improving the performance and the fault tolerance of mass storage units, this paper concentrates on the architectural issues of parallelizing I/O access and a disk array system by means of definition of a new, particularly flexible architecture, called Partial Dynamic Declustering, which is fault-tolerant and offers higher levels of performance and reliability than the solutions normally used. A fast distributed algorithm based on a dynamic structure and usable for the implementation of an efficient I/O subsystem manager is proposed. Particular attention is also paid to the definition of analytical models based on Stochastic Reward Petri nets in order to analyze the performance and reliability of the system proposed.

Keywords: parallel I/O, disk array, pario-bib

Abstract: We concentrate on the architectural issues of parallelizing I/O access in a disk array system by means of definition of a new, particularly flexible architecture, called partial dynamic declustering, which is fault-tolerant and offers higher levels of performance and reliability than the solutions normally used. A simulation analysis highlights the efficiency of the proposed solution in balancing the file system workload and demonstrates its validity in both cases of unbalanced loads and expansion of the system. Particular attention is also paid to the definition of analytical models, based on stochastic reward nets, in order to analyze the performance and reliability of the system. The response time distribution function is evaluated and a specific performance analysis with varying degrees of declustering and workload is carried out.

Keywords: parallel I/O, disk array, pario-bib

Abstract: The introduction of multiprocessor architectures into computer systems has further increased the gap between processing times and access times to mass memories, thus making the processes more and more I/O-bound. To provide higher performance levels (both transfer rate and I/O rate), disk array technology is based on the use of a number of logically interconnected disks of a small size, in order to replace disks which have a large capacity but are very expensive. With a view to improving the performance and fault tolerance of the mass storage units, this paper concentrates on the architectural issues of parallelizing I/O access in a disk array system by means of definition of a new, particularly flexible architecture, called Partial Dynamic Declustering, which is fault-tolerant and offers higher levels of performance and reliability than the solutions normally used. A fast distributed algorithm based on a dynamic structure and usable for the implementation of an efficient I/O subsystem manager is proposed and evaluated by a simulative analysis. A specific study also characterizes the system's performance with varying degrees of declustering and workload types (from the transactional to the scientific type). The results obtained allow us to obtain the optimal configuration of the system (number of disks per group) which will ensure the desired response time values for varying workloads.

Keywords: parallel I/O, pario-bib

Abstract: Clusters of workstations become more and more popular to power data server applications such as large scale Web sites or e-Commerce applications. There has been much research on scaling the front tiers (web servers and application servers) using clusters, but databases usually remain on large dedicated SMP machines. In this paper, we focus on the database tier using clusters of commodity hardware. Our approach consists of studying different replication strategies to achieve various degree of performance and fault tolerance.  Redundant Array of Inexpensive Databases (RAIDb) is to databases what RAID is to disks. In this paper, we focus on RAIDb-1 that offers full replication and RAIDb-2 that introduces partial replication, in which the user can define the degree of replication of each database table.  We present a Java implementation of RAIDb called Clustered JDBC or C-JDBC. C-JDBC achieves both database performance scalability and high availability at the middleware level without changing existing applications. We show, using the TPC-W benchmark, that partial replication (RAIDb-2) can offer better performance scalability (up to 25%) than full replication by allowing fine-grain control on replication. Distributing and restricting the replication of frequently written tables to a small set of backends reduces I/O usage and improves CPU utilization of each cluster node.

Keywords: replication strategies, RAIDb, database, pario-bib

Abstract: The paper considers he problem of parallel external sorting in the contex of a form of heterogeneous clusters. We introduce two algorithms and we compare them two another one that we have previously developed. Since most common sort algorithms assume high-speed random access to all intermediate memory, they are unsuitable if the values to be sorted don't fit in main memory. This is the case for cluster computing platforms which are made of standard, cheap and scarce components. For that class of computing resources a good use of I/O operations compatible with the requirements of load balancing and computational complexity are the key to success. We explore three techniques and show how they can be deployed for clusters with processor performances related by a multiplicative factor. We validate the approaches in showing experimental results for the load balancing factor.

Keywords: out-of-core, sorting, parallel I/O, load balancing, data distribution, pario-app, pario-bib

Abstract: We present a new computing approach for the parallelization on message-passing computer architectures of the DNAml algorithm, one of the most powerful tools available for constructing phylogenetic trees from DNA sequences. An analysis of the data dependencies of the method gave little chances to develop an efficient parallel approach. However, a careful run-time analysis of the behaviour of the algorithm allowed us to propose a very efficient parallel implementation based on the combination of advanced dynamic scheduling strategies, speculative running-time execution decisions and I/O buffering. In this work, we discuss specific Parallel Virtual Machine (PVM)-based implementations for a cluster of workstations and for Distributed Memory multiprocessors, with high performance results. The code can be obtained from our public-domain sites.

Keywords: parallel computers, run-time analysis, phylogenetic trees, DNAml program, source code, parallel I/O, pario-bib

Comment: They discuss the parallelization on message-passing computers of the {DNA}ml algorithm, a tool used to construct phylogenetic trees from {DNA} sequences. By performing a run-time analysis of the behavior of the algorithm they came up with an efficient parallel implementation based on dynamic scheduling strategies, speculative run-time execution decisions and I/O buffering. They use I/O buffering (prefetching) to fetch tasks that need to be processed. The parallel code was written in C using PVM for message passing and is avaialable via anonymous ftp at ftp.ac.uma.es.

Keywords: object-based storage, distributed file system, parallel file system, pario-bib

Comment: Describes an open-source project to develop an object-based file system for clusters. Related to the NASD project at CMU (http://www.pdl.cs.cmu.edu/NASD/).

Abstract: Because many scientific applications require large data processing, the importance of parallel I/O has been increasingly recognized. For collective I/O, one of the considerable features of parallel I/O, we suggest the subgroup method. It is the way of using collective I/O of MPI effectively in terms of application programs. From the experimental results, we could conclude that the subgroup method for collective I/O is more efficient than plain collective I/O.

Keywords: collective I/O, MPI subgroup, pario-bib

Keywords: parallel I/O, disk array, pario-bib, RAID

Comment: A framework for evaluating the reliability of RAIDs. They consider failure and repair rates that depend on the workload.

Abstract: Due to the growing popularity of emerging applications such as digital libraries, Video-On Demand, distance learning, and Internet World-Wide Web, multimedia servers with a large capacity and high performance storage subsystem are in high demand. Serial storage interfaces are emerging technologies designed to improve the performance of such storage subsystems. They provide high bandwidth, fault tolerance, fair bandwidth sharing and long distance connection capability. All of these issues are critical in designing a scalable and high performance storage subsystem. Some of the serial storage interfaces provide the spatial reuse feature which allows multiple concurrent transmissions. That is, multiple hosts can access disks concurrently with full link bandwidth if their access paths are disjoint. Spatial reuse provides a way to build a storage subsystem whose aggregate bandwidth may be scaled up with the number of hosts. However, it is not clear how much the performance of a storage subsystem could be improved by the spatial reuse with different configurations and traffic scenarios. Both limitation and capability of this scalability need to be investigated. To understand their fundamental performance characteristics, we derive an analytic model for the serial storage interfaces with the spatial reuse feature. Based on this model, we investigate the maximum aggregate throughput from different system configurations and load distributions. We show how the number of disks needed to saturate a loop varies with different number of hosts and different load scenarios. We also show how the load balancing by uniformly distributing the load to all the disks on a loop may incur high overhead. This is because the accesses to far away disks need to go through many links and consume the bandwidth of each link it goes through. The results show the achievable throughput may be reduced by more than half in some cases.

Keywords: I/O interface, I/O network, I/O architecture, parallel I/O, pario-bib

Abstract: There are two major challenges for a high-performance remote-sensing database. First, it must provide low-latency retrieval of very large volumes of spatio-temporal data. This requires effective declustering and placement of a multi-dimensional dataset onto a large disk farm. Second, the order of magnitude reduction in data-size due to post-processing makes it imperative, from a performance perspective, that the postprocessing be done on the machine that holds the data. This requires careful coordination of computation and data retrieval. This paper describes the design, implementation and evaluation of Titan, a parallel shared-nothing database designed for handling remote-sensing data. The computational platform for Titan is a 16-processor IBM SP-2 with four fast disks attached to each processor. Titan is currently operational and contains about 24 GB of AVHRR data from the NOAA-7 satellite. The experimental results show that Titan provides good performance for global queries and interactive response times for local queries.

Keywords: parallel databases, satellite imagery, remote sensing, parallel I/O, pario-bib

Keywords: parallel I/O, parallel file system, pario-bib

Comment: A more detailed spec of the datamesh architecture, specifying components and operations. It is a block server where blocks are associatively addressed by tags. Some search operations are supported, as are atomic tag-changing operations. See also cao:tickertaip, wilkes:datamesh1, wilkes:datamesh, wilkes:houses, wilkes:lessons.

Keywords: parallel I/O, pario-bib

Comment: See also bruce:chimp, chapple:pulgf, and chapple:pulgf-adv, for general information on CHIMP and PUL-GF. This document is an exploration of the potential ways to parallelize the underlying I/O support for the PUL-GF interface. They reason about tradeoffs in the number of servers, disks, and clients, but (as they note) without any performance evaluation to back it up. In particular, they argue that there should be one partition per disk, one server per disk, and probably one client to many servers, or many clients to many servers. A key assumption is that a traditional serial file system is the home location for files, and that files are ``converted'' into parallel files (or vice versa) by replicating or distributing them. Application could choose the number of servers (and hence disks) for each file. Hints could be provided about many things. Interesting idea to allow user hooks for cache prefetch and writeback functions. Support for variable-length records (``atoms'') is a key component. Segments of a file with different formats, e.g., a header and a matrix, may be separated into different components when the file is distributed into parallel form. See chapple:pulpf for info on the eventual realization of these ideas.

Keywords: parallel I/O, pario-bib

Comment: PUL is a set of libraries that run on top of the CHIMP portable message-passing library (see bruce:chimp). One of the PUL libraries is PUL-GF, to support file I/O. The underlying I/O support is not parallel (but see chapple:pario). The interface is parallel, however; in particular, it supports file modes similar to those used in many systems, which they call single, multi, random, and independent. Formatted and unformatted, synchronous and asynchronous. Very general multidimensional-array read and write functions. Ability to group multiple I/O requests into atomic units, though not a full transaction capability. See also chapple:pulgf-adv and chapple:pario.

Keywords: parallel I/O, pario-bib

Comment: See chapple:pulgf for a definition of PUL-GF. This document describes the internal client-server interface to PUL-GF, including ways that users can extend the functionality of PUL-GF. In particular, they give an example of how a new file format (a run-length encoded 2-d matrix) can be read and written transparently as if it were a plain matrix file. The extensibility is offered by run-time registration of user-defined interposition functions, to be called at key moments in the processing of a file I/O request. See also bruce:chimp and chapple:pario.

Keywords: parallel I/O, multiprocessor file system interface, pario-bib

Comment: See also chapple:pulgf and chapple:pario. An evolution of their parallel I/O interface. PUL-PF is a library on top of existing file systems. Every process is either a client or a server; servers write some portion of the file to a file in the file system. Servers can be divided into groups so that files need not be spread across all servers. There seems to be client caching, with consistency controlled differently depending on access mode; when necessary, the application must call get-token and send-token commands to serialize access to an atom. Independently of their single, multi, random, and independent mode, they can read or write the next, previous, current, or ``wild'' atom (wild means the next ``most available'' atom not yet read by this process). Most I/O is on atoms, but particles (pieces of atoms) can also be independently read and written. Hints are supported to specify access pattern (random or sequential, stride), file partitioning, mapping, atom size, or caching. In many of those cases it goes beyond a hint to the supply of a user-defined function, e.g., for cache-replacement algorithm.

Abstract: Previous implementations of out-of-core columnsort limit the problem size to $N \leq \sqrt{(M/P)^3 / 2}$, where $N$ is the number of records to sort, $P$ is the number of processors, and $M$ is the total number of records that the entire system can hold in its memory (so that $M/P$ is the number of records that a single processor can hold in its memory). We implemented two variations to out-of-core columnsort that relax this restriction. Subblock columnsort is based on an algorithmic modification of the underlying columnsort algorithm, and it improves the problem-size bound to $N \leq (M/P)^{5/3} / 4^{2/3}$ but at the cost of additional disk I/O\. $M$-columnsort changes the notion of the column size in columnsort.

Keywords: parallel I/O, sorting, out-of-core applications, pario-bib

Abstract: Sorting very large datasets is a key subroutine in almost any application that is built on top of a large database. Two ways to sort out-of-core data dominate the literature: merging-based algorithms and partitioning-based algorithms. Within these two paradigms, all the programs that sort out-of-core data on a cluster rely on assumptions about the input distribution. We propose a third way of out-of-core sorting: oblivious algorithms. In all, we have developed six programs that sort out-of-core data on a cluster. The first three programs, based completely on Leighton's columnsort algorithm, have a restriction on the maximum problem size that they can sort. The other three programs relax this restriction; two are based on our original algorithmic extensions to columnsort. We present experimental results to show that our algorithms perform well. To the best of our knowledge, the programs presented in this thesis are the first to sort out-of-core data on a cluster without making any simplifying assumptions about the distribution of the data to be sorted.

Keywords: out-of-core sorting, columnsort, cluster computing, parallel I/O, pario-bib

Comment: Doctoral dissertation. Advisor: Thomas H. Cormen

Abstract: Leighton's columnsort algorithm sorts on an $r \times s$ mesh, subject to the restrictions that $s$ is a divisor of $r$ and that $r \geq 2s^2$ (so that the mesh is tall and thin). We show how to mitigate both of these restrictions. One result is that the requirement that $s$ is a divisor of $r$ is unnecessary; columnsort sorts correctly whether or not $s$ divides $r$. We present two algorithms that, as long as $s$ is a perfect square, relax the restriction that $r \geq 2s^2$; both reduce the exponent of $s$ to $3/2$. One algorithm requires $r \geq 4s^{3/2}$ if $s$ divides $r$ and $r \geq 6s^{3/2}$ if $s$ does not divide $r$. The other algorithm requires $r \geq 4^{3/2}$, and it requires $s$ to be a divisor of $r$. Both algorithms have applications in increasing the maximum problem size in out-of-core sorting programs.

Keywords: parallel I/O, sorting, out-of-core applications, pario-bib

Abstract: In today's workstation based environment, applications such as design databases, multimedia databases, and knowledge bases do not fit well into the relational data processing framework. The object-oriented data model has been proposed to model and process such complex databases. Due to the nature of the supported applications, object-oriented database systems need efficient mechanisms for the retrieval of complex objects and the navigation along the semantic links among objects. Object clustering and buffering have been suggested as efficient mechanisms for the retrieval of complex objects. However, to improve the efficiency of the aforementioned operations, one has to look at the recent advances in storage technology. This paper is an attempt to investigate the feasibility of using parallel disks for object-oriented databases. It analyzes the conceptual changes needed to map the clustering and buffering schemes proposed on the new underlying architecture. The simulation and performance evaluation of the proposed leveled-clustering and mapping schemes utilizing parallel I/O disks are presented and analyzed.

Keywords: parallel I/O, disk array, object oriented database, pario-bib

Abstract: The complexity of parallel I/O systems imposes significant challenge in managing and utilizing the available system resources to meet application performance, portability and usability goals. We believe that a parallel I/O system that automatically selects efficient I/O plans for user applications is a solution to this problem. In this paper, we present such an automatic performance optimization approach for scientific applications performing collective I/O requests on multidimensional arrays. The approach is based on a high level description of the target workload and execution environment characteristics, and applies genetic algorithms to select high quality I/O plans. We have validated this approach in the Panda parallel I/O library. Our performance evaluations on the IBM SP show that this approach can select high quality I/O plans under a variety of system conditions with a low overhead, and the genetic algorithm-selected I/O plans are in general better than the default plans used in Panda.

Keywords: parallel I/O, performance optimization, genetic algorithm, pario-bib

Keywords: collective I/O, multiprocessor file system, parallel I/O, pario-bib

Keywords: parallel I/O, RAID, disk array, pario-bib

Comment: A experimental validation of the performance predictions of patterson:raid, plus some extensions. Confirms that RAID level 5 (rotated parity) is best for large read/writes, and RAID level 1 (mirroring) is best for small reads/writes.

Keywords: parallel I/O, RAID, disk striping, pario-bib

Comment: Choosing the optimal striping unit, i.e., size of contiguous data on each disk (bit, byte, block, etc.). A small striping unit is good for low-concurrency workloads since it increases the parallelism applied to each request, but a large striping unit can support high-concurrency workloads where each independent request depends on fewer disks. They do simulations to find throughput, and thus to pick the striping unit. They find equations for the best compromise striping unit based on the concurrency and the disk parameters, or on the disk parameters alone. Some key assumptions may limit applicability, but this is not addressed.

Abstract: To provide high performance for applications with a wide variety of i/o requirements and to support many different parallel platforms, the design of a parallel i/o system must provide for efficient utilization of available bandwidth both for disk traffic and for message passing. In this paper we discuss the message-passing scalability of the server-directed i/o architecture of Panda, a library for synchronized i/o of multidimensional arrays on parallel platforms. We show how to improve i/o performance in situations where message-passing is a bottleneck, by combining the server-directed i/o strategy for highly efficient use of available disk bandwidth with new mechanisms to minimize internal communication and computation overhead in Panda. We present experimental results that show that with these improvements, Panda will provide high i/o performance for a wider range of applications, such as applications running with slow interconnects, applications performing i/o operations on large numbers of arrays, or applications that require drastic data rearrangements as data are moved between memory and disk (e.g., array transposition). We also argue that in the future, the improved approach to message-passing will allow Panda to support applications that are not closely synchronized or that run in heterogeneous environments.

Keywords: parallel I/O, parallel file system, pario-bib

Comment: see seamons:panda. This paper goes further with some communication improvements.

Abstract: Parallel I/O systems typically consist of individual processors, communication networks, and a large number of disks. Managing and utilizing these resources to meet performance, portability and usability goals of applications has become a significant challenge. We believe that a parallel I/O system that automatically selects efficient I/O plans for user applications is a solution to this problem. In this paper, we present such an automatic performance optimization approach for scientific applications performing collective I/O requests on multidimensional arrays. Under our approach, as optimization engine in a parallel I/O system selects optimal I/O plans automatically without human intervention based on a description of the application I/O requests and the system configuration. To validate our hypothesis, we have built an optimizer that uses a rule-based and randomized search-based algorithms to select optimal parameter settings in Panda, a parallel I/O library for multidimensional arrays. Our performance results obtained from two IBM SPs with significantly different configurations show that the Panda optimizer is able to select high-quality I/O plans and deliver high performance under a variety of system configurations

Keywords: parallel I/O, Panda, portability, pario-bib

Abstract: We present an analytical performance model for Panda, a library for synchronized i/o of large multidimensional arrays on parallel and sequential platforms, and show how the Panda developers use this model to evaluate Panda's parallel i/o performance and guide future Panda development. The model validation shows that system developers can simplify performance analysis, identify potential performance bottlenecks, and study the design trade-offs for Panda on massively parallel platforms more easily than by conducting empirical experiments. More importantly, we show that the outputs of the performance model can be used to help make optimal plans for handling application i/o requests, the first step toward our long-term goal of automatically optimizing i/o request handling in Panda.

Keywords: performance modeling, parallel I/O, pario-bib

Comment: On Web and CDROM only. They derive a detailed but fairly simple model of the Panda 2.0.5 parallel I/O library, by carefully enumerating the costs involved in a collective I/O operation. They measure Panda, AIX, and MPI to obtain parameters, and then they validate the model by comparison with the actual Panda implementation running a basic benchmark and an actual application. The model predicts the benchmark performance very well, and is as much as 20% off on the performance of the application. They have embedded the performance model in a "simulator", which predicts the performance of a given sequence of collective I/O requests, and they plan to use it in future versions of Panda to formulate I/O plans by predicting the performance resulting from several different Panda parameter settings, and choosing the best.

Keywords: parallel I/O, RAID, disk array, pario-bib

Comment: Basically an updated version of patterson:raid and the prepublished version of gibson:failcorrect.

Keywords: parallel I/O, RAID, disk array, pario-bib

Keywords: RAID, disk array, parallel I/O, survey, pario-bib

Comment: An excellent overview of RAID concepts and technology. It starts from the beginning with a discussion of disk hardware, RAID basics, etc, and then goes on to discuss some of the more advanced features. They also describe a few RAID implementations. Basically, it is a perfect paper to read for folks new to RAID.

Keywords: parallel I/O, multiprocessor file system, pario-bib

Comment: A special back-end box for a Sun4 file server, that hooks a HIPPI network through a crossbar to fast memory, a parity engine, and a bunch of disks on SCSI. They pulled about 20 MB/s through it, basically disk-limited; with more disks they would hit 32-40 MB/s. Much improved over RAID-I, which was limited by the memory bandwidth of the Sun4 server.

Keywords: disk array, striping, RAID, pario-bib

Abstract: Parallel I/O systems typically consist of individual processors, communication networks, and a large number of disks. Managing and utilizing these resources to meet performance, portability, and usability goals of high performance scientific applications has become a significant challenge. For scientists, the problem is exacerbated by the need to retune the I/O portion of their code for each supercomputer platform where they obtain access. We believe that a parallel I/O system that automatically selects efficient I/O plans for user applications is a solution to this problem. The authors present such an approach for scientific applications performing collective I/O requests on multidimensional arrays. Under our approach, an optimization engine in a parallel I/O system selects high quality I/O plans without human intervention, based on a description of the application I/O requests and the system configuration. To validate our hypothesis, we have built an optimizer that uses rule based and randomized search based algorithms to tune parameter settings in Panda, a parallel I/O library for multidimensional arrays. Our performance results obtained from an IBM SP using an out-of-core matrix multiplication application show that the Panda optimizer is able to select high quality I/O plans and deliver high performance under a variety of system configurations with a small total optimization overhead.

Keywords: parallel I/O, pario-bib

Keywords: parallel I/O, disk array, performance evaluation, RAID, pario-bib

Comment: Measuring the performance of a RAID prototype with a Sun4/280, 28 disks on 7 SCSI strings, using 4 HBA controllers on a VME bus from the Sun. The found lots of bottlenecks really slowed them down. Under Sprite, the disks were the bottleneck for single disk I/O, single disk B/W, and string I/O. Sprite was a bottleneck for single disk I/O and String I/O. The host memory was a bottleneck for string B/W, HBA B/W, overall I/O, and overall B/W. With a simpler OS, that saved on data copying, they did better, but were still limited by the HBA, SCSI protocol, or the VME bus. Clearly they needed more parallelism in the busses and control system.

Keywords: RAID, disk array, network file system, parallel I/O, pario-bib

Comment: Part of jin:io-book; reformatted version of drapeau:raid-ii.

Keywords: parallel I/O, multimedia, tertiary storage, memory hierarchy, pario-bib

Comment: Part of a special issue.

Abstract: We present a collection of new techniques for designing and analyzing efficient external-memory algorithms for graph problems and illustrate how these techniques can be applied to a wide variety of specific problems. Our results include: \begin{itemize} \item Proximate-neighboring. We present a simple method for deriving external-memory lower bounds via reductions from a problem we call the ``proximate neighbors'' problem. We use this technique to derive non-trivial lower bounds for such problems as list ranking, expression tree evaluation, and connected components. \item PRAM simulation. We give methods for efficiently simulating PRAM computations in external memory, even for some cases in which the PRAM algorithm is not work-optimal. We apply this to derive a number of optimal (and simple) external-memory graph algorithms. \item Time-forward processing. We present a general technique for evaluating circuits (or ``circuit-like'' computations) in external memory. We also use this in a deterministic list ranking algorithm. \item Deterministic 3-coloring of a cycle. We give several optimal methods for 3-coloring a cycle, which can be used as a subroutine for finding large independent sets for list ranking. Our ideas go beyond a straightforward PRAM simulation, and may be of independent interest. \item External depth-first search. We discuss a method for performing depth first search and solving related problems efficiently in external memory. Our technique can be used in conjunction with ideas due to Ullman and Yannakakis in order to solve graph problems involving closed semi-ring computations even when their assumption that vertices fit in main memory does not hold. \end{itemize}

Our techniques apply to a number of problems, including list ranking, which we discuss in detail, finding Euler tours, expression-tree evaluation, centroid decomposition of a tree, least-common ancestors, minimum spanning tree verification, connected and biconnected components, minimum spanning forest, ear decomposition, topological sorting, reachability, graph drawing, and visibility representation.

Keywords: parallel I/O algorithm, graph algorithm, pario-bib

Abstract: Parallel scientific applications store and retrieve very large, structured datasets. Directly supporting these structured accesses is an important step in providing high-performance I/O solutions for these applications. High-level interfaces such as HDF5 and Parallel netCDF provide convenient APIs for accessing structured datasets, and the MPI-IO interface also supports efficient access to structured data. However, parallel file systems do not traditionally support such access. In this work, we present an implementation of structured data access support in the context of the Parallel Virtual File System (PVFS). We call this support "datatype I/O" because of its similarity to MPI datatypes. This support is built by using a reusable datatype-processing component from the MPICH2 MPI implementation. We describe how this component is leveraged to efficiently process structured data representations resulting from MPI-IO operations. We quantitatively assess the solution using three test applications. We also point to further optimizations in the processing path that could be leveraged for even more efficient operation.

Keywords: I/O interface, high-level libraries, PVFS, structured data representations, pario-bib

Comment: not read, don't have

Abstract: I/O performance remains a weakness of parallel computing systems today. While this weakness is partly attributed to rapid advances in other system components, I/O interfaces available to programmers and the I/O methods supported by file systems have traditionally not matched efficiently with the types of I/O operations that scientific applications perform, particularly noncontiguous accesses. The MPI-IO interface allows for rich descriptions of the I/O patterns desired for scientific applications and implementations such as ROMIO have taken advantage of this ability while remaining limited by underlying file system methods.

A method of noncontiguous data access, list I/O, was recently implemented in the Parallel Virtual File System (PVFS). We implement support for this interface in the ROMIO MPI-IO implementation. Through a suite of non-contiguous I/O tests we compared ROMIO list I/O to current methods of ROMIO noncontiguous access and found that the list I/O interface provides performance benefits in many noncontiguous cases.

Keywords: parallel I/O, MPI-IO, ROMIO, list I/O, noncontiguous access, pario-bib

Abstract: Processor-embedded disks, or smart disks, with their network interface controller, can in effect be viewed as processing elements with on-disk memory and secondary storage. The data sizes and access patterns of today's large I/O-intensive workloads require architectures whose processing power scales with increased storage capacity. To address this concern, we propose and evaluate disk-based distributed smart storage architectures. Based on analytically derived performance models, our evaluation with representative workloads show that offloading processing and performing point-to-point data communication improve performance over centralized architectures. Our results also demonstrate that distributed smart disk systems exhibit desirable scalability and can efficiently handle I/O-intensive workloads, such as commercial decision support database (TPC-H) queries, association rules mining, data clustering, and two-dimensional fast Fourier transform, among others. (15 refs.)

Keywords: processor-embedded disks, smart disks, analytic performance models, I/O workload, pario-bib

Keywords: parallel I/O, tape striping, pario-bib

Comment: URL points to tech report version. He points out two problems with tape striping: that it is difficult to keep tape drives synchronized due to physical variations and to bad-segment remapping in the tape, and that the start-up cost is very high making it difficult to get multiple tapes loaded and started at the same time. So he proposes a 'triangular interleaving' rather than the traditional round-robin interleaving, coupled with lots of buffering, to deal with these problems. He also proposes to use different striping factors for different files (movies), depending on access characteristics. He includes parameters for some tape robots.

Abstract: The paper presents an analytical model of a whole disk array architecture, XDAC, which consists of several major subsystems and features: the two-dimensional array structure; IO-bus with split transaction protocol; and cache for processing multiple I/O requests in parallel. Our modelling approach is based on a subsystem access time per request (SATPR) concept, in which we model for each subsystem the mean access time per disk array request. The model is fed with a given set of representative workload parameters and then used to conduct performance analysis for exploring the impact of fork/join synchronization as well as evaluating some architectural design issues of the XDAC system. Moreover, by comparing the SATPRs of subsystems, we can identify the bottleneck for performance improvements.

Keywords: disk array, performance evaluation, analytical model, parallel I/O, pario-bib

Abstract: Fine grained data distributions are widely used to balance computational loads across compute processes in parallel scientific applications. When a fine grained data distribution is used in memory, performance of I/O intensive applications can be limited not only by disk speed but also by message passing, because a large number of small messages may be generated by the implementation strategy used in the underlying parallel file system or parallel I/O library. Combining (or packetizing) a set of small messages into a large message is generally known to speed up parallel I/O. However, overall I/O performance is affected not only by small messages but also by other factors like cyclic block size and interconnect characteristics. We describe small message combination and communication scheduling for fine grained data distributions in the Panda parallel I/O library and analyze I/O performance on parallel platforms having different interconnects: IBM SP2, IBM workstation cluster connected by FDDI and Pentium II cluster connected by Myrinet.

Keywords: parallel I/O, pario-bib

Abstract: A cost-effective way to run a parallel application is to use existing workstations connected by a local area network such as Ethernet or FDDI. In this paper, we present an approach for parallel I/O of multidimensional arrays on small networks of workstations with a shared-media interconnect, using the Panda I/O library.

In such an environment, the message passing throughput per node is lower than the throughput obtainable from a fast disk and it is not easy for users to determine the configuration which will yield the best I/O performance.

We introduce an I/O strategy that exploits local data to reduce the amount of data that must be shipped across the network, present experimental results, and analyze the results using an analytical performance model and predict the best choice of I/O parameters.

Our experiments show that the new strategy results in a factor of 1.2-2.1 speedup in response time compared to the Panda version originally developed for the IBM SP2, depending on the array sizes, distributions and compute and I/O node meshes. Further, the performance model predicts the results within a 13% margin of error.

Keywords: parallel I/O, distributed system, pario-bib

Comment: They examine a system that supports nodes that are both compute and I/O nodes. The assumption is that the application is writing data to a new file, and does not care to which disks the data goes. They are trying to decide which nodes should be used for I/O, given the distribution of data on compute nodes and the distribution desired across disks. They use a Hungarian algorithm to solve a weighted optimization problem on a bipartite graph connecting I/O nodes to compute nodes, in an attempt to minimize the data flow across the network. But there is no attempt to make a decision that might be sensible for a future read operation that may want to read in a different pattern.

Abstract: With the increasing number of scientific applications manipulating huge amounts of data, effective high-level data management is an increasingly important problem. Unfortunately, so far the solutions to the high-level data management problem either require deep understanding of specific storage architectures and file layouts (as in high-performance file storage systems) or produce unsatisfactory I/O performance in exchange for ease-of-use and portability (as in relational DBMSs). In this paper we present a novel application development environment which is built around an active meta-data management system (MDMS) to handle high-level data in an effective manner. The key components of our three-tiered architecture are user application, the MDMS, and a hierarchical storage system (HSS). Our environment overcomes the performance problems of pure database-oriented solutions, while maintaining their advantages in terms of ease-of-use and portability. The high levels of performance are achieved by the MDMS, with the aid of user-specified, performance-oriented directives. Our environment supports a simple, easy-to-use yet powerful user interface, leaving the task of choosing appropriate I/O techniques for the application at hand to the MDMS. We discuss the importance of an active MDMS and show how the three components of our environment, namely the application, the MDMS, and the HSS, fit together. We also report performance numbers from our ongoing implementation and illustrate that significant improvements are made possible without undue programming effort.

Keywords: cluster computing, scientific computing, parallel I/O, data management, pario-bib

Abstract: With the increasing number of scientific applications manipulating huge amounts of data, effective data management is an increasingly important problem. Unfortunately, so far the solutions to this data management problem either require deep understanding of specific storage architectures and file layouts (as in high-performance file systems) or produce unsatisfactory I/O performance in exchange for ease-of-use and portability (as in relational DBMSs).

In this paper we present a new environment which is built around an active meta-data management system (MDMS). The key components of our three-tiered architecture are user application, the MDMS, and a hierarchical storage system (HSS). Our environment overcomes the performance problems of pure database-oriented solutions, while maintaining their advantages in terms of ease-of-use and portability.

The high levels of performance are achieved by the MDMS, with the aid of user-specified directives. Our environment supports a simple, easy-to-use yet powerful user interface, leaving the task of choosing appropriate I/O techniques to the MDMS. We discuss the importance of an active MDMS and show how the three components, namely application, the MDMS, and the HSS, fit together. We also report performance numbers from our initial implementation and illustrate that significant improvements are made possible without undue programming effort.

Keywords: cluster computing, scientific computing, parallel I/O, data management, pario-bib

Comment: They argue that existing parallel file systems are too low-level, they have their own set of I/O calls (non-portable), and policies are generally hard-coded into the system. Databases provide a portable layer on top of the file system, but they cannot provide high performance. They propose to "combine the advantages of file systems and databases, while avoiding their respective disadvantages." Their system is composed of a user program, a meta-data management system (MDMS), and a heirarchical storage system (HSS). The user program will query the MDMS to learn where in the HSS their data reside, what the performance of the storage system is, information about the acc data from the storage system, etc...

Keywords: parallel I/O, out-of-core, pario-bib

Comment: This TR overviews the PASSION project, and all its components: two-phase access, out-of-core support for structured and unstructured problems, data sieving, prefetching, caching, compiler and language support, file system support, virtual parallel file system, and parallel pipes. They reference many of their related papers in an extensive bibliography. See also singh:adopt, jadav:ioschedule, thakur:passion, thakur:runtime, bordawekar:efficient, thakur:out-of-core, delrosario:prospects, delrosario:two-phase, bordawekar:primitives, bordawekar:delta-fs.

Abstract: We are developing a runtime library which provides a number of routines to perform the I/O required in parallel applications in an efficient and convenient manner. This is part of a project called PASSION, which aims to provide software support for high-performance parallel I/O at the compiler, runtime and file system levels. The PASSION Runtime Library uses a high-level interface which makes it easy for the user to specify the I/O required in the program. The user only needs to specify what portion of the data structure needs to read from or written to the file, and the PASSION routines will perform all the necessary I/O efficiently. This paper gives an overview of the PASSION Runtime Library and describes in detail its high-level interface.

Keywords: parallel I/O, runtime library, pario-bib

Comment: See also choudhary:passion.

Keywords: file system, database, parallel I/O, pario-bib, dfk

Comment: A position paper for the Strategic Directions in Computer Research workshop at MIT in June 1996. See gibson:sdcr and wegner:sdcr.

Keywords: parallel I/O, multiprocessor file system, pario-bib

Comment: Language extensions to support parallel I/O. Compiler optimizations. Runtime library to support the compiler and interface with the native file system. Compiler would develop a mapping of data to the processor memories and to the disks, and then decide on I/O schedules to move data around, overlap I/O with computation, even move computation around to best fit what is available in memory at a given time. It can also help with checkpointing. Compiler should pass info to the runtime system, which in turn may need to pass info to the file system, to help with optimization. I/O scheduling includes reordering accesses; they even go so far as to propose doing seek optimization in the runtime library. Support for collective I/O. Extension of MPI to I/O, to take advantage of its support for asynchrony, scatter-gather, etc.\ On the way, they hope to work with the FS people to decide on the functional requirements of the file system. See also poole:sio-survey, bagrodia:sio-character, bershad:sio-os.

Abstract: We explore the method of combining the replication and parity approaches to tolerate multiple disk failures in a disk array. In addition to the conventional mirrored and chained declustering methods, a method based on the hybrid of mirrored-and-chained declustering is explored. A performance study that explores the effect of combining replication and parity approaches is conducted. It is experimentally shown that the proposed approach can lead to the most cost-effective solution if the objective is to sustain the same load as before the failures.

Keywords: fault tolerance, disk array, replication, declustering, parallel I/O, pario-bib

Comment: Consider hybrid chained and mirrored declustering.

Abstract: We discuss data production rates and their impact on the performance of scientific applications using parallel computers. On one hand, too high rates of data production can be overwhelming, exceeding logistical capacities for transfer, storage and analysis. On the other hand, the rate limiting step in a computationally-based study should be the human-guided analysis, not the calculation. We present performance data for a biomolecular simulation of the enzyme, acetylcholinesterase, which uses the parallel molecular dynamics program EulerGROMOS. The actual production rates are compared against a typical time frame for results analysis where we show that the rate limiting step is the simulation, and that to overcome this will require improved output rates.

Keywords: parallel I/O application, molecular dynamics, pario-bib

Comment: Note proceedings only on CD-ROM or WWW.

Keywords: parallel I/O algorithm, sorting, pario-bib

Comment: They present a new parallel sorting algorithm that allows overlap between disk, network, and processor. By pipelining the tasks, they can double the speed of sorting; best results, of course, when these three components take approximately equal time. The disk I/O is really only used to load the initial data set and write the output data set, rather than being used for an external sorting scheme. They obtain their gains by overlapping that disk I/O with the communication and processing.

Keywords: parallel I/O, HPF, compiler, pario-bib

Keywords: file system, parallel I/O, disk layout, pario-bib

Comment: These two guys from NCAR redid the block allocation strategy routine on the Cray. Current strategy uses round-robin among the disks, using a different disk for each allocation request. Each request looks for blocks on that disk, until it is satisfied or space runs out, and then goes to the next disk. It uses a free-block bitmap to find the blocks. Problem: too many extents, not enough contiguity. These guys tried first-bit and best-fit from all extents on all disks. First-fit had faster allocation time, of course, and both had much lower file fragmentation. They also used the vector hardware to search the bitmap for non-zero words.

Keywords: parallel architecture, parallel I/O, pario-bib

Comment: They argue for network-attached devices, and for making I/O devices and networks, instead of CPUs, more the center of architectural design.

Abstract: In order for I/O systems to achieve high performance in a parallel environment, they must either sacrifice client-side file caching, or keep caching and deal with complex coherency issues. The most common technique for dealing with cache coherency in multi-client file caching environments uses file locks to bypass the client-side cache. Aside from effectively disabling cache usage, file locking is sometimes unavailable on larger systems.

The high-level abstraction layer of MPI allows us to tackle cache coherency with additional information and coordination without using file locks. By approaching the cache coherency issue further up, the underlying I/O accesses can be modified in such a way as to ensure access to coherent data while satisfying the user's I/O request. We can effectively exploit the benefits of a file system's client-side cache while minimizing its management costs.

Keywords: client-side file caching, file locking, MPI, pario-bib

Abstract: This paper describes the functionality of ViC*, a compiler for a variant of the data-parallel language C* with support for out-of-core data. The compiler translates C* programs with shapes declared outofcore, which describe parallel data stored on disk. The compiler output is a SPMD-style program in standard C with I/O and library calls added to efficiently access out-of-core parallel data. The ViC* compiler also applies several program transformations to improve out-of-core data access.

Keywords: compiler, data-parallel programming, programming language, virtual memory, out of core, parallel I/O, pario-bib

Abstract: This paper describes the functionality of ViC*, a compiler for a variant of the data-parallel language C* with support for out-of-core data. The compiler translates C* programs with shapes declared outofcore, which describe parallel data stored on disk. The compiler output is a SPMD-style program in standard C with I/O and library calls added to efficiently access out-of-core parallel data. The ViC* compiler also applies several program transformations to improve out-of-core data layout and access.

Keywords: compiler, data-parallel programming, programming language, virtual memory, out of core, parallel I/O, pario-bib

Keywords: parallel computer architecture, shared memory, parallel I/O, pario-bib

Comment: The Convex Exemplar connects hypernodes, which are basically SMP nodes built from 8 HP PA-RISC CPUs, lots of RAM, and a crossbar switch, with their own implementation of the SCI interconnect. Hierarchical caching supports a global shared physical address space. Each hypernode can also have an I/O adapter, to which they can attach lots of different I/O devices. The I/O adapter has the capability to DMA directly into any memory in the system, even on other hypernodes. Each hypernode runs its own file-system server, which manages UNIX file systems on the devices of that hypernode. Striped file systems are supported in software, although it's not clear if they can stripe across hypernodes, or only within hypernodes, ie, whether (striped) file systems can span multiple hypernodes.

Keywords: parallel I/O, parallel file system, striping, pario-bib

Comment: Implementation of striped disks on the CONVEX. Uses partitions of normal device drivers. Kernel data structure knows about the interleaving granularity, the set of partitions, sizes, etc.

Keywords: multimedia, compression, data-parallel computing, Maspar, parallel I/O, pario-bib

Keywords: parallel I/O, database, disk caching, pario-bib

Comment: A database machine. Experimental/analytical model of a placement algorithm that declusters relations across several parallel, independent disks. The declustering is done on a subset of the disks, and the choices involved are the number of disks to decluster onto, which relations to put where, and whether a relation should be cache-resident. Communications overhead limits the usefulness of declustering in some cases, depending on the workload. See boral:bubba.

Keywords: parallel I/O, multiprocessor file system interface, pario-bib

Comment: Part of jin:io-book, revised version of corbett:mpi-overview.

Keywords: multiprocessor file system, Vesta, parallel I/O, pario-bib

Comment: Part of jin:io-book; reformatted version of corbett:jvesta.

Abstract: The Vesta parallel file system is designed to provide parallel file access to application programs running on multicomputers with parallel I/O subsystems. Vesta uses a new abstraction of files: a file is not a sequence of bytes, but rather it can be partitioned into multiple disjoint sequences that are accessed in parallel. The partitioning-which can also be changed dynamically-reduces the need for synchronization and coordination during the access. Some control over the layout of data is also provided, so the layout can be matched with the anticipated access patterns. The system is fully implemented and forms the basis for the AIX Parallel I/O File System on the IBM SP2. The implementation does not compromise scalability or parallelism. In fact, all data accesses are done directly to the I/O node that contains the requested data, without any indirection or access to shared metadata. Disk mapping and caching functions are confined to each I/O node, so there is no need to keep data coherent across nodes. Performance measurements shown good scalability with increased resources. Moreover, different access patterns are show to achieve similar performance.

Keywords: multiprocessor file system, Vesta, parallel I/O, pario-bib

Comment: See also corbett:pfs, corbett:vesta*, feitelson:pario. This is the ultimate Vesta reference. There seem to be only a few small things that are completely new over what's been published elsewhere, although this presentation is much more complete and polished.

Keywords: parallel I/O, message-passing, multiprocesor file system interface, pario-bib

Comment: Superceded by mpi-ioc:mpi-io5. See the MPI-IO Web page at http://parallel.nas.nasa.gov/MPI-IO/.

Keywords: parallel I/O, message-passing, multiprocesor file system interface, pario-bib

Comment: The goal is to design a standard file interface for SPMD message-passing programs. An earlier version of this specification was prost:mpi-io. Superceded by mpi-ioc:mpi-io5. See also the general MPI I/O web page at http://parallel.nas.nasa.gov/MPI-IO/.

Keywords: parallel I/O, message-passing, multiprocesor file system interface, pario-bib

Comment: Superceded by mpi-ioc:mpi-io5. See the MPI-IO Web page at http://parallel.nas.nasa.gov/MPI-IO/.

Abstract: Thanks to MPI, writing portable message passing parallel programs is almost a reality. One of the remaining problems is file I/O. Although parallel file systems support similar interfaces, the lack of a standard makes developing a truly portable program impossible. It is not feasible to develop large scientific applications from scratch for each generation of parallel machine, and, in the scientific world, a program is not considered truly portable unless it not only compiles, but also runs efficiently.

The MPI-IO interface is being proposed as an extension to the MPI standard to fill this need. MPI-IO supports a high-level interface to describe the partitioning of file data among processes, a collective interface describing complete transfers of global data structures between process memories and files, asynchronous I/O operations, allowing computation to be overlapped with I/O, and optimization of physical file layout on storage devices (disks).

Keywords: parallel I/O, multiprocessor file system interface, pario-bib

Comment: A more readable explanation of MPI-IO than the proposed-standard document corbett:mpi-io3. See polished book version, corbett:mpi-overview-book. See also the slides presented at IOPADS

Abstract: Thanks to MPI, writing portable message passing parallel programs is almost a reality. One of the remaining problems is file I/O. Although parallel file systems support similar interfaces, the lack of a standard makes developing a truly portable program impossible. It is not feasible to develop large scientific applications from scratch for each generation of parallel machine, and, in the scientific world, a program is not considered truly portable unless it not only compiles, but also runs efficiently.

The MPI-IO interface is being proposed as an extension to the MPI standard to fill this need. MPI-IO supports a high-level interface to describe the partitioning of file data among processes, a collective interface describing complete transfers of global data structures between process memories and files, asynchronous I/O operations, allowing computation to be overlapped with I/O, and optimization of physical file layout on storage devices (disks).

Keywords: parallel I/O, file system interface, pario-bib

Comment: Part of a whole book on parallel I/O; see iopads-book.

Abstract: Parallel computer architectures require innovative software solutions to utilize their capabilities. This statement is true for system software no less than for application programs. File system development for the IBM SP product line of computers started with the Vesta research project, which introduced the ideas of parallel access to partitioned files. This technology was then integrated with a conventional Advanced Interactive Executive (AIX) environment to create the IBM AIX Parallel I/O File System product. We describe the design and implementation of Vesta, including user interfaces and enhancements to the control environment needed to run the system. Changes to the basic design that were made as part of the AIX Parallel I/O File System are identified and justified.

Keywords: parallel file system, parallel I/O, Vesta, pario-bib

Comment: Probably the most authoritative Vesta/PIOFS paper yet. Good description of the system, motivations, etc. Not as much detail as some, like corbett:vesta-di.

Abstract: Row-Diagonal Parity (RDP) is a new algorithm for protecting against double disk failures. It stores all data unencoded, and uses only exclusive-or operations to compute parity. RDP is provably optimal in computational complexity, both during construction and reconstruction. Like other algorithms, it is optimal in the amount of redundant information stored and accessed. RDP works within a single stripe of blocks of sizes normally used by file systems, databases and disk arrays. It can be utilized in a fixed (RAID-4) or rotated (RAID-5) parity placement style. It is possible to extend the algorithm to encompass multiple RAID-4 or RAID-5 disk arrays in a single RDP disk array. It is possible to add disks to an existing RDP array without recalculating parity or moving data. Implementation results show that RDP performance can be made nearly equal to single parity RAID-4 and RAID-5 performance.

Keywords: fault tolerance, disk failures, algorithms, row-diagonal parity, RAID, pario-bib

Comment: Awarded best paper.

Keywords: parallel I/O, multiprocessor file system interface, pario-bib

Comment: Specs of the proposed SIO low-level interface for parallel file systems. Key features: linear file model, scatter-gather read and write calls (list of strided segments), asynch versions of all calls, extensive hint system. Naming structure is unspecified; no directories specified. Permissions left out. Some control over client caching and over disk layout. Each file has a (small) 'label', which is just a little space for application-controlled meta data. Optional extensions: collective read and write calls, fast copy.

Keywords: multiprocessor file system interface, parallel I/O, Vesta, pario-bib

Keywords: parallel I/O, multiprocessor file system, concurrent file checkpointing, multiprocessor file system interface, Vesta, pario-bib

Comment: See corbett:jvesta. Design of a file system for a message-passing MIMD multiprocessor to be used for scientific computing. Separate I/O nodes from compute nodes; I/O nodes and disks are viewed as a data-staging area. File system runs on I/O nodes only. Files declustered by record, among physical partitions, each residing on a separate disk, and each separately growable. Then the user maps logical partitions, one per process, on the file at open time. These are designed to be two-dimensional, so that mapping arrays of various strides and contiguities, with records as the basic unit, is easy. Various consistency and atomicity requirements. File checkpointing, really snapshotting, is built in. No client caching, no redundancy for reliability. See also corbett:vesta2, corbett:vesta3, feitelson:pario.

Abstract: The Vesta Parallel file system is designed to provide parallel file access to application programs running on multicomputers with parallel I/O subsystems. Vesta uses a new abstraction of files: a file is not a sequence of bytes, but rather it can be partitioned into multiple disjoint sequences that are accessed in parallel. The partitioning - which can also be changed dynamically - reduces the need for synchronization and coordiantion during the access. Some control over the layout of data is also provided, so the layout can be matched with the anticipated access patterns. The system is fully implemented, and is beginning to be used by application programmers. The implementation does not compromise scalability or parallelism. In fact, all data accesses are done directly to the I/O node that contains the requested data, without any indirection or access to shared metadata. There are no centralized control points in the system.

Keywords: parallel I/O, multiprocessor file system, file system interface, Vesta, pario-bib

Comment: See corbett:jvesta and corbett:vesta* for other background. Note that since this paper they have put Vesta on top of a raw disk (using 64 KB blocks) rather than on top of AIX-JFS. They describe here the structure of Vesta (2-d files, cells, subfiles, etc), the ordering of bytes within a subfile, hashing of the file name to find the file metadata, Xrefs instead of directories, caching, asynchronous I/O, prefetching, shared file pointers, concurrency control, and block-list structure. Many things, some visible to the user and some not, are new.

Keywords: multiprocessor file system, parallel I/O, Vesta, pario-bib

Comment: Complete user's manual of the Vesta file system. Impressive in its completeness (e.g., it has user quotas). Handy for its detailed description of the interface, but doesn't say much (of course) about the implementation.

Keywords: multiprocessor file system, file checkpointing, Vesta, pario-bib

Comment: See also corbett:jvesta, corbett:vesta, corbett:vesta3, feitelson:pario. A new abstraction and a new interface. Typical systems use transparent striping, and access modes. They believe that ``optimization requires control''. Need to be able to tell the system what you want. User-defined or default. Asynch I/O. Concurrency control. Checkpointing. Export/import to external storage. New abstraction: file is multiple sequences of records. Each process sees a logical partition of the file. Physical partition is one or more disks. Logical partition defined in terms of records. Can repartition without moving data. Rectilinear decompositions of file data to processors. They can do gather/scatter requests. Using logical partitions give system the knowledge that user's accesses are disjoint. Collective operations with consistency checks, vs. independent access. Collective open defines logical view, then synch, then check that partitions are disjoint. If not, then they have access modes to define semantics (more or less the same as other systems). Consider this a target for HPF, etc. Physical partitioning (record size and number of partitions) is defined at create time. Can they have different physical or logical partition sizes in the same file? Future: parallel pipelines, ``out-of-core'' backing store for HPF arrays, high-level operations, collective operations.

Keywords: parallel I/O, algorithm, pario-bib

Comment: Earlier version available as Dartmouth tech report PCS-TR93-193. But the most recent and complete version is Dartmouth PCS-TR94-223, cormen:bmmc-tr.

Keywords: parallel I/O algorithms, pario-bib

Comment: Supercedes cormen:bmmc.

Abstract: Although several algorithms have been developed for the Parallel Disk Model (PDM), few have been implemented. Consequently, little has been known about the accuracy of the PDM in measuring I/O time and total running time to perform an out-of-core computation. This paper analyzes timing results on multiple-disk platforms for two PDM algorithms, out-of-core radix sort and BMMC permutations, to determine the strengths and weaknesses of the PDM.

The results indicate the following. First, good PDM algorithms are usually not I/O bound. Second, of the four PDM parameters, one (problem size) is a good indicator of I/O time and running time, one (memory size) is a good indicator of I/O time but not necessarily running time, and the other two (block size and number of disks) do not necessarily indicate either I/O or running time. Third, because PDM algorithms tend not to be I/O bound, using asynchronous I/O can reduce I/O wait times significantly.

The software interface to the PDM is part of the ViC* run-time library. The interface is a set of wrappers that are designed to be both efficient and portable across several underlying file systems and target machines.

Keywords: parallel I/O, parallel I/O algorithm, compiler, pario-bib

Abstract: Although several algorithms have been developed for the Parallel Disk Model (PDM), few have been implemented. Consequently, little has been known about the accuracy of the PDM in measuring I/O time and total time to perform an out-of-core computation. This paper analyzes timing results on a uniprocessor with several disks for two PDM algorithms, out-of-core radix sort and BMMC permutations, to determine the strengths and weaknesses of the PDM.

The results indicate the following. First, good PDM algorithms are usually not I/O bound. Second, of the four PDM parameters, two (problem size and memory size) are good indicators of I/O time and running time, but the other two (block size and number of disks) are not. Third, because PDM algorithms tend not to be I/O bound, asynchronous I/O effectively hides I/O times.

The software interface to the PDM is part of the ViC* run-time library. The interface is a set of wrappers that are designed to be both efficient and portable across several parallel file systems and target machines.

Keywords: parallel I/O, parallel I/O algorithm, compiler, pario-bib

Comment: This used to be called cormen:early-vic but I renamed it because the paper will appear in parcomp.

Abstract: The fast Fourier transform (FFT) plays a key role in many areas of computational science and engineering. Although most one-dimensional FFT problems can be solved entirely in main memory, some important classes of applications require out-of-core techniques. For these, use of parallel I/O systems can improve performance considerably. This paper shows how to perform one-dimensional FFTs using a parallel disk system with independent disk accesses. We present both analytical and experimental results for performing out-of-core FFTs in two ways: using traditional virtual memory with demand paging, and using a provably asymptotically optimal algorithm for the parallel disk model (PDM) of I.S. Vitter and E.A.M. Shriver (1994). When run on a DEC 2100 server with a large memory and eight parallel disks, the optimal algorithm for the PDM runs up to 144.7 times faster than in-core methods under demand paging. Moreover, even including I/O costs, the normalized times for the optimal PDM algorithm are competitive, or better than, those for in-core methods even when they run entirely in memory.

Keywords: parallel I/O, out of core, scientific computing, FFT, pario-bib

Comment: see also cormen:fft2 and cormen:fft3. Part of a special issue.

Abstract: The Fast Fourier Transform (FFT) plays a key role in many areas of computational science and engineering. Although most one-dimensional FFT problems can be solved entirely in main memory, some important classes of applications require out-of-core techniques. For these, use of parallel I/O systems can improve performance considerably. This paper shows how to perform one-dimensional FFTs using a parallel disk system with independent disk accesses. We present both analytical and experimental results for performing out-of-core FFTs in two ways: using traditional virtual memory with demand paging, and using a provably asymptotically optimal algorithm for the Parallel Disk Model (PDM) of Vitter and Shriver. When run on a DEC 2100 server with a large memory and eight parallel disks, the optimal algorithm for the PDM runs up to 144.7 times faster than in-core methods under demand paging. Moreover, even including I/O costs, the normalized times for the optimal PDM algorithm are competitive, or better than, those for in-core methods even when they run entirely in memory.

Keywords: parallel I/O, out of core, scientific computing, FFT, pario-bib

Abstract: This paper extends an earlier out-of-core Fast Fourier Transform (FFT) method for a uniprocessor with the Parallel Disk Model (PDM) to use multiple processors. Four out-of-core multiprocessor methods are examined. Operationally, these methods differ in the size of "mini-butterfly" computed in memory and how the data are organized on the disks and in the distributed memory of the multiprocessor. The methods also perform differing amounts of I/O and communication. Two of them have the remarkable property that even though they are computing the FFT on a multiprocessor, all interprocessor communication occurs outside the mini-butterfly computations. Performance results on a small workstation cluster indicate that except for unusual combinations of problem size and memory size, the methods that do not perform interprocessor communication during the mini-butterfly computations require approximately 86% of the time of those that do. Moreover, the faster methods are much easier to implement.

Keywords: parallel I/O, out of core, scientific computing, FFT, pario-bib

Comment: Extends the work of cormen:fft.

Abstract: This paper extends an earlier out-of-core Fast Fourier Transform (FFT) method for a uniprocessor with the Parallel Disk Model (PDM) to use multiple processors. Four out-of-core multiprocessor methods are examined. Operationally, these methods differ in the size of "mini-butterfly" computed in memory and how the data are organized on the disks and in the distributed memory of the multiprocessor. The methods also perform differing amounts of I/O and communication. Two of them have the remarkable property that even though they are computing the FFT on a multiprocessor, all interprocessor communication occurs outside the mini-butterfly computations; communication that ordinarily occurs in a butterfly is folded into other data-movement operations. An analysis program shows that the two methods that use no butterfly communication usually use less communication overall than the other methods. The analysis program is fast enough that it can be invoked at run time to determine which of the four methods uses the least communication. One set of performance results on a small workstation cluster indicates that the methods without butterfly communication are approximately 9.5% faster. Moreover, they are much easier to implement.

Keywords: out of core, parallel I/O, pario-bib

Comment: They find a way to move the interprocessor communication involved in the out-of-core FFT into a single BMMC permutation between "super-levels", where each super-level involves log(M) stages of the FFT. This usually leads to less communication and to better overall performance. See also cormen:fft and cormen:fft2.

Abstract: FG is a programming environment for asynchronous programs that run on clusters and fit into a pipeline framework. It enables the programmer to write a series of synchronous functions and represents them as stages of an asynchronous pipeline. FG mitigates the high latency inherent in interprocessor communication and accessing the outer levels of the memory hierarchy. It overlaps separate pipeline stages that perform communication, computation, and I/O by running the stages asynchronously. Each stage maps to a thread. Buffers, whose sizes correspond to block sizes in the memory hierarchy, traverse the pipeline. FG makes such pipeline-structured parallel programs easier to write, smaller, and faster. FG offers several advantages over statically scheduled overlapping and dynamically scheduled overlapping via explicit calls to thread functions. First, it reduces coding and debugging time. Second, we find that it reduces code size by approximately 15-26%. Third, according to experimental results, it improves performance. Compared with programs that use static scheduling, FG-generated programs run approximately 61-69% faster on a 16-node Beowulf cluster. Compared with programs that make explicit calls for dynamically scheduled threads, FG-generated programs run slightly faster. Fourth, FG offers various design options and makes it easy for the programmer to explore different pipeline configurations.

Keywords: asynchronous I/O, pipelined I/O, pario-bib

Abstract: Several algorithms for parallel disk systems have appeared in the literature recently, and they are asymptotically optimal in terms of the number of disk accesses. Scalable systems with parallel disks must be able to run these algorithms. We present for the first time a list of capabilities that must be provided by the system to support these optimal algorithms: control over declustering, querying about the configuration, independent I/O, and turning off parity, file caching, and prefetching. We summarize recent theoretical and empirical work that justifies the need for these capabilities. In addition, we sketch an organization for a parallel file interface with low-level primitives and higher-level operations.

Keywords: parallel I/O, multiprocessor file systems, algorithm, file system interface, dfk, pario-bib

Comment: Describing the file system capabilities needed by parallel I/O algorithms to effectively use a parallel disk system. Revised as Dartmouth PCS-TR93-188 (updated).

Abstract: Several algorithms for parallel disk systems have appeared in the literature recently, and they are asymptotically optimal in terms of the number of disk accesses. Scalable systems with parallel disks must be able to run these algorithms. We present a list of capabilities that must be provided by the system to support these optimal algorithms: control over declustering, querying about the configuration, independent I/O, turning off file caching and prefetching, and bypassing parity. We summarize recent theoretical and empirical work that justifies the need for these capabilities.

Keywords: parallel I/O, multiprocessor file systems, algorithm, file system interface, dfk, pario-bib

Comment: Describing the file system capabilities needed by parallel I/O algorithms to effectively use a parallel disk system. Cite cormen:integrate.

Abstract: This paper presents asymptotically equal lower and upper bounds for the number of parallel I/O operations required to perform bit-matrix-multiply/complement (BMMC) permutations on the Parallel Disk Model proposed by J.S. Vitter and E.A.M. Shriver (1994). A BMMC permutation maps a source index to a target index by an affine transformation over GF(2), where the source and target indices are treated as bit vectors. The class of BMMC permutations includes many common permutations, such as matrix transposition (when dimensions are powers of 2), bit-reversal permutations, vector-reversal permutations, hypercube permutations, matrix reblocking, Gray-code permutations, and inverse Gray-code permutations. The upper bound improves upon the asymptotic bound in the previous best known BMMC algorithm and upon the constant factor in the previous best known bit-permute/complement (BPC) permutation algorithm. The algorithm achieving the upper bound uses basic linear-algebra techniques to factor the characteristic matrix for the BMMC permutation into a product of factors, each of which characterizes a permutation that can be performed in one pass over the data. The factoring uses new subclasses of BMMC permutations: memoryload-dispersal (MLD) permutations and their inverses. These subclasses extend the catalog of one-pass permutations. Although many BMMC permutations of practical interest fall into subclasses that might be explicitly invoked within the source code, this paper shows how to quickly detect whether a given vector of target addresses specifies a BMMC permutation. Thus, one can determine efficiently at run time whether a permutation to be performed is BMMC and then avoid the general-permutation algorithm and save parallel I/Os by using the BMMC permutation algorithm herein

Keywords: parallel I/O, parallel I/O algorithms, pario-bib

Keywords: scientific computing, out-of-core computation, parallel I/O, pario-bib

Comment: Part of a special issue on parallel and distributed I/O.

Keywords: parallel I/O algorithm, pario-bib

Comment: See also cormen:thesis.

Keywords: parallel I/O, algorithm, pario-bib

Comment: Lots of algorithms for out-of-core permutation problems. See also cormen:permute, cormen:integrate.

Abstract: This paper describes the functionality of ViC*, a compiler-like preprocessor for out-of-core C*. The input to ViC* is a C* program but with certain shapes declared \verb`outofcore`, which means that all parallel variables of these shapes reside on disk. The output is a standard C* program with the appropriate I/O and library calls added for efficient access to out-of-core parallel variables.

Keywords: compiler, out-of-core computation, parallel I/O, pario-bib

Abstract: We present a sort-first parallel system for out-of-core rendering of large models on cluster-based tiled displays. The system renders high-resolution images of large models at interactive frame rates using off-the-shelf PCs with small memory. Given a model, we use an out-of-core preprocessing algorithm to build an on-disk hierarchical representation for the model. At run time, each PC renders the image for a display tile, using an out-of-core rendering approach that employs multiple threads to overlap rendering, visibility computation, and disk operations. The system can operate in approximate mode for real-time rendering, or in conservative mode for rendering with guaranteed accuracy. Running our system in approximate mode on a cluster of 16 PCs each with 512 MB of main memory, we are able to render 12-megapixel images of a 13-million-triangle model with 99.3% of accuracy at 10.8 frames per second. Rendering such a large model at high resolutions and interactive frame rates would typically require expensive high-end graphics hardware. Our results show that a cluster of inexpensive PCs is an attractive alternative to those high-end systems. (36 refs.)

Keywords: cluster based tiled displays, out-of-core rendering, sort first parallel rendering, pario-bib

Keywords: cooperative caching, distributed file system, parallel I/O, pario-bib

Comment: Part of jin:io-book; reformatted version of cortes:cooperative.

Keywords: parallel file system, RAID, cluster computing, parallel I/O, pario-bib

Abstract: In this paper, we examine some of the important problems observed in the design of cooperative caches. Solutions to the coherence, load-balancing and fault-tolerance problems are presented. These solutions have been implemented as a part of PAFS, a parallel/distributed file system, and its performance has been compared to the one achieved by xFS. Using the comparison results, we have observed that the proposed ideas not only solve the main problems of cooperative caches, but also increase the overall system performance. Although the solutions presented in this paper were targeted to a parallel machine, reasonable good results have also been obtained for networks of workstations.

Keywords: cooperative caching, distributed file system, parallel I/O, pario-bib

Comment: They make the claim that it is better not to replicate data into local client caches, rather, it is better to simply make remote read and write requests to the cached block in whatever memory it may be. That reduces the overhead (space and time) of replication and coherency, and leads to better performance. They also present a range of parity-based fault-tolerance mechanisms, and a load-balancing technique that reassigns cache buffers to cache-manager processes.

Abstract: Disk arrays, or RAIDs, have become the solution to increase the capacity and bandwidth of most storage system, but their usage has some limitations because all the disks in the array have to be equal. Nowadays, assuming a homogeneous set of disks to build an array is becoming a not very realistic assumption in many environments, especially in low-cost clusters of workstations. It is difficult to find a disk with the same characteristics as the ones in the array and replacing or adding new disks breaks the homogeneity. In this paper, we propose two block-distribution algorithms (one for RAID0 and an extension for RAID5) that can be used to build disk arrays from a heterogeneous set of disks. We also show that arrays using this algorithm are able to serve many more disk requests per second than when blocks are distributed assuming that all disks have the lowest common speed, which is the solution currently being used. (C) 2003 Elsevier Science (USA). All rights reserved.

Keywords: AdaptRaid, block distribution, heterogeneity; RAID, parallel I/O, pario-bib

Abstract: RAIDs level 5 are one of the most widely used kind of disk array, but their usage has some limitations because all the disks in the array have to be equal. Nowadays, assuming a homogeneous set of disks to build an array is becoming a not very realistic assumption in many environments, especially in low-cost clusters of workstations. It is difficult to and a disk with the same characteristics as the ones in the array and replacing or adding new disks breaks the homogeneity. In this paper, we propose a block- distribution algorithm that can be used to build disk arrays from a heterogeneous set of disks. We also show that arrays using this algorithm are able to serve many more disk requests per second than when blocks are distributed assuming that all disks have the lowest common speed, which is the solution currently being used.

Keywords: parallel I/O, RAID, pario-bib

Comment: The web page for the project is http://people.ac.upc.es/toni/AdaptRaid.html

Abstract: Heterogeneous disk arrays are becoming a common configuration in many sites and specially in storage area networks (SAN). As new disks have different characteristics than old ones, adding new disks or replacing old ones ends up in a heterogeneous disk array. Current solutions to this kind of arrays do not take advantage of the improved characteristics of the new disks. In this paper, we present a block-distribution algorithm that takes advantage of these new characteristics and thus improves the performance and capacity of heterogeneous disk arrays compared to current solutions.

Keywords: disk array, parallel I/O, pario-bib

Comment: The technical report associated with this paper can be found at ftp://ftp.ac.upc.es/pub/reports/DAC/2000/UPC-DAC-2000-76.ps.Z

Abstract: Clusters of workstations are becoming a quite popular platform to run high-performance applications. This fact has stressed the need of high-performance storage systems for this kind of environments. In order to design such systems, we need adequate tools, which should be flexible enough to model a cluster of workstations. Currently available simulators do not allow heterogeneity (several kind of disks), hierarchies or resource sharing (among others), which are quite common in clusters. To fill this gap, we have designed and implemented HRaid, which is a very flexible and easy to use storage-system simulator. In this paper, we present this simulator, its main abstractions and some simple examples of how it can be used.

Keywords: simulation, RAID, disk array, storage system, heterogeneous system, parallel I/O, pario-bib

Abstract: Summary form only given. After these two decades, it is now a good time to go through all the done work and try to learn the important lessons all these parallel I/O initiatives have taught us. This paper aims at giving this global overview. The focus is not on commercial/academic systems/prototypes, but on the concepts that lay behind them. These concepts have normally been applied at different levels, and thus, such an overview can be of interest to many people ranging from the hardware design to the application implementation. Some of the most important concepts that are discussed are, among others, data placement (RAIDs, 2D and 3D files, ...), network architectures for parallel I/O (Network attached devices, SAN, ...), parallel caching and prefetching (cooperative caching, Informed caching and prefetching, ...), and interfaces (collective I/O, data distribution interfaces, ...).

Keywords: tutorial, parallel I/O overview, pario-bib

Comment: Tutorial given at Cluster 2004.

Abstract: A new cooperative caching mechanism, PACA, along with a caching algorithm, LRU-Interleaved, and an aggressive prefetching algorithm, Full-File-On-Open, are presented. The caching algorithm is especially targeted to parallel machines running a microkernel-based operating system. It avoids the cache coherence problem with no loss in performance. Comparing our algorithm with N-Chance Forwarding, in the above environment, better results have been obtained by LRU-Interleaved. We also evaluate an aggressive prefetching algorithm that highly increases read performance taking advantage of the huge caches cooperative caching offers.

Keywords: file caching, multiprocessor file system, cooperative caching, parallel I/O, pario-bib

Comment: Contact toni@ac.upc.es. See also the a longer version of the paper, cortes:paca-tr.

Keywords: file caching, multiprocessor file system, cooperative caching, parallel I/O, pario-bib

Comment: See cortes:paca.

Abstract: In this paper we describe PAFS, a new parallel/distributed file system. Within the whole file system, special interest is placed on the caching mechanism. We present a cooperative cache that has the advantages of cooperation and avoids the problems derived from the coherence mechanisms. Furthermore, this has been achieved with a reasonable gain in performance. In order to show the obtained performance, we present a comparison between PAFS and xFS (a file system that also implements a cooperative cache).

Keywords: file caching, multiprocessor file system, cooperative caching, cache coherence, parallel I/O, pario-bib

Comment: Contact toni@ac.upc.es.

Abstract: In this paper we present PAFS, a new parallel/distributed file system. Within the whole file system, special interest is placed on the caching and prefetching mechanisms. We present a cooperative cache that avoids the coherence problem while it continues to be highly scalable and achieves very good performance. We also present an aggressive prefetching algorithm that allows full utilization of the big caches offered by the cooperative cache mechanism. All the results presented in this paper have obtained through simulation using the Sprite workload.

Keywords: file caching, multiprocessor file system, cooperative caching, cache coherence, parallel I/O, pario-bib

Comment: A longer, more detailed version of cortes:pafs.

Abstract: Cooperative caches offer huge amounts of caching memory that is not always used as well as it could be. We might find blocks in the cache that have not been requested for many hours. These blocks will hardly improve the performance of the system while the buffers they occupy could be better used to speed-up the I/O operations. In this paper, we present a family of simple prefetching algorithms that increase the file-system performance significantly. Furthermore, we also present a way to make any simple prefetching algorithm into an aggressive one that controls its aggressiveness not to flood the cache unnecessarily. All these algorithms and mechanisms have proven to increase the performance of two state-of-the-art parallel/distributed file systems: PAFS and xFS.

Keywords: parallel I/O, file access pattern, prefetching, caching, simulation, pario-bib

Comment: They present algorithms for "linear aggressive prefetching" for systems using a cooperative cache. Two prediction schemes are used: a OBA (one block ahead) and IS_PPM (Interval and size -prediction by partial match). The aggressive prefetch algorithm continuosly prefetches data until a miss-prediction occurs. When a mis-prediction occurs, they realize that they were on the wrong path and start prefetching again from the mis-predicted block. To limit the aggressiveness of the prefetching, they only allow one block from each file to be prefetched at a time. If a single application is running, this forces a parallel reads to only utilize one disk at a time. They claim, however, that when many files are being accessed they achieve good disk utilization. They implemented the prefetching algorithms on the xFS anderson:serverless and PAFS cortes:pafs file systems. They used a trace-driven simulator DIMEMAS labarta:dip to obtain performance results for portions of the CHARISMA and Sprite workloads. The results show that using aggressive prefetching does not usually load the system more than a system with no prefetching, and sometimes, it even lowers the disk traffic.

Keywords: parallel I/O, file access pattern, prefetching, caching, pario-bib

Abstract: Redundant disk arrays are single fault tolerant, incorporating a layer of error handling not found in nonredundant disk systems. Recovery from these errors is complex, due in part to the large number of erroneous states the system may reach. The established approach to error recovery in disk systems is to transition directly from an erroneous state to completion. This technique, known as forward error recovery, relies upon the context in which an error occurs to determine the steps required to reach completion, which implies forward error recovery is design specific. Forward error recovery requires the enumeration of all erroneous states the system may reach and the construction of a forward path from each erroneous state. We propose a method of error recovery which does not rely upon the enumeration of erroneous states or the context in which errors occur. When an error is encountered, we advocate mechanized recovery to an error-free state from which an operation may be retried. Using a form of backward error recovery, we are able to manage the complexity of error recovery in redundant disk arrays without sacrificing performance.

Keywords: parallel I/O, disk array, RAID, redundancy, reliability, recovery, pario-bib

Comment: Also available in HTML format at http://www.cs.cmu.edu/Web/Groups/PDL/HTML-Papers/CMG94/c.fm.html.

Keywords: parallel I/O, disk array, RAID, redundancy, reliability, recovery, pario-bib

Keywords: parallel I/O, RAID, disk array, reliability, simulation, pario-bib

Comment: See expanded version gibson:raidframe-tr.

Keywords: parallel I/O, file system, network, pario-bib

Comment: See also coyne:storage.

Keywords: parallel I/O, file system, network, pario-bib

Comment: See also coyne:hpss. They describe the National Storage Laboratory at LLNL. Collaboration with many companies. The idea is to build a combined storage system from many disk and tape components that is networked to supercomputers. The philosophy is to separate control and data network traffic, so that the overall control can be managed by a (relatively) small computer, without the same computer needing to pump all of the data through it's CPU. The data would go directly from the devices to the client supercomputer. They also want to support multiple hierarchies of data storage, so that new technologies can be inserted without disrupting existing hierarchies. Access interface is layered so that high-level abstractions can be provided as well as low-level control for those who need it.

Abstract: Grand challenge applications have to process large amounts of data, and then require high performance IO systems. Cluster computing is a good alternative to proprietary system for building cost effective IO intensive platform: some cluster architectures won sorting benchmark (MinuteSort, Datamation)! Recent advances in IO component technologies (disk, controller and network) let us expect higher IO performance for data intensive applications on cluster. The counterpart of this evolution is that much stress is put on the different buses (memory, IO) of each node which cannot be scaled. In this paper we investigate a strategy we called READ2 (Remote Efficient Access to Distant Device) to reduce the stress. With READ2 any cluster node accesses directly to remote disk: the remote processor and the remote memory are removed from the control and data path: Inputs/Outputs don't interfere with the host processor and the host memory activity. With READ2 strategy, a cluster can be considered as a shared disk architecture instead of a shared nothing one. This papers describes an implementation of READ^2 on Myrinet Networks. First experimental results show IO performance improvement.

Keywords: parallel I/O, pario-bib

Abstract: Rapid increases in computing and communication performance are exacerbating the long-standing problem of performance-limited input/output. Indeed, for many otherwise scalable parallel applications, input/output is emerging as a major performance bottleneck. The design of scalable input/output systems depends critically on the input/output requirements and access patterns for this emerging class of large-scale parallel applications. However, hard data on the behavior of such applications is only now becoming available. In this paper, we describe the input/output requirements of three scalable parallel applications (electron scattering, terrain rendering, and quantum chemistry) on the Intel Paragon XP/S. As part of an ongoing parallel input/output characterization effort, we used instrumented versions of the application codes to capture and analyze input/output volume, request size distributions, and temporal request structure. Because complete traces of individual application input/output requests were captured, in-depth, off-line analyses were possible. In addition, we conducted informal interviews of the application developers to understand the relation between the codes' current and desired input/output structure. The results of our studies show a wide variety of temporal and spatial access patterns, including highly read-intensive and write-intensive phases, extremely large and extremely small request sizes, and both sequential and highly irregular access patterns. We conclude with a discussion of the broad spectrum of access patterns and their profound implications for parallel file caching and prefetching schemes.

Keywords: file access pattern, file system workload, workload characterization, parallel I/O, pario-bib

Comment: They use the Pablo instrumentation and analysis tools to instrument three scalable applications that use heavy I/O: electron scattering, terrain rendering, and quantum chemistry. They look at the volume of data moved, the timing of I/O, and the periodic nature of I/O. They do a little bit with the access patterns of data within each file. They found a HUGE variation in request sizes, amount of I/O, number of files, and so forth. Their primary conclusion is thus that file systems should be adaptable to different access patterns, preferably under control of the application. Note proceedings only available on CD-ROM or WWW.

Keywords: parallel I/O, out-of-core algorithm, computational geometry, data structure, pario-bib

Comment: See also the component papers vitter:survey, arge:lower, crauser:segment, grossi:crosstrees, toledo:survey. Not clear to what extent these papers are about *parallel* I/O.

Keywords: parallel I/O, disk architecture, disk array, pario-bib

Comment: Glossy from Cray describing their new disk subsystem: up two four controllers and up to four ``drives'', each of which actually have four spindles. Thus, a full subsystem has 16 disks. Each drive or controller sustains 9.6 MBytes/sec sustained, for a total of 38.4 MBytes/sec. Each drive has 4.8 GBytes, for a total of 19.2 Gbytes. Access time per drive is 2-46.6 msec, average 24 msec. They don't say how the 4 spindles within a driver are controlled or arranged.

Keywords: parallel I/O, parallel file system, pario-bib

Comment: Man pages for his Flex version of file interface. See crockett:par-files.

Keywords: parallel I/O, file access pattern, parallel file system, pario-bib

Comment: Two views of a file: global (for sequential programs) and internal (for parallel programs). Standardized forms for these views, for long-lived files. Temp files have specialized forms. The access types are sequential, partitioned, interleaved, and self-scheduled, plus global random and partitioned random. He relates these to their best storage patterns. No mention of prefetching. Buffer cache only needed for direct (random) access. The application must specify the access pattern desired.

Keywords: parallel I/O, disk array, pario-bib

Keywords: workload characterization, scientific computing, parallel programming, message passing, pario-bib

Comment: Some mention of I/O.

Keywords: workload characterization, scientific computing, parallel programming, message passing, pario-bib

Comment: Some mention of I/O, though only in a limited way. Average 1207B/MFlop. Some of the applications do I/O throughout their run (2400B/MFlop avg), while others only do I/O at the beginning or end (14B/MFlop avg). But I/O is bursty, so larger bandwidths are suggested. The applications are parallel programs running on Intel Delta, nCUBE/1, nCUBE/2, and are in C, FORTRAN, or both.

Abstract: Dynamic simulations based on time-varying inputs are extremely I/O intensive. This is shown by industrial applications generating environmental projections based on seasonal-to-interannual climate forecasts which have a compute to data-access ratio of O(n) leading to significant performance degradation. Exploitation of compression techniques such as Run-Length-Encoding (RLE) significantly reduces the I/O bottleneck and storage requirements. Unfortunately, traditional RLE algorithms do not perform well in a parallel-vector platform such as the Cray architecture. This paper describes the design and implementation of a new RLE algorithm based on data chunking and packing that exploits the Cray gather-scatter vector hardware and multiple processors. This innovative approach reduces I/O and file storage requirements on average by an order of magnitude. Data intensive applications such as the integration of environmental and global climate models now become practical in a realistic time-frame.

Keywords: parallel I/O application, compression, pario-bib

Comment: In a Special Issue on I/O in Parallel Applications, volume 12, numbers 3 and 4.

Keywords: parallel I/O, pario-bib

Keywords: parallel I/O, MIMD, pario-bib

Comment: Journalized version of debenedictis:pario, debenedictis:ncube, and delrosario:nCUBE.

Keywords: parallel file system, parallel I/O, pario-bib

Comment: Interesting paper. Describes their mechanism for mapping I/O so that the file system knows both the mapping of a data structure into memory and on the disks, so that it can do the permutation and send the right data to the right disk, and back again. Interesting Unix-compatible interface. Needs to be extended to handle complex formats.

Keywords: parallel I/O, multiprocessor file system, file system interface, pario-bib

Comment: Looks like they give the byte-level mapping, then do normal reads and writes; mapping routes the data to and from the correct place. But it does let you intermix comp with I/O. Elegant concept. Nice interface. Works best for cases where 1) data layout known in advance, data format is known, and mapping is regular enough for easy specification. I think that irregular or unknown mappings could still be done with a flat mapping.

Keywords: parallel I/O, Unix, pario-bib

Comment: A more polished version of his other papers with del Rosario. The mapping-based mechanism is released in nCUBE software 3.0. It does support shared file pointers for self-scheduled I/O, as well as support for variable-length records, and asynchronous I/O (although the primary mechanism is for synchronous, i.e., SPMD, I/O). The basic idea of scalable pipes (between programs, devices, etc.) with mappings that determine routings to units seems like a good idea.

Abstract: The Direct Access File System (DAFS) is a new, fast, and lightweight remote file system protocol. DAFS targets the data center by addressing the performance and functional needs of clusters of application servers. We call this the local file sharing environment. File access performance is improved by utilizing Direct Access Transports, such as InfiniBand, Remote Direct Data Placement, and the Virtual Interface Architecture. DAFS also enhances file sharing semantics compared to prior network file system protocols. Applications using DAFS through a user-space I/O library can bypass operating system overhead, further improving performance. We present performance measurements of an IP-based DAFS network, demonstrating the DAFS protocol's lower client CPU requirements over commodity Gigabit Ethernet. We also provide the first multiprocessor scaling results for a well-known application (GNU gzip) converted to use DAFS.

Keywords: direct access file system, dafs, remote dma, pario-bib

Keywords: parallel I/O, parallel file system, pario-bib

Comment: More detail on the mapping functions, and more flexible mapping functions (can be user specified, or some from a library). Striped disks, parallel pipes, graphics, and HIPPI supported.

Keywords: parallel I/O, survey, pario-bib

Comment: Nice summary of grand-challenge and other applications, and their I/O needs. Points out the need for quantitative studies of workloads. Comments on architectures, eg, the advent of per-node disk devices. OS problems include communication latency, data decomposition, interface, prefetching and caching, and checkpointing. Runtime system and compilers are important, particularly in reference to data-mapping and re-mapping (see delrosario:two-phase). Persistent object stores and networking are mentioned briefly.

Keywords: parallel I/O, multiprocessor file system, pario-bib

Keywords: parallel I/O, multiprocessor file system, pario-bib

Comment: They show performance measurements of various data distributions on an nCUBE and the Touchstone Delta, for reading matrix from a column-major file striped across disks, into some distribution across procs. Distributions that don't match the I/O distribution are really terrible, due to having more, smaller requests, and sometimes mismatching the stripe size (getting seg-like contention) or block size (reading partial blocks). They find it is better to read the file using the `best' distribution, then to reshuffle the data in memory. Big speedups.

Keywords: parallel I/O, parallel file system, heterogeneous, pario-bib

Comment: They are planning a parallel file system that is layered on top of standard workstation file systems, to be used by parallel applications on heterogeneous workstation clusters. All in user-level libraries, and on a per-application basis, application programs can distributed their data among many files on many machines. They plan to use a mapped interface like that of debenedictis:modular, and support efficient collective I/O in ways reminiscent of bennett:jovian and kotz:diskdir. Published as harry:vipfs.

Abstract: Many parallel application areas that exploit massive parallelism, such as climate modeling, require massive storage systems for the archival and retrieval of data sets. As such, advances in massively parallel computation must be coupled with advances in mass storage technology in order to satisfy I/O constraints of these applications. We demonstrate the effects of such I/O-computation disparity for a representative distributed information system, NASA's Earth Observing System Distributed Information System (EOSDIS). We use performance modeling to identify bottlenecks in EOSDIS for two representative user scenarios from climate change research.

Keywords: climate modeling, performance modeling, parallel I/O, pario-bib

Keywords: parallel I/O, database, GAMMA, pario-bib

Comment: Better to cite dewitt:gamma3. Multiprocessor (VAX) DBMS on a token ring with disk at each processor. They thought this was better than separating disks from processors by network since then network must handle all I/O rather than just what needs to move. Conjecture that shared memory might be best interconnection network. Relations are horizontally partitioned in some way, and each processor reads its own set and operates on them there.

Keywords: parallel I/O, database, performance analysis, Teradata, GAMMA, pario-bib

Comment: Compared Gamma with Teradata. Various operations on big relations. See fairly good linear speedup in many cases. They vary only one variable at a time. Their bottleneck was at the memory-network interface.

Keywords: parallel I/O, database, GAMMA, pario-bib

Comment: Almost identical to dewitt:gamma, with some updates. See that for comments, but cite this one. See also dewitt:gamma3 for a more recent paper.

Keywords: parallel I/O, database, GAMMA, pario-bib

Comment: An updated version of dewitt:gamma2, with elements of dewitt:gamma-dbm. Really only need to cite this one. This is the same basic idea as dewitt:gamma2, but after they ported the system from the VAXen to an iPSC/2. Speedup results good. Question: how about comparing it to a single-processor, single-disk system with increasing disk bandwidth? That is, how much of their speedup comes from the increasing disk bandwidth, and how much from the actual use of parallelism?

Keywords: database, parallel computing, parallel I/O, pario-bib

Comment: They point out that the comments of boral:critique - that database machines were doomed - did really not come true. Their new thesis is that specialized hardware is not necessary and has not been successful, but that parallel database systems are clearly succesful. In particular, they argue for shared-nothing layouts. They survey the state-of-the-art parallel DB systems. Earlier version in Computer Architecture News 12/90.

Keywords: parallel I/O, parallel database, external sorting, pario-bib

Comment: Comparing exact and probabilistic splitting for external sorting on a database. Model and experimental results from Gamma machine. Basically, the idea is to decide on a splitting vector, which defines $N$ buckets for an $N$-process program, and have each program read its initial segment of the data and send each element to the appropriate bucket (other process). All elements received are written to disks as small sorted runs. Then each process mergesorts its runs. Probabilistic split uses only a sample of the elements to define the vector.

Keywords: Carla, Bridge, multiprocessor file system, Butterfly, parallel I/O, pario-bib

Comment: See also dibble:*

Keywords: parallel I/O, sorting, merging, parallel file reference pattern, pario-bib

Comment: Based on Bridge file system (see dibble:bridge). Parallel external merge-sort tool. Sort file on each disk, then do a parallel merge. The merge is serialized by the token-passing mechanism, but the I/O time dominates. The key is to keep disks busy constantly. Uses some read-ahead, write-behind to control fluctuations in disk request timing. Analytical analysis of the algorithm lends insight and matches well with the timings. Locality is a big win in Bridge tools.

Keywords: parallel I/O, external sorting, merging, parallel file reference pattern, pario-bib

Comment: Subset of dibble:sort. Extra comments to distinguish from striping and RAID work. Good point that those projects are addressing a different bottleneck, and that they can provide essentially unlimited bandwidth to a single processor. Bridge could use those as individual file systems, parallelizing the overall file system, avoiding the software bottleneck. Using a very-reliable RAID at each node in Bridge could safeguard Bridge against failure for reasonable periods, removing reliability from Bridge level.

Keywords: parallel I/O, external sorting, merging, parallel file system, pario-bib

Comment: Also TR 334. Mostly covered by other papers, but includes good introduction, discussion of reliability and maintenance issues, and implementation. Short mention of prefetching implied that simple OBL was counter-productive, but later tool-specific buffering with read-ahead was often important. The three interfaces to the PIFS server are interesting. A fourth compromise might help make tools easier to write.

Abstract: In this paper, we evaluate the impact on performance of various implementation techniques for collective I/O operations, and we do so across four important parallel architectures. We show that a naive implementation of collective I/0 does not result in significant performance gains for any of the architectures, but that an optimized implementation does provide excellent performance across all of the platforms under study. Furthermore, we demonstrate that there exists a single implementation strategy that provides the best performance for all four computational platforms. Next, we evaluate implementation techniques for thread-based collective I/O operations. We show that the most obvious implementation technique, which is to spawn a thread to execute the whole collective I/O operation in the background, frequently provides the worst performance, often performing much worse than just executing the collective I/O routine entirely in the foreground. To improve performance, we explore an alternate approach where part of the collective I/O operation is performed in the background, and part is performed in the foreground. We demonstrate that this implementation technique can provide significant performance gains, offering up to a 50% improvement over implementations that do not attempt to overlap collective I/O and computation.

Keywords: parallel I/O, collective I/O, pario-bib, parallel architecture

Abstract: Java is quickly becoming the preferred language for writing distributed applications because of its inherent support for programming on distributed platforms. In particular, Java provides compile-time and run-time security, automatic garbage collection, inherent support for multithreading, support for persistent objects and object migration, and portability. Given these significant advantages of Java, there is a growing interest in using Java for high-performance computing applications. To be successful in the high-performance computing domain, however, Java must have the capability to efficiently handle the significant I/O requirements commonly found in high-performance computing applications.

While there has been significant research in high-performance I/O using languages such as C, C++, and Fortran, there has been relatively little research into the I/O capabilities of Java. In this paper, we evaluate the I/O capabilities of Java for high-performance computing. We examine several approaches that attempt to provide high-performance I/O-many of which are not obvious at first glance-and investigate their performance in both parallel and multithreaded environments. We also provide suggestions for expanding the I/O capabilities of Java to better support the needs of high-performance computing applications.

Keywords: parallel I/O, Java, pario-bib

Abstract: Massively parallel computers are increasingly being used to solve large, I/O intensive applications in many different fields. For such applications, the I/O requirements quite often present a significant obstacle in the way of achieving good performance, and an important area of current research is the development of techniques by which these costs can be reduced. One such approach is collective I/O, where the processors cooperatively develop an I/O strategy that reduces the number, and increases the size, of I/O requests, making a much better use of the I/O subsystem. Collective I/O has been shown to significantly reduce the cost of performing I/O in many large, parallel applications, and for this reason serves as an important base upon which we can explore other mechanisms which can further reduce these costs. One promising approach is to use threads to perform the collective I/O in the background while the main thread continues with other computation in the foreground.

In this paper, we explore the issues associated with implementing collective I/O in the background using threads. The most natural approach is to simply spawn off an I/O thread to perform the collective I/O in the background while the main thread continues with other computation. However, our research demonstrates that this approach is frequently the worst implementation option, often performing much more poorly than just executing collective I/O completely in the foreground. To improve the performance of thread-based collective I/O, we developed an alternate approach where part of the collective I/O operation is performed in the background, and part is performed in the foreground. We demonstrate that this new technique can significantly improve the performance of thread-based collective I/O, providing up to an 80% improvement over sequential collective I/O (where there is no attempt to overlap computation with I/O). Also, we discuss one very important application of this research which is the implementation of the split-collective parallel I/O operations defined in MPI 2.0.

Keywords: parallel I/O, multithread programming, collective I/O, disk-directed I/O, two-phase I/O, pario-bib

Comment: They examine an implementation of collective I/O in MPI2 such that the collective I/O is done in the background, using a thread, while the computation continues. They found that the performance can be quite disappointing, because of the competition for the CPU between the computational thread and the background thread executing the redistribution phase of the I/O operation. They get better results by doing the redistribution in the foreground, making the computation wait, and then doing the I/O in the background thread while the computation continues. Results from four major parallel platforms, but only for write operations.

Keywords: parallel I/O, neural network, pario-bib

Comment: An application that reads large files, sequentially, on CM2 with DataVault.

Keywords: scientific application, parallel I/O, ocean modeling, climate modeling, pario-bib

Comment: They describe the approaches taken to optimize an out-of-core parallel ocean model simulation on parallel distributed-memory machines. The original code used fixed size memory windows to store the in-core portions of dataset on the machine. The code used the same approach for machines that had enough memory to store the entire data set in-core, except rather than reading and writing to disk, the code copied to/from ramdisk (very copy intensive). The new code added an option to allow the entire dataset to be run in-core. Another place where the code could be optimized was in the writing of the dataset. For computational efficiency, the data was stored in memory as an array U(ix,iz,iy), but other applications needed the data stored on disk as U(ix,iy,iz). To optimize the I/O, the new code allocated additional processors to gather and re-organize and write the data to disk (much like Salvo).

Keywords: RAID, disk array, network file system, parallel I/O, pario-bib

Comment: See also chen:raid2. The only significant addition in this paper is a discussion of the performance of the RAID-II running an LFS file system.

Keywords: parallel I/O, pario-bib

Comment: RAID-3 striping across drives in a tape robot, using 3 data plus one parity. Tape-switch time is very high, ie, 4 minutes. Switching four tapes at the same time would only get a little overlap, because there is only one robot arm. Assume large request size. Striping is much faster when only one request is considered, but with many requests outstanding, response time goes way down due to limited concurrency. More readers with the same stripe group size alleviate the contention and allow concurrency. Faster readers is the most important thing to improve performance, more important than improving robot speed. As both speeds improve the benefit of striping diminishes. Seems like this could be expressed in a simple equation...

Keywords: intel, ncube, hypercube, multiprocessor architecture, performance, parallel I/O, pario-bib

Comment: An excellent paper presenting lots of detailed performance measurements on the iPSC/1, iPSC/2, iPSC/860, nCUBE 3200, and nCUBE 6400: arithmetic, FLOPS, communication, I/O. Tables of numbers provide details needed for simulation. iPSC/860 definitely is fastest, but way out of balance wrt communication vs. computation. Number of message hops is not so important in newer machines.

Keywords: parallel I/O, scheduling, pario-bib

Comment: They note that the set of data transfers in a parallel I/O architecture can be expressed as a graph coloring problem. Realistically, a centralized solution is not possible because the information is inherently distributed. So they develop some distributed algorithms and experimentally compare them to the centralized algorithm. They get within 5% and do better than earlier algorithms.

Abstract: A growing imbalance in CPU (central processing unit) and I/O (input/output) speeds has led to a communications bottleneck in distributed architectures, especially for data intensive applications such as multimedia information systems, databases, and grand challenge problems. Our solution is to schedule parallel I/O operations explicitly. We present a class of decentralized scheduling algorithms that eliminate contention for I/O ports while maintaining an efficient use of bandwidth. These algorithms based on edge coloring and matching of bipartite graphs, rely upon simple heuristics to obtain shorter schedules. We use simulation to evaluate the ability of our algorithms to obtain near optimal solutions in a distributed context, and compare our work with that of other researchers. Our results show that our algorithms produce schedules within 5 of the optimal schedule, a substantial improvement over existing algorithms. (20 refs.)

Keywords: randomized edge coloring, scheduling algorithms, bipartite graphs, parallel I/O, pario-bib

Keywords: parallel I/O algorithms, pario-bib

Comment: They devise some decentralized algorithms to generate schedules for data transfers between a set of clients and a set of servers when the complete set of transfers is known in advance, and the clients and servers are fairly tightly synchronized. They concentrate on the limitation that clients and servers may each only participate in one transfer at any given moment; interconnect bandwidth is not an issue. Their simulations show that their algorithms come within 20% of optimal.

Abstract: The cost of data transfers, and in particular of I/O operations, is a growing problem in parallel computing. This performance bottleneck is especially severe for data-intensive applications such as multimedia information systems, databases, and Grand Challenge problems. A promising approach to alleviating this bottleneck is to schedule parallel I/O operations explicitly.

Although centralized algorithms for batch scheduling of parallel I/O operations have previously been developed, they are not be appropriate for all applications and architectures. We develop a class of decentralized algorithms for scheduling parallel I/O operations, where the objective is to reduce the time required to complete a given set of transfers. These algorithms, based on edge-coloring and matching of bipartite graphs, rely upon simple heuristics to obtain shorter schedules. We present simulation results indicating that the best of our algorithms can produce schedules whose length (or makespan) is within 2 - 20% of the optimal schedule, a substantial improvement on previous decentralized algorithms. We discuss theoretical and experimental work in progress and possible extensions.

Keywords: parallel I/O, distributed scheduling algorithm, pario-bib

Comment: Part of a whole book on parallel I/O; see iopads-book.

Keywords: parallel computer architecture, MIMD, pario-bib

Comment: Basically the same architecture as the nCUBE/2, scaled up. Eight to 65K processors, each 50 MIPS and 100 DP MFLOPS, initially 50 MHz. RISC. 16 hypercube channels and 2 I/O channels per processor. CPU chip includes MMU, TLB, I- and D-cache, hypercube and I/O channels, and memory interface. The channels have DMA support built-in (5 usec startup overhead, worst-case end-to-end latency 10 usec), and can talk directly to the memory interface or to the cache. 64-bit virtual address space, with 48 bits implemented. Hardware support for distributed virtual memory. Separate 16-node hypercube is used for I/O processing, with up to 400 disks attached. Packaging includes multi-chip module with DRAMs stacked directly on the CPU chip, fluid-cooled, so that an entire node is one package, with the 18 network links as essentially its only external connections.

Keywords: parallel I/O, disk caching, parallel file system, log-structured file system, Intel iPSC/2, pario-bib

Comment: Essentially a small literature survey. No new ideas here, but it is a reasonable overview of the situation. Mentions caching, striping, disk layout optimization, log-structured file systems, and Bridge and Intel CFS. Plugs their ``Swift'' architecture (see cabrera:pario).

Keywords: parallel I/O, parallel architecture, performance evaluation, pario-bib

Comment: See el-ghazawi:mpio.

Keywords: parallel I/O, parallel architecture, performance evaluation, pario-bib

Comment: See el-ghazawi:mp1.

Keywords: parallel file system, parallel I/O, pario-bib

Comment: See also elford:ppfs-tr, huber:ppfs.

Keywords: parallel file system, parallel I/O, pario-bib

Comment: See also elford:ppfs-detail, huber:ppfs-scenarios, huber:ppfs.

Keywords: trends, disk technology, disk array, parallel I/O, pario-bib

Keywords: Carla, multiprocessor file system, Bridge, Butterfly, parallel I/O, pario-bib

Abstract: The problem of providing file I/O to parallel programs has been largely neglected in the development of multiprocessor systems. There are two essential elements of any file system design intended for a highly parallel environment: parallel I/O and effective caching schemes. This paper concentrates on the second aspect of file system design and specifically, on the question of whether prefetching blocks of the file into the block cache can effectively reduce overall execution time of a parallel computation, even under favorable assumptions.

Experiments have been conducted with an interleaved file system testbed on the Butterfly Plus multiprocessor. Results of these experiments suggest that 1) the hit ratio, the accepted measure in traditional caching studies, may not be an adequate measure of performance when the workload consists of parallel computations and parallel file access patterns, 2) caching with prefetching can significantly improve the hit ratio and the average time to perform an I/O operation, and 3) an improvement in overall execution time has been observed in most cases. In spite of these gains, prefetching sometimes results in increased execution times (a negative result, given the optimistic nature of the study).

We explore why is it not trivial to translate savings on individual I/O requests into consistently better overall performance and identify the key problems that need to be addressed in order to improve the potential of prefetching techniques in this environment.

Keywords: dfk, parallel file system, prefetching, disk caching, MIMD, parallel I/O, pario-bib

Abstract: NonStop SQL is an implementation of ANSI/ISO SQL on Tandem Computer Systems. In its second release, NonStop SQL transparently and automatically implements parallelism within an SQL statement by exploiting Tandem's multiprocessor architecture. For basic queries on a uniform database, it achieves performance that is near-linear with respect to the number of processors and disks used. The authors describe benchmarks demonstrating these results and the technology used to achieve them.

Keywords: parallel database, parallel architecture, parallel I/O, pario-bib

Comment: They (briefly) describe the Tandem NonStop system, including their disk nodes (which contain CPU, memory, and disk) and their use. A query involves sending a request to all the disk nodes, who independently read the appropriate data from their local disk, filter out all the interesting records, and send only those interesting records to the originator for processing. This is an early example of smart (programmable) I/O nodes.

Keywords: multiprocessor architecture, pario-bib

Comment: A nice summary of the Paragon architecture and OS. Some information that is not found in Intel's technical summary, and with much less marketing hype. But, it was written in April 1993 with a look to the future, so it may represent things that are not ready yet. Network interface allows user-mode msgs, DMA direct to user space if receive has been posted; else there is a new queue for every possible sending processor. They plan to expand the nodes to 4-processors and 64-128 MB. PFS stripes across RAIDs. Now SCSI-1 with 5 MB/s, later 10 MB/s SCSI-2, then 20 MB/s fast SCSI-2. See also intel:paragon.

Abstract: As network latency drops below disk latency, access time to a remote disk will begin to approach local disk access time. The performance of I/O may then be improved by spreading disk pages across several remote disk servers and accessing disk pages in parallel. To research this we have prototyped a data page server called a Page File. This persistent data type provides a set of methods to access disk pages stored on a cluster of remote machines acting as disk servers. The goal is to improve the throughput of database management system or other I/O intensive application by accessing pages from remote disks and incurring disk latency in parallel. This report describes the conceptual foundation and the methods of access for our prototype.

Keywords: parallel I/O, network, virtual memory, parallel database, pario-bib

Comment: An early document on a system under development. It declusters pages of a file across many page servers, and provides an abstraction of a linearly ordered collection of pages. The intended use is by database systems. As it stands now, there is little here other than block declustering, and thus, nothing new to the I/O community. Perhaps later they will develop interesting new caching or prefetching strategies.

Abstract: Remotely sensed imagery has been used for developing and validating various studies regarding land cover dynamics. However, the large amounts of imagery collected by the satellites are largely contaminated by the effects of atmospheric particles. The objective of atmospheric correction is to retrieve the surface reflectance from remotely sensed imagery by removing the atmospheric effects. We introduce a number of computational techniques that lead to a substantial speedup of an atmospheric correction algorithm based on using look-up tables. Excluding I/O time, the previous known implementation processes one pixel at a time and requires about 2.63 seconds per pixel on a SPARC-10 machine, while our implementation is based on processing the whole image and takes about 4-20 microseconds per pixel on the same machine. We also develop a parallel version of our algorithm that is scalable in terms of both computation and I/O. Experimental results obtained show that a Thematic Mapper (TM) image (36 MB per band, 5 bands need to be corrected) can be handled in less than 4.3 minutes on a 32-node CM-5 machine, including I/O time.

Keywords: remote sensing, parallel I/O application, pario-bib

Comment: Note proceedings only on CD-ROM or WWW.

Keywords: multiprocessor file system, parallel I/O, Vesta, pario-bib

Comment: Part of jin:io-book. Excellent survey. Reformatted version of feitelson:pario.

Abstract: Applications executing on massively parallel supercomputers require a high aggregate bandwidth of I/O with low latency. This requirement cannot be satisfied by an external file server. Once solution is to employ an internal parallel I/O subsystem, in which I/O nodes with DASD are linked wo the same interconnection network that connects the compute nodes. The option of increasing the number of I/O nodes together with the number of compute nodes allows for a balanced architecture. Indeed, most multicomputer vendors provide internal parallel I/O subsystems as part of their product offerings. However, these systems typically attempt to preserve a Unix-compatible interface, hiding or abstracting the parallelism. New interfaces may be required to fully utilize the capabilities of Parallel I/O.

Keywords: multiprocessor file system, parallel I/O, Vesta, pario-bib

Comment: A very nice survey of multiprocessor file systems issues. Published version of feitelson:pario-tr.

Keywords: multiprocessor file system, parallel I/O, Vesta, pario-bib, OS94W

Comment: A very nice survey of multiprocessor file systems issues. They make a good point that I/O needs would increase if I/O capabilities increase, because people would output more interations, more complete data sets, etc.\ They make the case for internal file systems, the use of dedicated I/O nodes, the attachment of every RAID to two I/O nodes for reliability, the Vesta interface, and user control over the view of a parallel file. See also corbett:vesta*. Published as feitelson:pario.

Keywords: parallel I/O, pario-bib

Comment: How to deal with stdin/stdout on a parallel processor. Basically, each task is given its own window, where the user can see the output and type input to that task. Then, they have a window for LEDs, ie, little squares, one for each task. The square changes color depending on the situation. The default is to turn green when output is available, red when waiting for input, and white when the window is currently open. Clicking on these opens the appropriate window, so there is some control over which windows you are watching. They also provide a programmer interface to allow the programmer to control the LED color.

Keywords: parallel I/O, multiprocessor file system, Vesta, pario-bib

Comment: See feitelson:vesta-perf-tr.

Keywords: parallel I/O, multiprocessor file system, Vesta, pario-bib

Comment: Cite feitelson:vesta-perf. A good performance study of Vesta running on an SP-1. See corbett:jvesta for ultimate reference. In all, Vesta performed very well both for single-node and multiple-node performance. I wish that they had tried some very small BSUs; at one point they tried 16-byte BSUs and the performance looked very poor. Section on I/O vectors is confusing.

Keywords: parallel I/O, parallel I/O interface, pario-bib

Comment: Part of jin:io-book.

Abstract: A fault tolerant parallel virtual file system is designed and implemented to provide high I/O performance and high reliability. A queuing model is used to analyze in detail the average response time when multiple clients access the system. The results show that I/O response time is with a function of several operational parameters. It decreases with the increase in I/O buffer hit rate for read requests, write buffer size for write requests and number of server nodes in the parallel file system, while higher I/O requests arrival rate increases I/O response time.

Keywords: fault-tolerance, PVFS, perforamance analysis, pario-bib

Abstract: Without any additional cost, all the disks on the nodes of a cluster can be connected together through CEFT-PVFS, an RAID-10 style parallel file system, to provide a multi-GB/s parallel I/O performance. I/O response time is one of the most important measures of quality of service for a client. When multiple clients submit data-intensive jobs at the same time, the response time experienced by the user is an indicator of the power of the cluster. In this paper, a queuing model is used to analyze in detail the average response time when multiple clients access CEFT-PVFS. The results reveal that response time is with a function of several operational parameters. The results show that I/O response time decreases with the increases in I/O buffer hit rate for read requests, write buffer size for write requests and the number of server nodes in the parallel file system, while the higher the I/O requests arrival rate, the longer the I/O response time. On the other hand, the collective power of a large cluster supported by CEFT-PVFS is shown to be able to sustain a steady and stable I/O response time for a relatively large range of the request arrival rate.

Keywords: PVFS, parallel I/O, I/O response time, pario-bib

Abstract: In a previous work [Ferragina-Grossi, ACM STOC 95], we proposed a text indexing data structure for secondary storage, which we called SB-tree, that combines the best of B-trees and suffix arrays, overcoming the limitations of inverted files, suffix arrays, suffix trees, and prefix B-trees. In this paper we study the performance of SB-trees in a practical setting, performing a set of searching and updating experiments. Improved performance was obtained by a new space efficient and alphabet-independent organization of the internal nodes of the SB-tree, and a new batch insertion procedure that avoids thrashing.

Keywords: out-of-core algorithm, parallel I/O, pario-bib

Keywords: out-of-core algorithm, parallel I/O, pario-bib

Abstract: Processing and analyzing large volumes of data plays an increasingly important role in many domains of scientific research. High-level language and compiler support for developing applications that analyze and process such datasets has, however, been lacking so far.

In this paper, we present a set of language extensions and a prototype compiler for supporting high-level object-oriented programming of data intensive reduction operations over multidimensional data. We have chosen a dialect of Java with data-parallel extensions for specifying a collection of objects, a parallel for loop, and reduction variables as our source high-level language. Our compiler analyzes parallel loops and optimizes the processing of datasets through the use of an existing run-time system, called active data repository (ADR). We show how loop fission followed by interprocedural static program slicing can be used by the compiler to extract required information for the run-time system. We present the design of a compiler/run-time interface which allows the compiler to effectively utilize the existing run-time system.

A prototype compiler incorporating these techniques has been developed using the Titanium front-end from Berkeley. We have evaluated this compiler by comparing the performance of compiler generated code with hand customized ADR code for three templates, from the areas of digital microscopy and scientific simulations. Our experimental results show that the performance of compiler generated versions is, on the average 21% lower, and in all cases within a factor of two, of the performance of hand coded versions.

Keywords: parallel I/O, parallel applications, data parallel, pario-bib

Keywords: pario-bib, application

Comment: Best Application Paper award.

This paper describes a client/server application that emulates a high power light microscope. They use wavelet compression to reduce the size of each of the electronic slides and they use a parallel data server much like the ones used for sattelite image data (see chang:titan) to service data requests.

Keywords: parallel I/O, multiprocessor file system, benchmark, pario-bib

Comment: See also carter:benchmark. Some preliminary results from one of their benchmarks. Note: ``I was only using a single Cray disk with a maximum transfer rate of 9.6MBytes/sec.'' - Fineberg.

Abstract: MPI-IO provides a demonstrably efficient portable parallel Input/Output interface, compatible with the MPI standard. PMPIO is a "reference implementation" of MPI-IO, developed at NASA Ames Research Center. To date, PMPIO has been ported to the IBM SP-2, SGI and Sun shared memory workstations, the Intel Paragon, and the Cray J90. Preliminary results using the PMPIO implementation of MPI-IO show an improvement of as much as a factor of 20 on the NAS BTIO benchmark compared to a Fortran based implementation. We show comparative results on the SP-2 Paragon, and SGI architectures.

Keywords: parallel I/O, pario-bib

Keywords: parallel I/O, hypercube, parallel file system, pario-bib

Comment: For hypercube-like architectures. Interleaved files, though flexible. Separate network for I/O, maybe not hypercube. I/O is blocked and buffered - no coherency or prefetching issues discussed. Buffered close to point of use. Parallel access is ok. Broadcast supported? I/O nodes distinguished from comp nodes. I/O hooked to front-end too. See hadimioglu:fs and hadimioglu:hyperfs

Abstract: Increased computer networking has sparked a resurgence of the `on-line' revolution of the 1970's, making ever larger amounts of data available on a world wide basis and placing greater demands on the performance and availability of tertiary storage systems. In this paper, we argue for a new approach to tertiary storage system architecture that is obtained by coupling multiple small and inexpensive `building block' libraries (or jukeboxes) together to create larger tertiary storage systems. We call the resulting system a RAIL and show that it has performance and availability characteristics superior to conventional tertiary storage systems, for almost the same dollar/megabyte cost. A RAIL system is the tertiary storage equivalent of a fixed magnetic disk RAID storage system, but with several additional features that enable the ideas of data striping and redundancy to function efficiently on dismountable media and robotic media mounting systems. We present the architecture of such a system called Starfish I and describe the implementation of a prototype. We also introduce the idea of creating a log-structured library array (LSLA) on top of a RAIL architecture (StarFish II) and show how it can have write performance equivalent to that of secondary storage, and improved read performance along with other advantages such as easier compression and the elimination of the 4*RAID/RAIL write penalty.

Keywords: parallel I/O, redundant data, striping, tertiary storage, pario-bib

Comment: Part of a special issue.

Keywords: parallel I/O, pario-bib

Comment: Part of jin:io-book; reformatted version of nieplocha:arrays.

Keywords: computational science, chemistry, parallel I/O, pario-bib

Comment: A library package for computational chemistry programs. It supports out-of-core arrays. See also nieplocha:chemio.

Keywords: parallel I/O, parallel database, multiprocessor file system, climate model, grand challenge, tertiary storage, archival storage, RAID, tape robot, pario-bib

Comment: Includes the slides from many presenters covering climate modeling, data requirements for climate models, archival storage systems, multiprocessor file systems, and so forth. NCAR data storage growth rates (p. 54), 500 bytes per MFlop, or about 8 TB/year with Y/MP-8. Average file length 26.2 MB. Migration across both storage hierarchy and generations of media. LLNL researcher: typical 50-year, 3-dimensional model with 5-degree resolution will produce 75 GB of output. Attendee list included.

Abstract: As high-speed networks make it easier to use distributed resources, it becomes increasingly common that applications and their data are not colocated. Users have traditionally addressed this problem by manually staging data to and from remote computers. We argue instead for a remote I/O paradigm in which programs use familiar parallel I/O interfaces to access remote filesystems. In addition to simplifying remote execution, remote I/O can improve performance relative to staging by overlapping computation and data transfer or by reducing communication requirements. However, remote I/O also introduces new technical challenges in the areas of portability, performance, and integration with distributed computing systems. We propose techniques designed to address these challenges and describe a remote I/O library called RIO that we are developing to evaluate the effectiveness of these techniques. RIO addresses issues of portability by adopting the quasi-standard MPI-IO interface and by defining a RIO device and RIO server within the ADIO abstract I/O device architecture. It addresses performance issues by providing traditional I/O optimizations such as asynchronous operations and through implementation techniques such as buffering and message forwarding to offload communication overheads. Microbenchmarks and application experiments demonstrate that our techniques can improve turnaround time relative to staging.

Keywords: parallel I/O, distributed file system, pario-bib

Comment: They want to support users that have datasets at different locations in the Internet, but need to access the data at supercomputer parallel machines. Rather than staging data in and out, they want to provide remote access. Issues: naming, dynamic loads, heterogeneity, security, fault-tolerance. All traffic goes through a 'forwarder node' that funnels all the traffic into the network. They use URLs for pathnames (e.g., "x-rio://..."). They find that non-blocking ops are important, as is collective I/O. They think that buffering will be important. Limited experiments.

Keywords: hypercube, pario-bib

Comment: See fox:cubix for parallel I/O.

Keywords: parallel file system, hypercube, pario-bib

Comment: Parallel I/O control, called CUBIX. Interesting method. Depends a lot on ``loose synchronization'', which is sortof SIMD-like.

Abstract: Between 1994 and 1997, researchers at Southwest Research Institute (SwRI) investigated methods for distributing parallel computation and data visualization under the support of an internally funded Research Initiative Program entitled the Advanced Visualization Technology Project (AVTP). A hierarchical data cache architecture was developed to provide a flexible interface between the modeling or simulation computational processes and data visualization programs. Compared to conventional post facto data visualization approaches, this data cache structure provides many advantages including simultaneous data access by multiple visualization clients, comparison of experimental and simulated data, and visual analysis of computer simulation as computation proceeds.

However, since the data cache was resident on a single workstation, this approach did not address the issue of scalability of methods for avoiding the data storage bottleneck by distributing the data across multiple networked workstations. Scalability through distributed database approaches is being investigated as part of the Applied Visualization using Advanced Network Technology Infrastructure (AVANTI) project.

This paper describes a methodology currently under development that is intended to avoid bottlenecks that typically arise as the result of data consumers (e.g. visualization applications) that must access and process large amounts of data that has been generated and resides on other hosts, and which must pass through a central data cache prior to being used by the data consumer. The methodology is based on a fundamental paradigm that the end result (visualization) rendered by a data consumer can, in many cases, be produced using a reduced data set that has been distilled or filtered from the original data set.

In the most basic case, the filtered data used as input to the data consumer may simply be a proper subset of massive data sets that have been distributed among hosts. For the general case, however, the filtered data may bear no resemblance to the original data since it is the result of processing the raw data set and distilling it to its visual "essence", i.e. the minimal data set that is absolutely required by the data consumer in order to perform the required rendering function. Data distribution bottlenecks for visualization applications are thereby reduced by avoiding the transfer of large amounts of raw data in favor of considerably distilled visual data.

There are, of course, computational costs associated with this approach since raw data must be processed into its visual essence, but these computational costs may be distributed among multiple processors. It should be realized, however, that, in general, these computational costs would exist any way since, for the visualization to be performed, there must be a transformation between the raw data and the visualization primitives (e.g. line segments, polygon vertices, etc.) to be rendered. The main principal put forth by this paper is that if data distribution bottlenecks are to be minimized, the amount of raw data transferred should be reduced by employing data filtering processes that can be distributed among multiple hosts.

The complete paper demonstrates, both analytically and experimentally, that this approach becomes increasingly effective (scalable) as the computational expense associated with the data filtering transformation rises.

Keywords: distributed computing, filters, grid, input/output, parallel I/O, pario-bib, app-pario

Comment: The goal of their work is to improve the performance of data visualization applications which use remote the data generators (disk or running application) and data consumers (the visualization station) for visualzation applications. They deal with network bottlenecks by using a distributed-redundant data cache to hold intermediate data between the data generator and the data consumer. They also reduce network traffic by applying data filters to the data at the distributed cache processors. The main argument is that since the data must be filtered before it is visualized, it makes more sense to perform the filter at the data cache so the computation can be distributed and to reduce the amount of data that needs to be transferred to the data consumer.

Keywords: parallel I/O, multiprocessor file system, pario-bib

Comment: The complete paper on the SPIFFI parallel file system. It seems to be much like Intel CFS from the programmer's point of view, with a few new file modes, user-selectable striping granularity. Their Paragon, though a source of problems, had a disk on every node (though they do not take advantage of that in this work). They have a buffer pool on each I/O node, which does prefetching in a somewhat novel way.

Keywords: parallel file system, multimedia, video server, pario-bib

Comment: See also freedman:spiffi. They simulate their video-on-demand server. Their model is a cluster of workstation servers, connected by a network to video-display terminals. The terminals just have a circular buffer queue that they fill by making requests to the server, and drain by uncompressing MPEG and displaying video. The servers manage a buffer pool and a set of striped disks. All videos are striped across all disks. They use dual LRU lists in the server buffer pool: one for used blocks, and one for prefetched blocks (``love prefetching''). They use a ``real-time'' disk scheduling algorithm that prioritizes requests by their deadlines (or anticipated deadline in case of a prefetch). Their metric is maximum number of terminals that can be supported without glitches. They plan to implement their system on a workstation cluster.

Keywords: interactive visualization, multi-resolution visualization, adaptive visualization, scientific application, parallel octrees, pario-bib

Comment: They describe a technique that combines heirarchical data reduction methods with parallel computing to allow "interactive exploration of large data sets while retaining full-resolution capabilities." They point out that visualization of large data sets requires a post-processing step to reduce the size, or sophisticated rendering algorithms that work with the full resolution. There method combines the two techniques.

Keywords: parallel I/O, multiprocessor file system, pario-bib

Comment: Proposes the min_SAR, max_SAR, and ratio phi as measures of aggregate file system bandwidth. Has to do with load balance issues; how well the file system balances between competing nodes in a heavy-use period.

Keywords: parallel I/O, Intel iPSC/2, pario-bib

Keywords: parallel I/O, Intel iPSC/2, pario-bib

Keywords: parallel I/O, Intel iPSC/2, disk caching, prefetching, pario-bib

Comment: Nice study of performance of existing CFS system on 32-node + 4 I/O-node iPSC/2. They show big improvements due to declustering, preallocation, caching, and prefetching. See also pratt:twofs.

Keywords: parallel I/O, pario-bib

Comment: They give a useful overview of the I/O requirements of many applications codes, in terms of input, output, scratch files, debugging, and checkpointing. They also describe their architecture-independent I/O interface that provides calls to read and write entire arrays, with some flexibility in the format and distribution of the array. Curious centralized control method. Limited performance evaluation. They're trying to keep the I/O media, file layout, and I/O architecture transparent to the user. Implementation decides which processors actually do read/write. Data formatted or unformatted; file sequential or parallel; can specify distributed arrays with ghost points. Runs on lots of platforms; will also be implementing on IBM SP-1 with disk per node, 128 nodes. Their package is freely available via ftp. Future: buffer-size experiments, unstructured data, use parallel file internally and then seqeuentialize on close.

Keywords: parallel I/O, pario-bib

Comment: Part of jin:io-book; reformatted version of galbreath:applio.

Keywords: disk array, RAID, parallel I/O, pario-bib, survey

Keywords: parallel I/O, disk striping, load balancing, pario-bib

Comment: Using trace-driven simulation to compare dynamic load-balancing techniques in databases that span several disk drives, with the inherent load-balancing of striping. Their traces were from two Oracle databases on two different NCR systems. They found that striping, with its essentially random block-by-block load balancing, does a better job of avoiding short-term load imbalances than the ``manual'' load-balancing does.

Abstract: This article describes an implementation of MPI-IO using a new parallel file system, called Expand (Expandable Parallel File System), which is based on NFS servers. Expand combines multiple NFS servers to create a distributed partition where files are striped. Expand requires no changes to the NFS server and uses RPC operations to provide parallel access to the same file. Expand is also independent of the clients, because all operations are implemented using RPC and NFS protocols. Using this system, we can join heterogeneous servers (Linux, Solaris, Windows 2000, etc.) to provide a parallel and distributed partition. The article describes the design, implementation and evaluation of Expand with MPI-IO. This evaluation has been made in Linux clusters and compares Expand and PVFS.

Keywords: parallel file system, parallel I/O, pario-bib

Keywords: parallel I/O, disk striping, reliability, disk array, pario-bib

Comment: Reliability of striped filesystems may not be as bad as you think. Parity disks help. Performance improvements limited to small number of disks ($n<10$). Good point: efficiency of striping will increase as the gap between CPU/memory performance and disk speed and file size widens. Reliability may be better if measured in terms of performing a task in time T, since the striped version may take less time. This gives disks less opportunity to fail during that period. Also consider the CPU failure mode, and its use over less time.

Abstract: In recent years, many commercial Massively Parallel Processor (MPP) systems have been available to the computing community. These systems provide very high processing power (up to hundreds of GFLOPs), and can scale efficiently with the number of processors. However, many scientific and commercial applications that run on these multiprocessors may not experience significant benefit in terms of speedup and are bottlenecked by their I/O requirements. Although these multiprocessors may be configured with sufficient I/O hardware, the file system software often fails to provide the available I/O bandwidth to the application, and causes severe performance degradation for I/O intensive applications.

A highly efficient parallel file system has been implemented on Intel's Teraflops (TFLOPS) machine and provides a sustained I/O bandwidth of 1 GB/sec. This file system provides almost 95% of the available raw hardware I/O bandwidth and the I/O bandwidth scales proportional to the available I/O nodes.

Intel's TFLOPS machine is the first Accelerated Strategic Computing Initiative (ASCI) machine that DOE has acquired. This computer is 10 times more powerful than the fastest machine today, and will be used primarily to simulate nuclear testing and to ensure the safety and effectiveness of the nation's nuclear weapons stockpile.

This machine contains over 9000 Intel's Pentium Pro processors, and will provide a peak CPU performance of 1.8 teraflops. This papers presents the I/O design and architecture of Intel's TFLOPS supercomputer, describes the Cougar OS I/O and its interface with the Intel's Parallel File System.

Keywords: parallel file system, intel, ASCI Red, pario-bib

Comment: Describes the parallel file system for ASCI Red. The paper is only available as HTML

Abstract: Bulk Synchronous Parallel ML or BSML is a functional data-parallel language for programming bulk synchronous parallel (BSP) algorithms. The execution time can be estimated and dead-locks and indeterminism are avoided. For large scale applications where parallel processing is helpful and where the total amount of data often exceeds the total main memory available, parallel disk I/O becomes a necessity. We present here a library of I/O features for BSML and its cost model.

Keywords: parallel I/O, parallel ML, BSML, data parallel language, pario-bib

Keywords: parallel I/O, multimedia, pario-bib

Comment: Part of jin:io-book; reformatted version of gennart:comparing.

Abstract: Multimedia interfaces increase the need for large image databases, capable of storing and reading streams of data with strict synchronicity and isochronicity requirements. In order to fulfil these requirements, we use a parallel image server architecture which relies on arrays of intelligent disk nodes, each disk node being composed of one processor and one or more disks. This contribution analyzes through simulation the real-time behavior of two multiprocessor multi-disk architectures: GigaView and the Unix workstation cluster. GigaView incorporates point-to-point communication between processing units and the workstation cluster supports communication through a shared bus-and-memory architecture. For a standard multimedia server architecture consisting of 8 disks and 4 disk-node processors, we evaluate stream frame access times under various parameters such as load factors, frame size, stream throughput and synchronicity requirements. We compare the behavior of GigaView and the workstation cluster in terms of delay and delay jitter

Keywords: parallel I/O, multimedia, pario-bib

Keywords: parallel I/O, IBM SP2, pario-bib

Keywords: multimedia, parallel I/O, pario-bib

Comment: Part of jin:io-book; reformatted version of ghandeharizadeh:mitra.

Abstract: Mitra is a scalable storage manager that supports the display of continuous media data types, e.g., audio and video clips. It is a software based system that employs off-the-shelf hardware components. Its present hardware platform is a cluster of multi-disk workstations, connected using an ATM switch. Mitra supports the display of a mix of media types. To reduce the cost of storage, it supports a hierarchical organization of storage devices and stages the frequently accessed objects on the magnetic disks. For the number of displays to scale as a function of additional disks, Mitra employs staggered striping. It implements three strategies to maximize the number of simultaneous displays supported by each disk. First, the EVEREST file system allows different files (corresponding to objects of different media types) to be retrieved at different block size granularities. Second, the FIXB algorithm recognizes the different zones of a disk and guarantees a continuous display while harnessing the average disk transfer rate. Third, Mitra implements the Grouped Sweeping Scheme (GSS) to minimize the impact of disk seeks on the available disk bandwidth.

In addition to reporting on implementation details of Mitra, we present performance results that demonstrate the scalability characteristics of the system. We compare the obtained results with theoretical expectations based on the bandwidth of participating disks. Mitra attains between 65% to 100% of the theoretical expectations.

Keywords: multimedia, parallel I/O, pario-bib

Comment: This paper describes the continous media server Mita. Mita runs on a cluster of multi-disk HP 9000/735 workstations. Each workstation consists of 80 Mbytes of memory and four disks. They implement ''staggered striping'' of the data in which disks are clustered based on media type and treated as a single logical unit. Data is then striped across the logical disk cluster in a round-robin fashion. They present performance results as a function of total number of disks and the number of disks in a cluster.

Keywords: parallel I/O, multimedia, pario-bib

Comment: Part of a special issue.

Abstract: We have designed and implemented the Google File System, a scalable distributed file system for large distributed data-intensive applications. It provides fault tolerance while running on inexpensive commodity hardware, and it delivers high aggregate performance to a large number of clients. While sharing many of the same goals as previous distributed file systems, our design has been driven by observations of our application workloads and technological environment, both current and anticipated, that reflect a marked departure from some earlier file system assumptions. This has led us to re-examine traditional choices and explore radically different design points.

The file system has successfully met our storage needs. It is widely deployed within Google as the storage platform for the generation and processing of data used by our service as well as research and development efforts that require large data sets. The largest cluster to date provides hundreds of terabytes of storage across thousands of disks on over a thousand machines, and it is concurrently accessed by hundreds of clients.

In this paper, we present file system interface extensions designed to support distributed applications, discuss many aspects of our design, and report measurements from both micro-benchmarks and real world use.

Keywords: distributed file system, pario-bib

Keywords: parallel I/O, MIMD, multiprocessor architecture, hypercube, pario-bib

Comment: Given a hypercube that has I/O nodes scattered throughout, they compare a plain one to one that has the I/O nodes also interconnected with a half-size hypercube. They show that this has better performance because the I/O traffic does not interfere with normal inter-PE traffic. See also ghosh:pario.

Keywords: multiprocessor file system, parallel I/O, pario-bib

Comment: They simulate a 128-node hypercube with 16 I/O nodes attached at uniformly distributed points. They compare two architectures: one with a separate I/O network, and another without a separate I/O network. When there, the extra network is used to route I/O packets from the originating I/O node to the I/O node closest to the destination processing node (or vice versa). They run simulations under workloads with differing amounts of locality, and experiment with different bandwidths for the links. They conclude that the extra network helps. But they never make the (proper, fair) comparison where the total network bandwidth is held constant. See also ghosh:hyper.

Keywords: parallel I/O, RAID, redundancy, reliability, pario-bib

Keywords: parallel I/O, disk array, disk striping, reliability, RAID, pario-bib

Comment: Excellent book. Good source for discussion of the access gap and transfer gap, disk lifetimes, parity methods, reliability analysis, and generally the case for RAIDs. Page 220 he briefly discusses multiprocessor I/O architecture.

Keywords: network-attached storage, storage architecture, parallel I/O, pario-bib

Comment: Part of jin:io-book; reformatted version of gibson:storage.

Abstract: The most difficult and frequently most important challenge for high performance file access is the achievement of low latency cache misses. We propose to explore the utility and feasibility of using file access hints to schedule overlapped prefetching of file data. Hints may be issued explicitly by programmers, automatically by compilers, speculatively by parent tasks such as shells and makes, or historically by previously profiled executions. Our research will also address the thorny issues of hint specification, memory resource management, imprecise and incorrect hints, and appropriate interfaces for propogating hints through and to effected application, operating system, file system, and device specific modules. We begin our research with a detailed examination of two applications with large potential for improvement: compilation of multiple module software systems and scientific simulation using very large grid state files.

Keywords: file system, prefetching, pario-bib

Comment: A relatively early TIP report with nothing really new over patterson:tip.

Abstract: The ever increasing need for I/O bandwidth will be met with ever larger arrays of disks. These arrays require redundancy to protect against data loss. This paper examines alternative choices for encodings, or codes, that reliably store information in disk arrays. Codes are selected to maximize mean time to data loss or minimize disks containing redundant data, but are all constrained to minimize performance penalties associated with updating information or recovering from catastrophic disk failures. We also codes that give highly reliable data storage with low redundant data overhead for arrays of 1000 information disks.

Keywords: parallel I/O, disk array, RAID, reliability, pario-bib

Comment: See gibson:raid for comments since it is the same.

Abstract: By providing direct data transfer between storage and client, network-attached storage devices have the potential to improve scalability for existing distributed file systems (by removing the server as a bottleneck) and bandwidth for new parallel and distributed file systems (through network striping and more efficient data paths). Together, these advantages influence a large enough fraction of the storage market to make commodity network-attached storage feasible. Realizing the technology's full potential requires careful consideration across a wide range of file system, networking and security issues. This paper contrasts two network-attached storage architectures-(1) Networked SCSI disks (NetSCSI) are network attached storage devices with minimal changes from the familiar SCSI interface, while (2) Network-Attached Secure Disks (NASD) are drives that support independent client access to drive object services. To estimate the potential performance benefits of these architectures, we develop an analytic model and perform trace-driven replay experiments based on AFS and NFS traces. Our results suggest that NetSCSI can reduce file server load during a burst of NFS or AFS activity by about 30%. With the NASD architecture, server load (during burst activity) can be reduced by a factor of up to five for AFS and up to ten for NFS.

Keywords: NASD, network-attached disks, distributed file system, parallel file system, security, secure disks, pario-bib

Comment: Essentially, the conference (and subsequent) journal version of gibson:nasd-tr. The studies that use simple analytical models (based on measured workloads of NFS and AFS file managers) to compare performance of NASD to SAD (storage-attached disks) and NetSCSI are often cited as justification for the NASD and object-based storage approaches.

Abstract: By providing direct data transfer between storage and client, network-attached storage devices have the potential to improve scalability (by removing the server as a bottleneck) and performance (through network striping and shorter data paths). Realizing the technology's full potential requires careful consideration across a wide range of file system, networking and security issues. To address these issues, this paper presents two new network-attached storage architectures. (1) Networked SCSI disks (NetSCSI) are network-attached storage devices with minimal changes from the familiar SCSI interface (2) Network-attached secure disks (NASD) are drives that support independent client access to drive provided object services. For both architectures, we present a sketch of repartitionings of distributed file system functionality, including a security framework whose strongest levels use tamper resistant processing in the disks to provide action authorization and data privacy even when the drive is in a physically insecure location.

Using AFS and NFS, trace results suggest that NetSCSI can reduce file server load during a burst of AFS activity by a factor of about 2; for the NASD architecture, server load (during burst activity) can be reduced by a factor of about 4 for AFS and 10 for NFS.

Keywords: parallel I/O, network attached storage, distributed file systems, computer security, network attached secure disks, NASD, capability system, pario-bib

Comment: They outline their rationale for the idea of Network-attached Secure Disks (NASD). Basically the idea is to develop disk drives that attach right to the LAN, rather than to a file server, and allow clients to access the disks directly for many of the simpler file system actions (read and write file data, read file attributes), and only contact the server for more complex activities (opening and creating files, changing attributes). This removes the load from file servers, which are getting too slow to move large amounts of data needed by large installations. Issues include security, of course, which they solve with encryption (for privacy) and time-limited capabilities (keys) given out by the server to authenticated clients, which the clients show to the disk to gain access. They compare the performance of NASD, using a simple analytical model and parameters obtained from measuring real NFS and AFS implementations, to the performance of SAD (server-attached disks) and NetSCSI (a hybrid approach that involves the server in every operation but allows data to flow directly from disk to and from the network).

Keywords: parallel I/O, RAID, reliability, disk array, pario-bib

Comment: Design of parity encodings to handle more than one bit failure in any group. Their 2-bit correcting codes are good enough for 1000-disk RAIDs that 3-bit correction is not needed.

Keywords: parallel I/O, RAID, disk array, reliability, simulation, pario-bib

Comment: Short version appeared as courtright:raidframe. Pretty neat idea. They provide a way to express the sequence of disk-access operations in a RAID controller using directed acyclic graphs, and a library that can `execute' these graphs either in a simulation or in an software-RAID implementation. The big benefit is that it is faster, easier, and less error-prone to implement various RAID management policies.

Keywords: parallel I/O, RAID, disk array, multiprocessor file system, file prefetching, file caching, cache consistency, pario-bib

Keywords: parallel I/O, RAID, disk array, multiprocessor file system, file prefetching, file caching, cache consistency, pario-bib

Comment: An overview of research being done in Garth's group. Touches on work in RAID disk arrays, parallel file systems, and prefetching. I think gibson:scotch-tr is nearly the same.

Abstract: We discuss the strategic directions and challenges in the management and use of storage systems-those components of computer systems responsible for the storage and retrieval of data. The performance gap between main and secondary memories shows no imminent sign of vanishing, and thus continuing research into storage I/O will be essential to reap the full benefit from the advances occurring in many other areas of computer science. In this report we identify a few strategic research goals and possible thrusts to meet those goals.

Keywords: supercomputing, data storage, database, parallel I/O, pario-bib

Comment: A more reliable, but limited-access, URL is http://www.acm.org/pubs/citations/journals/surveys/1996-28-4/p779-gibson/

Keywords: network-attached storage, storage architecture, parallel I/O, pario-bib

Abstract: Storage systems are continuing to grow, and they are become shared resources with the advent of I/O networks like FibreChannel. Managing these resources to meet performance and resiliency goals is becoming a significant challenge. We believe that completely automatic, attribute-managed storage is the way to address this issue. Our approach is based on declarative specifications of both application workloads and device characteristics. These are combined by a matching engine to generate a load-assignment that provides optimal performance and meets availability guarantees, at minimum cost.

Keywords: I/O architecture, disk array, RAID, file system, storage system, pario-bib

Comment: This is just a 4-page position paper. See also shriver:slides.

Keywords: video server, multimedia, parallel I/O, pario-bib

Comment: An approach called adaptive piggybacking groups together streams that are watching the same video, but at slightly different times, so that they can share the I/O streams.

Abstract: In recent years advances in computational speed have been the main focus of research and development in high performance computing. In comparison, the improvement in I/O performance has been modest. Faster processing speeds have created a need for faster I/O as well as for the storage and retrieval of vast amounts of data. The technology needed to develop these mass storage systems exists today. Robotic storage libraries are vital components of such systems. However, they normally exhibit high latency and long transmission times. We analyze the performance of robotic storage libraries and study striping as a technique for improving response time. Although striping has been extensively studied in the content of disk arrays, the architectural differences between robotic storage libraries and arrays of disks suggest that a separate study of striping techniques in such libraries would be beneficial.

Keywords: mass storage, parallel I/O, pario-bib

Keywords: parallel I/O, multimedia, survey, pario-bib

Comment: Part of a special issue.

Abstract: In this paper, we give new techniques for designing efficient algorithms for computational geometry problems that are too large to be solved in internal memory, and we use these techniques to develop optimal and practical algorithms for a number of important large-scale problems in computational geometry. Our algorithms are optimal for a wide range of two-level and hierarchical multilevel memory models, including parallel models. The algorithms are optimal in terms of both I/O cost and internal computation.

Our results are built on four fundamental techniques: distribution sweeping, a generic method for externalizing plane-sweep algorithms; persistent B-trees, for which we have both on-line and off-line methods; batch filtering, a general method for performing $K$ simultaneous external-memory searches in any data structure that can be modeled as a planar layered dag; and external marriage-before-conquest, an external-memory analog of the well-known technique of Kirkpatrick and Seidel. Using these techniques we are able to solve a very large number of problems in computational geometry, including batched range queries, 2-d and 3-d convex hull construction, planar point location, range queries, finding all nearest neighbors for a set of planar points, rectangle intersection/union reporting, computing the visibility of segments from a point, performing ray-shooting queries in constructive solid geometry (CSG) models, as well as several geometric dominance problems.

These results are significant because large-scale problems involving geometric data are ubiquitous in spatial databases, geographic information systems (GIS), constraint logic programming, object oriented databases, statistics, virtual reality systems, and graphics. This work makes a big step, both theoretically and in practice, towards the effective management and manipulation of geometric data in external memory, which is an essential component of these applications.

Keywords: computational geometry, parallel I/O algorithm, pario-bib

Keywords: parallel I/O, RAID, disk array, pario-bib

Keywords: parallel I/O, pario-bib

Keywords: parallel I/O, object-oriented, distributed data structures, runtime library, pario-bib

Comment: URL is for tech report version. They have a language called pC++ that allows object-parallel programming. They have a library called d/streams for I/O of distributed arrays. pC++/streams is a combination. You open a file, specify the in-memory distribution, read from the stream, and then extract some variables. Likewise, you insert some variables (into the stream buffer), then write it. They manage the distribution, and they store necessary metadata to reassemble the data structure when reading. Variables can be arbitrary classes, with $>>$ and $<<$ overloaded as the insert and extract operators. Performance is reasonable on Intel Paragon and SGI Challenge.

Abstract: Technology trends promise to give us processors with pico-second clock speeds. These pico-processors will spend much of their time waiting for information from the storage hierarchy. I believe this will force us to adopt a data-flow programming model. Similar trends will bless us with peta-byte online stores with exa-byte near-line stores. One large disk manufacture claims it costs 8$/year to manage a megabyte of online storage. That is 8 Billion dollars per year to manage a petabyte. Automating storage management is one of our major challenges. This talk covers these technology trends, surveys the current status of commercial software tools (aka database systems), their peak performance and price performance. It then poses four major challenges: total-cost-of ownership, long-term archiving, reliably storing exabytes, and data mining on petabyte databases.

Keywords: parallel computing, computer architecture, parallel I/O, pario-bib, memory hierarchy, distributed computing, database, object oriented

Comment: Very interesting talk. URL points to PowerPoint slides.

Keywords: disk striping, reliability, pario-bib

Comment: Parity striping, a variation of RAID 5, is just a different way of mapping blocks to disks. It groups parity blocks into extents, and does not stripe the data blocks. A logical disk is mostly contained in one physical disk, plus a parity region in another disk. Good for transaction processing workloads. Has the low cost/GByte of RAID, the reliability of RAID, without the high transfer rate of RAID, but with much better requests/second throughput than RAID 5. (But 40% worse than mirrors.) So it is a compromise between RAID and mirrors. BUT, see mourad:raid.

Keywords: parallel I/O, parallel file system, object-oriented, file system interface, Intel iPSC/2, pario-bib

Comment: See also grimshaw:elfs. They hope to provide high bandwidth and low latency, reduce the cognitive burden on the programmer, and manage proliferation of data formats and architectural changes. Details of the plan to make an extensible OO interface to file system. Objects each have a separate thread of control, so they can do asynchronous activity like prefetching and caching in the background, and support multiple outstanding requests. The Mentat object system makes it easy for them to support pipelining of I/O with I/O and computation in the user program. Let the user choose type of consistency needed. See grimshaw:objects for more results.

Keywords: parallel I/O, parallel file system, object-oriented, file system interface, pario-bib

Comment: Full paper grimshaw:ELFSTR. Really neat idea. Uses OO interface to file system, which is mostly in user mode. The object classes represent particular access patterns (e.g., 2-D matrix) in the file, and hide the actual structure of the file. The object knows enough to taylor the cache and prefetch algorithms to the semantics. Class inheritance allows layering.

Keywords: parallel I/O, multiprocessor file system, file system interface, pario-bib

Comment: Not much new from ELFS TR. A better citation than grimshaw:ELFS though. Does give CFS performance results. Note on 721 he says that CFS prefetches into ``local memory from which to satisfy future user requests that never come.'' This happens if the local access pattern isn't purely sequential, as in an interleaved pattern.

Abstract: Highly parallel applications often use either highly parallel file systems or large numbers of independent disks. Either approach can provide the high data rates necessary for parallel applications. However, the failure of a single disk or server can render the data useless. Conventional techniques, such as those based on applying erasure correcting codes to each file write, are prohibitively expensive for massively parallel scientific applications because of the granularity of access at which the codes are applied. In this paper we demonstrate a scalable method for recovering from single disk failures that is optimized for typical scientific data sets. This approach exploits coarser-grained (but precise) semantics to reduce the overhead of constructing recovery data and makes use of parallel computation (proportional to the data size and independent of number of processors) to construct data. Experiments are presented showing the efficiency of this approach on a cluster with independent disks, and a technique is described for hiding the creation of redundant data within the MPI-IO implementation.

Keywords: fault-tolerance, single-disk failures, MPI-IO, pario-bib

Abstract: The Message Passing Interface (MPI) specification is widely used for solving significant scientific and engineering problems on parallel computers. There exist more than a dozen implementations on computer platforms ranging from IBM SP-2 supercomputers to clusters of PCs running Windows NT or Linux ("Beowulf" machines). The initial MPI Standard document, MPI-1, was recently updated by the MPI Forum. The new version, MPI-2, contains both significant enhancements to the existing MPI core and new features.

Using MPI is a completely up-to-date version of the authors' 1994 introduction to the core functions of MPI. It adds material on the new C++ and Fortran 90 bindings for MPI throughout the book. It contains greater discussion of datatype extents, the most frequently misunderstood feature of MPI-1, as well as material on the new extensions to basic MPI functionality added by the MPI-2 Forum in the area of MPI datatypes and collective operations.

Using MPI-2 covers the new extensions to basic MPI. These include parallel I/O, remote memory access operations, and dynamic process management. The volume also includes material on tuning MPI applications for high performance on modern MPI implementations.

Keywords: parallel computing, message passing, parallel I/O, multiprocessor file system interface, pario-bib

Comment: Has a large chapter on MPI-IO with lots of example programs.

Keywords: parallel I/O, parallel architecture, networking, pario-bib

Comment: They examine the characteristics of a system that has I/O nodes which interface between the internal interconnection network of a distributed-memory MIMD machine and some external network, such as HIPPI. They build a simple model to show how different components affect the I/O throughput. They show the performance of their iWarp-HIPPI interface. They conclude that the I/O nodes must have sufficient memory bandwidth to support multiple data streams coming from several compute nodes, being combined into a single faster external network, or vice versa. They need to support scatter/gather, because the data is often distributed in small pieces. For the same reason, they need to have low per-message overhead. The internal network routing must allow multiple paths between compute nodes and the I/O nodes, to avoid congestion.

Keywords: out-of-core algorithm, data structure, pario-bib

Comment: See also the component papers vitter:survey, arge:lower, crauser:segment, grossi:crosstrees, toledo:survey. Not clear to what extent these papers are about *parallel* I/O.

Keywords: mass storage, parallel I/O, pario-bib

Comment: Part of jin:io-book; reformatted version of grossman:library.

Abstract: Requirements for a high-performance, scalable digital library of multimedia data are presented together with a layered architecture for a system that addresses the requirements. The approach is to view digital data as persistent collections of complex objects and to use lightweight object management to manage this data. To scale as the amount of data increases, the object management component is layered over a storage management component. The storage management component supports hierarchical storage, third-party data transfer and parallel input-output. Several issues that arise from the interface between the storage management and object management components are discussed. The authors have developed a prototype of a digital library using this design. Two key components of the prototype are AIM Net and HPSS. AIM Net is a persistent object manager and is a product of Oak Park Research. HPSS is the High Performance Storage System, developed by a collaboration including IBM Government Systems and several national labs.

Keywords: mass storage, parallel I/O, pario-bib

Abstract: This paper presents a framework for synthesizing efficient out-of-core programs for block recursive algorithms such as the fast Fourier transform (FFT) and Batcher's bitonic sort. The block recursive algorithms conside red in this paper are described using tensor (Kronecker) product and other matrix operations. The algebraic properties of the matrix representation are used to derive efficient out-of-core programs. These programs are targeted towards a two-level disk model which allows HPF supported cyclic(B) data distribution on a disk array. The effectiveness of our approach is demonstrated through an example out-of-core FFT program implemented on a work-station.

Keywords: parallel I/O algorithm, pario-bib

Keywords: parallel computing, scientific computing, weather prediction, global climate model, parallel I/O, pario-bib

Comment: There is some discussion of I/O issues. This weather code does some out-of-core work, to communicate data between time steps. They also dump a 'history' file every simulated day, and periodic checkpoint files. They are flexible about the layout of the history file, assuming postprocessing will clean it up. The I/O is not too much trouble on the Cray C90, where they get 350 MBps to the SSD for the out-of-core data. The history I/O is no problem. On distributed-memory machines with no SSD, out-of-core was impractical and the history file was only written once per simulated month. 'The most significant weakness in the distributed-memory implementation is the treatment of I/O, [due to] file system maturity....' See hammond:atmosphere and jones:skyhi in the same issue.

Keywords: network congestion, parallel tcp streams, transport protocols, pario-bib

Keywords: network congestion, parallel tcp streams, fairness, transport protocols, pario-bib

Comment: Also see earlier hacker:parallel-tcp and hacker:effects

Abstract: This paper examines the effects of using parallel TCP flows to improve end-to-end network performance for distributed data intensive applications. A series of transmission experiments were conducted over a wide-area network to assess how parallel flows improve throughput, and to understand the number of flows necessary to improve throughput while avoiding congestion. An empirical throughput expression for parallel flows based on experimental data is presented, and guidelines for the use of parallel flows are discussed. (45 refs.)

Keywords: network congestion, parallel tcp streams, transport protocols, pario-bib

Keywords: hypercube, multiprocessor file system, pario-bib

Comment: An early paper describing a proposed file system for hypercubes. The writing is almost impenetrable. Confusing and not at all clear what they propose. See also hadimioglu:hyperfs and flynn:hyper-fs.

Keywords: multiprocessor file system, parallel I/O, hypercube, pario-bib

Comment: Describes a hypercube file system based on I/O nodes and processor nodes. A few results from a hypercube simulator. See hadimioglu:fs and flynn:hyper-fs.

Keywords: parallel computing, scientific computing, weather prediction, global climate model, parallel I/O, pario-bib

Comment: They discuss a weather code that runs on the CM-5. The code writes a history file, dumping some data every timestep, and periodically a restart file. They found that CM-5 Fortran met their needs, although required huge buffers to get much scalability. They want to see a single, shared file-system image from all processors, have the file format be independent of processor count, use portable conventional interface, and have throughput scale with the number of computation processors. See also hack:ncar and jones:skyhi in the same issue.

Keywords: parallel I/O, parallel file system, heterogeneous, pario-bib

Comment: See delrosario:vipfs-tr for an earlier version. Also appears as NPAC report SCCS-686.

Abstract: In order to attain portability when using message passing on a distributed memory system, a portable message passing system must be used as well as other portable system support services. MPI[1] addresses the message passing problem. To date, there are no standards for system services and I/O. A library developed at NOAA's Forecast Systems Laboratory (FSL) known as the Nearest Neighbor Tool[2] (NNT) provides a high level portable interface to interprocess communications for finite difference approximation numerical weather prediction (NWP) models. In order to achieve portability, MPI is used to support interprocess communications. The other services are provided by the lower level library developed at NOAA/FSL known as the Scalable Runtime System (SRS). The principle focus of this paper is SRS.

Keywords: parallel I/O, runtime library, pario-bib

Comment: They describe the runtime system that supports the Nearest-Neighbor Tool (NNT), which they use to parallelize weather-prediction codes. This paper gives a vague overview of the I/O support. The interface sounds fairly typical, as does the underlying structure (server processes, cache processes, etc). Sounds like it is in its early stages, but is useful for many applications.

Keywords: parallel I/O, distributed file system, disk striping, pario-bib

Comment: Part of jin:io-book; reformatted version of hartman:zebra3.

Keywords: disk striping, distributed file system, pario-bib

Comment: Not a parallel file system, but worth comparing to Swift. Certainly, a similar idea could be used in a multiprocessor. Cite hartman:zebra3.

Keywords: file system, disk striping, distributed file system, RAID, log-structured file system, parallel I/O, pario-bib

Comment: Zebra stripes across network servers, but not on a file-by-file basis. Instead they use LFS ideas to stripe a per-client log across all file servers. Each client can then compute a parity block for each strip that it writes. They store ``deltas'', changes in block locations, in with the data, and also send them to the (central) file manager. The file manager and stripe cleaner are key state managers, that keep track of where blocks are located, and of stripe utilizations. Performance numbers limited to small-scale tests. This paper has more details than hartman:zebra, and performance numbers (but not with real workloads or stripe cleaner). Some tricky consistency issues.

Keywords: parallel I/O, distributed file system, disk striping, pario-bib

Keywords: parallel I/O, Linda, data parallel, nCUBE, parallel graphics, heterogeneous computing, pario-bib

Comment: C*-Linda is basically a combination of C* and C-Linda. The model is that of several SIMD modules interacting in a MIMD fashion through a Linda tuple space. The modules are created using eval, as in Linda. In this case, the compiler statically assigns each eval to a separate subcube on an nCUBE 3200, although they also talk about multiprogramming several modules on a subcube (not supported by VERTEX). They envision having separate modules running on the nCUBE's graphics processors, or having the file system directly talk to the tuple space, to support I/O. They also envision talking to modules elsewhere on a network, e.g., a workstation, through the tuple space. They reject the idea of sharing memory between modules due to the lack of synchrony between modules, and message passing because it is error-prone.

Keywords: hypercube, parallel architecture, nCUBE, pario-bib

Comment: Description of the first nCUBE, the NCUBE/ten. Good historical background about hypercubes. Talks about their design choices. Says a little about the file system - basically just a way of mounting disks on top of each other, within the nCUBE and to other nCUBEs.

Keywords: parallel I/O, multiprocessor file system, pario-bib

Comment: An overview of the issues in designing a parallel file system, along with some early ideas for their own file system. They aim for a general-purpose system, and characterize the workload into three classes: independent, much like a timesharing system; cooperative-agents, like that expected by most current MIMD file systems; and single-agent, for data-parallel programs where a ``master'' process issues single large requests on behalf of many processes. Their design is heavily weighted to the assumption of shared memory, and in particular to a randomized shared memory (like RP3), so they don't worry about locality much. They say little about their interface, although they intend to stick to a Unix interface - and Unix semantics - as much as possible. The file system is essentially represented by a collection of shared data structures and many threads to manipulate those structures.

Abstract: Distributed-memory systems have traditionally had great difficulty performing network I/O at rates proportional to their computational power. The problem is that the network interface has to support network I/O for a supercomputer, using computational and memory bandwidth resources similar to those of a workstation. As a result, the network interface becomes a bottleneck. We implemented an architecture for network I/O for the iWarp system with the following two key characteristics: first, application-specific tasks are off-loaded from the network interface to the distributed-memory system, and second, these tasks are performed in close cooperation with the application. The network interface has been used by several applications for over a year. In this paper we describe the network interface software that manages the communication between the iWarp distributed-memory system and the network interface, we validate the main features of our network interface architecture based on application experience, and we discuss how this architecture can be used by other distributed-memory systems.

Keywords: parallel network I/O, pario-bib

Comment: Parallel network I/O on the iWARP. Note proceedings only on CD-ROM and WWW.

Keywords: parallel I/O, pario-bib

Comment: Scalable I/O initiative intends to build a testbed. At Argonne, they have a 128-node SP-1 with a high-speed switch. 96 are compute nodes, 32 are I/O nodes (128 MB RAM, 1 GB local disk, FibreChannel port). FibreChannel connects to RS/6000 which has 256 MB RAM, two 80 MB/s busses, and a HIPPI interface to a 220 GB RAID (level 1 or 5) and 6.4 TB tape robot. They run UniTree on all this. They use multiple files to get parallelism. FibreChannel with TCP/IP is the limiting factor. note they are focussing more on the external connectivity issues rather than on the internal file system.

Keywords: parallel I/O, disk media, optical disk, holographic storage, trends, tape storage, parallel transfer disk, disk striping, pario-bib

Comment: A good overview of the current state-of-the art in March 1991, including particular numbers and vendor names. They discuss disk media (density, rotation, etc.), parallel transfer disks, disk arrays, parity and RAID, HiPPI, tape archives, optical memory, and holographic storage. Rotation speeds can increase as diameter goes down. Density increases are often offset by slower head settling times. Disk arrays will hit their ``heydey'' in the 1990s. Trend toward network-attached storage devices, that don't need a computer as a server.

Keywords: parallel I/O, virtual memory, SIMD, multiprocessor file system, pario-bib

Comment: He has an MPL (Maspar C) preprocessor that inserts code to allow you to make plural vectors and arrays pageable. The preprocessor inserts checks before every access to see whether you have that data in memory, and if not, to page it in. The preprocessor is supported by a run-time library. No compiler, OS, or hardware mods.

Keywords: parallel I/O, file system, pario-bib

Comment: Ways to arrange 2-d images on disk arrays that have multiple processors (like Datamesh), so that retrieval time for images or subimages is minimized.

Keywords: parallel I/O benchmarks, MPI-IO, pario-app, pario-bib

Abstract: In this paper, we present a mechanism able to predict the performance a given workload will achieve when running on a given storage device. This mechanism is composed by two modules. The first one is able to reproduce its behavior later on, without a new execution, even when the storage drives or data placement are modified. The second module is a drive modeler that is able to learn how storage drive works in an automatic way, just executing some synthetic tests. Once we have the workload and drive models, we can predict how well that application will perform on the selected storage device or devices or when the data placement is modified. The results presented in this paper will show that this prediction system achieves errors below 10% when compared to the real performance obtained. It is important to notice that the two modules will treat both the application and the storage device as black and will need no previous information about them. (20 refs.)

Keywords: performance prediction, data placement, storage device modeling, parallel I/O, pario-bib

Comment: Could not find a URL. See for proceedings information.

Keywords: parallel architecture, SIMD, MIMD, parallel I/O, pario-bib

Comment: A good basic citation for the CM-5 architecture. A little bit about I/O.

Keywords: parallel database, concurrency control, deadlock, parallel I/O, pario-bib

Comment: Most interesting to me in this paper is their discussion of the ``container model,'' in which they claim they allow the processors to be driven by the I/O devices. What it boils down to is a producer-consumer queue of containers, each of which contains a task (some tuples and presumably some instruction about what to do with them). The disks put data into containers and stick them on the queue; the processors repeatedly pull containers (tasks) from the queue and process them. They don't describe the activity of the disks in much detail. See kitsuregawa:sdc.

Abstract: Recently, a new server-less architecture is proposed for building low-cost yet scalable video streaming systems. Compare to conventional client-server-based video streaming systems, this server-less architecture does not need any dedicated video server and yet is highly scalable. Video data are distributed among user hosts and these hosts cooperate to stream video data to one another. Thus as new hosts join the system, they also add streaming and storage capacity to absorb the added streaming load. This study investigates the data reorganization problem when growing a server-less video streaming system. Specifically, as video data are distributed among user hosts, these data will need to be redistributed to newly joined hosts to utilize their storage and streaming capacity. This study presents a new data reorganization algorithm that allows controllable tradeoff between data reorganization overhead and streaming load balance.

Keywords: data reorganization, video on demand, video streaming, pario-bib

Abstract: We describe and evaluate a strategy for declustering the parity encoding in a redundant disk array. This declustered parity organization balances cost against data reliability and performance during failure recovery. It is targeted at highly-available parity-based arrays for use in continuous- operation systems. It improves on standard parity organizations by reducing the additional load on surviving disks during the reconstruction of a failed disk's contents. This yields higher user throughput during recovery, and/or shorter recovery time.

We first address the generalized parity layout problem, basing our solution on balanced incomplete and complete block designs. A software implementation of declustering is then evaluated using a disk array simulator under a highly concurrent workload comprised of small user accesses. We show that declustered parity penalizes user response time while a disk is being repaired (before and during its recovery) less than comparable non-declustered (RAID5) organizations without any penalty to user response time in the fault-free state.

We then show that previously proposed modifications to a simple, single-sweep reconstruction algorithm further decrease user response times during recovery, but, contrary to previous suggestions, the inclusion of these modifications may, for many configurations, also slow the reconstruction process. This result arises from the simple model of disk access performance used in previous work, which did not consider throughput variations due to positioning delays.

Keywords: parity, declustering, striping, disk array, redundancy, reliability, pario-bib

Abstract: The performance of traditional RAID Level 5 arrays is, for many applications, unacceptably poor while one of its constituent disks is non-functional. This paper describes and evaluates mechanisms by which this disk array failure-recovery performance can be improved. The two key issues addressed are the data layout, the mapping by which data and parity blocks are assigned to physical disk blocks in an array, and the reconstruction algorithm, which is the technique used to recover data that is lost when a component disk fails.

The data layout techniques this paper investigates are instantiations of the declustered parity organization, a derivative of RAID Level 5 that allows a system to trade some of its data capacity for improved failure-recovery performance. We show that our instantiations of parity declustering improve the failure-mode performance of an array significantly, and that a parity-declustered architecture is preferable to an equivalent-size multiple-group RAID Level 5 organization in environments where failure-recovery performance is important. The presented analyses also include comparisons to a RAID Level 1 (mirrored disks) approach.

With respect to reconstruction algorithms, this paper describes and briefly evaluates two alternatives stripe-oriented reconstruction and disk-oriented reconstruction, and establishes that the latter is preferable as it provides faster reconstruction. The paper then revisits a set of previously-proposed reconstruction optimizations, evaluating their efficacy when used in conjunction with the disk-oriented algorithm. The paper concludes with a section on the reliability versus capacity trade-off that must be addressed when designing large arrays.

Keywords: parallel I/O, disk array, RAID, redundancy, reliability, pario-bib

Abstract: This paper describes and evaluates two algorithms for performing on-line failure recovery (data reconstruction) in redundant disk arrays. It presents an implementation of disk-oriented reconstruction, a data recovery algorithm that allows the reconstruction process to absorb essentially all the disk bandwidth not consumed by the user processes, and then compares this algorithm to a previously proposed parallel stripe-oriented approach. The disk-oriented approach yields better overall failure-recovery performance.

The paper evaluates performance via detailed simulation on two different disk array architectures: the RAID level 5 organization, and the declustered parity organization. The benefits of the disk-oriented algorithm can be achieved using controller or host buffer memory no larger than the size of three disk tracks per disk in the array. This paper also investigates the tradeoffs involved in selecting the size of the disk accesses used by the failure recovery process.

Keywords: parallel I/O, disk array, RAID, redundancy, reliability, pario-bib

Abstract: There exists a wide variety of applications in which data availability must be continuous, that is, where the system is never taken off-line and any interruption in the accessibility of stored data causes significant disruption in the service provided by the application. Examples include on-line transaction processing systems such as airline reservation systems, and automated teller networks in banking systems. In addition, there exist many applications for which a high degree of data availability is important, but continuous operation is not required. An example is a research and development environment, where access to a centrally-stored CAD system is often necessary to make progress on a design project. These applications and many others mandate both high performance and high availability from their storage subsystems.

Parity-based redundant disk arrays are very attractive storage alternatives for these systems because they offer both low cost per megabyte and high data reliability. Unfortunately such systems exhibit poor availability characteristics; their performance is severely degraded in the presence of a disk failure. This dissertation addresses the design of parity-based redundant disk arrays that offer dramatically higher levels of performance in the presence of failure than systems comprising the current state of the art.

We consider two primary aspects of the failure-recovery problem: the organization of the data and redundancy in the array, and the algorithm used to recover the lost data. We apply results from combinatorial theory to generate data and parity organizations that minimize performance degradation during failure recovery by evenly distributing all failure-induced workload over a larger-than-minimal collection of disks. We develop a reconstruction algorithm that is able to absorb for failure-recovery essentially all of the array's bandwidth that is not absorbed by the application process(es). Additionally, we develop a design for a redundant disk array targeted at extremely high availability through extremely fast failure recovery. This development also demonstrates the generality of the presented techniques.

Keywords: parallel I/O, disk arrays, RAID, redundancy, reliability, pario-bib

Comment: Garth Gibson, advisor.

Keywords: parallel I/O, pario-bib

Comment: A short summary of disk I/O issues: disk technology, latency reduction, parallel I/O, etc..

Keywords: disk array, reliability, parallel I/O, pario-bib

Comment: Chained declustering has cost like mirroring, since it replicates each block, but has better load increase during failure than mirrors, interleaved declustering, or RAID. (Or parity striping (my guess)). Has reliability between that of mirrors and RAID, and much better than interleaved declustering. Would also be much easier in a distributed environment. See hsiao:diskrep.

Keywords: disk array, reliability, disk mirroring, parallel I/O, pario-bib

Comment: Compares mirrored disks (MD) with interleaved declustering (ID) with chained declustering (CD). ID and CD found to have much better performance in normal and failure modes. See hsiao:decluster.

Keywords: disk array, reliability, disk mirroring, parallel I/O, pario-bib

Comment: See hsiao:diskrep.

Keywords: multimedia server, video on demand, pario-bib

Keywords: parallel I/O, disk cache, disk striping, disk aray, pario-bib

Comment: Part of jin:io-book; reformatted version of hu:rapid-cache.

Abstract: This paper presents a new cache architecture called RAPID-Cache for Redundant, Asymmetrically Parallel, and Inexpensive Disk Cache. A typical RAPID-Cache consists of two redundant write buffers on top of a disk system. One of the buffers is a primary cache made of RAM or NVRAM and the other is a backup cache containing a two level hierarchy: a small NVRAM buffer on top of a log disk. The backup cache has nearly equivalent write performance as the primary RAM cache, while the read performance of the backup cache is not as critical because normal read operations are performed through the primary RAM cache and reads from the backup cache happen only during error recovery periods. The RAPID-Cache presents an asymmetric architecture with a fast-write-fast-read RAM being a primary cache and a fast-write-slow-read NVRAM-disk hierarchy being a backup cache. The asymmetric cache architecture allows cost-effective designs for very large write caches for high-end disk I/O systems that would otherwise have to use dual-copy, costly NVRAM caches. It also makes it possible to implement reliable write caching for low-end disk I/O systems since the RAPID-Cache makes use of inexpensive disks to perform reliable caching. Our analysis and trace-driven simulation results show that the RAPID-Cache has significant reliability/cost advantages over conventional single NVRAM write caches and has great cost advantages over dual-copy NVRAM caches. The RAPID-Cache architecture opens a new dimension for disk system designers to exercise trade-offs among performance, reliability and cost.

Keywords: parallel I/O, disk cache, disk striping, disk aray, pario-bib

Abstract: It has been demonstrated that simulated annealing provides high-quality results for the data clustering problem. However, existing simulated annealing schemes are memory-based algorithms; they are not suited for solving large problems such as data clustering which typically are too big to fit in the memory space in its entirety. Various buffer replacement policies, assuming either temporal or spatial locality, are not useful in this case since simulated annealing is based on a randomized search process. Poor locality of references will cause the memory to thrash because too many replacements are required. This phenomenon will incur excessive disk accesses and force the machine to run at the speed of the I/O subsystem. In this paper, we formulate the data clustering problem as a graph partition problem (GPP), and propose a decomposition-based approach to address the issue of excessive disk accesses during annealing. We apply the statistical sampling technique to randomly select subgraphs of the GPP into memory for annealing. Both the analytical and experimental studies indicate that the decomposition-based approach can dramatically reduce the costly disk I/O activities while obtaining excellent optimized results.

Keywords: out of core, information retrieval, parallel I/O, pario-bib

Keywords: parallel file system, parallel I/O, pario-bib

Comment: Part of jin:io-book, revised version of huber:ppfs.

Abstract: The I/O problem is described in the context of parallel scientific applications. A user-level input/output library, PPFS, is introduced to address these issues. The design and implementation of PPFS are presented. Some simple performance benchmarks are reported. Experiments on two production-scale applications are given.

Keywords: parallel file system, pario-bib

Comment: He describes the design and implementation of PPFS, along with some experimental results. PPFS is a C++ library and a set of servers that implement a parallel file system on top of unix on a cluster or a Paragon. Interesting features of PPFS include: files are a sequence of records (fixed size or variable size), read_next and read_any operations, a no-extend option to reduce overhead of maintaining file-size information, client and server caching, intermediate caching agents for consistency, prefetching and write behind, and user-defined declustering and indexing policies. User-defined changes actually have to be precompiled into the server programs. Good results in comparison to PFS on the Paragon, though that doesn't say much. They are porting it to the SP-2.

Abstract: Rapid increases in processor performance over the past decade have outstripped performance improvements in input/output devices, increasing the importance of input/output performance to overall system performance. Further, experience has shown that the performance of parallel input/output systems is particularly sensitive to data placement and data management policies, making good choices critical. To explore this vast design space, we have developed a user-level library, the Portable Parallel File System (PPFS), which supports rapid experimentation and exploration. The PPFS includes a rich application interface, allowing the application to advertise access patterns, control caching and prefetching, and even control data placement. PPFS is both extensible and portable, making possible a wide range of experiments on a broad variety of platforms and configurations. Our initial experiments, based on simple benchmarks and two application programs, show that tailoring policies to input/output access patterns yields significant performance benefits, often improving performance by nearly an order of magnitude.

Keywords: parallel file system, parallel I/O, pario-bib

Keywords: parallel file system, parallel I/O, pario-bib

Comment: See also elford:ppfs-tr, huber:ppfs.

Abstract: Rapid increases in processor performance over the past decade have outstripped performance improvements in input/output devices, increasing the importance of input/output performance to overall system performance. Further, experience has shown that the performance of parallel data management policies, making good choices critical. To explore this vast design space, we have developed a user-level library, the Portable Parallel File System (PPFS), which supports rapid experimentation and exploration. The PPFS includes a rich application interface, allowing the application to advertise access patterns, control caching and prefetching, and even control data placement. PPFS is both extensible and portable, making possible a wide range of experiments on a broad variety of platforms and configurations. Our initial experiments, based on on simple benchmarks and two application programs, show that tailoring policies to input/output access patterns yields significant performance benefits, often improving performance by nearly an order of magnitude.

Keywords: parallel file system, pario-bib

Comment: They have built a user-level library that implements a parallel file system on top of a set of vanilla Unix file systems. Their goals include flexibility and portability, so they can use PPFS to explore issues in parallel I/O. They allow the application to have lots of control over data distribution, cache and prefetch policies, etc. They support fixed- and variable-length records. They support client, server, and shared caches. This TR includes syntax and specs for all functions. They include performance for synthetic benchmarks and application codes, compared with Intel Paragon PFS (which is admittedly not a very tough competitor).

Abstract: Increasing requirements in HPC led to improvements of CPU power, but bandwidth of I/O subsystems does not keep up with the performance of processors any more. This problem is commonly known as the I/O bottleneck. Additionally, new and stimulating data-intensive problems in biology, physics, astronomy, space exploration, and human genom research arise, which bring new high-performance applications dealing with massive data spread over globally distributed storage resources. Therefore research in HPC focuses more on I/O systems: all leading hardware vendors of multiprocessor systems provided powerful concurrent I/O subsystems. In accordance researchers focus on the design of appropriate programming tools and models to take advantage of the available hardware resources. Numerous projects about this topic have appeared, from which a large and unmanageable quantity of publications have come. These publications concern themselves to a large extent with very special problems. Due to the time of their appearance the few overview papers deal with Parallel I/O or Cluster I/O. Substantial progress has been made in these research areas since then. Grid Computing has emerged as an important new field, distinguished from conventional Distributed Computing by its focus on large-scale resource sharing, innovative applications and, in some cases, high-performance orientation. Over the past five years, research and development efforts within the Grid community have produced protocols, services and tools that address precisely the challenges that arise when we try to build Grids, I/O being an important part of it. Therefore our work gives an overview of I/O in HPC.

Keywords: parallel i/o, cluster i/o, grid i/o, distributed computing, pario-bib

Comment: Like stockinger:dictionary, this master's thesis categorizes and describes a large set of parallel I/O-related projects and applications.

Keywords: parallel I/O, I/O, pario-bib

Comment: Does FORTRAN format conversion in software in parallel or in hardware, to obtain good speedups for lots of programs. However he found that increasing the I/O bandwidth was the most significant change that could be made in the parallel program.

Abstract: Recently, there have been many efforts to get high performance in cluster computing with inexpensive PCs connected through high-speed networks. Some of them were to provide high bandwidth and parallelism in file service using a distributed file system. Other researches for distributed file systems include the cooperative cache that reduces servers' load and improves overall performance. The cooperative cache shares file caches among clients so that a client can request a file to another client, not to the server, through inter-client message passing. In various distributed file systems, PVFS (Parallel Virtual File System) provides high performance with parallel I/O in Linux widely used in cluster computing. However, PVFS doesn't support any file cache facility. This paper describes the design and implementation of the cooperative cache for PVFS (Coopc-PVFS). We show the efficiency of Coopc-PVFS in comparison to original PVFS. As a result, the response time of Coopc-PVFS is shorter than or similar to that of original PVFS.

Keywords: PVFS, cooperative cache, pario-bib

Abstract: A new RAID-x (redundant array of inexpensive disks at level x) architecture is presented for distributed I/O processing on a serverless cluster of computers. The RAID-x architecture is based on a new concept of orthogonal striping and mirroring (OSM) across all distributed disks in the cluster. The primary advantages of this OSM approach lie in: (1) a significant improvement in parallel I/O bandwidth, (2) hiding disk mirroring overhead in the background, and (3) greatly enhanced scalability and reliability in cluster computing applications. All claimed advantages are substantiated with benchmark performance results on the Trojans cluster built at USC in 1999. Throughout the paper, we discuss the issues of scalable I/O performance, enhanced system reliability, and striped checkpointing on distributed RAID-x in a serverless cluster environment.

Keywords: parallel I/O, disk array, disk striping, RAID, pario-bib

Keywords: disk array, multimedia, parallel I/O, pario-bib

Comment: Petri-net model of disk array using Information-Dispersal Algorithm (IDA) to stripe data. Continuous-media workload.

Keywords: multiprocessor architecture, parallel I/O, pario-bib

Comment: See also information about Vesta file system, corbett:vesta.

Keywords: file access pattern, parallel I/O, Intel iPSC/2, hypercube, pario-bib

Comment: Lists several examples and the amount and types of data they require, and how much bandwidth. Fluid flow modeling, Molecular modeling, Seismic processing, and Tactical and strategic systems.

Keywords: parallel I/O, hypercube, Intel iPSC/2, pario-bib

Comment: Simple overview, not much detail. See intel:ipsc2, pierce:pario, asbury:fortranio. Separate I/O nodes from compute nodes. Each I/O node has a SCSI bus to the disks, and communicates with other nodes in the system via Direct-Connect hypercube routing.

Keywords: parallel architecture, parallel I/O, Intel, pario-bib

Comment: Not a bad glossy. See also esser:paragon.

Keywords: parallel I/O, hypercube, Intel iPSC/2, pario-bib

Keywords: parallel I/O, parallel I/O architecture, parallel I/O algorithm, multiprocessor file system, workload characterization, parallel file access pattern, pario-bib

Comment: A book containing papers from IOPADS '94 and IOPADS '95, plus several survey/tutorial papers. See the bib entries with cross-ref to iopads-book.

Keywords: parallel I/O, multiprocessor file system, pario-bib

Comment: The entire proceedings is about parallel I/O.

Keywords: parallel I/O, multiprocessor file system, pario-bib

Comment: The entire proceedings is about parallel I/O.

Abstract: We provide an overview of different file system architectures. We show the influence of I/O access pattern studies results on file system design. We present techniques, algorithms and data structures used in file system implementations. We overview issues related to both local and distributed file systems. We describe distributed file system architectures for different kinds of network connectivity: tightly-connected networks (clusters and supercomputers), loosely-connected networks (computational grids) or disconnected computers (mobile computing). File system architectures for both network-attached and computer-attached storage are reviewed. We show how the parallel file systems address the requirements of I/O bound parallel applications. Different file sharing semantics in distributed and parallel file systems are explored. We also present how efficient metadata management can be realized in journaled file systems.

Keywords: survey, file system architecture, pario-bib

Abstract: This paper presents Clusterfile, a parallel file system that provides parallel file access on a cluster of computers. We introduce a file partitioning model that has been used in the design of Clusterfile. The model uses a data representation that is optimized for multidimensional array partitioning while allowing arbitrary partitions. The paper shows how the file model can be employed for file partitioning into both physical subfiles and logical views. We also present how the conversion between two partitions of the same file is implemented using a general memory redistribution algorithm. We show how we use the algorithm to optimize non-contiguous read and write operations. The experimental results include performance comparisons with the Parallel Virtual File System (PVFS) and an MPI-IO implementation for PVFS.

Keywords: parallel file system, parallel I/O, pario-bib

Abstract: This paper presents the integration of two collective I/O techniques into the Clusterfile parallel file system : disk-directed I/O and two-phase I/O. We show that global cooperative cache management improves the collective I/O performance. The solution focuses on integrating disk parallelism with other types of parallelism: memory (by buffering and caching on several nodes), network (by parallel I/O scheduling strategies) and processors (by redistributing the I/O related computation over several nodes). The performance results show considerable throughput increases over ROMIO's extended two-phase I/O.

Keywords: disk-directed I/O, two-phase I/O, clusterfile parallel file system, cooperative cache, pario-bib

Abstract: This paper presents view I/O, a non-contiguous parallel I/O technique. We show that the linear file model may be an unsuitable abstraction for non-contiguous I/O optimizations. Additionally, the poor cooperation between a file system and an I/O library like MPI-IO may drastically affect the performance. View I/O has detailed knowledge about parallel structure of a file and about the potential access pattern and exploits it in order to improve performance. The access overhead is reduced by using a strategy "declare once, use several times" and by file off-set compaction. We compare and contrast view I/O with other non-contiguous I/O methods. Our measurements on a cluster of computers indicate a significant performance improvement over other approaches. (15 refs.)

Keywords: non-contiguous I/O, parallel file structure, pario-bib

Keywords: parallel file system, pario-bib

Comment: File system in the PIMOS operating system for the PIM (Parallel Inference Machine) in the Fifth Generation Computer Systems project in Japan. Paper design, no results yet. Uses disks that are attached directly to the computational processors. Significant in that it does use client caches in a parallel file system. Caches are kept coherent with a centralized directory-based protocol for exclusive-writer, multiple-reader semantics, supporting sequential consistency. Disk management includes logging to surivive crashes. Bitmap free list with buddy system to support full, 1/2, and 1/4 blocks. Trick to avoid constant update of on-disk free list. My suspicion is that cache coherence protocol may be expensive, especially in larger systems.

Abstract: One of the key components of a multi-user multimedia-on-demand system is the data server. Digitalization of traditionally analog data such as video and audio, and the feasibility of obtaining network bandwidths above the gigabit-per-second range are two important advances that have made possible the realization, in the near future, of interactive distributed multimedia systems. Secondary-to-main memory I/O technology has not kept pace with advances in networking, main memory and CPU processing power. Consequently, the performance of the server has a direct bearing on the overall performance of such a system.

In this paper we present a high-performance solution to the I/O retrieval problem in a distributed multimedia system. Parallelism of data retrieval is achieved by striping the data across multiple disks. We identify the different components that contribute to media data retrieval delay. The variable delays among these have a great bearing on the server throughput under varying load conditions. We present a buffering scheme to minimize these variations. We have implemented our model on the Intel Paragon parallel computer. The results of component-wise instrumentation of the server operation are presented and analyzed. We present experimental results that demonstrate the efficacy of the buffering scheme. Based on our experiments, a dynamic admission control policy that takes server workload into account is proposed.

Keywords: parallel I/O, I/O scheduling, multimedia, video on demand, pario-bib

Comment: Much more detailed than jadav:media-on-demand. Here they present less survey information, and all the details on their Paragon implementation/simulation. They experiment with many tradeoffs, and propose and evaluate several scheduling and admission-control algorithms.

Abstract: One of the key components of a multi user multimedia on demand system is the data server. Digitization of traditionally analog data such as video and audio, and the feasibility of obtaining network bandwidths above the gigabit per second range are two important advances that have made possible the realization, in the near future, of interactive distributed multimedia systems. Secondary-to-main memory I/O technology has not kept pace with advances in networking, main memory and CPU processing power. Consequently, the performance of the server has a direct bearing on the overall performance of such a system. We develop a model for the architecture of a server for such a system. Parallelism of data retrieval is achieved by striping the data across multiple disks. The performance of any server ultimately depends on the data access patterns. Two modifications of the basic retrieval algorithm are presented to exploit data access patterns in order to improve system throughput and response time. A complementary information caching optimization is discussed. Finally, we present performance results of these algorithms on the IBM SP1 and Intel Paragon parallel computers.

Keywords: parallel I/O, multimedia, pario-bib

Comment: Journal version is jadav:j-ioschedule? See also jadav:media-on-demand. [Comments based on a much earlier version.] They propose I/O scheduling algorithms for multimedia file servers. They assume an MIMD architecture with no shared memory and with a disk on every node. One node is essentially a manager for new requests. Another set are interface nodes, each managing the data flow for a few multimedia data streams. The majority are server nodes, responsible just for fetching their data from disk and sending it to the interface nodes. The interface nodes assemble data from the server nodes into a data stream, and send it on out to the client. They describe algorithms for scheduling requests from the interface node to the server node, and for sending data out to the client. They also describe an algorithm for determining whether the system can accept a new request.

Keywords: multimedia, scheduling, parallel I/O, pario-bib

Comment: See also jadav:ioschedule, jadav:j-ioschedule.

Keywords: parallel I/O, I/O scheduling, multimedia, video on demand, pario-bib

Comment: Conference version is jadav:ioschedule; similar abstract. See jadav:media-on-demand.

Abstract: A server for an interactive distributed multimedia system may require thousands of gigabytes of storage space and high I/O bandwidth. In order to maximize system utilization, and thus minimize cost, the load must be balanced among the server's disks, interconnection network and scheduler. Many algorithms for maximizing retrieval capacity from the storage system have been proposed. This paper presents techniques for improving server capacity by assigning media requests to the nodes of a server so as to balance the load on the interconnection network and the scheduling nodes. Five policies for dynamic request assignment are developed. An important factor that affects data retrieval in a high-performance continuous media server is the degree of parallelism of data retrieval. The performance of the dynamic policies on an implementation of a server model developed earlier is presented for two values of the degree of parallelism.

Keywords: multimedia, parallel I/O, pario-bib

Abstract: This paper discusses the architectural requirements of a multimedia-on-demand system, with special emphasis on the media server. Although high-performance computers are the best choice for building media-on-demand servers, implementation poses many difficulties. We conclude with a discussion of the open issues regarding the design and implementation of the server.

Keywords: parallel I/O, multimedia, video on demand, pario-bib

Comment: A survery of the issues involved in designing a media-on-demand server (they really focus on temporal data like video and audio). They do have a few results comparing various granularities for disk-requests and network messages, which seem to be from an Intel Paragon implementation, although they do not describe the experimental setup. See jadav:evaluation, jadav:j-ioschedule, jadav:ioschedule.

Keywords: wireless communication, mobile computing, RAID, parallel I/O, pario-bib

Comment: They discuss the idea of broadcasting a disk's data over the air, so PDAs can 'read' the disk by waiting for the necessary data to come along. Good for read-only or write-rarely disks. They discuss the idea of dividing the air into multiple (frequency or time) tracks and 'striping' data across the tracks for better bandwidth and reliability.

Abstract: The I/O bottleneck in parallel computer systems has recently begun receiving increasing interest. Most attention has focused on improving the performance of I/O devices using fairly low-level parallelism in techniques such as disk striping and interleaving. Widely applicable solutions, however, will require an integrated approach which addresses the problem at multiple system levels, including applications, systems software, and architecture. We propose that within the context of such an integrated approach, scheduling parallel I/O operations will become increasingly attractive and can potentially provide substantial performance benefits.

We describe a simple I/O scheduling problem and present approximate algorithms for its solution. The costs of using these algorithms in terms of execution time, and the benefits in terms of reduced time to complete a batch of I/O operations, are compared with the situations in which no scheduling is used, and in which an optimal scheduling algorithm is used. The comparison is performed both theoretically and experimentally. We have found that, in exchange for a small execution time overhead, the approximate scheduling algorithms can provide substantial improvements in I/O completion times.

Keywords: network, graph coloring, multiprocessor file system, resource allocation, scheduling, parallel I/O, pario-bib

Comment: See also jain:pario

Keywords: parallel I/O, shared memory, scheduling, pario-bib

Comment: An algorithm to schedule (off-line) a set of transfers between P procs and D disks, such that no proc or disk does more than one request at a time, and no more than K transfers are concurrent (due to channel limits), with integer arbitrary-length transfers that are preemptable (ie segmentable). Much faster than previous algorithms. Problems, IMHO: off-line is only good for batch executions with known needs (ok for big collective I/Os I suppose). All k channels are usable by all proc-disk pairs, may not be realistic. No accomodation for big difference in disk and channel time, ie, disk probably can't do a channel transfer every time unit. Allows transfers in any order, which means disk seeks could be bad. No cost for preemption of a transfer, which could mean more message overhead if more messages are needed to do a given transfer. Assumes all transfers have predictable time. Still, it could be useful in some situations, esp. where order really doesn't matter.

Abstract: We sketch the reasons for the I/O bottleneck in parallel and distributed systems, pointing out that it can be viewed as a special case of a general bottleneck that arises at all levels of the memory hierarchy. We argue that because of its severity, the I/O bottleneck deserves systematic attention at all levels of system design. We then present a survey of the issues raised by the I/O bottleneck in five key areas of parallel and distributed systems: applications, algorithms, compilers, operating systems and architecture. Finally, we address some of the trends we observe emerging in new paradigms of parallel and distributed computing: the convergence of networking and I/O, I/O for massively distributed ``global information systems'' such as the World Wide Web, and I/O for mobile computing and wireless communications. These considerations suggest exciting new research directions in I/O for parallel and distributed systems in the years to come.

Keywords: parallel I/O, out-of-core, pario-bib

Comment: Part of a whole book on parallel I/O; see iopads-book.

Keywords: parallel I/O, file access pattern, multiprocessor file system, pario-bib

Comment: He looks at the effect of I/O traffic on memory access in a multistage network, and custom mappings of file data to disks to support non-sequential I/O. He considers both the traditional ``multiuser'' workload and the case where a application accesses a single file in parallel. Assumes a dance-hall shared-memory MIMD base architecture (CEDAR). Disks are attached either to the memory or processor side of the network, and in either case require four network traversals per read/write operation. Nice summary of previous parallel I/O architectures, and characterization of the workload. Main conclusions: the network is not an inherent bottleneck, but I/O traffic can cause up to 50% loss in memory traffic bandwidth, and bursts of I/O can saturate the network. For a high I/O request rate (eg, all procs active), spread each request over a small number of disks (eg, one), whereas for a low I/O request rate (eg, one proc active) spread each request over lots of disks (eg, all). This avoids cache thrashing when multiple procs hit on one disk node. However, if they are all reading the same data, then there is no cache thrashing and you want to maximize parallelism across disks. When accessing disjoint parts of the same file, it is sometimes better to have one proc do all the accesses, because this avoids out-of-order requests that spoil prefetching, and it avoids contention by multiple procs. No single file-to-disk mapping worked for everything; interleaved (striped) worked well for most sequential patterns, but ``sequential'' (partitioned) mappings worked better for multiple-process loads that tend to focus each process on a disk, eg, an interleaved pattern where the stride is equal to the number of disks. Thus, if your pattern can get you disk locality, use a mapping that will provide it.

Keywords: parallel I/O, pario-bib

Comment: Ways to distribute data across multiple disks to speed information retrieval, given an inverted index. Based on a shared-everything multiprocessor model.

Keywords: RAID, disk array, parallel file system, caching, prefetching, multiprocessor file system, parallel I/O applications, parallel I/O, pario-bib

Comment: An excellent collection of papers that were mostly published earlier.

Abstract: To access a RAID (redundant arrays of inexpensive disks), the disk stripe size greatly affects the performance of the disk array. In this article, we present a performance model to analyze the effects of striping with different stripe sizes in a RAID. The model can be applied to optimize the stripe size. Compared with previous approaches, our model is simpler to apply and more accurately reveals the real performance. Both system designers and users can apply the model to support parallel I/O events

Keywords: parallel I/O, RAID, disk striping, pario-bib

Keywords: parallel I/O, multiprocessor file system, pario-bib

Comment: If you insert blocks into a striped file, you mess up the nice striping. So he breaks the file into striped extents, and keeps track of the extents with a distributed B-tree index. Deletions also fit into the same scheme.

Abstract: The evolution of system architectures and system configurations has created the need for a new supercomputer system interconnect. Attributes required of the new interconnect include commonality among system and subsystem types, scalability, low latency, high bandwidth, a high level of resiliency, and flexibility. Cray Research Inc. is developing a new system channel to meet these interconnect requirements in future systems. The channel has a ring-based architecture, but can also function as a point-to-point link. It integrates control and data on a single, physical path while providing low latency and variance for control messages. Extensive features for client isolation, diagnostic capabilities, and fault tolerance have been incorporated into the design. The attributes and features of this channel are discussed along with implementation and protocol specifics.

Keywords: mass storage, I/O architecture, I/O interconnect, supercomputer, parallel I/O, pario-bib

Comment: About the Cray Research SCX channel, capable of 1200 MB/s peak and 900 MB/s delivered throughput.

Keywords: computational physics, parallel I/O, pario-bib

Comment: Old paper on the need for large memory and fast paging and I/O in out-of-core solutions to 3-d seismic modeling. They used 4-way parallel I/O to support their job. Needed to transfer a 3-d matrix in and out of memory by rows, columns, and vertical columns. Stored in block-structured form to improve locality on the disk.

Keywords: mass storage, parallel I/O, multiprocessor file system interface, pario-bib

Keywords: parallel computing, scientific computing, weather prediction, global climate model, parallel I/O, pario-bib

Comment: They talk about a weather code. There's a bit about the parallel I/O issues. They periodically write a restart file, and they write out several types of data files. They write out the data in any order, with a little mini-header in each chunk that describes the chunk. I/O was not a significant percentage of their run time on either the CM5 or C90. See hammond:atmosphere and hack:ncar in the same issue.

Abstract: Buffer management for a D-disk parallel I/O system is considered in the context of randomized placement of data on the disks. A simple prefetching and caching algorithm PHASE-LRU using bounded lookahead is described and analyzed. It is shown that PHASE-LRU performs an expected number of I/Os that is within a factor Theta(log D/log log D) of the number performed by an optimal off-line algorithm. In contrast, any deterministic buffer management algorithm with the same amount of lookahead must do at least Omega(rootD) times the number of I/Os of the optimal. (C) 2004 Elsevier B.V. All rights reserved.

Keywords: parallel I/O, prefetching, data placement, caching, buffer management, analysis, algorithms, randomization, pario-bib

Abstract: We address the problem of prefetching and caching in a parallel I/O system and present a new algorithm for parallel disk scheduling. Traditional buffer management algorithms that minimize the number of block misses are substantially suboptimal in a parallel I/O system where multiple I/Os can proceed simultaneously. We show that in the offline case, where a priori knowledge of all the requests is available, PC-OPT performs the minimum number of I/Os to service the given I/O requests. This is the first parallel I/O scheduling algorithm that is provably offline optimal in the parallel disk model. In the online case, we study the context of global L-block lookahead, which gives the buffer management algorithm a lookahead consisting of L distinct requests. We show that the competitive ratio of PC-OPT, with global L-block lookahead, is Θ (M - L + D), when L < M, and Θ (M D / L), when L > M, where the number of disks is D and buffer size is M.

Keywords: parallel I/O, file prefetching, pario-bib

Abstract: We address the problem of prefetching and caching in a parallel I/O system and present a new algorithm for optimal parallel-disk scheduling. Traditional buffer management algorithms that minimize the number of I/O disk accesses, are substantially suboptimal in a parallel I/O system where multiple I/Os can proceed simultaneously.

We present a new algorithm Super for parallel-disk I/O scheduling. We show that in the off-line case, where a priori knowledge of all the requests is available, Super performs the minimum number of I/Os to service the given I/O requests. This is the first parallel I/O scheduling algorithm that is provably offline optimal. In the on-line case, we study Super in the context of global L-block lookahead, which gives the buffer management algorithm a lookahead consisting of L distinct requests. We show that the competitive ratio of Super, with global L-block lookahead, is Theta(M-L+D), when L < M, and Theta(MD/L), when L >= M, where the number of disks is D and buffer size is M.

Keywords: parallel I/O, prefetch, disk cache, pario-bib

Abstract: We present an optimal algorithm, L-OPT, for prefetching and I/O scheduling in parallel I/O systems using a read-once model of block reference. The algorithm uses knowledge of the next L block references, L-block lookahead, to schedule I/Os in an on-line manner. It uses a dynamic priority assignment scheme to decide when blocks should be prefetched, so as to minimize the total number of I/Os. The parallel disk model of an I/O system is used to study the performance of L-OPT. We show that L-OPT is comparable to the best on-line algorithm with the same amount of lookahead; the ratio of the length of its schedule to the length of the optimal schedule is within a constant factor of the best possible. Specifically, we show that the competitive ratio of L-OPT is $Ω(\sqrt{MD/L})$ which matches the lower bound on the competitive ratio of any prefetching algorithm with L-block lookahead. In addition we show that when the lookahead consists of the entire reference string, L-OPT performs the minimum possible number of I/Os; hence L-OPT is the optimal off-line algorithm. Finally, using synthetic traces we empirically study the performance characteristics of L-OPT.

Keywords: disk scheduling, parallel I/O, pario-bib

Keywords: parallel I/O, multimedia, multiprocessor file system, pario-bib

Comment: Hook a video-display system to the compute node of an SP-1 running Vesta, and then use Vesta file system to serve the video.

Abstract: Video on Demand (VoD) servers are expected to serve hundreds of customers with as many, or more, movie videos. Such an environment requires large storage capacity and real-time, high-bandwidth transmission capabilities. Massive striping of videos across disk arrays is a viable means to store large amounts of video data and, through parallelism of file access, achieve the needed bandwidth. The Vesta Parallel File System facilitates parallel access from an application to files distributed across a set of I/O processors, each with a set of attached disks. Given Vesta's parallel file access capabilities, this paper examines a number of issues pertaining to the implementation of VoD services on top of Vesta. We develop a prototype VoD experimentation environment on an IBM SP-1 and analyze Vesta's performance in video data retrieval for real-time playback. Specifically, we explore the impact of concurrent video streams competing for I/O node resources, cache effects, and video striping across multiple I/O nodes.

Keywords: parallel I/O, parallel file system, video on demand, multimedia, pario-bib

Comment: Part of a whole book on parallel I/O; see iopads-book.

Keywords: parallel file system, parallel I/O, pario-bib

Abstract: any large-scale applications have significant I/O requirements as well as computational and memory requirements. Unfortunately, the limited number of I/O nodes provided in a typical configuration of the modern message-passing distributed-memory architectures such as the Intel Paragon and the IBM SP-2 limits the I/O performance of these applications severely. In this paper, we examine some software optimization techniques and evaluate their effects in five different I/O-intensive codes from both small and large application domains. Our goals in this study are twofold. First, we want to understand the behavior of large-scale data-intensive applications and the impact of I/O subsystems on their performance and vice versa. Second, and more importantly, we strive to determine the solutions for improving the applications' performance by a mix of software techniques. Our results reveal that different applications can benefit from different optimizations. For example, we found that some applications benefit from file layout optimizations, whereas others take advantage of collective I/O. A combination of architectural and software solutions is normally needed to obtain good I/O performance. For example, we show that with a limited number of I/O resources, it is possible to obtain good performance by using appropriate software optimizations. We also show that beyond a certain level, imbalance in the architecture results in performance degradation even when using optimized software, thereby indicating the necessity of an increase in I/O resources.

Keywords: parallel I/O, parallel application, pario-bib

Abstract: Parallel machines are an important part of the scientific application developer's tool box and the computational and processing demands placed on these machines are rapidly increasing. Many scientific applications tend to perform high volume data storage, data retrieval and data processing, which demands high performance from the I/O subsystem. In this paper, we conduct an experimental study of the I/O performed by the Hartree-Fock (HF) method, as implemented using a fully distributed data approach in the NWChem parallel computational chemistry package. We use PASSION, a parallel and scalable I/O library and it's optmizations such as prefetching to improve the I/O performance of the HF application and we also present extensive experimental results of the same. The effects of both application-related factors and system-related factors on the application's I/O performance are studied in detail. We rank the optimizations based on the significance and impact on the performance of HF's I/O phase as: I. efficient interface to the file system, II. prefetching optimization, and III. buffering. The results show that within the limits of our experimental parameters, application-related factors are more effective on the overall I/O behavior of this application. We obtained up to 95% improvement in I/O time and 43% improvement in the overall application performance with these optimizations.

Keywords: parallel I/O, scientific computing, pario-bib

Comment: No page numbers: proceedings on CDROM and web only.

Abstract: Many scientific applications tend to perform high volume data storage, data retrieval and data processing, which demands high performance from the I/O subsystem. The focus and contribution of this work is to study the I/O behavior of the Hartree-Fock method using PASSION. HF's I/O phases can contribute up to 62.34% of the total execution time. We reduce the execution time and I/O time up to 54% and 6% respectively of that of the original case through PASSION and its optimizations. Additionally, we categorize the factors that affect the I/O performance of HF into key application-related parameters and key system-related parameters. Based on extensive empirical results and within our experimental space, we order the parameters according to their impact on HF's I/O performance as follows: efficient interface, prefetching, buffering, number of I/O nodes, striping factor and striping unit. We conclude that application-related factors have a more significant effect on HF's I/O performance than the system-related factors within our experimental space.

Keywords: parallel I/O application, pario-bib

Comment: In a Special Issue on I/O in Parallel Applications, volume 12, numbers 3 and 4.

Abstract: Current approaches to parallel I/O demand extensive user effort to obtain acceptable performance. This is in part due to difficulties in understanding the characteristics of a wide variety of I/O devices and in part due to inherent complexity of I/O software. While parallel I/O systems provide users with environments where persistent data sets can be shared between parallel processors, the ultimate performance of I/O-intensive codes depends largely on the relation between data access patterns exhibited by parallel processors and storage patterns of data in files and on disks. In cases where access patterns and storage patterns match, we can exploit parallel I/O hardware by allowing each processor to perform independent parallel I/O. In order to keep performance decent under circumstances in which data access patterns and storage patterns do not match, several I/O optimization techniques have been developed in recent years. Collective I/O is such an optimization technique that enables each processor to do I/O on behalf of other processors if doing so improves the overall performance. While it is generally accepted that collective I/O and its variants can bring impressive improvements as far as the I/O performance is concerned, it is difficult for the programmer to use collective I/O in an optimal manner. In this paper, we propose and evaluate a compiler-directed collective I/O approach which detects the opportunities for collective I/O and inserts the necessary I/O calls in the code automatically. An important characteristic of the approach is that instead of applying collective I/O indiscriminately, it uses collective I/O selectively only in cases where independent parallel I/O would not be possible or would lead to an excessive number of I/O calls. The approach involves compiler-directed access pattern and storage pattern detection schemes that work on a multiple application environment. We have implemented the necessary algorithms in a source-to-source translator and within a stand-alone tool. Our experimental results on an SGI/Cray Origin 2000 multiprocessor machine demonstrate that our compiler-directed collective I/O scheme performs very well on different setups built using nine applications from several scientific benchmarks. We have also observed that the I/O performance of our approach is only 5.23 percent worse than an optimal scheme.

Keywords: parallel I/O, collective I/O, compiler, pario-bib

Abstract: Since many of large scale computational problems usually deal with large quantities of data, optimizing the performance of I/O subsystems of massively parallel machines is an important challenge for system designers. We describe data access reorganization strategies for efficient compilation of out-of-core data-parallel programs on distributed memory machines. Our analytical approach and experimental results indicate that the optimizations introduced in this paper can reduce the amount of time spent in I/O by as much as an order of magnitude on both uniprocessors and multicomputers.

Keywords: parallel I/O, compiler, out-of-core, pario-bib

Abstract: This paper presents compiler algorithms to optimize out-of-core programs. These algorithms consider loop and data layout transformations in a unified framework. The performance of an out-of-core loop nest containing many references can be improved by a combination of restructuring the loops and file layouts. This approach considers array references one-by-one and attempts to optimize each reference for parallelism and locality. When there are references for which parallelism optimizations do not work, communication is vectorized so that data transfer can be performed before the innermost tiling loop. Preliminary results from hand-compiles on IBM SP-2 and Intel Paragon show that this approach reduces the execution time, improves the bandwidth speedup and overall speedup. In addition, we extend the base algorithm to work with file layout constraints and show how it can be used for optimizing programs consisting of multiple loop nests.

Keywords: compiler, out of core, parallel I/O, pario-bib

Keywords: parallel I/O, compiler, out of core, pario-bib

Keywords: parallel I/O, compiler, pario-bib

Abstract: Programs accessing disk-resident arrays perform poorly in general due to excessive number of I/O calls and insufficient help from compilers. In this paper, in order to alleviate this problem, we propose a series of compiler optimizations. Both the analytical approach we use and the experimental results provide strong evidence that our method is very effective on uniprocessors for out-of-core nests whose data sizes far exceed the size of available memory.

Keywords: parallel I/O, compiler, out-of-core, pario-bib

Keywords: out of core, parallel I/O, pario-bib

Abstract: This paper describes optimization techniques for translating out-of-core programs written in a data parallel language to message passing node programs with explicit parallel I/O. We demonstrate that straightforward extension of in-core compilation techniques does not work well for out-of-core programs. We then describe how the compiler can optimize the code by (1) determining appropriate file layouts for out-of-core arrays, (2) permuting the loops in the nest(s) to allow efficient file access, and (3) partitioning the available node memory among references based on I/O cost estimation. Our experimental results indicate that these optimizations can reduce the amount of time spent in I/O by as much as an order of magnitude.

Keywords: compiler, data-parallel, out-of-core, parallel I/O, pario-bib

Abstract: This paper describes a framework by which an out-of-core stencil program written in a data-parallel language can be translated into node programs in a distributed-memory message-passing machine with explicit I/O and communication. We focus on a technique called \emph{Data Space Tiling} to group data elements into slabs that can fit into memories of processors. Methods to choose \emph{legal} tile shapes under several constraints and deadlock-free scheduling of tiles are investigated. Our approach is \emph{unified} in the sense that it can be applied to both FORALL loops and the loops that involve flow-dependences.

Keywords: parallel I/O, compiler, out-of-core, pario-bib

Keywords: interprocess communication, parallel I/O, pario-bib

Comment: A parallel version of the Unix 'pipe' feature, for connecting the output of one program to multiple other programs or files. Implemented on Solaris. Performance results.

Keywords: scientific database, parallel I/O, pario-bib

Comment: See also karpovich:case-study. That is a subset of this paper.

Keywords: scientific database, parallel I/O, pario-bib

Comment: Apparently a subset of karpovich:bottleneck. They store a sparse, multidimensional data set (radio astronomy data) as a set of tagged data values, ie, as a set of tuples, each with several keys and a data value. They use a PLOP format to partition each dimension into slices, so that each intersection of the slices forms a bucket. They decide on the splits based on a preliminary statistical survey of the data. Bucket overflow is handled by chaining. Then, they evaluate various kinds of queries, ie, multidimensional range queries, for their performance. In this workload queries (reads) are much more common than updates (writes).

Keywords: parallel I/O, multiprocessor file system interface, object oriented, pario-bib

Comment: See also grimshaw:elfs, grimshaw:ELFSTR, grimshaw:objects, and karpovich:*. This is also available as UVA TR CS-94-28. This paper focuses more on the objec-oriented nature of ELFS than on its ability to support parallel I/O. It also describes two classes they've developed, one for 2d dense matrices and another for range queries on multidimensional sparse data sets. It does have some new performance numbers for ELFS on Intel CFS.

Keywords: parallel I/O, RAID, disk array, disk striping, pario-bib

Comment: Part of jin:io-book; reformatted version of katz:diskarch.

Keywords: parallel I/O, RAID, disk array, disk striping, pario-bib

Comment: Good review of the background of disks and I/O architectures, but a shorter RAID presentation than patterson:RAID. Also addresses controller structure. Good ref for the I/O crisis background, though they don't use that term here. Good taxonomy of previous array techniques.

Keywords: parallel I/O, RAID, Sprite, reliability, disk striping, disk array, pario-bib

Comment: Early RAID project paper. Describes the Berkeley team's plan to use an array of small (100M) hard disks as an I/O server for network file service, transaction processing, and supercomputer I/O. Considering performance, reliability, and flexibility. Initially hooked to their SPUR multiprocessor, using Sprite operating system, new filesystem. Either asynchronous striped or independent operation. Supercomputer I/O is characterized as sequential, minimum latency, low throughput. Use of parity disks to boost reliability. Files may be striped across one or more disks and extend over several sectors, thus a two-dimensional filesystem; striping need not involve all disks.

Keywords: distributed file system, supercomputer file system, file striping, RAID, parallel I/O, pario-bib

Comment: Comments on the emerging trend of file systems for mainframes and supercomputers that are not attached directly to the computer, but instead to a network attached to the computer. Avoiding data copying seems to be a critical issue in the OS and controllers, for disk and network interfaces. Describes RAID-II prototype.

Keywords: parallel I/O, RAID, reliability, disk array, pario-bib

Comment: A short summary of the RAID project. Some more up-to-date info, like that they have completed the first prototype with 8 SCSI strings and 32 disks.

Abstract: This paper reports on part of an on-going analysis of parallel systems for commercial users. The particular focus of this paper is on the requirements that commercial users, in particular users with financial database systems, have of parallel systems.

The issues of concern to such users differ from those of concern to science and engineering users. Performance of the parallel system is not the only, or indeed primary, reason for moving to such systems for commercial users. Infra-structure issues are important, such as system availability and inter-working with existing systems.

These issues are discussed in the context of a banking customer's requirements. The various technical concerns that these requirements impose are discussed in terms of commercially available systems.

Keywords: parallel architecture, parallel I/O, databases, commercial requirements, pario-bib

Keywords: parallel I/O, out-of-core, pario-bib

Comment: They describe a project they are beginning, which attempts to have the compiler analyze a program that uses large arrays, and insert explicit I/O statements to move data from those arrays to and from disk. This is seen as an alternative to OS and hardware virtual memory, and is likely to provide much better performance (so show their initial results). Their focus is on overlapping I/O and computation.

Abstract: Parallel single-level store (PSLS) systems integrate a shared virtual memory and a parallel file system. They provide programmers with a global address space including both memory and file data. PSLS systems implemented in a cluster thus represent a natural support for long-running parallel applications, combining both the natural shared memory programming model and a large and efficient file system. However, the need to tolerate failures in such a system increases with the size of applications. We present a highly-available parallel single level store system (HA-PSLS), which smoothly integrates a backward error recovery high-availability mechanism into a PSLS system. Our system is able to tolerate multiple transient failures, a single permanent failure, and power cut failures affecting the whole cluster, without requiring any specialized hardware. For this purpose, HA-PSLS relies on a high degree of integration (and reusability) of high-availability and standard features. A prototype integrating our high-availability support has been implemented and we show some performance results. (24 refs.)

Keywords: parallel single level store, high-availability, fault tolerance, checkpointing, replication, integration, parallel file systems, shared virtual memory, pario-bib

Abstract: We describe the effects of a new user-level library for the Galley Parallel File System. This library allows some pre-existing sequential programs to make use of the Galley Parallel File System with minimal modification. It permits programs to efficiently use the parallel file system because the user-level library groups accesses together. We examine the performance of our library, and we show how code needs to be modified to use the library.

Keywords: multiprocessor file system interface, run-time library, parallel file system, parallel I/O, pario-bib, dfk

Keywords: disk interleaving, parallel I/O, performance modeling, pario-bib

Comment: As opposed to synchronous disk interleaving, where disks are rotationally synchronous and one access is processed at a time. They develop a performance model and validate it with traces of a database system's disk accesses. Average access delay on each disk can be approximated by a normal distribution.

Keywords: parallel I/O, disk striping, scientific computing, algorithm, pario-bib

Keywords: parallel I/O, disk striping, file access pattern, disk array, pario-bib

Comment: Uniprocessor interleaving techniques. Good case for interleaving. Probably better to reference kim:interleaving and kim:fft. Discusses an 3D FFT algorithm in which the matrix is broken into subblocks that are accessed in layers. The layers are stored so this is either contiguous or with a regular stride, in fairly large chunks.

Keywords: parallel I/O, disk striping, disk array, pario-bib

Keywords: disk prefetching, parallel I/O, pario-bib

Comment: They do a theoretical analysis of prefetching and caching in uniprocessor, single- and multi-disk situations, given that they know the complete access sequence; their measure is not hit rate but rather overall execution time. They found some algorithms that are close to optimal.

Abstract: High-performance I/O systems depend on prefetching and caching in order to deliver good performance to applications. These two techniques have generally been considered in isolation, even though there are significant interactions between them; a block prefetched too early reduces the effectiveness of the cache, while a block cached too long reduces the effectiveness of prefetching. In this paper we study the effects of several combined prefetching and caching strategies for systems with multiple disks. Using disk-accurate trace-driven simulation, we explore the performance characteristics of each of the algorithms in cases in which applications provide full advance knowledge of accesses using hints. Some of the strategies have been published with theoretical performance bounds, and some are components of systems that have been built. One is a new algorithm that combines the desirable characteristics of the others. We find that when performance is limited by I/O stalls, aggressive prefetching helps to alleviate the problem; that more conservative prefetching is appropriate when significant I/O stalls are not present; and that a single, simple strategy is capable of doing both.

Keywords: parallel I/O, tracing, prefetch, trace-driven simulation, pario-bib

Keywords: parallel database, parallel I/O, pario-bib

Comment: Most interesting to me in this paper is their discussion of the ``container model,'' in which they claim they allow the processors to be driven by the I/O devices. See hirano:deadlock.

Abstract: We have developed a threaded parallel data streaming approach using Globus to transfer multi-terabyte simulation data from a remote supercomputer to the scientist's home analysis/visualization cluster, as the simulation executes, with negligible overhead. Data transfer experiments show that this concurrent data transfer approach is more favorable compared with writing to local disk and then transferring this data to be post-processed. The present approach is conducive to using the grid to pipeline the simulation with post-processing and visualization. We have applied this method to the Gyrokinetic Toroidal Code (GTC), a 3-dimensional particle-in-cell code used to study micro-turbulence in magnetic confinement fusion from first principles plasma theory.

Keywords: grid, parallel data streams, hydrodynamics, application, parallel I/O, pario-app, pario-bib

Comment: published on the web

Abstract: In this paper, we describe the design and implementation of the Platform Independent Parallel Solver (PIPSolver) package for the out-of-core (OOC) solution of complex dense linear systems. Our approach is unique in that it allows essentially all of RAM to be filled with the current portion of the matrix (slab) to be updated and factored, thereby greatly improving the computation to I/O ratio over previous approaches. This work could be viewed in part as an exercise in maximal code reuse: By formulating the OOC LU factorization just right, we managed to reuse essentially all of a very robust and efficient incore solver, leading directly to a very robust and efficient OOC solver. Experiences and performance are reported for the Cray T3D system.

Keywords: out-of-core algorithm, parallel I/O, pario-bib

Abstract: Mission to Planet Earth (MTPE) is a long-term NASA research mission to study the processes leading to global climate change. The EOS Data and Information System (EOSDIS) is the component within MTPE that will provide the Earth science community with easy, affordable, and reliable access to Earth science data. EOSDIS is a distributed system, with major facilities at eight Distributed Active Archive Centers (DAACs) located throughout the United States. At the DAACs the Science Data Processing Segment (SDPS) will receive, process, archive, and manage all data. It is estimated that several hundred gigaflops of processing power will be required to process and archive the several terabytes of new data that will be generated and distributed daily. Thousands of science users and perhaps several hundred thousand nonscience users will access the system.

Keywords: mass storage, I/O architecture, parallel I/O, pario-bib

Abstract: Scientific applications are increasingly being implemented on massively parallel supercomputers. Many of these applications have intense I/O demands, as well as massive computational requirements. This paper is essentially an annotated bibliography of papers and other sources of information about scientific applications using parallel I/O. It will be updated periodically.

Keywords: parallel I/O application, file access patterns, dfk, pario-bib

Abstract: Many scientific applications that run on today's multiprocessors, such as weather forecasting and seismic analysis, are bottlenecked by their file-I/O needs. Even if the multiprocessor is configured with sufficient I/O hardware, the file-system software often fails to provide the available bandwidth to the application. Although libraries and enhanced file-system interfaces can make a significant improvement, we believe that fundamental changes are needed in the file-server software. We propose a new technique, disk-directed I/O, to allow the disk servers to determine the flow of data for maximum performance. Our simulations show that tremendous performance gains are possible both for simple reads and writes and for an out-of-core application. Indeed, our disk-directed I/O technique provided consistent high performance that was largely independent of data distribution, obtained up to 93% of peak disk bandwidth, and was as much as 18 times faster than the traditional technique.

Keywords: parallel I/O, multiprocessor file system, file system caching, dfk, pario-bib

Comment: In jin:io-book, reprinted from kotz:jdiskdir.

Abstract: Improvements in the processing speed of multiprocessors are outpacing improvements in the speed of disk hardware. Parallel disk I/O subsystems have been proposed as one way to close the gap between processor and disk speeds. In a previous paper we showed that prefetching and caching have the potential to deliver the performance benefits of parallel file systems to parallel applications. In this paper we describe experiments with practical prefetching policies that base decisions only on on-line reference history, and that can be implemented efficiently. We also test the ability of these policies across a range of architectural parameters.

Keywords: dfk, parallel file system, prefetching, disk caching, parallel I/O, MIMD, pario-bib

Comment: Reformatted version of kotz:jpractical. In jin:io-book.

Abstract: Many scientific applications that run on today's multiprocessors are bottlenecked by their file I/O needs. Even if the multiprocessor is configured with sufficient I/O hardware, the file-system software often fails to provide the available bandwidth to the application. Although libraries and improved file-system interfaces can make a significant improvement, we believe that fundamental changes are needed in the file-server software. We propose a new technique, disk-directed I/O, that flips the usual relationship between server and client to allow the disks (actually, disk servers) to determine the flow of data for maximum performance. Our simulations show that tremendous performance gains are possible. Indeed, disk-directed I/O provided consistent high performance that was largely independent of data distribution, and close to the maximum disk bandwidth.

Keywords: parallel I/O, multiprocessor file system, file system caching, pario-bib, dfk

Comment: This paper also appeared in Bulletin of the IEEE Technical Committee on Operating Systems and Application Environments, Autumn 1994, pp. 29-42. Also available at http://www.usenix.org/publications/library/proceedings/osdi/kotz.html.

SEE TECH REPORT kotz:diskdir-tr. Please note that the tech report contains newer numbers than those in the OSDI version, although the conclusions have not changed.

Abstract: Many scientific applications that run on today's multiprocessors are bottlenecked by their file I/O needs. Even if the multiprocessor is configured with sufficient I/O hardware, the file-system software often fails to provide the available bandwidth to the application. Although libraries and improved file-system interfaces can make a significant improvement, we believe that fundamental changes are needed in the file-server software. We propose a new technique, disk-directed I/O, that flips the usual relationship between server and client to allow the disks (actually, disk servers) to determine the flow of data for maximum performance. Our simulations show that tremendous performance gains are possible. Indeed, disk-directed I/O provided consistent high performance that was largely independent of data distribution, and close to the maximum disk bandwidth.

Keywords: parallel I/O, multiprocessor file system, file system caching, dfk, pario-bib

Comment: Short version appeared in OSDI'94. Please note that the revised tech report contains newer numbers than those in the OSDI version, although the conclusions have not changed.

Abstract: Many scientific applications that run on today's multiprocessors are bottlenecked by their file I/O needs. Even if the multiprocessor is configured with sufficient I/O hardware, the file-system software often fails to provide the available bandwidth to the application. Although libraries and improved file-system interfaces can make a significant improvement, we believe that fundamental changes are needed in the file-server software. We propose a new technique, disk-directed I/O, that flips the usual relationship between server and client to allow the disks (actually, disk servers) to determine the flow of data for maximum performance. Our simulations show that tremendous performance gains are possible. Indeed, disk-directed I/O provided consistent high performance that was largely independent of data distribution, and close to the maximum disk bandwidth.

Keywords: parallel I/O, multiprocessor file system, file system caching, pario-bib, dfk

Comment: Same as kotz:diskdir.

SEE TECH REPORT kotz:diskdir-tr. Please note that the tech report contains newer numbers than those in the OSDI version, although the conclusions have not changed.

Abstract: We sketch the reasons for the I/O bottleneck in parallel and distributed systems, pointing out that it can be viewed as a special case of a general bottleneck that arises at all levels of the memory hierarchy. We argue that because of its severity, the I/O bottleneck deserves systematic attention at all levels of system design. We then present a survey of the issues raised by the I/O bottleneck in six key areas of parallel and distributed systems: applications, algorithms, languages and compilers, run-time libraries, operating systems, and architecture.

Keywords: survey, parallel I/O, pario-bib, dfk

Abstract: As parallel computers are increasingly used to run scientific applications with large data sets, and as processor speeds continue to increase, it becomes more important to provide fast, effective parallel file systems for data storage and for temporary files. In an earlier work we demonstrated that a technique we call disk-directed I/O has the potential to provide consistent high performance for large, collective, structured I/O requests. In this paper we expand on this potential by demonstrating the ability of a disk-directed I/O system to read irregular subsets of data from a file, and to filter and distribute incoming data according to data-dependent functions.

Keywords: parallel I/O, multiprocessor file systems, dfk, pario-bib

Abstract: As parallel computers are increasingly used to run scientific applications with large data sets, and as processor speeds continue to increase, it becomes more important to provide fast, effective parallel file systems for data storage and for temporary files. In an earlier work we demonstrated that a technique we call disk-directed I/O has the potential to provide consistent high performance for large, collective, structured I/O requests. In this paper we expand on this potential by demonstrating the ability of a disk-directed I/O system to read irregular subsets of data from a file, and to filter and distribute incoming data according to data-dependent functions.

Keywords: parallel I/O, multiprocessor file systems, dfk, pario-bib

Abstract: As parallel systems move into the production scientific-computing world, the emphasis will be on cost-effective solutions that provide high throughput for a mix of applications. Cost-effective solutions demand that a system make effective use of all of its resources. Many MIMD multiprocessors today, however, distinguish between ``compute'' and ``I/O'' nodes, the latter having attached disks and being dedicated to running the file-system server. This static division of responsibilities simplifies system management but does not necessarily lead to the best performance in workloads that need a different balance of computation and I/O.

Of course, computational processes sharing a node with a file-system service may receive less CPU time, network bandwidth, and memory bandwidth than they would on a computation-only node. In this paper we begin to examine this issue experimentally. We found that high-performance I/O does not necessarily require substantial CPU time, leaving plenty of time for application computation. There were some complex file-system requests, however, which left little CPU time available to the application. (The impact on network and memory bandwidth still needs to be determined.) For applications (or users) that cannot tolerate an occasional interruption, we recommend that they continue to use only compute nodes. For tolerant applications needing more cycles than those provided by the compute nodes, we recommend that they take full advantage of both compute and I/O nodes for computation, and that operating systems should make this possible.

Keywords: parallel I/O, multiprocessor file system, dfk, pario-bib

Abstract: Most MIMD multiprocessors today are configured with two distinct types of processor nodes: those that have disks attached, which are dedicated to file I/O, and those that do not have disks attached, which are used for running applications. Several architectural trends have led some to propose configuring systems so that all processors are used for application processing, even those with disks attached. We examine this idea experimentally, focusing on the impact of remote I/O requests on local computational processes. We found that in an efficient file system the I/O processors can transfer data at near peak speeds with little CPU overhead, leaving substantial CPU power for running applications. On the other hand, we found that some complex file-system features could require substantial CPU overhead. Thus, for a multiprocessor system to obtain good I/O and computational performance on a mix of applications, the file system (both operating system and libraries) must be prepared to adapt their policies to changing conditions.

Keywords: parallel I/O, multiprocessor file system, dfk, pario-bib

Abstract: Many scientific applications for high-performance multiprocessors have tremendous I/O requirements. As a result, the I/O system is often the limiting factor of application performance. Several new parallel file systems have been developed in recent years, each promising better performance for some class of parallel applications. As we gain experience with parallel computing, and parallel file systems in particular, it becomes increasingly clear that a single solution does not suit all applications. For example, it appears to be impossible to find a single appropriate interface, caching policy, file structure, or disk management strategy. Furthermore, the proliferation of file-system interfaces and abstractions make application portability a significant problem.

We propose that the traditional functionality of parallel file systems be separated into two components: a fixed core that is standard on all platforms, encapsulating only primitive abstractions and interfaces, and a set of high-level libraries to provide a variety of abstractions and application-programmer interfaces (APIs). We think of this approach as the ``RISC'' of parallel file-system design.

We present our current and next-generation file systems as examples of this structure. Their features, such as a three-dimensional file structure, strided read and write interfaces, and I/O-node programs, are specifically designed with the flexibility and performance necessary to support a wide range of applications.

Keywords: parallel I/O, multiprocessor file system, dfk, pario-bib

Comment: A position paper.

Abstract: As we gain experience with parallel file systems, it becomes increasingly clear that a single solution does not suit all applications. For example, it appears to be impossible to find a single appropriate interface, caching policy, file structure, or disk-management strategy. Furthermore, the proliferation of file-system interfaces and abstractions make applications difficult to port.

We propose that the traditional functionality of parallel file systems be separated into two components: a fixed core that is standard on all platforms, encapsulating only primitive abstractions and interfaces, and a set of high-level libraries to provide a variety of abstractions and application-programmer interfaces (APIs).

We present our current and next-generation file systems as examples of this structure. Their features, such as a three-dimensional file structure, strided read and write interfaces, and I/O-node programs, re specifically designed with the flexibility and performance necessary to support a wide range of applications.

Keywords: parallel I/O, multiprocessor file system, dfk, pario-bib

Comment: Nearly identical to kotz:flexibility. The only changes are the format, a shorter abstract, and updates to Section 7 and the references.

Abstract: Increasingly, file systems for multiprocessors are designed with parallel access to multiple disks, to keep I/O from becoming a serious bottleneck for parallel applications. Although file system software can transparently provide high-performance access to parallel disks, a new file system interface is needed to facilitate parallel access to a file from a parallel application. We describe the difficulties faced when using the conventional (Unix-like) interface in parallel applications, and then outline ways to extend the conventional interface to provide convenient access to the file for parallel programs, while retaining the traditional interface for programs that have no need for explicitly parallel file access. Our interface includes a single naming scheme, a multiopen operation, local and global file pointers, mapped file pointers, logical records, multifiles, and logical coercion for backward compatibility.

Keywords: dfk, parallel I/O, multiprocessor file system, file system interface, pario-bib

Comment: See also lake:pario for implementation of some of the ideas.

Abstract: Increasingly, file systems for multiprocessors are designed with parallel access to multiple disks, to keep I/O from becoming a serious bottleneck for parallel applications. Although file system software can transparently provide high-performance access to parallel disks, a new file system interface is needed to facilitate parallel access to a file from a parallel application. We describe the difficulties faced when using the conventional (Unix-like) interface in parallel applications, and then outline ways to extend the conventional interface to provide convenient access to the file for parallel programs, while retaining the traditional interface for programs that have no need for explicitly parallel file access. Our interface includes a single naming scheme, a multiopen operation, local and global file pointers, mapped file pointers, logical records, multifiles, and logical coercion for backward compatibility.

Keywords: dfk, parallel I/O, multiprocessor file system, file system interface, pario-bib

Comment: See also lake:pario for implementation of some of the ideas.

Keywords: dfk, parallel I/O, multiprocessor file system, file system interface, pario-bib

Comment: Short paper (2 pages).

Abstract: In other papers I propose the idea of disk-directed I/O for multiprocessor file systems. Those papers focus on the performance advantages and capabilities of disk-directed I/O, but say little about the application-programmer's interface or about the interface between the compute processors and I/O processors. In this short note I discuss the requirements for these interfaces, and look at many existing interfaces for parallel file systems. I conclude that many of the existing interfaces could be adapted for use in a disk-directed I/O system.

Keywords: disk-directed I/O, parallel I/O, multiprocessor filesystem interfaces, pario-bib, dfk

Comment: See also kotz:jdiskdir, kotz:expand, and kotz:lu.

Keywords: parallel I/O, multiprocessor file system, dfk, pario-bib

Comment: A bibliography of many references on parallel I/O and multiprocessor file-systems issues. As of the fifth edition, it is available on the WWW in HTML format.

Abstract: Many scientific applications that run on today's multiprocessors, such as weather forecasting and seismic analysis, are bottlenecked by their file-I/O needs. Even if the multiprocessor is configured with sufficient I/O hardware, the file-system software often fails to provide the available bandwidth to the application. Although libraries and enhanced file-system interfaces can make a significant improvement, we believe that fundamental changes are needed in the file-server software. We propose a new technique, disk-directed I/O, to allow the disk servers to determine the flow of data for maximum performance. Our simulations show that tremendous performance gains are possible both for simple reads and writes and for an out-of-core application. Indeed, our disk-directed I/O technique provided consistent high performance that was largely independent of data distribution, obtained up to 93% of peak disk bandwidth, and was as much as 18 times faster than the traditional technique.

Keywords: parallel I/O, multiprocessor file system, file system caching, dfk, pario-bib

Comment: This paper is a substantial revision of the diskdir-tr version: all of the experiments have been re-done, using a better-tuned version of the file systems (see kotz:tuning), and adding two-phase I/O to all comparisons. It also incorporates some of the material from kotz:expand and kotz:int-ddio. Also available at http://www.acm.org/pubs/citations/journals/tocs/1997-15-1/p41-kotz/.

Abstract: Improvements in the processing speed of multiprocessors are outpacing improvements in the speed of disk hardware. Parallel disk I/O subsystems have been proposed as one way to close the gap between processor and disk speeds. In a previous paper we showed that prefetching and caching have the potential to deliver the performance benefits of parallel file systems to parallel applications. In this paper we describe experiments with practical prefetching policies that base decisions only on on-line reference history, and that can be implemented efficiently. We also test the ability of these policies across a range of architectural parameters.

Keywords: dfk, parallel file system, prefetching, disk caching, parallel I/O, MIMD, pario-bib

Comment: See also kotz:jwriteback, kotz:fsint2, cormen:integrate.

Keywords: parallel file system, file access pattern, multiprocessor file system workload, parallel I/O, pario-bib, dfk

Abstract: Improvements in the processing speed of multiprocessors are outpacing improvements in the speed of disk hardware. Parallel disk I/O subsystems have been proposed as one way to close the gap between processor and disk speeds. Such parallel disk systems require parallel file system software to avoid performance-limiting bottlenecks. We discuss cache management techniques that can be used in a parallel file system implementation for multiprocessors with scientific workloads. We examine several writeback policies, and give results of experiments that test their performance.

Keywords: dfk, parallel file system, disk caching, parallel I/O, MIMD, pario-bib

Comment: See kotz:jpractical, kotz:fsint2, cormen:integrate.

Abstract: New file systems are critical to obtain good I/O performance on large multiprocessors. Several researchers have suggested the use of collective file-system operations, in which all processes in an application cooperate in each I/O request. Others have suggested that the traditional low-level interface (read, write, seek) be augmented with various higher-level requests (e.g., read matrix). Collective, high-level requests permit a technique called disk-directed I/O to significantly improve performance over traditional file systems and interfaces, at least on simple I/O benchmarks. In this paper, we present the results of experiments with an ``out-of-core'' LU-decomposition program. Although its collective interface was awkward in some places, and forced additional synchronization, disk-directed I/O was able to obtain much better overall performance than the traditional system.

Keywords: parallel I/O, numerical analysis, dfk, pario-bib

Abstract: New file systems are critical to obtain good I/O performance on large multiprocessors. Several researchers have suggested the use of collective file-system operations, in which all processes in an application cooperate in each I/O request. Others have suggested that the traditional low-level interface (read, write, seek) be augmented with various higher-level requests (e.g., read matrix), allowing the programmer to express a complex transfer in a single (perhaps collective) request. Collective, high-level requests permit techniques like two-phase I/O and disk-directed I/O to significantly improve performance over traditional file systems and interfaces. Neither of these techniques have been tested on anything other than simple benchmarks that read or write matrices. Many applications, however, intersperse computation and I/O to work with data sets that cannot fit in main memory. In this paper, we present the results of experiments with an ``out-of-core'' LU-decomposition program, comparing a traditional interface and file system with a system that has a high-level, collective interface and disk-directed I/O. We found that a collective interface was awkward in some places, and forced additional synchronization. Nonetheless, disk-directed I/O was able to obtain much better performance than the traditional system.

Keywords: parallel I/O, numerical analysis, dfk, pario-bib

Abstract: The computational performance of multiprocessors continues to improve by leaps and bounds, fueled in part by rapid improvements in processor and interconnection technology. I/O performance thus becomes ever more critical, to avoid becoming the bottleneck of system performance. In this paper we provide an introduction to I/O architectural issues in multiprocessors, with a focus on disk subsystems. While we discuss examples from actual architectures and provide pointers to interesting research in the literature, we do not attempt to provide a comprehensive survey. We concentrate on a study of the architectural design issues, and the effects of different design alternatives.

Keywords: parallel I/O, multiprocessor file system, pario-bib, dfk

Comment: Invited paper. Part of a whole book on parallel I/O; see iopads-book.

Abstract: Parallel disk subsystems have been proposed as one way to close the gap between processor and disk speeds. In a previous paper we showed that prefetching and caching have the potential to deliver the performance benefits of parallel file systems to parallel applications. In this paper we describe experiments with practical prefetching policies, and show that prefetching can be implemented efficiently even for the more complex parallel file access patterns. We test these policies across a range of architectural parameters.

Keywords: dfk, parallel file system, prefetching, disk caching, parallel I/O, MIMD, OS93W extra, OS92W, pario-bib

Comment: Short form of primary thesis results. See kotz:jwriteback, kotz:fsint2, cormen:integrate.

Abstract: The problem of providing file I/O to parallel programs has been largely neglected in the development of multiprocessor systems. There are two essential elements of any file system design intended for a highly parallel environment: parallel I/O and effective caching schemes. This paper concentrates on the second aspect of file system design and specifically, on the question of whether prefetching blocks of the file into the block cache can effectively reduce overall execution time of a parallel computation, even under favorable assumptions.

Experiments have been conducted with an interleaved file system testbed on the Butterfly Plus multiprocessor. Results of these experiments suggest that 1) the hit ratio, the accepted measure in traditional caching studies, may not be an adequate measure of performance when the workload consists of parallel computations and parallel file access patterns, 2) caching with prefetching can significantly improve the hit ratio and the average time to perform an I/O operation, and 3) an improvement in overall execution time has been observed in most cases. In spite of these gains, prefetching sometimes results in increased execution times (a negative result, given the optimistic nature of the study).

We explore why is it not trivial to translate savings on individual I/O requests into consistently better overall performance and identify the key problems that need to be addressed in order to improve the potential of prefetching techniques in this environment.

Keywords: dfk, parallel file system, prefetching, MIMD, disk caching, parallel I/O, pario-bib

Abstract: The increasing speed of the most powerful computers, especially multiprocessors, makes it difficult to provide sufficient I/O bandwidth to keep them running at full speed for the largest problems. Trends show that the difference in the speed of disk hardware and the speed of processors is increasing, with I/O severely limiting the performance of otherwise fast machines. This widening access-time gap is known as the ``I/O bottleneck crisis.'' One solution to the crisis, suggested by many researchers, is to use many disks in parallel to increase the overall bandwidth.

This dissertation studies some of the file system issues needed to get high performance from parallel disk systems, since parallel hardware alone cannot guarantee good performance. The target systems are large MIMD multiprocessors used for scientific applications, with large files spread over multiple disks attached in parallel. The focus is on automatic caching and prefetching techniques. We show that caching and prefetching can transparently provide the power of parallel disk hardware to both sequential and parallel applications using a conventional file system interface. We also propose a new file system interface (compatible with the conventional interface) that could make it easier to use parallel disks effectively.

Our methodology is a mixture of implementation and simulation, using a software testbed that we built to run on a BBN GP1000 multiprocessor. The testbed simulates the disks and fully implements the caching and prefetching policies. Using a synthetic workload as input, we use the testbed in an extensive set of experiments. The results show that prefetching and caching improved the performance of parallel file systems, often dramatically.

Keywords: dfk, parallel file system, prefetching, MIMD, disk caching, parallel I/O, pario-bib

Comment: Published as kotz:prefetch, kotz:jwriteback, kotz:jpractical, kotz:fsint2.

Keywords: parallel I/O, multiprocessor file system, performance, survey, dfk, pario-bib

Comment: A brief note on the reported performance of existing file systems (Intel CFS, nCUBE, CM-2, CM-5, and Cray). Many have disappointingly low absolute throughput, in MB/s.

Abstract: STARFISH is a parallel file-system simulator we built for our research into the concept of disk-directed I/O. In this report, we detail steps taken to tune the file systems supported by STARFISH, which include a traditional parallel file system (with caching) and a disk-directed I/O system. In particular, we now support two-phase I/O, use smarter disk scheduling, increased the maximum number of outstanding requests that a compute processor may make to each disk, and added gather/scatter block transfer. We also present results of the experiments driving the tuning effort.

Keywords: parallel I/O, multiprocessor file system, dfk, pario-bib

Comment: Reports on some new changes to the STARFISH simulator that implements traditional caching and disk-directed I/O. This is meant mainly as a companion to kotz:jdiskdir. See also kotz:jdiskdir, kotz:diskdir, kotz:expand.

Abstract: Multiprocessors have permitted astounding increases in computational performance, but many cannot meet the intense I/O requirements of some scientific applications. An important component of any solution to this I/O bottleneck is a parallel file system that can provide high-bandwidth access to tremendous amounts of data in parallel to hundreds or thousands of processors.

Most successful systems are based on a solid understanding of the characteristics of the expected workload, but until now there have been no comprehensive workload characterizations of multiprocessor file systems. We began the CHARISMA project in an attempt to fill that gap. We instrumented the common node library on the iPSC/860 at NASA Ames to record all file-related activity over a two-week period. Our instrumentation is different from previous efforts in that it collects information about every read and write request and about the mix of jobs running in the machine (rather than from selected applications).

The trace analysis in this paper leads to many recommendations for designers of multiprocessor file systems. First, the file system should support simultaneous access to many different files by many jobs. Second, it should expect to see many small requests, predominantly sequential and regular access patterns (although of a different form than in uniprocessors), little or no concurrent file-sharing between jobs, significant byte- and block-sharing between processes within jobs, and strong interprocess locality. Third, our trace-driven simulations showed that these characteristics led to great success in caching, both at the compute nodes and at the I/O nodes. Finally, we recommend supporting strided I/O requests in the file-system interface, to reduce overhead and allow more performance optimization by the file system.

Keywords: parallel file system, file access pattern, multiprocessor file system workload, parallel I/O, pario-bib, dfk

Comment: Also at http://www.acm.org/pubs/citations/proceedings/supercomputing/198354/p640-kotz and http://computer.org/conferen/sc94/kotz.html

Abstract: Multiprocessors have permitted astounding increases in computational performance, but many cannot meet the intense I/O requirements of some scientific applications. An important component of any solution to this I/O bottleneck is a parallel file system that can provide high-bandwidth access to tremendous amounts of data in parallel to hundreds or thousands of processors.

Most successful systems are based on a solid understanding of the characteristics of the expected workload, but until now there have been no comprehensive workload characterizations of multiprocessor file systems. We began the CHARISMA project in an attempt to fill that gap. We instrumented the common node library on the iPSC/860 at NASA Ames to record all file-related activity over a two-week period. Our instrumentation is different from previous efforts in that it collects information about every read and write request and about the mix of jobs running in the machine (rather than from selected applications).

The trace analysis in this paper leads to many recommendations for designers of multiprocessor file systems. First, the file system should support simultaneous access to many different files by many jobs. Second, it should expect to see many small requests, predominantly sequential and regular access patterns (although of a different form than in uniprocessors), little or no concurrent file-sharing between jobs, significant byte- and block-sharing between processes within jobs, and strong interprocess locality. Third, our trace-driven simulations showed that these characteristics led to great success in caching, both at the compute nodes and at the I/O nodes. Finally, we recommend supporting strided I/O requests in the file-system interface, to reduce overhead and allow more performance optimization by the file system.

Keywords: parallel file system, file access pattern, multiprocessor file system workload, parallel I/O, pario-bib, dfk

Abstract: Improvements in the processing speed of multiprocessors are outpacing improvements in the speed of disk hardware. Parallel disk I/O subsystems have been proposed as one way to close the gap between processor and disk speeds. Such parallel disk systems require parallel file system software to avoid performance-limiting bottlenecks. We discuss cache management techniques that can be used in a parallel file system implementation. We examine several writeback policies, and give results of experiments that test their performance.

Keywords: dfk, parallel file system, disk caching, parallel I/O, MIMD, pario-bib

Comment: See also kotz:jpractical, kotz:fsint2, cormen:integrate.

Abstract: The most frequently used part of MPI-2 is MPI I/O. Due to the complexity of parallel programming in general, and of handling parallel I/O in particular, there is a need for tools that support the application development process. There axe many situations where incorrect usage of MPI by the application programmer can be automatically detected. In this paper we describe the MARMOT tool that uncovers some of these errors and we also analyze to what extent it is possible to do so for MPI I/O.

Keywords: MPI I/O, error detection, performance analysis, MARMOT, pario-bib

Keywords: memory-mapped file, file system, parallel I/O, pario-bib

Abstract: This paper describes the design and implementation of a new application level I/O facility, called the Alloc Stream Facility. The Alloc Stream Facility has several key advantages. First, performance is substantially improved as a result of a) the structure of the facility that allows it to take advantage of system specific features like mapped files, and b) a reduction in data copying and the number of I/O system calls. Second, the facility is designed for multi-threaded applications running on multiprocessors and allows for a high degree of concurrency. Finally, the facility can support a variety of I/O interfaces, including stdio, emulated Unix I/O, ASI, and C++ streams, in a way that allows applications to freely intermix calls to the different interfaces, resulting in improved code reusability.

We show that on several Unix workstation platforms the performance of Unix applications using the Alloc Stream Facility can be substantially better that when the applications use the original I/O facilities.

Keywords: memory-mapped file, file system, parallel I/O, pario-bib

Comment: See also krieger:mapped. ``This is an extended version of the paper with the same title in the March, 1994 edition of IEEE Computer.'' A 3-level interface structure: interface, backplane, and stream-specific modules. Different interfaces available: unix, stdio, ASI (theirs), C++. Common backplane. Stream-specific implementations that export operations like salloc and sfree, which return pointers to data buffers. ASI exports that interface to the user, for maximum efficiency. Performance is best when using mapped files as underlying implementation. Many stdio or unix apps are faster only after relinking. ASI is even faster. In addition to better performance, also get multithreading support, multiple interfaces, and extensibility.

Abstract: The {H{\sc urricane}} File System (HFS) is a new file system being developed for large-scale shared memory multiprocessors with distributed disks. The main goal of this file system is scalability; that is, the file system is designed to handle demands that are expected to grow linearly with the number of processors in the system. To achieve this goal, HFS is designed using a new structuring technique called Hierarchical Clustering. HFS is also designed to be flexible in supporting a variety of policies for managing file data and for managing file system state. This flexibility is necessary to support in a scalable fashion the diverse workloads we expect for a multiprocessor file system.

Keywords: multiprocessor file system, parallel I/O, operating system, shared memory, pario-bib

Comment: This paper is now out of date; see krieger:thesis. Designed for scalability on the hierarchical clustering model (see unrau:cluster), the Hurricane File System for NUMA shared-memory MIMD machines. Each cluster has its own full file system, which communicates with those in other clusters. Pieces are name server, open-file server, and block-file server. On first access, the file is mapped into the application space. VM system calls BFS to arrange transfers. Open questions: policies for file state management, block distribution, caching, and prefetching. Object-oriented approach used to allow for flexibility and extendability. Local disk file systems are log-structured.

Abstract: The Hurricane File System (HFS) is designed for (potentially large-scale) shared memory multiprocessors. Its architecture is based on the principle that, in order to maximize performance for applications with diverse requirements, a file system must support a wide variety of file structures, file system policies and I/O interfaces. Files in HFS are implemented using simple building blocks composed in potentially complex ways. This approach yields great flexibility, allowing an application to customize the structure and policies of a file to exactly meet its requirements. For example, a file's structure can be optimized for concurrent random-access write-only operations by ten processes. Similarly, the prefetching, locking, and file cache management policies can all be chosen to match an application's access pattern. In contrast, most existing parallel file systems support a single file structure and a small set of policies.

We have implemented HFS as part of the Hurricane operating system running on the Hector shared memory multiprocessor. We demonstrate that the flexibility of HFS comes with little processing or I/O overhead. We also show that for a number of file access patterns HFS is able to deliver to the applications the full I/O bandwidth of the disks on our system.

Keywords: parallel I/O, parallel file system, object-oriented, pario-bib

Comment: A published form of krieger:hfs and the thesis krieger:thesis. Their main point is that the file system is constructed from building-block objects. When you create a file you choose a few building blocks, for example, a replication block that mirrors the file, and some distribution blocks that distribute each replica across a set of disks. When you open the file you plug in some more building blocks, e.g., to do prefetching or to provide the kind of interface that you want to use. They point out that this flexibility is critical to be able to get good performance, because different file-access patterns need different structures and policies. They found that mapped files minimize copying costs and improve performance. They were able to obtain full disk bandwidth. Great paper.

Abstract: The Hurricane File System (HFS) is designed for (potentially large-scale) shared-memory multiprocessors. Its architecture is based on the principle that, in order to maximize performance for applications with diverse requirements, a file system must support a wide variety of file structures, file system policies, and I/O interfaces. Files in HFS are implemented using simple building blocks composed in potentially complex ways. This approach yields great flexibility, allowing an application to customize the structure and policies of a file to exactly meet its requirements. As an extreme example, HFS allows a file's structure to be optimized for concurrent random-access write-only operations by 10 threads, something no other file system can do. Similarly, the prefetching, locking, and file cache management policies can all be chosen to match an application's access pattern. In contrast, most parallel file systems support a single file structure and a small set of policies. We have implemented HFS as part of the Hurricane operating system running on the Hector shared-memory multiprocessor. We demonstrate that the flexibility of HFS comes with little processing or I/O overhead. We also show that for a number of file access patterns, HFS is able to deliver to the applications the full I/O bandwidth of the disks on our system.

Keywords: parallel I/O, parallel file system, object-oriented, pario-bib

Abstract: The Hurricane File System (HFS) is designed for large-scale, shared-memory multiprocessors. Its architecture is based on the principle that a file system must support a wide variety of file structures, file system policies and I/O interfaces to maximize performance for a wide variety of applications. HFS uses a novel, object-oriented building-block approach to provide the flexibility needed to support this variety of file structures, policies, and I/O interfaces. File structures can be defined in HFS that optimize for sequential or random access, read-only, write-only or read/write access, sparse or dense data, large or small file sizes, and different degrees of application concurrency. Policies that can be defined on a per-file or per-open instance basis include locking policies, prefetching policies, compression/decompression policies and file cache management policies. In contrast, most existing file systems have been designed to support a single file structure and a small set of policies.

We have implemented large portions of HFS as part of the Hurricane operating system running on the Hector shared-memory multiprocessor. We demonstrate that the flexibility of HFS comes with little processing or I/O overhead. Also, we show that HFS is able to deliver the full I/O bandwidth of the disks on our system to the applications.

Keywords: parallel I/O, multiprocesor file system, shared memory, memory-mapped I/O, pario-bib

Comment: Excellent work. HFS uses an object-oriented building-block approach to provide flexible, scalable high performance. Indeed, HFS appears to be one of the most flexible parallel file systems available, allowing users to independently control (or redefine) policies for prefetching, caching, redundancy and fault tolerance, and declustering.

Keywords: parallel I/O, parallel file system, parallel I/O, performance measurement, pario-bib

Comment: Short measurements of CM-2 Datavault. Faster if you access through Paris. Can get nearly full 32 MB/s bandwidth. Problem in its ability to use multiple CMIO busses.

Keywords: parallel I/O, multiprocessor file system, pario-bib

Comment: Essentially a (short) combination of krystynak:datavault and nitzberg:cfs.

Keywords: parallel file system interface, pario-bib

Comment: Experience making libc reentrant, adding semaphores, etc., on a Convex. Some problems with I/O. Added semaphores and private memory to make libc calls reentrant, i.e., callable in parallel by multiple threads.

Keywords: parallel I/O, pario-bib

Comment: A visualization tool, now long gone, for display of CHARISMA trace files. See nieuwejaar:workload for details of CHARISMA.

Abstract: The paper summarizes our experiences using the Panda parallel I/O library with the H3expresso numerical relativity code on the Cornell SP2. Two performance issues are described: providing efficient off-loading of output data, and satisfying users' desire to dedicate fewer nodes to I/O. We explore the tradeoffs between potential solutions, and present performance results for our approaches. We also show that Panda's high level interface, which allows the user to request input or output of a set of arrays with a single command, is a good match for H3expresso's needs

Keywords: application experience, parallel input/output, parallel I/O, performance issues, multiprocessor file system interface, pario-bib

Abstract: Large simulations which run for hundreds of hours on parallel computers often periodically generate snapshots of states, which are later post-processed to visualize the simulated physical phenomenon. For many applications, fast I/O during post-processing, which is dependent on an efficient organization of data on disk, is as important as minimizing computation-time I/O. In this paper we propose optimizations to support efficient parallel I/O for scientific simulations and subsequent visualizations. We present an ordering mechanism to linearize data on disk, a performance model to help to choose a proper stripe unit size, and a scheduling algorithm to minimize communication contention. Our experiments on an IBM SP show that the combination of these strategies provides a 20-25% performance boost.

Keywords: scientific computing, simulation, parallel I/O, pario-bib

Keywords: scientific applications, query-based interface, parallel I/O, pario-bib

Comment: They describe an architecture for accessing data in scientific datasets by performing range queries (a multidimensional bounding box) over the data. This type of access mechanism is useful for applications like satellite imaging.

Keywords: parallel I/O, parallel architecture, multiprocessor file system, pario-bib

Comment: They measure the performance of the CM-5 Scalable File System using synthetic benchmarks. They compare CM-Fortran with CMMD. The hardware-dependent (``physical'') modes were much faster than the generic-format modes, which have to reorder data between the processor distribution and the disk distribution. The network turned out to be a bottleneck for the performance when reordering was needed. They conclude that more user control over the I/O would be very helpful.

Keywords: parallel I/O, sorting, pario-bib

Comment: Examines external sorting techniques such as merge sort, tag sort, multi-pass distribution sort, and one-pass distribution sort. The model is one where I/O complexity is included, assuming a linear seek time distribution and a cost of 1/2 rotation for each seek. Parallel I/O or computing are not considered until the distribution sorts. Architectural model on page 58.

Abstract: Recent studies have demonstrated that a significant number of I/O operations are performed by a number of classes of different parallel applications. Appropriate I/O management strategies are required however for harnessing the power of parallel I/O. This paper focuses on two I/O management issues that affect system performance in multiprogrammed parallel environments. Characterization of I/O behavior of parallel applications in terms of four different models is discussed first, followed by an investigation of the performance of a number of different data distribution strategies. Using computer simulations this research shows that I/O characteristics of applications and data distribution have an important effect on system performance. Applications that can simultaneously do computation and I/O, plus strategies that can incorporate centralized I/O management are found to be beneficial for a multiprogrammed parallel environment.

Keywords: parallel I/O, pario-bib

Comment: See majumdar:management.

Abstract: Recent studies have demonstrated that significant I/O is performed by a number of parallel applications. In addition to running these applications on multiple processors, the parallelization of I/O operations and the use of multiple disk drives are required for achieving high system performance. This research is concerned with the effective management of parallel I/O by using appropriate I/O scheduling strategies. Based on a simulation model the performance of a number of scheduling policies are investigated. Using I/O characteristics of jobs such as the total outstanding I/O demand is observed to be useful in devising effective scheduling strategies.

Keywords: parallel I/O, scheduling, pario-bib

Keywords: parallel I/O, MIMD, multiprocessor file system, pario-bib

Comment: They describe the I/O system for the Myrias SPS-3 parallel computer. The SPS is a no-remote-access (NORMA) machine with a software shared memory abstraction. They provide a standard C/FORTRAN I/O interface, with a few extensions. The user's parallel program is considered a client, and an I/O processor (IOP) is the server. No striping across IOPs, which makes it relatively simple for them to have the server manage the shared file pointer. Their extensions allow atomic, file-pointer update, returning the actual position where I/O occurred, and atomic access to fixed- and variable-length records. They have three protocols, for different transfer sizes; small using simple request/response; medium using sliding window; and large using scatter/gather and special hardware double buffering at the IOP. They use scatter/gather DMA, and page-table fiddling, for messaging. Performance is 89-96% of hardware peak, limited by IOP's VME backplane.

Keywords: parallel I/O, algorithms, pario-bib

Abstract: As the number of nodes in cluster systems continues to grow, leveraging scalable algorithms in all aspects of such systems becomes key to maintaining performance. While scalable algorithms have been applied successfully in some areas of parallel I/O, many operations are still performed in an uncoordinated manner. In this work we consider, in three file system scenarios, the possibilities for applying scalable algorithms to the many operations that make up the MPI-IO interface. From this evaluation we extract a set of file system characteristics that aid in developing scalable MPI-IO implementations.

Keywords: scalability analysis, MPI-IO, pario-bib

Keywords: pvfs2, parallel file system, pario-bib

Abstract: The paper describes a new interconnection network for massively parallel systems, referred to as star-connected cycles (SCC). The SCC graph presents an I/O-bounded structure that results in several advantages over variable degree graphs like the star and the hypercube. The description of the SCC graph includes issues such as labelling of nodes, degree, diameter and symmetry. The paper also presents an optimal routeing algorithm for the SCC and efficient broadcasting algorithms with O(n) running time, with n being the dimensionality of the graph. A comparison with the cube-connected cycles (CCC) and other interconnection networks is included, indicating that, for even n, an n-SCC and a CCC of similar sizes have about the same diameter. In addition, it is shown that one-port broadcasting in an n-SCC graph can be accomplished with a running time better than or equal to that required by an n-star containing (n-1) times fewer nodes.

Keywords: parallel I/O, parallel computer architecture, pario-bib

Keywords: srb, performance-related optimization, pario, pario-bib

Comment: SRB data transfer optimization on cluster storage servers. If disk-bound, the system transfers from server to disks are broken so that protocol processing and disk transfer are pipelined. If network bound, stripe transfer from multiple clients to multiple servers. No mention of remote execution.

Keywords: distributed file system, multiprocessor file system, pario-bib

Comment: See also broom:acacia, broom:impl, mutisya:cache, and broom:cap. The Acacia file system has file access modes that are much like those in Intel CFS and TMC CMMD. By default all processes have their own file pointer, but they can switch to another mode either all together or in row- or column-subsets. The other modes include a replicated mode (where all read or write the same data), and a variety of shared modes, with arbitrary, fixed, or unspecified ordering among processors, and with fixed or variable-sized records. They also have a parallel-open operation, support for logical records, control over the striping width (number of disks) and height (block size), and control over of redundancy. A prototype is running.

Keywords: parallel I/O, disk array, RAID, pario-bib

Comment: An early paper, perhaps the earliest, that describes the techniques that later became RAID. Lawlor notes how to use parity to recover data lost due to disk crash, as in RAID3, addresses the read-before-write problem by caching the old data block as well as the new data block, and shows how two-dimensional parity can protect against two or more failures.

Keywords: RAID, disk array, reliability, parallel I/O, pario-bib

Comment: Part of jin:io-book; reformatted version of lee:jparity.

Keywords: parallel I/O, distributed file system, declustering, reliability, pario-bib

Comment: Part of jin:io-book; reformatted version of lee:petal.

Keywords: parallel I/O, file prefetching, pario-bib

Keywords: parallel I/O, pario-bib

Abstract: We address the problem of assigning nonpartitioned files in a parallel I/O system where the file accesses exhibit Poisson arrival rates and fixed service times. We present two new file assignment algorithms based on open queuing networks which aim at minimizing simultaneously the load balance across all disks, as well as the variance of the service time at each disk. We first present an off-line algorithm, Sort Partition, which assigns to each disk files with similar access time. Next, we show that, assuming that a perfectly balanced file assignment can be found for a given set of files, Sort Partition will find the one with minimal mean response time. We then present an on-line algorithm, Hybrid Partition, that assigns groups of files with similar service times in successive intervals while guaranteeing that the load imbalance at any point does not exceed a certain threshold. We report on synthetic experiments which exhibit skew in file accesses and sizes and we compare the performance of our new algorithms with the vanilla greedy file allocation algorithm.

Keywords: parallel I/O, parallel file system, pario-bib

Keywords: parallel I/O, disk striping, performance, pario-bib

Comment: Details of their prototype. Defines terms like stripe unit. Explores ways to lay out parity. Does performance simulations. Describes ops needed in device driver. Good to read if you plan to implement a RAID. Results: small R+W, or high loads, don't care about parity placement; in low load, there are different best cases for large R+W. Best all-around is left-symmetric. See also lee:parity.

Keywords: RAID, reliability, parallel I/O, disk striping, pario-bib

Comment: Journal version of lee:parity.

Abstract: In this paper we propose user-controllable I/O operations and explore the effects of them with some synthetic access patterns. The operations allow users to determine a file structure matching the access patterns, control the layout and distribution of data blocks on physical disks, and present various access patterns with a minimum number of I/O operations. The operations do not use a file pointer to access data as in typical file systems, which eliminates the overhead of managing the offset of the file, making it easy to share data and reducing the number of I/O operations.

Keywords: logical disks, parallel I/O, pario-bib

Keywords: parallel I/O, prefetching, disk array, pario-bib

Keywords: RAID, disk array, reliability, parallel I/O, pario-bib

Comment: Interesting comparison of several parity placement schemes. Boils down to two basic choices, depending on whether read performance or write performance is more important to you.

Keywords: parallel I/O, distributed file system, declustering, reliability, pario-bib

Comment: They are trying to build a file server that is easier to manage than most of today's distributed file systems, because disks are cheap but management is expensive. They describe a distributed file server that spreads blocks of all files across many disks and many servers. They use chained declustering so that they can survive loss of server or disk. They dynamically balance load. They dynamically reconfigure when new virtual disks are created or new physical disks are added. They've built it all and are now going to look at possible file systems that can take advantage of the features of Petal.

Keywords: disk array, parallel I/O, RAID, analytic model, pario-bib

Abstract: This paper presents communication-efficient algorithms for the external data redistribution problem. Deterministic lower bounds and upper bounds are presented for the number of I/O operations, communication time and the memory requirements of external redistribution. Our algorithms differ from most other algorithms presented for out-of-core applications in that it is optimal (within a small constant factor) not only in the number of I/O operations, but also in the time taken for communication. A coarse-grained MIMD architecture with I/O subsystems attached to each processor is assumed, but the results are expected to be applicable over a wider variety of architectures.

Keywords: parallel I/O algorithm, out-of-core, pario-bib

Comment: See shankar:transport for the underlying communication primitives.

Abstract: The paper describes a parallel file object environment to support distributed array store on shared nothing distributed computing environments. Our environment enables programmers to extend the concept of array distribution from memory levels to file levels. It allows parallel I/O according to the distribution of objects in an application. When objects are read and/or written by multiple applications using different distributions, we present a novel scheme to help programmers to select the best data distribution pattern according to minimum amount of remote data movements for the store of array objects on distributed file systems.

Keywords: parallel I/O, object oriented, distributed memory, pario-bib

Abstract: This paper presents the design of UPIO, a software for user-controllable parallel input and output. UPIO is designed to maximize I/O performance for scientific applications on MIMD multicomputers. The most important features of UPIO are: It supports a domain-specific file model and a variety of application interfaces to present numerous access patterns. UPIO provides user-contollerable I/O operations that allow users to control data access, file structure, and data distribution. The domain-specific file model and user controllability give low I/O overhead and allow programmers to exploit the aggregate bandwidth of parallel disks.

Keywords: parallel I/O, pario-bib

Comment: They describe an interface that seems to allow easier access for programmers that want to map matrices onto parallel files. The concepts are not well explained, so it's hard to really understand what is new and different. They make no explicit comparison with other advanced interfaces like that in Vesta or Galley. No performance results.

Abstract: In many different areas of computing, problems can arise which are too large to fit in main memory. For these problems, the I/O cost of moving data between main memory and secondary storage (for example, disks) becomes a significant bottleneck affecting the performance of the program.

Since most algorithms do not take into account the size of main memory, new algorithms have been developed to optimize the number of I/O's performed. This paper details the implementation of one such algorithm, for external-memory depth-first search.

Depth-first search is a basic tool for solving many problems in graph theory, and since graph theory is applicable for many large computational problems, it is important to make sure that such a basic tool is designed to avoid the bottleneck of main memory to secondary storage I/O's.

The algorithm whose implementation is described in this paper is sketched out in an extended abstract by Chiang et al. We attempt to improve the given algorithm by minimizing I/O's performed, and to extend the algorithm by finding disjoint trees, and by classifying all the edges in the problem.

Keywords: out-of-core algorithm, parallel I/O, pario-bib

Comment: Senior honors thesis. Advisor: Tom Cormen.

Abstract: The paper shows the implementation of a 3D simulation code for turbulent flow and combustion processes in full-scale utility boilers on an Intel Paragon XP/S computer. For the portable parallelization, an explicit approach is chosen using a domain decomposition method for the static subdivision of the numerical grid together with the SPMD programming model. The measured speedup for the presented case using a coarse grid is good, although some numerical requirements restrict the implemented message passing to strongly synchronized communication. On the Paragon, the NX message passing library is used for the computations. Furthermore, MPI and PVM are applied and their pros and cons on this computer are described. In addition to the basic message passing techniques for local and global communication, other possibilities are investigated. Besides the applicability of the vectorizing capability of the compiler, the influence of the I/O performance during computations is demonstrated. The scalability of the parallel application is presented for a refined discretization.

Keywords: parallel I/O, application, pario-bib

Keywords: parallel file system, distributed shared memory, DSM, COMA, pario-bib

Comment: Basically, cooperative shared memory with a backing store.

Keywords: parallel I/O algorithm, pario-bib

Abstract: This paper presents a framework of using resource metrics to characterize the various models of parallel computation. Our framework reflects the approach of recent models to abstract architectural details into several generic parameters, which we call resource metrics. We examine the different resource metrics chosen by different parallel models, categorizing the models into four classes: the basic synchronous models, and extensions of the basic models which more accurately reflect practical machines by incorporating notions of asynchrony, communication cost and memory hierarchy. We then present a new parallel computation model, the LogP-HMM model, as an illustration of design principles based on the framework of resource metrics. The LogP-HMM model extends an existing parameterized network model (LogP) with a sequential hierarchical memory model (HMM) characterizing each processor. The result accurately captures both network communication costs and the effects of multileveled memory such as local cache and I/O. We examine the potential utility of our model in the design of near optimal sorting and FFT algorithms.

Keywords: parallel I/O algorithm, pario-bib

Abstract: In this paper, we present a framework for synthesizing I/O efficient out-of-core programs for block recursive algorithms, such as the fast Fourier transform (FFT) and block matrix transposition algorithms. Our framework uses an algebraic representation which is based on tensor products and other matrix operations. The programs are optimized for the striped Vitter and Shriver's two-level memory model in which data can be distributed using various cyclic(B) distributions in contrast to the normally used physical track distribution cyclic(B_d), where B_d is the physical disk block size.

We first introduce tensor bases to capture the semantics of block-cyclic data distributions of out-of-core data and also data access patterns to out-of-core data. We then present program generation techniques for tensor products and matrix transposition. We accurately represent the number of parallel I/O operations required for the synthesized programs for tensor products and matrix transposition as a function of tensor bases and data distributions. We introduce an algorithm to determine the data distribution which optimizes the performance of the synthesized programs. Further, we formalize the procedure of synthesizing efficient out-of-core programs for tensor product formulas with various block-cyclic distributions as a dynamic programming problem.

We demonstrate the effectiveness of our approach through several examples. We show that the choice of an appropriate data distribution can reduce the number of passes to access out-of-core data by as large as eight times for a tensor product, and the dynamic programming approach can largely reduce the number of passes to access out-of-core data for the overall tensor product formulas.

Keywords: parallel I/O, out-of-core algorithm, pario-bib

Abstract: This paper presents a framework for synthesizing I/O-efficient out-of-core programs for block recursive algorithms, such as the fast Fourier transform and matrix transpositions. the programs are synthesized from tensor (Kronecker) product representations of algorithms. These programs are optimized for a striped two-level memory model where in the out-of-core data can have block-cyclic distributions on multiple disks.

Keywords: parallel I/O algorithm, pario-bib

Abstract: In this paper, we present a framework for synthesizing I/O efficient out-of-core programs for block recursive algorithms, such as the fast Fourier transform (FFT) and block matrix transposition algorithms. Our framework uses an algebraic representation which is based on tensor products and other matrix operations. The programs are optimized for the striped Vitter and Shriver's two-level memory model in which data can be distributed using various cyclic(B) distributions in contrast to the normally used physical track distribution cyclic(B_d), where B_d is the physical disk block size.

We first introduce tensor bases to capture the semantics of block-cyclic data distributions of out-of-core data and also data access patterns to out-of-core data. We then present program generation techniques for tensor products and matrix transposition. We accurately represent the number of parallel I/O operations required for the synthesized programs for tensor products and matrix transposition as a function of tensor bases and data distributions. We introduce an algorithm to determine the data distribution which optimizes the performance of the synthesized programs. Further, we formalize the procedure of synthesizing efficient out-of-core programs for tensor product formulas with various block-cyclic distributions as a dynamic programming problem.

We demonstrate the effectiveness of our approach through several examples. We show that the choice of an appropriate data distribution can reduce the number of passes to access out-of-core data by as large as eight times for a tensor product, and the dynamic programming approach can largely reduce the number of passes to access out-of-core data for the overall tensor product formulas.

Keywords: parallel I/O algorithm, pario-bib

Abstract: For concurrent I/O operations, atomicity defines the results in the overlapping file regions simultaneously read/written by requesting processes. Atomicity has been well studied at the file system level, such as POSIX standard. In this paper, we investigate the problems arising from the implementation of MPI atomicity for concurrent overlapping write access and provide two programming solutions. Since the MPI definition of atomicity differs from the POSIX one, an implementation that simply relies on the POSIX file systems does not guarantee correct MPI semantics. To have a correct implementation of atomic I/O in MPI, we examine the efficiency of three approaches: 1) file locking, 2) graph-coloring, and 3) process-rank ordering. Performance complexity for these methods are analyzed and their experimental results are presented for file systems including NFS, SGI's XFS, and IBM's GPFS.

Keywords: MPI, concurrent I/O operations, overlapping write access, atomic I/O operations, pario-bib

Abstract: Dedicated cluster parallel computers (DCPCs) are emerging as low-cost high performance environments for many important applications in science and engineering. A significant class of applications that perform well on a DCPC are coarse-grain applications that involve large amounts of file I/O. Current research in parallel file systems for distributed systems is providing a mechanism for adapting these applications to the DCPC environment. We present the Parallel Virtual File System (PVFS), a system that provides disk striping across multiple nodes in a distributed parallel computer and file partitioning among tasks in a parallel program. PVFS is unique among similar systems in that it uses a stream-based approach that represents each file access with a single set of request parameters and decouples the number of network messages from details of the file striping and partitioning. PVFS also provides support for efficient collective file accesses and allows overlapping file partitions. We present results of early performance experiments that show PVFS achieves excellent speedups in accessing moderately sized file segments.

Keywords: parallel I/O, cluster computing, parallel file system, pario-bib

Keywords: parallel I/O, workstation cluster, text retrieval, pario-bib

Comment: They implement a parallel text retrieval application on a cluster of DEC 5000 workstations.

Abstract: In this paper, we study I/O server placement for optimizing parallel I/O performance on switch-based clusters, which typically adopt irregular network topologies to allow construction of scalable systems with incremental expansion capability. Finding optimal solution to this problem is computationally intractable. We quantified the number of messages travelling through each network link by a workload function, and developed three heuristic algorithms to find good solutions based on the values of the workload function. Our simulation results demonstrate performance advantage of our algorithms over a number of algorithms commonly used in existing parallel systems. In particular, the load-balance-based algorithm is superior to the other algorithms in most cases, with improvement ratio of 10% to 95 in terms of parallel I/O throughput.

Keywords: I/O server placement, network topologies, switch-based clusters, pario-bib

Keywords: disk array, log-structured file system, RAID, parallel I/O, pario-bib

Comment: Part of jin:io-book.

Keywords: parallel I/O, pario-bib

Keywords: parallel I/O, disk striping, disk array, pario-bib

Keywords: parallel I/O, replication, file system, disk mirroring, disk shadowing, pario-bib

Comment: A look at shadowed disks. If you have $k$ disks set up to read from the disk with the shortest seek, but write to all disks, you have increased reliability, read time like the min of the seeks, and write time like the max of the seeks. It appears that with increasing $k$ you can get good performance. But this paper clearly shows, since writes move all disk heads to the same location, that the effective value of $k$ is actually quite low. Only 4-10 disks are likely to be useful for most traffic loads.

Abstract: It is now recognized that a high level of I/O performance is crucial in making effective use of parallel machines for many scientific application codes. This paper considers the I/O requirements in one particular scientific application area; 3D modelling of continental shelf sea regions. We identify some of the scientific aims which drive the model development, and the consequent impact on the I/O needs. As a case study we take a parallel production code running a simulation of the North Sea on a Cray T3D platform and investigate the I/O performance in dealing with the dominant I/O component; dumping of results data to disk. In order to place the performance issues in a more general framework we construct a simple theoretical model of I/O requirements, and use this to probe the impact of available I/O performance on current and proposed scientific objectives.

Keywords: parallel I/O application, pario-bib

Comment: In a Special Issue on I/O in Parallel Applications, volume 12, numbers 3 and 4.

Keywords: RAID, disk array, parallel I/O, distributed file system, pario-bib

Comment: One of the features of this system is the way they develop and execute transaction plans as little scripts that are built by the client, sent to the servers, and then executed by interpreters.

Keywords: parallel I/O, multiprocessor file system, pario-bib

Comment: They took the Unix file system from SunOS and extended it to run on the CM-5. This involved handling non-power-of-two block sizes, parallel I/O calls, large file sizes, and more encouragement for extents to be allocated. The hardware is particularly suited to RAID 3 with a 16 byte striping unit, although in theory the software could do anything it wants. Geared to data-parallel model. Proc nodes (PNs) contact the timesharing daemon (TSD) on the control processor (CP), who gets block lists from the file system, which runs on one of the CPs. The TSD then arranges with the disk storage nodes (DSNs) to do the transfer directly with the PNs. Each DSN has 8 MB of buffer space, 8 disk drives, 4 SCSI busses, and a SPARC as controller. Partition managers mount non-local sfs via NFS. Performance results good. Up to 185 MB/s on 118 (2 MB/s) disks.