A data grid is an architecture or set of services that allows users to access, modify and transfer extremely large amounts of geographically distributed data for research purposes.Allcock, Bill; Chervenak, Ann; Foster, Ian; et al. Data Grid tools: enabling science on big distributed data Data grids make this possible through a host of middleware applications and services that pull together data and resources from multiple administrative domains and then present it to users upon request.
The data in a data grid can be located at a single site or multiple sites where each site can be its own administrative domain governed by a set of security restrictions as to who may access the data.Venugopal, Srikumar; Buyya, Rajkumar; Ramamohanarao, Kotagiri. A taxonomy of data grids for distributed data sharing - management and processing p.37 Likewise, multiple replicas of the data may be distributed throughout the grid outside their original administrative domain and the security restrictions placed on the original data for who may access it must be equally applied to the replicas.Shorfuzzaman, Mohammad; Graham, Peter; Eskicioglu, Rasit. Adaptive replica placement in hierarchical data grids. p.15 Specifically developed data grid middleware is what handles the integration between users and the data they request by controlling access while making it available as efficiently as possible.
Middleware Middleware provides all the services and applications necessary for efficient management of datasets and files within the data grid while providing users quick access to the datasets and files.Padala, Pradeep. A survey of data middleware for Grid systems p.1 There is a number of concepts and tools that must be available to make a data grid operationally viable. However, at the same time not all data grids require the same capabilities and services because of differences in access requirements, security and location of resources in comparison to users. In any case, most data grids will have similar middleware services that provide for a universal name space, data transport service, data access service, data replication and resource management service. When taken together, they are key to the data grids functional capabilities.
Universal namespace Since sources of data within the data grid will consist of data from multiple separate systems and networks using different file naming conventions, it would be difficult for a user to locate data within the data grid and know they retrieved what they needed based solely on existing physical file names (PFNs). A universal or unified name space makes it possible to create logical file names (LFNs) that can be referenced within the data grid that map to PFNs.Padala, Pradeep. A survey of data middleware for Grid systems When an LFN is requested or queried, all matching PFNs are returned to include possible replicas of the requested data. The end user can then choose from the returned results the most appropriate replica to use. This service is usually provided as part of a management system known as a Storage Resource Broker (SRB).Arcot, Rajasekar; Wan, Michael; Moore, Reagan; Schroeder, Wayne; Kremenek. Storage resource broker – managing distributed data in a grid Information about the locations of files and mappings between the LFNs and PFNs may be stored in a metadata or replica catalogue.Venugopal, Srikumar; Buyya, Rajkumar; Ramamohanarao, Kotagiri. A taxonomy of data grids for distributed data sharing - management and processing p.11 The replica catalogue would contain information about LFNs that map to multiple replica PFNs.
Data transport service Another middleware service is that of providing for data transport or data transfer. Data transport will encompass multiple functions that are not just limited to the transfer of bits, to include such items as fault tolerance and data access.Coetzee, Serena. Reference model for a data grid approach to address data in a dynamic SDI p.16 Fault tolerance can be achieved in a data grid by providing mechanisms that ensures data transfer will resume after each interruption until all requested data is received.Venugopal, Srikumar; Buyya, Rajkumar; Ramamohanarao, Kotagiri. A taxonomy of data grids for distributed data sharing - management and processing p.21 There are multiple possible methods that might be used to include starting the entire transmission over from the beginning of the data to resuming from where the transfer was interrupted. As an example, GridFTP provides for fault tolerance by sending data from the last acknowledged byte without starting the entire transfer from the beginning.
The data transport service also provides for the low-level access and connections between hosts for file transfer.Allcock, Bill; Foster, Ian; Nefedova, Veronika; Chervenak, Ann; Deelman, Ewa; Kesselman, Carl. High-performance remote access to climate simulation data: A challenge problem for data grid technologies. The data transport service may use any number of modes to implement the transfer to include parallel data transfer where two or more data streams are used over the same channel or striped data transfer where two or more steams access different blocks of the file for simultaneous transfer to also using the underlying built-in capabilities of the network hardware or specifically developed protocols to support faster transfer speeds.Izmailov, Rauf; Ganguly, Samrat; Tu, Nan. Fast parallel file replication in data grid p.2 The data transport service might optionally include a network overlay function to facilitate the routing and transfer of data as well as file I/O functions that allow users to see remote files as if they were local to their system. The data transport service hides the complexity of access and transfer between the different systems to the user so it appears as one unified data source.
Data access service Data access services work hand in hand with the data transfer service to provide security, access controls and management of any data transfers within the data grid.Raman, Vijayshankar; Narang, Inderpal; Crone, chris; Hass, Laura; Malaika, Susan. Services for data access and data processing on grids Security services provide mechanisms for authentication of users to ensure they are properly identified. Common forms of security for authentication can include the use of passwords or Kerberos (protocol). Authorization services are the mechanisms that control what the user is able to access after being identified through authentication. Common forms of authorization mechanisms can be as simple as file permissions. However, need for more stringent controlled access to data is done using Access Control Lists (ACLs), Role-Based Access Control (RBAC) and Tasked-Based Authorization Controls (TBAC).Thomas, R. K. and Sandhu R. S. Task-based authorization controls (tbac): a family of models for active and enterprise-oriented authorization management These types of controls can be used to provide granular access to files to include limits on access times, duration of access to granular controls that determine which files can be read or written to. The final data access service that might be present to protect the confidentiality of the data transport is encryption.Sreelatha, Malempati. Grid based approach for data confidentiality. p.1 The most common form of encryption for this task has been the use of SSL while in transport. While all of these access services operate within the data grid, access services within the various administrative domains that host the datasets will still stay in place to enforce access rules. The data grid access services must be in step with the administrative domains access services for this to work.
Data replication service To meet the needs for scalability, fast access and user collaboration, most data grids support replication of datasets to points within the distributed storage architecture.Chervenak, Ann; Schuler, Robert; Kesselman, Carl; Koranda, Scott; Moe, Brian. Wide area data replication for scientific collaborations The use of replicas allows multiple users faster access to datasets and the preservation of bandwidth since replicas can often be placed strategically close to or within sites where users need them. However, replication of datasets and creation of replicas is bound by the availability of storage within sites and bandwidth between sites. The replication and creation of replica datasets is controlled by a replica management system. The replica management system determines user needs for replicas based on input requests and creates them based on availability of storage and bandwidth.Lamehamedi, Houda; Szymanski, Boleslaw; Shentu, Zujun; Deelman, Ewa. Data replication strategies in grid environments All replicas are then cataloged or added to a directory based on the data grid as to their location for query by users. In order to perform the tasks undertaken by the replica management system, it needs to be able to manage the underlying storage infrastructure. The data management system will also ensure the timely updates of changes to replicas are propagated to all nodes.
Replication update strategy There are a number of ways the replication management system can handle the updates of replicas. The updates may be designed around a centralized model where a single master replica updates all others, or a decentralized model, where all peers update each other. The topology of node placement may also influence the updates of replicas. If a hierarchy topology is used then updates would flow in a tree like structure through specific paths. In a flat topology it is entirely a matter of the peer relationships between nodes as to how updates take place. In a hybrid topology consisting of both flat and hierarchy topologies updates may take place through specific paths and between peers.
Replication placement strategy There are a number of ways the replication management system can handle the creation and placement of replicas to best serve the user community. If the storage architecture supports replica placement with sufficient site storage, then it becomes a matter of the needs of the users who access the datasets and a strategy for placement of replicas. There have been numerous strategies proposed and tested on how to best manage replica placement of datasets within the data grid to meet user requirements. There is not one universal strategy that fits every requirement the best. It is a matter of the type of data grid and user community requirements for access that will determine the best strategy to use. Replicas can even be created where the files are encrypted for confidentiality that would be useful in a research project dealing with medical files.Kranthi, G. and Rekha, D. Shashi. Protected data objects replication in data grid p.40 The following section contains several strategies for replica placement.
Dynamic replication