SURAgrid Environment Variables
Version 1.0 - January 2006

Background

SURAgrid is a consortium of organizations collaborating and combining resources to help bring grid technology to the level of seamless, shared infrastructure. SURAgrid goals include:

• The development of scalable infrastructure that leverages local institutional identity and authorization while managing access to shared resources across institutional boundaries;
• Promoting the use of this infrastructure for the broad research and education community;
• Providing forum for participating institutions to gain additional experience with grid technology and to promote collaborative project development.

Participants contribute to each of these goals in alignment with institutional objectives, expertise and available resources. Working groups are established as needed to meet milestones that SURAgrid participants are most critical, valuable and timely. Working group members’ energy, expertise and enthusiasm contribute directly to the meeting of these milestones and, with that, the success of SURAgrid.

Working group members for the development of SURAgrid Environment Variables v1.0 include: Ashok Adiga, Texas Advanced Computing Center (TACC); Victor Bolet, Georgia State University; Shawn McKee, University of Michigan; Jerry Perez, Texas Tech University; John-Paul Robinson, University of Alabama at Birmingham; Warren Smith, Texas Advanced Computing Center (TACC), UT Austin; Judith Utley, (formerly) Old Dominion University; Mary Fran Yafchak, Southeastern Universities Research Association (SURA)

Introduction

One of the problems to be overcome in grid computing is that every computer system is different. But, for applications to be executed on many different systems, the applications must be able to run on these systems with no user intervention. This means that the differences between computer systems must be hidden or that information must be provided to the application so that it can adapt to the system it is executing on.

The SURAgrid is beginning to address this problem with this document that describes the environment variables that should be set on every computer system on the SURAgrid. In exploring this topic, the working group considered approaches in use by similar large-scale grid initiatives, notably the TeraGrid and Open Science Grid, as well lessons learned from grid development and deployment on their own campuses. After a number of discussions, we have selected an initial, minimal set of environment variables that should be set on systems that are part of SURAgrid. The availability of these environment variables begins to allow SURAgrid applications to adapt to the system on which they are running.

Scope

There are a variety of computer systems that make up the SURAgrid. These include individual compute and data servers, loosely coupled clusters, tightly coupled clusters, and so on. Many clusters also have a system where users log in to submit jobs to the compute nodes, compile codes, run small applications, and so on.

SURAgrid users can run applications on any of these systems. Therefore, the environment variables that we describe in this document must be set on all of them. Further, these variables are only useful if they are set by default in the user’s environment with no action on the user’s part: The first time a user logs in to a system or runs an application on a system, these environment variables are set.

Environment Variables

SURAgrid currently consists of a set of Linux systems. We recommend that the environments on these systems be configured in a typical way so that environment variables such as HOME, PATH, LIBPATH, MANPATH, and so on are set appropriately.

Since the SURAgrid requires that Globus be installed on each system, we require that the GLOBUS_LOCATION environment variable be set for all users.

Note: We do not require that the PATH, LIBPATH, etc. include Globus programs and libraries. We do not require this because the user can source globus_user_env.sh or globus_user.csh in $GLOBUS_LOCATION/etc/.

All of the SURAgrid environment variables will use the SURAGRID_ prefix. We recommend that SURAgrid systems be configured so that all of these environment variables are set. We require that if an environment variable is not applicable to the system, it is not set. This implies that the user/application should test for the existence of SURAgrid environment variables before using them.

We define the following environment variables specific to SURAgrid:

SURAGRID_SCRATCH

This environment variable contains the location of temporary scratch disk space that is local to the system and is available to the user. The user should have permission to write to $SURAGRID_SCRATCH and that directory should be private to the user. We require that a user be able to deny others the ability to read or modify the files in this directory. We recommend that by default, only the user can read or modify files in this directory. An example of a value for SURAGRID_SCRATCH is /scratch/johndoe. The purpose of this disk space is to store temporary files while applications are executing. For example, partial results, checkpoints, and so on.

Note: This scratch space must not be mounted as scratch space on other systems. If it was, the processes of a parallel application that write to $SURAGRID_SCRATCH could access the same file and overwrite each other.

SURAGRID_SHARED_SCRATCH

This environment variable contains the location of temporary scratch disk space that is mounted across the entire cluster (including login node) and is available to the user. The user should have permission to write to $SURAGRID_SHARED_SCRATCH and that directory should be private to the user. An example of a value for SURAGRID_SHARED_SCRATCH is /work/johndoe. We require that a user be able to deny others the ability to read or modify the files in this directory. We recommend that by default, only the user can read or modify files in this directory. The purpose of this disk space is to store files that are needed by all of the processes of a parallel application during an execution of that application. For example, executable programs, input data, input configuration, output data, and so on.

SURAGRID_SHARED_PROJECTS

This environment variable contains the location of disk space that is mounted across the entire cluster (including the login node) and is available to different projects. A project is a group of users that are working on a common set of applications and data. We recommend that users not be able to write to $SURAGRID_SHARED_PROJECTS. Each project should have a subdirectory in this directory that they can access as $SURAGRID_SHARED_PROJECTS/<project>. An example project directory is /projects/compbio where SURAGRID_SHARED_PROJECTS is /projects and the project is compbio. We recommend that the site create these subdirectories rather than letting users create them. We require that a project be able to deny non-project members the ability to read or modify their project files. We recommend that by default, only project members can read and modify the project directory. The purpose of this disk space is to store files that are needed by the members of the project. For example, executable programs, project-specific configuration files, common data sets, interesting output data, and so on.

Note: Each project within SURAgrid should have a unique name. A formal processes for assigning these names and for creating project directories has not been determined. At the current time, sending an email to the suragrid@sura.org list providing your project name and asking that a project directory be created is probably sufficient.

Future Considerations

This set of environment variable recommendations and requirements will be incorporated into the current process of resource and application deployment on SURAgrid. However, it is understood that grid technology and grid applications are still evolving, as is SURAgrid, and the specification and management of the grid environment will need to evolve accordingly. The version 1.0 specification provides a starting point for resource and application providers to begin exercising the capabilities of SURAgrid and, from that, learn more about what is needed in the future. As the need for additional or modified requirements and recommendations becomes clear, the specification will be revisited to develop the next version.

The current working group is considering the following areas be addressed in version 2.0, in addition to others that might become evident over time. If you have environment configuration issues that you would like to see considered in future versions, please send email to suragrid@sura.org (if a member of the SURAgrid community) or to Mary Fran Yafchak, SURA IT Program Coordinator and SURAgrid Project Manager, maryfran@sura.org.

For consideration in version 2:

• Appropriate meta-data about the resources at each site, which can be used to help schedule and allocate resources.
• An expanded list of environment variables, many of which may be optional, that can aid applications and users in optimizing their job submissions.
• Recommendations about file systems and data storage, accessibility, and management.
• How and when to modify environment variables as new applications are added to SURAgrid, including whether it is expected that applications will adapt to current requirements, or if mechanisms/processes can be developed such that applications influence or create new environment variables as needed.