Data: How to Initiate Transfers: GridFTP Clients

GridFTP is a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth, wide-area networks. TeraGrid has three clients which utilize gridFTP (click on the client name to see more information and examples of that method):

• tgcp
• globus-url-copy
• uberftp

http://www.teragrid.org/userinfo/data/gridftp.php

http://www.teragrid.org/userinfo/data/

The evolution of storage systems

by R. J. T. Morris and B. J. Truskowski
	Storage systems are built by taking the basic capability of a storage device, such as the hard disk drive, and adding layers of hardware and software to obtain a highly reliable, high-performance, and easily managed system. We explain in this paper how storage systems have evolved over five decades to meet changing customer needs. First, we briefly trace the development of the control unit, RAID (redundant array of independent disks) technologies, copy services, and basic storage management technologies. Then, we describe how the emergence of low-cost local area data networking has allowed the development of network-attached storage (NAS) and storage area network (SAN) technologies, and we explain how block virtualization and SAN file systems are necessary to fully reap the benefits of these technologies. We also discuss how the recent trend in storage systems toward managing complexity, ease-of-use, and lowering the total cost of ownership has led to the development of autonomic storage. We conclude with our assessment of the current state-of-the-art by presenting a set of challenges driving research and development efforts in storage systems.

http://www.research.ibm.com/journal/sj/422/morris.html

Google Summer of Code

Google Summer of Code 2008 is on! Over the past three years, the program has brought together over 1500 students and 2000 mentors from 90 countries worldwide, all for the love of code. We look forward to welcoming more new contributors and projects this year.


** Globus: Google Summer of Code 2008 Ideas

New execution and data transfer providers

Globus project: Swift

Description: The Java CoG kit provides an abstraction for process execution and file transfer (for example, local execution, local filesystem copy, over ssh/scp, GridFTP, GRAM2, GRAM4, direct submission to the PBS scheduling system). Execution and transfer providers can then be used by higher level applications such as Swift in order to move data to execution sites and to perform application execution without needing to be particularly aware of how that execution and transfer is happening. An interesting project might be to implement a provider for some existing execution or transfer mechanisms so that they could be used as part of CoG.

Requires: Decent Java programming skills. A favourite execution or data transfer mechanism

Mentor: Ben Clifford

Integration of GridFTP with Freeloader storage system

Globus project: GridFTP

Description: GridFTP is a high-performance, secure, reliable data transfer protocol optimized for high-bandwidth wide-area networks. It is based on the Internet FTP protocol, and it defines extensions for high performance operation and security. Striped data transfer (aka cluster-to-cluster data transfer) is a key feature that utilizes multiple CPUs and NICs to achieve higher performance. In striped mode, however, GridFTP assumes the support of a high-performance parallel file system, a relatively expensive resource. Freeloader is a storage system that aggregates the idle storage space from workstations connected to a local area network to build a high-performance data store. FreeLoader breaks files into chunks and distributes these chunks across the storage nodes. This accelerates read/write operations as they can benefit from the parallel access to multiple disks. This project aims to integrate GridFTP and FreeLoader to reduce the cost and increase the performance of GridFTP deployments.

Links related to this project

• GridFTP
• Freeloader

** Project Ideas from the Ohio Supercomputer Center

1. Improved scalability in pbsdcp scatter implementation
   Mentor: Troy Baer
   Programming Language(s): Perl, C with MPI
   License: GPL

   pbsdcp is a distributed copy command for PBS and TORQUE batch environments that is part of OSC's pbstools package. It is used to copy files between shared directories (e.g. NFS home directories) and local storage on a set of compute nodes (e.g. /tmp). It has two modes of operation: scatter, in which files in a shared directory are copied into local file systems on each of the compute nodes; and gather, in which files in local file systems on each of compute nodes are collected into a shared directory

   The scatter mode in pbsdcp is currently implemented in a rather naive fashion: for each node, it forks an rcp on the source files with a destination directory on that node's local storage. This means that the amount of data which must be transferred from the shared storage scales linearly with the number of nodes. We would like to replace that implementation with something more scalable, such as a tree-based or store-and-forward distribution scheme. Moreover, we would like this to use MPI for communication between nodes if possible, so that the high-performance Infiniband and Myrinet networks in our (and similar) clusters will be used for as much of the data transfer as possible.

   1. Improved scalability in all
      Mentor: Rick Mohr
      Programming Langauge(s): C
      License: GPL

      all is a distributed shell command built on top of rsh used by OSC and other sites. It allows commands to be run on either all or an arbitrary subset of the nodes in a cluster, either sequentially or in parallel.

      The parallel mode of all currently has a scalability problem on clusters larger than about 200 nodes. Because all uses rsh and rsh wants to use privileged ports (i.e. port numbers below 1024), parallel executions of all run out of the necessary ports for node counts above 200 or so. One solution to this problem would be "chunking" or "batching"; that is, starting up at most a fixed number (say 128) of rsh connections and then only starting more once the first few rshes have completed. (Similar logic can be seen in OSC's parallel command processor.)

      Alternately, a project to add some of all's features, such as its relatively simple syntax and PBS/TORQUE integration, to another distributed shell command such as pdsh would also be considered.

** dev:sahana_gsoc08_ideas

SystemTap

SystemTap provides free software (GPL) infrastructure to simplify the gathering of information about the running Linux system. This assists diagnosis of a performance or functional problem. SystemTap eliminates the need for the developer to go through the tedious and disruptive instrument, recompile, install, and reboot sequence that may be otherwise required to collect data.

SystemTap provides a simple command line interface and scripting language for writing instrumentation for a live running kernel. We are publishing samples, as well as enlarging the internal "tapset" script library to aid reuse and abstraction. We also plan to support probing userspace applications. We are investigating interfacing Systemtap with similar tools such as Frysk, Oprofile and LTT.

Current project members include Red Hat, IBM, Intel, and Hitachi.

Scripts & Tools
http://sourceware.org/systemtap/wiki/ScriptsTools

DTrace Network Providers

from : http://opensolaris.org/os/community/dtrace/NetworkProvider/

The following is a design proposal for a collection of DTrace Networking Providers. These providers aim to provide networking observability and troubleshooting information for Solaris users. The first prototype TCP provider was demonstrated at CEC 2006.

#dtrace -n 'tcp:::receive /args[2]->tcp_dport == 80/ {
       @pkts[args[1]->ip_daddr] = count();
}'
dtrace: description 'tcp:::receive' matched 1 probe
^C

 192.168.1.8                                                       9
 fe80::214:4fff:fe3b:76c8                                         12
 192.168.1.51                                                     32
 10.1.70.16                                                       83
 192.168.7.3                                                     121
 192.168.101.101                           

more DTrace

Brendan Gregg's Homepage

http://brendangregg.com/

Top Ten DTrace (D) Scripts

DTrace is a comprehensive and flexible dynamic tracing facility built into the Solaris Operating System. DTrace allows dynamic instrumentation of a running Solaris system, which can assist with answering questions like "which process is chewing up CPU 38," or "which user is causing the cross-call activity on CPU 6," or "which setuid binaries are being executed?"

DTrace uses a scripting language called "D," which uses a syntax very similar to C and Awk. Several amazing D scripts have been developed and distributed through the Internet, so I thought I would share my favorite D scripts in a Letterman like "Top 10" format:

http://prefetch.net/articles/solaris.dtracetopten.html

Observing I/O Behavior with the DTraceToolkit

http://prefetch.net/articles/observeiodtk.html


DTrace user's guide http://docs.sun.com/app/docs/doc/817-6223/

DTrace Toolkit http://www.opensolaris.org/os/community/dtrace/dtracetoolkit/


DTrace presentation [FIRST look at this]

http://www.nbl.fi/~nbl97/solaris/dtrace/dtt_present.pdf

OpenSolaris Community: DTrace

OpenSolaris Community: DTrace

http://www.opensolaris.org/os/community/dtrace/

Endorsed projects

Chime Visualization Tool for DTrace
DTrace Provider for NFSv4
Mozilla DTrace

An Overview of DTrace

DTrace is a comprehensive dynamic tracing framework for the Solaris™ Operating Environment. DTrace provides a powerful infrastructure to permit administrators, developers, and service personnel to concisely answer arbitrary questions about the behavior of the operating system and user programs.

The Solaris™ Dynamic Tracing Guide describes how to use DTrace to observe, debug and tune system behavior. The Solaris™ Dynamic Tracing (DTrace) Guide (here), also includes a complete reference for bundled DTrace observability tools and the D programming language.

For Users:

• dynamically enable and manage thousands of probes
• dynamically associate predicates and actions with probes
• dynamically manage trace buffers and probe overhead
• examine trace data from a live system or from a system crash dump

For Solaris™ Developers:

• implement new trace data providers that plug into DTrace
• implement trace data consumers that provide data display
• implement tools that configure DTrace probes

No Bad Dogs - DTrace - Bryan Cantrill

No Bad Dogs How to Make a Dog-Slow System Sit Up and Speak

http://research.sun.com/minds

--from WIKI

Bryan M. Cantrill is an engineer at Sun Microsystems. Cantrill graduated from Brown University, B.Sc. in computer science. He was born in Colorado where he attained the rank of Eagle Scout.

In 2005 Bryan Cantrill was named one of the 35 Top Young Innovators by Technology Review, MIT's magazine. Cantrill was included in the TR35 list for his development of DTrace, a function of the OS Solaris 10 that provides a non-invasive means for real-time tracing and diagnosis of software.

DTrace:

from [http://en.wikipedia.org/wiki/DTrace]

DTrace is a comprehensive dynamic tracing framework created by Sun Microsystems. It was released under the Common Development and Distribution License (CDDL) in January 2005 and included in Sun's Solaris 10 for troubleshooting system problems in real time. DTrace was the first component of the OpenSolaris project to have its source code released under the CDDL.

DTrace is designed to give operational insights that allow users to tune and troubleshoot applications and the OS itself. Special consideration has been taken to make it safe to use in a production environment. For example, there is minimal probe effect when tracing is underway, and no performance impact associated with any disabled probe; this is important since there are tens of thousands of DTrace probes that can be enabled.

Tracing programs (also referred to as scripts) can be written using the D programming language (not to be confused with other programming languages named "D"). The language is a subset of C with added functions and variables specific to tracing. D programs most resemble awk programs in structure; they consist of a set of actions rather than a top-down structured program. In a DTrace program, one or more probes (instrumentation points) are enabled; whenever the condition for the probe is met, the action associated with the probe in the DTrace program is executed (the probe "fires").

DTrace was designed and implemented by Bryan Cantrill, Mike Shapiro, and Adam Leventhal. The authors received recognition in 2005 for the innovations in DTrace from InfoWorld and Technology Review.^[1][2] DTrace won the top prize in the Wall Street Journal's 2006 Technology Innovation Awards competition.^[3]

DTrace implementations require tight integration with the operating system kernel. Although DTrace was initially written for Solaris, its source code is freely available as part of the OpenSolaris project, and work is in progress to port it to FreeBSD (in which there has been initial success^[4] as a substitute for the ktrace utility; further work on DTrace integration into FreeBSD is being sponsored by Cisco) and QNX ^[5]. Apple has included DTrace in Mac OS X 10.5 "Leopard" with a GUI called Instruments;^[6] Unlike other platforms that DTrace is supported on, Mac OS X has a flag (P_LNOATTACH) that a program may set that disallows tracing of that process by debugging utilities such as DTrace and gdb. This can affect tracing of other system information, as unrelated probes that should fire while a program with this flag set is running will fail to do so.^[7]

Using DTrace

Typical usage of DTrace consists of invoking it from the command line, providing one or more arguments. Here a list of command line with examples arguments.

# New processes with arguments,
dtrace -n 'proc:::exec-success { trace(curpsinfo->pr_psargs); }'

# Files opened by process,
dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'

# Syscall count by program,
dtrace -n 'syscall:::entry { @num[execname] = count(); }'

# Syscall count by syscall,
dtrace -n 'syscall:::entry { @num[probefunc] = count(); }'

# Syscall count by process,
dtrace -n 'syscall:::entry { @num[pid,execname] = count(); }'

# Disk size by process,
dtrace -n 'io:::start { printf("%d %s %d",pid,execname,args[0]->b_bcount); }'  

# Pages paged in by process,
dtrace -n 'vminfo:::pgpgin { @pg[execname] = sum(arg0); }'

Bioinformatic in Purdue

from [Kihara Bioinformatics Laboratory News]

• 3D-Surfer, a server for rapid protein 3D structure search is released. For a user-uploaded, or selected PDB file, the server searches protein 3D structures with a similar surface shape in the entire PDB database in less than a minute. The conventional RMSD can be also computed to retrieved structures. Characteristic local surfaces, i.e. pockets and protrusions, are identified by our VisGrid algorithm.
• "Combining gene sequence similarity and textual information for gene function annotation" by Luo Si, D. Yu, D. Kihara, & F. Yi, accepted for Information Retrieval.

3D-SURFER is web-based software for protein surface comparison and analysis. The server integrates various repertoire of methods to assist in high throughput screening and visualization of protein surface comparisons. It takes less than a second to perform an exhaustive comparison between a single protein surface to all protein structures in the current PDB (80,394 single chain structures of 34,088 total PDB entries). Conveniently, the web interface also renders animated protein rotations, displays CATH codes [Orengo CA, Structure, 1997], and structure alignment calculations using the Combinatorial Extension (CE) algorithm [Shindyalov IN and Bourne PE, Protein Eng, 1998]