Twenty-Plus Years of Netlib and NA-Net, Part I
April 12, 2006
Figure 1. Netlib usage at the University of Tennessee, Knoxville, and Oak Ridge National Laboratory. The labels on the vertical axis correspond to different mechanisms for downloading.
Jack Dongarra, Gene Golub, Eric Grosse, Cleve Moler, and Keith Moore
In 1984, Eric Grosse of Bell Labs in Murray Hill, New Jersey, and Jack Dongarra, then of Argonne National Laboratory, established the original Netlib software repositories. Each repository made a collection of high-quality mathematical software available via electronic mail---the Bell Labs server provided access via UUCP protocols, and the Argonne server via Internet mail. The Netlib collection quickly spread to other servers around the world. As Internet protocol networks became ubiquitous, other access methods were added; today, though the electronic-mail interface is still supported, most access is via the World Wide Web. One of the first "open source" efforts, Netlib helped establish the trend of software distribution via the network, and it remains popular today.
In this article, the first of two parts, some of the people responsible for Netlib over the years trace the history of the repositories. In the second part, which is scheduled for the next issue of SIAM News, the authors tell the related story of the development of NA-Net and, coincident with its recent change in editorship, the NA Digest. Pointing to the importance of these largely volunteer efforts to progress in computational science, the authors conclude with a discussion of concerns about ongoing support for such activities.
The Revolutionary Side of Netlib
Netlib was created to facilitate quick, easy, and efficient on-demand access to public-domain computational software of interest to the scientific computing community. Distribution of this software was by electronic mail. To request files from either the Bell Labs or the Argonne server, a user would send one or more commands as message text to the server's e-mail address. For example, a user who sent the message "send dqag from quadpack"---either to research!Netlib (UUCP) or to netlib@anl-mcs (Internet)---would receive in reply the "dqag" subroutine from the "quadpack" package, along with any additional routines needed for use of that subroutine.
In today's world of pervasive high-speed Internet access, it can be difficult to appreciate the environment in which Netlib was hatched and, in fact, the revolutionary nature of Netlib. The ARPAnet had been in existence for several years, and software was commonly shared via the ARPAnet's FTP protocol. The ARPAnet served a limited community, however, and the Internet protocols adopted by the ARPAnet in 1981 were only beginning to see widespread utside use by the mid-1980s. Software could also be uploaded to or downloaded from "bulletin board systems," or BBSs, but users could incur significant long-distance telephone charges to peruse distant servers.
When Netlib was introduced, distribution of public-domain software often took the form of transmission of physical media, such as magnetic tapes, through the postal service. This was difficult not only because of the time and expense of handling the physical media, but also because of the lack of any widely used standards for writing magnetic tapes. Each computing platform wrote tapes in its native format; because of differences in word size, record size, and organization of the data on tape, reading foreign tapes could be quite difficult and labor intensive. ARPAnet e-mail systems, by necessity, standardized very early on a common message format to be used across all platforms; UUCP mail used a similar format. The standardization made electronic mail a superior medium for the exchange of source code, at least for small files, and Netlib made effective use of it.
Netlib appeared at a time when e-mail was beginning to be widely available within the user community. By opting for e-mail distribution, Netlib allowed users to acquire software routines without the lengthy delays and in some cases costly procedures encountered with physical media. Because the service was free of charge and because requests could be submitted directly to the repositories, users were able to acquire software without the administrative approval that might have been required for formal requests to the software providers or for the costs of writing and shipping magnetic tapes.
Collection Criteria and Contents
Netlib limited its scope to mathematical software and related information of interest for scientific computing. An effort was also made to limit the collection to software of demonstrably high quality. The fact that packages like LINPACK and EISPACK were available for Netlib's initial collection helped set the standard for future additions. More generally, the numerical analysis community's tradition of producing robust software to address clearly defined problems made it somewhat easier to establish a high-quality library for scientific computing than for some other areas. By having all contributions reviewed by an editorial board, Netlib also provided some assurance of the quality and stability of the software, along with some indication that the software was up to date.
In addition to LINPACK and EISPACK, the initial collection contained MINPACK, FNLIB, routines from the book Computer Methods for Mathematical Computations by Forsythe, Malcolm, and Moler, and QUADPACK, along with a collection of "golden oldies." Additional codes were soon added--from ACM's Transactions on Mathematical Software (TOMS), as well as some benchmarking codes, a biharmonic solver, a multiprecision arithmetic package, and the Basic Linear Algebra Subprograms, among others.
Server Locations and Mirroring
After going online on January 3, 1984, the Bell Labs Netlib server handled software distribution within Bell Labs during the spring. Both the Bell Labs and the Argonne servers were in public use by July of that year.
The Argonne server was physically moved to Oak Ridge National Laboratory in October 1989, when Jack Dongarra moved to Tennessee. With the growing popularity of the Netlib collection, it became advantageous to mirror the collection outside North America; to minimize the differences between servers, however, the number of official mirrors was deliberately kept small. A European mirror was established in Oslo in December 1989, and an Australian mirror was in place by 1989. In May 1993 the original Sequent server that had been in service at Argonne was replaced by two SparcStation 2 machines---one at ORNL and the other at the University of Tennessee in Knoxville. These servers were closely synchronized, and both were configured to accept e-mail requests sent to [email protected] domain. The ORNL server was retired in 1997.
The official servers today are those at netlib.bell-labs.com in Murray Hill, netlib.org at the University of Tennessee, the UK Mirror Service in Kent, and netlib.no in Bergen, Norway. These servers are synchronized to one another via a process described in [2].
Many unofficial mirrors have been created since the introduction of anonymous FTP access to the primary Netlib servers; the mirrors vary greatly with regard to the organization and currency of their collections.
Among additional sites that mirror Netlib are:
Netlib.uow.edu.au, Australia [ftp]
AARNet (Australia/New Zealand only) [ftp]
unicamp.br, Brasil
Netlib.amss.ac.cn, China [ftp]
Daresbury Lab, England
Codiciel, France
ENSEEIHT, France
ZIB, Germany
Athens, Greece
Univ Thrace, Greece
CASPUR, Italy
C.D.S., Italy
CILEA, Italy
Phase AIST, Japan
Ewha w. Univ., Korea
ChgNet, Russia
NCHC, Taiwan
freesoftware.com, USA west
These sites are not checked as closely as the official mirrors.
Access Protocols
Netlib was created in a world that had no ubiquitous Internet, but rather an admixture of incompatible networks, each with different hosts, services, and means of addressing. In many cases the networks were interconnected by mail gateways, which could be both difficult to use (because messages had to be explicitly routed through the gateways) and unreliable. Netlib initially used text-based e-mail, the only service common to all of those networks, to provide access to items in its collection; with wider deployment of Internet protocols, other, easier-to-use methods became more feasible.
Servers for the Internet's File Transfer Protocol (FTP) were provided at Bell Labs in August 1991, and soon afterward at ORNL. The Xnetlib browser, first deployed in December 1991, allowed users to peruse and download items from the ORNL server via a graphical user interface for the X Window System [1]. In response to the growing popularity of the Web, HTTP Netlib servers were established in October 1993 at UTK and ORNL; Gopher was enabled in November 1993. Although the e-mail interface is still supported, HTTP, and to a lesser extent FTP, quickly eclipsed all other sources of Netlib traffic.
Usage Patterns
At least for the UTK and ORNL servers, use of Netlib increased steadily during the period from 1985 to 2000. In recent years, traffic levels have varied (see Figure 1). Some of the recent variation may be due to increased use of crawlers and mirrors. Approximately 80% of the current traffic appears to be from individual users, with 7% from mirrors and 12% from Web crawlers, which are presumably building indices for use by search engines.
Impact on the Community
Although large amounts of mathematical software are commercially or freely available, not all of it is of high quality. It can also be difficult to locate appropriate software with Web search engines---keywords available for searching may be lacking or may not match the vocabulary used by searchers.
Netlib (http://www.Netlib.org/), the numerical analysis community's solution to these problems, is a moderated collection of high-quality software maintained by experts in the field, and organized and catalogued with appropriate metadata to enable easy searching. For the past twenty-some years, numerical analysts have contributed to and managed the collection, which continues to evolve, with new contributions of state-of-the-art software by leading researchers in scientific computing.
Netlib has been instrumental in promoting software reuse within the computational science and mathematics communities. Netlib pioneered the idea of network-delivered software and has remained at the forefront of delivery technologies. The evolution of Netlib tracked the emergence of every important Internet application protocol, from e-mail through FTP and HTTP. Additionally, Netlib showed that it is possible to manage such collections in a distributed manner, in terms of both infrastructure and administration. The Netlib collection itself has fallen behind the times, although much of it remains state-of-the-art. The fact that it still exists at all testifies to its good foundations, given the limited ongoing maintenance.
Acknowledgments
The authors thank Don Fike for assistance with Netlib statistics, Mark Crispin and Mark Kent for information about early NA-Net implementation, and Mel Ciment, who was at the National Science Foundation and provided seed funding for Netlib and NA-Net when they first started.
References
[1] J. Dongarra, T. Rowan, and R. Wade, Software distribution using XNETLIB, Trans. Math. Software, 21:1 (1995).
[2] E. Grosse, Repository mirroring, Trans. Math. Software, 21:1 (1995).
Jack Dongarra is a University Distinguished Professor in the Department of Computer Science and director of the Innovative Computing Laboratory and the Center for Information Technology Research at the University of Tennessee, in Knoxville; he is also a distinguished member of the research staff and Mathematics Division at Oak Ridge National Laboratory. Gene Golub is the Fletcher Jones Professor of Computer Science at Stanford University. Eric Grosse is a member of the research staff at Bell Laboratories, Lucent Technologies. Cleve Moler, SIAM's president-elect, is the chief scientist and founder of The MathWorks. Keith Moore is a member of the research staff at the Innovative Computer Laboratory at UTK.
