ACADEMY OF SCIENCES OF THE USSR LENINGRAD NUCLEAR PHYSICS INSTITUTE ________________ A.E.Shevel THE ORGANIZATION OF THE COMMUNAL COMPUTATION AT LENINGRAD NUCLEAR PHYSICS INSTITUTE IN 1991 AND IN THE NEAREST FUTURE Leningrad 1991 “„Š 681.142.4 N 1752 Abstract This paper describes the current structure of the computer centre and the future changes in the communal computation technics at LNPI. The future direction of the centre life is discussed in this paper. ----------------------------- Leningrad Nuclear Physics Institute (LNPI) consists of four main parts: neutron physics part concentrated around the nuclear reactor; high energy physics (HEP), centered on the synchrocyclotron, a biophysics department and theoretical division. Shared computation resources are mainly housed in the institute's computer centre. The computer centre has the following computer facility: - EC-1046 HEP computer cluster (see fig. 1); - CM-1700 nuclear reactor computer cluster (see fig. 2); - CM-IZOT computer cluster (see fig. 3); - a number of IBM PC compatible computers. About 70 persons work at LNPI computer centre performing - 2 - +------------------------------------------------------------+ | Installed in Installed in | | December of 1988 July of 1989 | | +----------------+ +----------------+ | | | Tape drives | | Tape drives | | | | 800/1600/6250 | | 800/1600/6250 | | | | 9 tracks | | 9 tracks | | | | 8 units | | 8 units | | | +-------+--------+ +-------+--------+ | | | +-------+ | | | +-------+--------+ | Disks | +-------+--------+ | | | EC-1046-1 +-----+ 4 GB +------+ EC-1046-2 | | | | 6 I/O channels | +-------+ | 6 I/O channels | | | | 8 MB core | | 8 MB core | | | | 1.5 MIPS | | 1.5 MIPS | | | +-------+--------+ +-------+--------+ | | | | | | +-------+---------+ +-------+---------+ | | | Controller | | Controller | | | | to link | | to link | | | | an experimental | | an experimental | | | | mini-computers | | mini-computers | | | | (up to 16) | +-----------------+ | | +-----------------+ | | About 70 terminals are attached to EC-1046-1 and | | EC-1046-2 through various control units. | | The cluster has about 150 registered users. | | This computer cluster service is available round the | | clock seven days a week throughout the year. | +------------------------------------------------------------+ Fig. 1. The schematic diagram of the EC-1046 computer cluster. the computer operations, developing and investigating the new computer components: - 3 - - 10 computer operators; - 8 specialists on mechanical repairing; - 18 hardware engineers; - 20 software specialists; - 10 technicians. The computer centre staff is devided on a number of teams: - to maintain the reactor computer cluster; +---------------------------------------------------------+ | Installed in 1988 Installed in 1989 | | +--------------+ +--------------+ | | | CM-1700-1 | | CM-1700-2 | | | | 2 MB core | | 4 MB core | | | | 100 MB disks | | 100 MB disks | | | | tape drives | +--------------+ | | | 800/1600 bpi | | | +--------------+ | | About 8 terminal lines are used now. | | About 20 users are registered. | | The structure of the CM-1700 is like the VAX 11/730. | | This computer cluster service is also available 24 | | hours a day in periods when the nuclear reactor is | | in operation. | +---------------------------------------------------------+ Fig. 2. The scheme of reactor computer cluster. - to maintain EC-1046 computer cluster; - to maintain the CM-IZOT cluster; - to maintain PCs; - 4 - - to maintain terminal network; - to develop and investigate the new program systems and new computer structures. It is easy to see that the last team plays a key role in planning of the computer centre future. It is worth noting that the above teams are housed in various institute buildings. This structure is not necessarily ideal, but is rather thanks to the local circumstances. +---------------------------------------------------------+ | Installed in 1989 Installed in 1990 | | +--------------+ +--------------+ | | | IZOT-1080 | | CM-1700 | | | | 5 MB core | | 2 MB core | | | | 400 MB disks | | 60 MB disks | | | | tape drives | +--------------+ | | | 800/1600 bpi | | | +--------------+ | | About 5 terminal lines are used now. | | About 20 users are registered. | | The structure of the CM-1700 is like the VAX 11/730. | | The structure of the IZOT is the same as CM-1700. | | This computer cluster service is available around | | the clock five days per week. | +---------------------------------------------------------+ Fig. 3. The scheme of the IZOT-CM computer cluster. As the HEP computer cluster is not only the most powerful one but also provides computer resources to the bulk of the institute users (about 80%), the remainder of this paper discusses only this cluster. In addition, the computational - 5 - problems are significantly simpler then those solved anywhere at LNPI. The HEP computer cluster (see fig. 1) is used mainly for the off-line computing (about 95%) but also provides an on-line support for the experiments on the synchrocyclotron (5%). The hardware In addition to the information that is presented in figure 1, the following points are of interest. The computer EC-1046 has the structure of an IBM/370. About a dozen terminals are available in the three public terminal rooms. The remainder is installed in the laboratory rooms. Many of the terminals have the dot serial printers attached to enable the printing of listings and screen dumps. The software There is no doubt that software is a significant component of computer systems. The HEP computer cluster runs under VM/CMS. The OS/SVS is used as a guest operating system. Most of on-line applications run under OS/SVS. The follwing key software packages are used on the HEP cluster EC-1046. These packages provide the base for the main computer utilisation: - CERN program library; - on-line package; - 6 - - batch facility package; - copy/restore facility package; - archive package. CERN program library is the commonly known application package. We try to support the reasonable subset of the total CERN program library to meet an experimentalist's needs. The on-line package supports mainly the physicists using the LNPI HEP experimental environment. The package permits to acquire of the data from experiment installations and provides the following facilities: the start of the data acquisition; the stop of the data acquisition; the play back from tape; the call of an user supplied analyse and/or check program and so on. The on-line package was developed by the system investigation team [1] and comprises some 15000 lines in assembler and PL/I. The batch package provides batch facilities for the users of EC-1046 in VM/CMS operating system. The main facilities of this package are similar to (but not the same) the SLAC BATCH system. Our batch package was developed and supplied to the computer centre by the colleagues from ITEP (Moscow). The copy/restore package is used to maintain the system and user data integrity. This package was written by the system team mainly in REXX and assembler [2] and contains some 6000 lines of code. The archive package [3] is useful facility for the users and professional programmers to maintain the private archives of text files. The package is avalable under VM/CMS and OS/SVS. The package offers the following: - 7 - - to copy any file(s) to tape; - to have many copies of the file with the same name on the one tape; - to restore any file(s) from tape with explicit version number, date of copying and others parameters. Every file on the archive tape is written as a standard OS/SVS file with the standard tape labels. The last feature makes the files to be written on an archive tape being available on another computer installation which has no such archive package. The package was developed by system team. The package consists of ten thousands lines in REXX, assembler and OS/SVS Job Control Language. As it is pointed out elsewhere, the physicists prefer to use the professional computation workstations based on various microprocessors [4,5]. The same trend is seen at LNPI. A number of the physical laboratories of our institute try to organize their own microcomputer facility based on IBM PC compatible computers (intel 80286, 80386). LNPI currently has about 150 such computers. Most of those computers run under MS DOS. In an attempt to overcome the contradiction between old fasion computer clusters and PCs we have established the special link between EC-1046 cluster and PCs. That special link permits the attachment a personal computer to EC-1046 through the EC-7970 control unit. The owner of PC can gain the following service: - PC can be used as ordinary terminal connected to terminal station EC-7970; - 8 - - PC can be used as a normal PC with the additional possibility of declaring and using a virtual floppy disk on the EC-1046 disks; - the PC can be used as autonomous computer with possibility to transfer any file(s) between itself and EC-1046 with a transfer speed of about 4 KB/sec. The above facilities can be performed by the following steps: - the addition of an adaptor link module in PC; - the loading of a special system program in MS DOS to emulate the needed functions. EC-1046 has the appropriate system programs to maintain such mode of work. The problems The main problems of the HEP cluster are listed below. Firstly, there is inadequate computer power and disk space. Secondly, the computer installation suffers from reliability problems. For example, the operating system is rebooted about once a day. As a result the computer centre requires exessive number of staff, that results in large growth in the cost of the operation of the computer. Now there are some technical limitation thanks to computer structure of EC-1046. For instance, the addressing space is limited by value of 16 MB per virtual machine. Of course we need the network service. Now LNPI has the - 9 - email service with the limited access which depends on the poor networking. Finally, the computer centre has financial problems. The last problems appeared due to the desire of physicists to have a personal computer instead of communal computation facility. The above problems urge us to plan new computer facilities and new approaches. I suppose the EC-1046 computer cluster will be in operation not more than 2-3 years. The new directions This autumn (1991) LNPI computer centre have introduced into operation the first at LNPI laboratory Unix station based on IBM PC AT with the following parameters: - processor intel 80386/387 clock 20/25 MHz - RAM 8 MB - HD 340 MB - FD 1.2 MB (5.25") - streamer 60 MB - additional terminal lines 4*RS-232. It is supposed that from 2 to 10 users can be registered on this Unix station. This station will be dedicated for the small Monte Carlo tasks, modelling of the small experiments and the text preparing. At last we try to transfer the CERN program library in PC/AT under Unix. The question arises - "Why IBM PC AT?" At this point we have to take into account the present computer situation. IBM PC compatible computers are in the wide use at USSR and at LNPI. It is very simple to buy the computer and the additional - 10 - accessorities and to find the repairing people and so on. On the other hand we feel the shortcomings of that decision: - insufficient computer power; - incompatibility with our colleagues at CERN for example. Where are we going from here? It is not difficult to understand the physicists's desire to have the personal computer workstation with unlimited power. However we need to emphasize the peculairities of the computer station power growth especially in the HEP area. Firstly, every physicist uses large common program libraries. The total volume of common libraries and data may be estimated by the value of the hundreds megabytes or one gigabyte. Secondly, as a rule the computer activity in HEP has a collective character. Many people should work on the computer together in order to achieve the considerable results. Finally, all the physicists's desires are limited by the financial resources which are available at the moment. Some conclusions As it was mentioned above the age of personal computers has came at LNPI. Approximately the same process can be seen in other institutes [4,6]. There is no doubt the age of EC-1046 or similar computers is near completion. Hence now we should regard the new situation. - 11 - In many places PC AT/286 and PC AT/386 will possibly be well enough especially in biophysics department, neutron physics and LNPI CAD/CAM applications. In other areas, particularly, in HEP and theory, such PCs can be regarded only as a temporary and intermediate stage decision. These differences depend on the real computation needs in those areas. Anyway I am sure that LNPI computer centre should take the participation in the setup of the small computer stations in physics and other divisions. In touch of HEP cluster it is possible to affirm that technical conditions and scientific aims in the computing area are considerably more clear then elsewhere at LNPI. Therefore we can shape a bit more detailed future of the HEP computer architecture. Since the future HEP experimets will mainly be planned in collaboration with other physics research centres it is useful to grow up the computer power to be compatible. Of course, the number of computer station will increase in both cases: will the computer centre be careful about it or not. In order to optimize the spending of money I would like to suggest the possible computer architecture. This architecture is shown in fig. 4. That structure composes the existing resources of the physics teams and planned computers. We should take into account the distance between the HEP buildings (about 500 m). The ethernet cable will embrace one building and others will be attached by the serial radial lines. - 12 - +----------------------------------------------------------+ | | | +-----+ - | | | | | WS1-WSn are various | | | WS1 +----+ computer workstation | | | | | (PC AT/286, PC AT/386 | | +-----+ | or more power stations.) | | . It is useful to impose | | . +---------+ the limitation 6-10 WSs. | | . | Printer | | | +-----+ +--+ | Main WSs are main compu- | | | | | | server | ters with parameters: | | | WSn +----+ +---------+ - 32-64 MB core | | | | | - 20 GB disks | | +-----+ | +---------+ - 20-30 MFLOPs. | | | | Main | The network is 10 Mboud | | +---------+ +--+ WS 2 | ethernet. | | | Main | | | | The main WSs should be | | | WS 1 +--+ +---------+ equiped by the radial | | | | | serial lines with 38000 | | +---------+ - bauds or more to attach | | remote PCs. | | | +----------------------------------------------------------+ Fig. 4. The scheme of the future computer architecture of the LNPI HEP cluster. The physics teams may have computers of any type. The main computer power will be used only by those users who need it. The others will use their own local PC. The main computer stations must have two or four large screens to analyze complex data. The local PCs can also have large monitors. One of the possible orders of work is as follows. A lot of work will be done at local PC: small computation, preparation of papers (reports, mails, project and so on). - 13 - When the more higher computer power is required a physicist will send the job to the main computer station. The result of any job will be routed to owners on local PC. If you need the powerful computer with interactive facilities you have to go to main computer and use the main monitor. The software on that computer installation plays an essential role. The Unix was chosen as a basic operating system thanks to the obvious fact that Unix is a standard 'de facto'. We plan to try to move in Intel 80x86 such application program tools as PAW and CSPACK [6]. It is useful to install the additional system components such as components of SHIFT project [7,8]. Of course, this computer project will require a lot of money. As it is forseen the total installation cost will be in the limits of one million US dollars. Acknowledgement I should emphasize the help and useful discussion of the computation future with A. Lodkin (LNPI), A. Oreshkin (LNPI), H. Renshall (CERN), B. Segal (CERN), V. Schegelsky (LNPI). I should express the special acknowledgement for the reasonable help to Jamie Shiers (CERN). - 14 - Literature 1. B.U. Sokolovski, A.N. Lodkin, A.E. Shevel The program tools for maintaining of experiment on high energy physics with EC computers (russian). LNPI preprint number 1063, may 1985. 2. S.A. Kulikov, A.E. Shevel The method to organize the backup copy at the computer installation under VM/CMS with a large number of a disk drives (russian). LNPI preprint number 1609, june 1990. 3. I.I. Gracheva, A.N. Lodkin, A.E. Shevel The ogranization of the archive program system developed under OS MVT to store the program modules and listings on a magnetic tape (russian). LNPI preprint number 914, december 1983. 4. Shingo Ichii Workstations in the Computing Environment for High Energy Physics Proceedings of the Simposium on Networking Japan, KEK, February 6-7, 1989, pp. 134-143. 5. D.O. Williams Is the role of the mainframe terminated? Mainframes versus Workstations CN report # CN/01/3 March 1991 CERN, Geneva, Switzerland - 15 - 6. CSPACK Client-Server Routines and Utilities Version 1.20 (May 1991) J. Shiers/CN M. Gussens/CN CERN Geneva, Switzerland 7. J. Baund, J. Bunn, F. Cane, D. Foster, F. Hemmer, E. Jagel, G. Lee, L. Robertson, B. Segal, A. Trannoy, I. Zacharov SHIFT. The Scalable Heterogeneous Integrated Facility for HEP computing. CERN, Geneva, March 2, 1991 8. J. Baund, J. Bunn, F. Cane, D. Foster, F. Hemmer, E. Jagel, G. Lee, L. Robertson, B. Segal, A. Trannoy, I. Zacharov SHIFT. User Guide and Reference Manual. Version 1.1 CERN, Geneva, January 30, 1991