| This is Data Move page for
ClusterGate.RU Moving (copying, replication) large volume of data over WAN |
- PHEDEX
complicated data
transfer system over unreliable WAN. The PhEDEx components are:
- Transfer management database (TMDB), currently version is 2.
- Transfer agents that manage the movement of files between sites. This also includes agents to migrate files to mass storage, to manage local mass storage stager pools and stage in files efficiently based on transfer demand, and to calculate file checksums when necessary before transfers.
- Management agents, in particular the allocator agent which assigns files to destinations based on site data subscriptions, and agents to maintains file transfer topology routing information.
- Tools to manage transfer requests; CMS/RefDB/PubDB specific.
- Local agents for managing files locally, for instance as files arrive from a transfer request or a production farm, including any processing that needs to be done before they can be made available for transfer: massaging information, merging files, registering files into the catalogues, injecting into TMDB.
- Web monitoring tools.
- BBFTP -- utility for bulk data transfer. bbFTP is
a file transfer software. It implements its own transfer protocol, which is
optimized for large files (larger than 2GB) and secure as it does not read the
password in a file and encrypts the connection information.
bbFTP main features are:
- Encoded username and password at connection
- SSH and Certificate authentication modules
- Multi-stream transfer
- Big windows as defined in RFC1323
- On-the-fly data compression
- Automatic retry
- Customizable time-outs
- Transfer simulation
- AFS authentication integration
- RFIO interface
- BBCP another tool for bulk data tranfer
- mirrordir (Mirrordir version 0.10.49
(International - builtin encryption) this utility to copy directory
from one machine to another one. Mirrordir makes a minimal set of changes to
the directory
to make it identical to the directory . Mirrordir dives into subdirectories recursively and duplicates all types of files exactly. - GridFTP Grid/Globus data transfer tool. Client part is known as globus-url-copy.
|
Please email the portalmaster@pnpi.spb.ru with questions or comments. |
| Our smart sponsors: |
|