High Performance Computing
High Performance and High Throughput Cluster
HPC(High Performance Clusters) and HTC(High Throughput Cluster) have been used for a long time for scientific research and commercial activities. Since the beginning these clusters have relied on heavy configuration both on the server as well as nodes.
We can design new hybrid clusters which will work on the existing models where in the nodes connected through fast interconnects in the local network or over the internet can contribute their computing resources. Also this cluster will have a dedicated mode in which the computers will be connected to each other and dedicated solely to the cluster. The cluster will have one remote boot server based on the LTSP(Linux Terminal Server Project). The remote server will be responsible for pushing right operating system to the node according its configuration. Thus tailored operating systems will be run on the node computers optimized to deliver full performance. The remote server will also have all the configurations required by the node.
This cluster system will also have a dynamic distributed file system. Any machine booting on the cluster with permanent storage will automatically become a part of the distributed file system. There will be a central file server which will be responsible for managing data distribution and backup. This server will decide where the data is to be stored and also push a copy of the data to the backup server. When a node boots in the cluster its storage will be reinitialized and the data will be shared from other nodes or from the backup server which will send it the data of a node that recently went down. New node will read and write data on the distributed file system and while going down notify the central server. The central server will distribute the data that was available on this node to the other nodes on the server and the cluster will resume normal operations.
Maintenance on this type of a cluster will be very easy as the nodes can be added or removed on the go and their is no special configuration required for adding new nodes or expanding the storage of the cluster on a whole.
I am still working on the project and have completed the LTSP integration. I would like to have your support. This is an open source GPL v2 program.
The repository is at http://newerahpc.googlecode.com
NeweraHPC(http://newerahpc.googlecode.com) is a simple scheduler for scheduling tasks on a cluster of heterogeneous clusters. The program is completely opensource written under the GPLv2 license.
With the advancement of processors there is not much need of single system image and similar clusters. But the job can be done very efficiently using a simple scheduler that can fragment a job into smaller pieces and distribute it over the cluster of computers and then collect the results back. NeweraHPC does the same.
It is a very simple library based on C++ and uses posix libraries thus compatible with all the available UNIX platforms. It has all the basic services required like a simple garbage collector and can handle memory fragmentations very well.
There are two ways jobs can be distributed, first method is the traditional way wherein we can use some external application to process data and the data is fed in the form of independent files which are then written to a standard file which is then sent back to node which requested the job.
The second method is called NXI(Newera Extensible Interface) which is in the form of dynamic link libraries and contains docking functions. These functions are given data which is then processed and returned back to the node. This method is very efficient and embarrassingly parallel application frameworks can be written through it.
There are two versions of the program available in the repository, version 2 and 3.
Version 2 program is the older one and has the NXI implemented in it. It has been depreciated and it will be ported soon to the version 3. There is a test suite available with the program which can calculate the pi value to a million decimal places.
Version 3 contains the external binary implementation and is more sophisticated and will soon have the ability to work with NXI.
For more refere to http://www.newerahpc.com