Creating low-cost "learning clusters" for HPC
Crazy Eddie's Used Systems
Up to this point, I've mostly focused on the software aspect of clusters – programming and cluster tools. However, the first two letters of HPC stand for "high" and "performance," which means understanding and tuning the hardware for the cluster. Toward this end, you need to find some hardware to build your own cluster.
An easy way to get started is to find some older hardware that is no longer being used. Friends and family are good sources, and you can sometimes even find good deals at yard sales, although "buyer beware" is a truth when you are in someone's garage buying a used PC. A number of websites also sell reasonable used systems. (Google is your friend, but be sure you read reviews before buying from a website.) A few stores also sell used hardware, but their prices can be a bit on the high side because you usually get some sort of warranty with them.
If you choose older hardware, I'll assume you want to use them without the virtualization I discussed previously. The HPC world calls this using "bare metal." Although you can use just one system to learn about cluster programming, just as you could use your laptop or desktop, as explained in the first section, ideally, you should have access to two or more systems and a basic switch. Figure 1 shows an outline of a really basic cluster with three compute nodes. (I've shown three compute nodes here, but you can get by with just one.)
The most important item is the Head Node, also called the master node; it is the brains of the cluster. In this particular setup, it also contains some storage for users, as shown by the disk to the right of the head node. The head node also contains the cluster tools used to provision other nodes, called compute nodes, and to run and control applications.
Note that the head node has two network connections. The first is to the Outside World, which is the connection into the cluster and, often, network device eth0. The second is from the head node to the cluster switch, which is a private network, so many times it will have an address of 192.168.x.x (don't forget that the head node has two Ethernet interfaces). This network is the only way the head node accesses the cluster nodes. You can use a single compute node or several. The only real requirements is that they can run a reasonable distribution.
The details on how you create a cluster from a head node, a switch, and a compute node, are usually found with the documentation of the cluster tool you are learning to use. Although it's beyond the scope of this article to cover them, look around, because a number of articles discuss how to set up and configure hardware for the various cluster tools.
Low-Cost Build-Your-Own
The use of older, used systems borrowed or bought from friends and family is a good way to get started learning about clusters and "hacking" the hardware, but sometimes there really is nothing like designing your own systems, so you have control of the hardware and know precisely what is in each node. This means going shopping.
When learning about clusters, my aim is to control the costs as much as possible. The prices I'll be quoting are from Newegg [16] at the time of writing (your mileage may vary), although you might be able to find cheaper prices. Also, just because I'm listing these items does not mean I endorse the associated companies (or the website). I just looked for the least expensive products in the various categories.
Table 1 lists the components for a simple head node with a reasonable amount of memory (4GB) and a dual-core processor. Note that these prices don't include tax and delivery (those vary too much to be estimated accurately). I included a DVD burner in the head node, but this is not strictly necessary. Overall, roughly US$ 250 will get you a dual-core head node.
Table 1
Dual-Core Node
Name | Description | Quantity | Price (US$) | Extended Price |
---|---|---|---|---|
AMD A4-3300 processor |
2.5GHz dual-core processor with AMD Radeon HD 6410D graphics |
1 |
$39.99 |
$39.99 |
ASRock A55M-V2 FM1 motherboard |
MicroATX Socket FM1 motherboard; 2x240-pin DDR3 2400+ (OC)/1866/1600/1333/1066/800 memory; 16GB maximum memory, dual-channel; 1 PCIe 2.0x16 slot, 1 PCI slot; 2x SATA 3Gbps connectors, RAID-0/1/10; Realtek 8105E GigE NIC |
1 |
$42.99 |
$42.99 |
1866MHz dual-channel memory |
2x 2GB = 4GB total memory |
1 |
$39.99 |
$39.99 |
ZION XON-720P mATX/ITX slim desktop case |
300W power supply |
1 |
$34.99 |
$34.99 |
250GB SATA 3.0Gbps disk |
– |
1 |
$59.99 |
$59.99 |
DVD +/-RW burner |
– |
1 |
$14.99 |
$14.99 |
GigE network card |
– |
1 |
$15.00 |
$15.00 |
Total |
– |
– |
– |
$247.94 |
A compute node version of the head node doesn't need a DVD burner or the second NIC, bringing the price down to US$ 218 per node. If you want to be more daring and go diskless, then the price is US$ 158 for a compute node.
Out of curiosity, I also configured a quad-core server to get an idea how much the extra cores cost. I used the same basic configuration as the dual-core, but I used the least expensive quad-core processor I could find. With the extra cores, I doubled the memory to 8GB using 4 DIMM slots instead of two, so each core has 2GB of memory (same as the dual-core system). Table 2 shows the configuration I used. Roughly US$ 370 will get you a quad-core head node.
Table 2
Quad-Core Head Node
Name | Description | Quantity | Price (US$) | Extended Price |
---|---|---|---|---|
AMD A6-3650 processor |
2.6GHz quad-core processor with AMD Radeon HD 6530D graphics |
1 |
$69.99 |
$69.99 |
ASRock A75 PRO4-M FM1 motherboard |
microATX Socket FM1 motherboard; 4x240-pin DDR3 2400+ (OC)/1866 (OC)/1600 (OC)/1333/1066/800 memory; 32GB maximum memory, dual-channel; 2 PCIe 2.0x16 slots (x16, x4), 2 PCI slots; 5x SATA 6Gbps connectors, RAID-0/1/10; Realtek 8111E GigE NIC |
1 |
$74.99 |
$74.99 |
1600MHz dual-channel memory |
4x4GB = 16GB total memory |
2 |
$49.99 |
$99.98 |
ZION XON-720P mATX/ITX slim desktop case |
300W power supply |
1 |
$34.99 |
$34.99 |
250GB SATA 3.0Gbps disk |
– |
1 |
$59.99 |
$59.99 |
DVD +/-RW burner |
– |
1 |
$14.99 |
$14.99 |
GigE network card |
– |
1 |
$15.00 |
$15.00 |
Total |
– |
– |
– |
$369.93 |
For the compute nodes, you don't need the second NIC or the DVD burner, resulting in a price of US$ 340. If you go diskless, you can get the price down to US$ 280.
You can get a small five-port GigE switch for less than US$ 20.00. Throw in some Ethernet cables at about US$ 2 per cable, and you get the following system totals:
- Five-node dual-core processor cluster: US$ 1,150 @ US$ 115/core
- Five-node dual-core processor, diskless cluster: US$ 910 @ US$ 91/core
- Five-node quad-core processor cluster: US$ 1,760 @ US$ 88/core
- Five-node quad-core processor, diskless cluster: US$ 1,520 @ US$ 76/core
These pricing games can be fun, but in all seriousness, they give you an idea how much money you would need to build your five-node cluster. (I limited it to five total nodes because of the size of the GigE switch.) Building your own systems means you know exactly what is in the cluster and how to tune it for the best possible performance.
Summary
As you discover HPC computing, you will likely want to learn about using HPC clusters. The two fundamental things to learn about clusters are writing applications and running or administering clusters. In this article, I walked through some options, starting with building and running applications on what you have today – your laptop or desktop computer – and highlighting options for further investigation.
One of the quickest and easiest ways to really get started is to use virtualization, which is sort of the antithesis of HPC: Virtualization takes a single system and makes it appear to be several separate systems; HPC tries to take several systems and make them appear to be a single system. You can use your laptop or desktop to run a number of VMs and then use them to learn about HPC applications or administering clusters. In doing so, you not only learn about HPC, you learn about virtualization for Linux, which is not a bad skill to have.
The next step up from using VMs on your laptop or desktop is to use VMs from a cloud provider such as Amazon. Cloud VMs give you a great deal of freedom in choosing the level of performance, as well as the number of VMs, you want. It also gives you another kind of freedom: Once you are done with the system, you just turn it off and it goes away. Cloud VMs can also be fairly cost effective because you only pay for the time you use them. However, you also don't have the benefit of owning the hardware and learning more about the details of configuring it and tuning applications for it.
Finally, I went over some options for your own hardware, starting with used or older systems, which can give you a very quick start configuring, managing, and running a cluster using bare metal. Of course, nothing is as fun as building the systems yourself, so I covered the option of building a cluster with a head node, a private network, and one to five compute nodes. Going this route allows you to understand what truly goes into the system, so you can tune it to run applications or tune the applications for the hardware.
If you are brand new to HPC, I would suggest first learning how to program for clusters, because this helps you understand how parallel applications work and gives you a much better understanding of clusters.
I recommend choosing the easiest route; spinning up some VMs on your laptop or desktop is not a bad way to go because you learn about virtualization while you are learning about HPC. Once you have written an application or two and understand how they work, and if you are interested in learning how to build or manage clusters, you should move on and experiment with cluster tools. Again, the easiest way to get started is to use VMs on your laptop or desktop. I know some cluster tool developers that do this every day, illustrating the usefulness of such an approach, even though nothing beats building your own hardware.
Infos
- Beowulf mailing list: http://www.beowulf.org/mailman/listinfo/beowulf
- Learning clusters: http://www.beowulf.org/pipermail/beowulf/2013-May/031587.html
- Creating a CentOS KVM virtual machine: http://www.techotopia.com/index.php/Creating_a_CentOS_KVM_Virtual_Machine
- Virtualization with KVM on a CentOS 6.2 server: http://www.howtoforge.com/virtualization-with-kvm-on-a-centos-6.2-server
- KVM and CentOS-6: http://wiki.centos.org/HowTos/KVM
- KVM virt-manager: http://www.cyberciti.biz/faq/kvm-virt-manager-install-centos-linux-guest/
- CentOS KVM install: http://www.forwardingplane.net/2013/03/centos-kvm-install-quick-start-to-a-vm/
- Warewulf series by Jeff Layton: http://www.admin-magazine.com/HPC/Articles/Warewulf-Cluster-Manager-Master-and-Compute-Nodes
- Amazon EC2: http://aws.amazon.com/hpc-applications/
- Amazon EC2 video: http://www.youtube.com/embed/YfCgK1bmCjw
- StarCluster: http://star.mit.edu/cluster/index.html
- StarCluster How-To: http://www.admin-magazine.com/HPC/Articles/StarCluster-Toolkit-Virtualization-Meets-HPC
- StarCluster slide deck: http://www.slideshare.net/chrisdag/running-mit-starcluster-on-amazon-spot-instances
- "Lustre on AWS" by Robert Read, Intel High Performance Data Division, April 16, 2013: http://www.opensfs.org/wp-content/uploads/2013/04/Read_final_LUG-2013-Lustre-on-AWS2.pdf
- EC2 instance types: http://aws.amazon.com/ec2/instance-types/
- Newegg: http://www.newegg.com/
« Previous 1 2
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
AlmaLinux OS Kitten 10 Gives Power Users a Sneak Preview
If you're looking to kick the tires of AlmaLinux's upstream version, the developers have a purrfect solution.
-
Gnome 47.1 Released with a Few Fixes
The latest release of the Gnome desktop is all about fixing a few nagging issues and not about bringing new features into the mix.
-
System76 Unveils an Ampere-Powered Thelio Desktop
If you're looking for a new desktop system for developing autonomous driving and software-defined vehicle solutions. System76 has you covered.
-
VirtualBox 7.1.4 Includes Initial Support for Linux kernel 6.12
The latest version of VirtualBox has arrived and it not only adds initial support for kernel 6.12 but another feature that will make using the virtual machine tool much easier.
-
New Slimbook EVO with Raw AMD Ryzen Power
If you're looking for serious power in a 14" ultrabook that is powered by Linux, Slimbook has just the thing for you.
-
The Gnome Foundation Struggling to Stay Afloat
The foundation behind the Gnome desktop environment is having to go through some serious belt-tightening due to continued financial problems.
-
Thousands of Linux Servers Infected with Stealth Malware Since 2021
Perfctl is capable of remaining undetected, which makes it dangerous and hard to mitigate.
-
Halcyon Creates Anti-Ransomware Protection for Linux
As more Linux systems are targeted by ransomware, Halcyon is stepping up its protection.
-
Valve and Arch Linux Announce Collaboration
Valve and Arch have come together for two projects that will have a serious impact on the Linux distribution.
-
Hacker Successfully Runs Linux on a CPU from the Early ‘70s
From the office of "Look what I can do," Dmitry Grinberg was able to get Linux running on a processor that was created in 1971.