HomeLab Adventures: The Expansioning

IMAG0218_picmonkeyed

Supermicro X8DT3

Over the last few months, I have been building out more and more virtual machines in my HomeLab ESX Cluster.  Its time to expand.

First came a four node Ceph Cluster, then an OpenStack Juno environment, and then an OpenStack Icehouse environment. Plus, I still needed to build out a Docker/Kubernetes test environment. It started to become apparent that I needed to add more capacity. Especially memory.

I figured it was time to dive into my hardware closet and see if I had any decent hardware. Luckily I was able to find a Supermicro X8DT3, and a couple of Xeon X5550s. I was off to a good start, but what I needed most was RAM. A bit more digging and I was able to find a box of 4gb Dimms , I needed 12 total… I found exactly that.

IMAG0224_picmonkeyed

1/2 of Memory Dimms Installed

Next, I needed to find a case. I did not feel like dropping $100+ on a new E-ATX case, especially if I did not know if my motherboard was actually working. Luckily I found the a used Cooler Master XM on Ebay. $60 shipped was worth the risk. It was not even used, and was in great shape. You can see the original sticker still in place over the dual hot swap drive bay.

coolermaster_pic1_picmonkeyed

Somebody Donated this Beast to the Salvation Army

I also needed a power supply, but I was not looking forward to dropping $100+ on a massive modular PSU. Instead I picked up a mid-range 650W power supply off Amazon along with a 8pin CPU splitter. 650W should be plenty to push two 90w Xeons. I read somewhere that DDR3 memory is fairly power efficient, around 2-5 watts per dim. I figured I was safe since I was not going to stuff this new box full of disks. I have a freenas box for that.

Once all my parts arrived, I spent an evening cobbling everything together.  Much to my surprise the system booted without issue. A firmware upgrade was in order, so I finished that off in no time flat.

IMAG0226_picmonkeyed

It Lives

A bit more digging through the pile turned up an LSI-8888elp raid controller. I plan to run two reclaimed 250GB drives in raid1 config (with a cold spare still in the closet) to give me a bit of local storage, and an ssd drive for the OS.

Had the pleasure of working with this beauty on my test bench. Makes system building much more enjoyable.

IMAG0225_picmonkeyed

Next up — install ESXi and add to the cluster. This will bring my total ESXI server count to three, perfect for a true cluster. All systems have same motherboards, CPUs, and memory configuration.

HomeLab: How to Resolve Supermicro x8dti Fan Revving Issues

Supermicro-Hyper-Speed-6027AX-TRF-Front-Straight

Its Winter here in the Atlanta area, and the temperatures have been dropping down close to freezing. Likewise, the temperatures in my basement Homelab have been dipping below 66 degrees Fahrenheit, and apparently this makes my Supermicro Servers a bit unhappy.

A bit of background. When I set out to build my lab I decided to transfer my Supermicro X8DTI boards out of their stock rack mount enclosures (like what you see above) and into Standard EATX Towers. Running my boards in these towers allowed me to use large 120mm and 140mm fans for cooling. Sound wise this is a huge improvement over the stock 80mm fans used in the default enclosure as these larger fans can spin much slower than stock and still keep the system cool.

On several occasions I have been working in my office and have heard the fans in my lab servers revving up and back down again. At first I thought that maybe a fan was failing, and that one of my systems was overheating. Or that one of my systems had sucked in a bit too much basement dirt and dust. However neither were the case.

Specifically what was happening was this… The systems were running cool, so the fans would spin down to a low rpm and the system would then throw a low rpm threshold alert and spin the fan back up.When this occurs the system switches into some sort of “Critical Cooling Mode” and spins all the fans up to 100% for a few seconds. Rinse and repeat a few dozen times and you hear what almost sounds like an intoxicated neighbor playing with his new weedeater.

Using the IPMIitool command from my Linux desktop first logged into IPMI controller on my systems and checked to make sure that the fans were actually working properly. SDR is short for Sensor Data Repository

# ipmitool -H 10.1.0.104 -U admin -P <password> sdr list
Fan2 | 2176 RPM | ok
Fan3 | 340 RPM | ok
Fan4 | no reading | ns
Fan5 | 544 RPM | ok
Fan6 | 340 RPM | ok
…truncated…

Fan6 above is spinning mighty slow. Slow enough to drop below the Lower Non-Critical Value threshold. Rather than increase the speed of the fan as I have seen others do, I decided to lower the thresholds for this fan with the command below. Since Fan5 is also a bigun’ I decided to preemptively adjust its thresholds as well.

ipmitool -H 10.1.0.104 -U admin -P <password> sensor thres Fan4 lower 100 200 300
ipmitool -H 10.1.0.104 -U admin -P <password> sensor thres Fan5 lower 100 200 300

MegaRaid Cards Via CLI

Megaman_8bit

MegaRAID® is LSI’s line of SATA/SAS Storage Controller.

MegaCLI

MegaCLI is the Linux console based management utility for LSI SAS controllers. Honestly its a pretty crummy command when compared to HP’s command line tool, but that’s often what you are stuck with when you buy Dell or Supermicro.

Note that I am running the 64 bit version of MegaCLI which is installed in /opt/MegaRAID/MegaCli and is called MegaCli64. On 32 bit systems its called MegaCLI.

The command below will dump out a bunch of info, but if you look for the section labeled “Device Present” you can see failed/degrated drives. In this case I have one failed drive out of 4 total drives

./MegaCli64 -AdpAllInfo -aALL

 Device Present
================
Virtual Drives    : 2
  Degraded        : 1
  Offline         : 0
Physical Devices  : 4
  Disks           : 3
  Critical Disks  : 0
  Failed Disks    : 0

For more specific disk information run the following command.

./MegaCli64 -LDPDInfo -aAll

Using the command above I can see more information on the drive with the failed submirror

Virtual Disk: 1 (Target Id: 1)
Name:
RAID Level: Primary-1, Secondary-0, RAID Level Qualifier-0
Size:59.125 GB
State: Degraded
Stripe Size: 64 KB
Number Of Drives:2

Megactl

According to Sourceforce, Megactl is.. “is a small collection of programs for examining configuration and status of LSI megaraid adapters, especially Dell PERC RAID adapters, and attached storage devices.”  Get it here.

In my this case I am running medasasctl which makes it a bit easier to see the failed drive.  In the example below I can see two virtual disks (both raid1), but only 3 physical, which indicates that one of my submirrors has failed.

megactl-0.4.1]# ./megasasctl
a0       LSI MegaRAID SAS 9260-8i encl:1 ldrv:2  batt:FAULT, unknown charge state
a0d0       29GiB RAID 1   1×2  optimal
a0d1       59GiB RAID 1   1×2  DEGRADED
a0e252s0    29GiB  a0d0  online 
a0e252s1    29GiB  a0d0  online 
a0e252s2    59GiB  a0d1  online