Asus RT-AC66U – Installing the ipkg Command

asus-rt-ac66u

Note: The RT-AC66U and the RT-AC66R are identical other than their product number. The RT-AC66R is the product sold through Best Buy and RT-AC66U is the product sold directly from ASUS.

IPKG is a cli utility used for package management. It is required to have ipkg installed if you want to configure your router for SNMP,  install BIND, or view I/O to a usb disk as you will need to install software packages that are not included in the stock firmware.

Note: I am running the ASUSWRT-Merlin firmware, you can find it here. The stock firmware may be a bit different, so you might not see the exact screens as shown below.

Prerequisite : USB Drive

To install ipkg you will need to install Download Master.  To Install Download Master you will need to have a USB drive plugged into the router and mounted. In the image below you can see my USB disk, labeled as “SMI USB DISK”

asus-0

Troubleshooting USB Drives

I had a bit of an issue here as I was trying to mount a drive that was formatted as EXT4. Apparently this was not supported. See error below.

EXT3-fs: sda1: couldn’t mount because of unsupported optional features (240).
EXT2-fs: sda1: couldn’t mount because of unsupported optional features (240).

I pulled the drive, reformatted as EXT3 and was off and running.

usb 1-1.1: USB disconnect, address 4
usb 1-1.2: new high speed USB device using ehci_hcd and address 5
usb 1-1.2: configuration #1 chosen from 1 choice
scsi2 : SCSI emulation for USB Mass Storage devices
scsi 2:0:0:0: Direct-Access SMI USB DISK 1100 PQ: 0 ANSI: 4
sd 2:0:0:0: [sda] 8196096 512-byte hardware sectors (4196 MB)
sd 2:0:0:0: [sda] Write Protect is off
sd 2:0:0:0: [sda] Mode Sense: 43 00 00 00
sd 2:0:0:0: [sda] Assuming drive cache: write through
sd 2:0:0:0: [sda] Assuming drive cache: write through
sda: sda1
sd 2:0:0:0: [sda] Attached SCSI removable disk
sd 2:0:0:0: Attached scsi generic sg0 type 0

You can find the list of supported formats and partition sizes here. I would suggest EXT3. NTFS and FAT32 are supported, but I have heard that you might run into issues.

Installing Download Master

Now that you have your USB disk mounted you can install Download Master.

Select USB Applications on the left pane. Then click on Download Master. In the example below, you can see that I have already performed the install.

asus-2

You should see the screen below once Download Master is installed.

asus1

Accessing the Router via the CLI

Now you can either telnet or SSH to your router using its LAN IP address.  Note that telnet is the default protocol, however you can enable SSH by clicking on “Administration” in the lower left pane, and then clicking the “System” tab. Under “SSH Daemon” select “Enable SSH”.

asus-3

Ipkg installs to the path shown below.

# which ipkg
/opt/bin/ipkg

A guide to using ipkg can be found here.

 

 

HomeLab Adventures: The Expansioning

IMAG0218_picmonkeyed

Supermicro X8DT3

Over the last few months, I have been building out more and more virtual machines in my HomeLab ESX Cluster.  Its time to expand.

First came a four node Ceph Cluster, then an OpenStack Juno environment, and then an OpenStack Icehouse environment. Plus, I still needed to build out a Docker/Kubernetes test environment. It started to become apparent that I needed to add more capacity. Especially memory.

I figured it was time to dive into my hardware closet and see if I had any decent hardware. Luckily I was able to find a Supermicro X8DT3, and a couple of Xeon X5550s. I was off to a good start, but what I needed most was RAM. A bit more digging and I was able to find a box of 4gb Dimms , I needed 12 total… I found exactly that.

IMAG0224_picmonkeyed

1/2 of Memory Dimms Installed

Next, I needed to find a case. I did not feel like dropping $100+ on a new E-ATX case, especially if I did not know if my motherboard was actually working. Luckily I found the a used Cooler Master XM on Ebay. $60 shipped was worth the risk. It was not even used, and was in great shape. You can see the original sticker still in place over the dual hot swap drive bay.

coolermaster_pic1_picmonkeyed

Somebody Donated this Beast to the Salvation Army

I also needed a power supply, but I was not looking forward to dropping $100+ on a massive modular PSU. Instead I picked up a mid-range 650W power supply off Amazon along with a 8pin CPU splitter. 650W should be plenty to push two 90w Xeons. I read somewhere that DDR3 memory is fairly power efficient, around 2-5 watts per dim. I figured I was safe since I was not going to stuff this new box full of disks. I have a freenas box for that.

Once all my parts arrived, I spent an evening cobbling everything together.  Much to my surprise the system booted without issue. A firmware upgrade was in order, so I finished that off in no time flat.

IMAG0226_picmonkeyed

It Lives

A bit more digging through the pile turned up an LSI-8888elp raid controller. I plan to run two reclaimed 250GB drives in raid1 config (with a cold spare still in the closet) to give me a bit of local storage, and an ssd drive for the OS.

Had the pleasure of working with this beauty on my test bench. Makes system building much more enjoyable.

IMAG0225_picmonkeyed

Next up — install ESXi and add to the cluster. This will bring my total ESXI server count to three, perfect for a true cluster. All systems have same motherboards, CPUs, and memory configuration.

Insync: The Powerful Google Drive Client For Linux

Insync

Insync is a very powerful and full featured Google Drive client for Windows, Mac, and Linux. I ran across Insync when I was looking for a Google Drive client for Linux after I kicked Dropbox to the curb and switching over to Google Drive for all my cloud storage needs.

In all honestly, if it wasn’t for the Insync client I do not think that I would have made the switch at all, as Google does not even offer a basic GUI client for Linux. Really Google?

PSA: The integration that Google Drive provides into Google Photos, Google Music, Google Docs, and Gmail is well worth the switch from Dropbox in my opinion, and 1TB for only 9 bucks a month is hard to beat (100GB is only $1.99 a month). Just having a Gmail account gives you access to 15GB of free space… so there is no reason not to give it a try.

Ok now back to the topic at hand.

Note that a personal license of the Insync Google Drive client is not free, rather it costs $15. However you can download and try it risk free and without entering any credit card info. This one time fee for a personal license allows you to run and install Insync on multiple machines. Currently I have it installed on 4 separate Linux workstations/laptops. Its well worth cash.

Installing Insync is very easy and well documented so I am not going to go into that topic here. Rather lets talk about using Insync on Linux.

Continue reading

How to Fix the Buffalo Linkstation NAS – Partition Not Found Error

Buffalo-LinkStation-220-1024x576-e90e950d78fb74df

Recently I picked up a Buffalo Linkstation 220 to play around with at home as I felt that I could use a bit of additional storage to play around with. Note that this previous statement is pretty much a lie. I have tons of storage, and was really just looking for an new toy to play around with. Basically I just had a few disks laying around that I wanted to put to use.

However, much to my dismay the I was unable to configure the device once I shoved in the disks, powered it up, and connected to it with the Buffalo Smart Phone Navigator. I figured that this was not a big deal however, so I tried the installable Windows App from my Windows 7 Vm. The Buffalo NAS Navigator was also able to connect to the device, however the device showed that it was currently booted in what was called “Emergency Mode”. Not sensing a real emergency, I did not panic.

See borrowed image below.

buffalo_unbrick_001

Fortunately the site that I borrowed the above image from (here) and this site (here) give advice on how to fix the issue. First step is to download the Buffalo Linkstation Firmware Updater that you can get here. Both pages advise you to modify the LSUpdater.ini file. However their instructions did not work for me. The exact changes, and the LSUpdater.ini in its entirety are below.


[Application]
Title = BUFFALO LinkStation Series Updater Ver.1.62
WaitReboot = 1200
WaitFormat = 600
WaitFileSend = 600
WaitDiscover = 120

[Target]
ProductID = 0x80000000
ProductID2 = 0x0000001D
ProductID3 = 0x0000300D
ProductID4 = 0x0000300E
ProductID5 = 0x00003011

Name = LinkStation

[Flags]
VersionCheck = 0
NoFormatting = 0

[SpecialFlags]
Debug = 1

At this point you launch the updater again, and select “Update“. This fully partitions the drives and then updates the firmware. This process takes a while, so be patient. Now you can launch the NAS Navigator and configure the device.

HomeLab: How to Resolve Supermicro x8dti Fan Revving Issues

Supermicro-Hyper-Speed-6027AX-TRF-Front-Straight

Its Winter here in the Atlanta area, and the temperatures have been dropping down close to freezing. Likewise, the temperatures in my basement Homelab have been dipping below 66 degrees Fahrenheit, and apparently this makes my Supermicro Servers a bit unhappy.

A bit of background. When I set out to build my lab I decided to transfer my Supermicro X8DTI boards out of their stock rack mount enclosures (like what you see above) and into Standard EATX Towers. Running my boards in these towers allowed me to use large 120mm and 140mm fans for cooling. Sound wise this is a huge improvement over the stock 80mm fans used in the default enclosure as these larger fans can spin much slower than stock and still keep the system cool.

On several occasions I have been working in my office and have heard the fans in my lab servers revving up and back down again. At first I thought that maybe a fan was failing, and that one of my systems was overheating. Or that one of my systems had sucked in a bit too much basement dirt and dust. However neither were the case.

Specifically what was happening was this… The systems were running cool, so the fans would spin down to a low rpm and the system would then throw a low rpm threshold alert and spin the fan back up.When this occurs the system switches into some sort of “Critical Cooling Mode” and spins all the fans up to 100% for a few seconds. Rinse and repeat a few dozen times and you hear what almost sounds like an intoxicated neighbor playing with his new weedeater.

Using the IPMIitool command from my Linux desktop first logged into IPMI controller on my systems and checked to make sure that the fans were actually working properly. SDR is short for Sensor Data Repository

# ipmitool -H 10.1.0.104 -U admin -P <password> sdr list
Fan2 | 2176 RPM | ok
Fan3 | 340 RPM | ok
Fan4 | no reading | ns
Fan5 | 544 RPM | ok
Fan6 | 340 RPM | ok
…truncated…

Fan6 above is spinning mighty slow. Slow enough to drop below the Lower Non-Critical Value threshold. Rather than increase the speed of the fan as I have seen others do, I decided to lower the thresholds for this fan with the command below. Since Fan5 is also a bigun’ I decided to preemptively adjust its thresholds as well.

ipmitool -H 10.1.0.104 -U admin -P <password> sensor thres Fan4 lower 100 200 300
ipmitool -H 10.1.0.104 -U admin -P <password> sensor thres Fan5 lower 100 200 300

Configure Syslog Logging Levels on the Asus RT-AC66U Router


4614_WizardStressToy_1

So here is a quick little one that I figured out the other day. Having just setup a Splunk server at home I wanted to make sure that I was not going to hit the data limit of 500mb a day for the free version of Splunk. I figured out pretty fast that my ASUS RT-AC66U was a very chatty-cathy when it came to syslog… sending me all sorts of very raw data that I was, at least at first, not so sure I was interested in indexing. So I hit the cli and started poking around.

First off, before we jump in, let’s make sure that we are all on the same page. First thing to note is that I am running the custom Merlin firmware, however that doubt that the stock firmware is much different. Second, let’s make sure that we all know how to configure syslog on our Asus.

To setup forwarding syslog to a remote syslog server, you first client on “Administration” in the “Advanced Settings” panel on the left. Then select the “System” tab near the top of the page. Scroll down to “Miscellaneous”. This section is shown below. Enter the IP address of your syslog server (or Splunk server in this case) in the “Remote Log Server” field.

syslog_asus

Now lets get down to the business of adjusting our logging level. First you need to ssh into your router.

Note that it appears that by default the log level is set to 7.

admin@RT-AC66U: # nvram show | grep log_level
log_level=7

Now before you get too excited, I am actually not sure that the main log level adheres to rfc5424. I have yet to find any published documentation from Asus to confirm this. However, according to this guy’s blog, this configuration might be a bit less chatty. Note that there are a few additional settings here which you can play around with. With these settings, I am assuming that 1 is on, and 0 if off. I am still experimenting.

admin@RT-AC66U: # nvram set log_level=2
admin@RT-AC66U: # nvram set log_enable=1
admin@RT-AC66U: # nvram set log_rejected=1
admin@RT-AC66U: # nvram set log_dropped=1
admin@RT-AC66U: # nvram set log_accepted=0

Now lets save our change and reboot

admin@RT-AC66U: # nvram commit
admin@RT-AC66U: # reboot

Note that there also is a vpn_loglevel=3 setting that can be configured via nvram. This setting might be useful to those running a VPN server on their router.

Disk I/O Monitoring on the Asus RT-AC66U Router

asus-rt-ac66uThe Asus RT-AC66U, like many home routers that are on the market today, allow you to connect a USB drive to one of its onboard USB ports and share this disk out to your network.  Via the RT-ACC66U, you can share your NAS disk via CIFS or NFS. My configuration has a 1TB unmirrored drive used for temporary scratch storage, and as a network landing area for files that I want to backup.

Note that this is my 4th article on hacking the RT-AC66U. You can check out my other articles below if you are interested.

As you must already know, the Asus RT-AC66U runs Busybox, which is a very small but powerful embedded Linux distro. Because of this there are a lot of familiar commands available via the CLI. However, don’t get to comfortable, as this is still a very foreign land.

Note that this article assumes that you have ssh or telnet working and can log into your RT-AC66U via the CLI.

As I have stated before, you can use the ipgk command to search for and install packages. In the example below I searched for iostat, but found dstat instead. Either one was fine for my purposes…. at least initially.

admin@RT-AC66U:/tmp/home/root# ipkg list | grep iostat
dstat – 0.7.0-1 – dstat is a versatile replacement for vmstat, iostat, netstat, nfsstat, and ifstat

Now that I know what to install, I need to install it.

admin@RT-AC66U:/tmp/home/root# ipkg install dstat
Installing dstat (0.7.0-1) to /opt/…
Downloading http://ipkg.nslu2-linux.org/feeds/optware/oleg/cross/stable/dstat_0.7.0-1_mipsel.ipk
Configuring dstat
Successfully terminated.

Now that dstat is installed lets run it. The switches “rad” enable i/o stats and enable disk stats. The “-D” option allows us to specify a disk by name.

admin@RT-AC66U:/tmp/home/root# dstat -rad -D sda

The command above output what you see below.

dstat-screen-shot

As I mentioned above, dstat is most definitely, a very useful command. However, so far I have not been able to figure out how to get it to display the percentage utilized for a drive, which is rather easy to do with iostat.