Rocking a 16 core ARM workstation for data analysis and NAS – HoneyComb LX2K

I preordered the HoneyComb LX2K workstation board last October and I has finally arrived and I finally got time to build my ARM workstation. The process is challenging and fun!

Building of the machine

The LX2K is a very capable 16-core ARM workstation board selling at 750$ (550$ when I ordered, early access price). Providing some impressive specs for it’s price. Even when comparing to consumer grade products!

CPU	ARM Cortex A72 x16
RAM	Up to 64GB @ 3200MT/s
Storage	SATA 3.0 x4 NVMe PCIe 3.0×4 x1 On board 64GB EMMc Micro SD reader
PCIe	PCIe 3.0×8 open slot x1
NIC	GbE x1 10G Fiber x4
USB	USB 3.0 x4 USB 2.0 x4
Misc	USB Serial console x1 Built in L2 Ethernet

Just to appropriate how good the specs are. The main CPU – NXP LX2160A is comparable or outperforms a Ryzen5 2400G (~150US$ as of writing) while using less power and the a 10G Fiber NIC with the same capabilities costs 330$. That makes 70$ for the rest of the stuff. Or 270$ if you buy it now. In any case is a good deal. Remember, this is an (edge) server class hardware.

The box I received containing the board.

I forget to also take a photo of the board itself (You’ll see the board later on, in the case). Anyway, it’s a mini-ITX form factored board with a tiny heat sink attached. So small that I’m suspecting the fan might have to work very hard to keep the CPU cool and making the classic server noise. So I got myself a sound suppression case among other parts to begin testing the board.

Case	Cooler Master Sliencio 425
PSU	Delta NX550
SSD	ADATA SX 8200 Pro 512G
RAM	ADATA 32G 2666MT/s

Delta Eletronics is a local company whom design and build very high quality PSUs for other companies. And they do sell their PSU under their own brand in Taiwan. So they are my choice to go when building computers. And the other parts are just random picks I made. The only bet I made was the RAM I used wasn’t on the tested RAM list.

Getting the OS working

Since this isn’t an x86 machine. I was expecting troubles getting a custom Linux working on it. And it was a whole lot of trouble…

SolidRun, the manufacturer of LX2K, does provide build scripts for Ubuntu and they promise Debian and switching to the mainline kernel. But, of course, I want to run Arch Linux. And their official Ubuntu build targets 3200MT/s memory while I’m using a 2600MT/s one (yes, the DDR clock is determined by the boot loader, unlike on x86). So I’ve to build my own.

The process is quite boring to write so I’m glossing over it. It involves finding out why trying to build an OS image using an Xavier board over SSHFS because I don’t want to build GCC is a bad idea, why using Docker is better than forcing some debian-only dependencies onto my Arch system and figuring out what all the magic-number means. After that, getting Arch Linux to work is quite easy. Their build script expects a directory containing the root filesystem in a certain location. I just replaced it with Arch Linux ARM‘s ARMv8 Genetic build and tune a few FS related numbers. Bam! it works!

Arch Linux on LX2K!

Now time to install the image onto the NVMe Drive. SolidRun have documented how its done via UBoot.

load mmc 0:1 0xa4000000 ubuntu-core.img
nvme scan
nvme write 0xa4000000 0 262144
nvme write 0xac000000 262144 262144
nvme write 0xb4000000 524288 204800

However they never specify how the parameters are calculated nor UBoot’s documentation being anywhere helpful. The worst problem being that my Arch image is about 4x bigger then the minimal Ubuntu generate by the build script. – I gave up after days of trying, none of the values matched any reasonable formula and any larger value then the shown ones crash uboot. Fortunately, It seems that the uboot command are writing ~1.8GB of data into NVMe – writing 204800+262144 blocks of 4096 bites per block (typical SSD block size). I should be good in such case. Even if somehow that breaks a part of the FS. I can hope for the basic utilities (kernel, init, network and pacman) still being functional and just reinstall every package.

That legit works! Arch booted normally. Then I expend the root partition to the entire disk and run through the normal Arch Linux installation process. Setting up local, fstab, among other stuff. Oh, and remember to lock the kernel version or make a backup of the NXP OoT kernel and point uboot to use it before running system update. The build script installs the kernel at /boot/Image, which is the same place Arch Linux ARM stores the kernel. As mainline kernel won’t boot on the board now. You need to keep NXP’s kernel.

System tuning

With the system fully working. It’s time to change a few parameters here and there.

First of all, I noticed the CPU governor is set to “performance” on boot when messing with kconfig. But I never bothered to rebuild the entire image and reinstall everything. So a small script is written to set the governor to “onemand” and setting peripherals to a power saving mode (generated using powertop). Then ask systemd to execute the script right after most initialization is done.

#!/usr/bin/bash

# Set CPU to only use high frequency when needed
cpupower frequency-set -g ondemand

# Enable power saving features for peripherals
# SATA powersaves
for f in /sys/class/scsi_host/host*/link_power_management_policy; do
	echo 'min_power' > f;
done

# NVMe Disk
echo 'auto' > '/sys/bus/pci/devices/0000:01:00.0/power/control';

# GPU (RX 570)
echo 'auto' > '/sys/bus/pci/devices/0001:01:00.0/power/control';

# VM writeback
echo '1500' > '/proc/sys/vm/dirty_writeback_centisecs';

At this point, I have noticed the CPU fan spinning at max speed even though the fan controller supports PWM and cat /sys/class/hwmon/hwmon1/fan1_fault shows 0. A few hours of head scratching later, I still have no clue why the fan speed isn’t under control. That’s when I had a look into the board. Ahh yeah… they attached a 3-pin low speed cable to the 4 pin fan header. Which, the 4th pin is the PWM signal.

The included 3pin cable.

This is a bad sign… Having a low speed cable means the default is too high. I can connect the fan directly to the PWM header on the board. Now the fan control works as expected. Originally the fan ran at 9000RPM constantly. It doped to ~7000RPM idle. Just that I’ve to tolerate some noise under load (11000RPM!) until I can get replacement parts.

Getting ROOT and Jupyter working

I use ROOT, a data analysis framework by CERN, a lot in work and for personal projects. However Arch Linux’s ROOT package isn’t available for ARM. Building from source it is.

git clone https://github.com/root-project/root.git
mkdir build
cd build
cmake ..
make -j16

And just figure out the dependencies on-the-fly. I also enabled C++17 support, TMVA on CPU, GSL, etc.. that I can benefit from. Then sudo make install to install ROOT to the system.

One of the issue with ROOT is it doesn’t work with Jupyter by default. You run root --notebook to launch a ROOT capable Jupyter notebook. It su*ks. The JupyROOT directory (I don’t know why this isn’t a repo by itself) documents how to make the ROOT C++ kernel available under Jupyter.

cd /path/to/root/source/build
source bin/thisroot.sh

cp -r $ROOTSYS/etc/notebook/kernels/root ~/.local/share/jupyter/kernels

Then you can launch Jupyter as usual.

jupyter lab

ROOT in Jupyter Lab

Web services and reverse proxy

I need more than just Jupyter on my server. I also want a dashboard, vscode-server, NextCloud among other services to make a truly awesome server. All of these services speaks the HTTP protocol. Yet you can’t bind all of them to port 443. The solution is to use a reverse HTTP proxy that listens on port 443 then forward the request to the appropriate application.

Nginx is my default solution to build a reverse proxy. Nginx gas lower latency comparing to Apache. Installed it via sudo pacman -S nginx. Configured it to forward anything under /jupyter to my Jupyer server running on locahost:xxxx (I changed the port and only binding it to localhost for security). The same for the dashboard, NextCloud, etc… all of them are hosted either on localhost or using a Unix Domain Socket. Also I’ve to get Jupyter to start at boot and loading ROOT (recall my current installation requires me a shell script beforehand). Well… you launch sh from systemd, I guess? That’s my best solution anyway.

[Unit]
Description=JupyterLab
After=syslog.target network.target

[Service]
User=my_user
WorkingDirectory=/home/my_user/
ExecStart=sh -c "source ~/Documents/root/build/bin/thisroot.sh; /usr/bin/jupyter lab --ip localhost --port xxxx"
StandardOutput=syslog
StandardError=syslog
Restart=on-failure

[Install]
WantedBy=multi-user.target

I properly should find a way to get ROOT properly installed. Good enough for now.

The Pros and Cons

I experimented with other stuff that’s too boring to write. But TL;DR:

• The entire system runs <60 watt under full load
• Good performance. Outperforms x86 from time to time
• 10GBit SFP ports
• Great driver support
  • Haven’t ran into any driver issue
• Most packages available in Arch Linux ARM

And the bads

• Have to build a few packages from scratch
• Don’t run distributed FS on it
  • They rely on fast cores, not the core count
  • I got 6MB/s using MoosFS w/ 2 HDD…
• Mainline kernel still WIP
  • Have to build kernel modules from scratch if needed

Issue and bugs

With all the work I’ve done. There’s still two issues I’ve with the server. Good none of them are deal breakers.

U-Boot does not auto boot without a serial connection

As stupid as it looks. For some very strange reason. UBoot does not automatically boot Linux after a few seconds of no keyboard presses. I don’t know why nor have the ability to investigate. I stuck a 1st Gen Raspberry Pi next to the server and connect power and USB to it. Problem solved.

Fan running fast

As I found right after finishing building the OS image. The board runs very loud. So I ordered a few low speed fan cables with different resistance to find a suitable one. – I found 57Ohm resistance work well limiting the maximum speed to 8700RPM while the idle speed is still 7000PRM for reasons.

Mainline Linux

The build scripts build the Linux 4.19 kernel. Which is a bit old in the Arch standard. I’d really want to run an up to date Linux 5.6. Epically considering Purism have been upstreaming a ton for power related patches for their i.MX 8 processor (the LX2610A on board is a far cousin of it). New features like better ZFS and Btrfs would make huge differences.

That’s it. How I build myself a working ARM server. Feel free to comment or contact me if you have questions.

Advertisement