I preordered the HoneyComb LX2K workstation board last October and I has finally arrived and I finally got time to build my ARM workstation. The process is challenging and fun!
Building of the machine
The LX2K is a very capable 16-core ARM workstation board selling at 750$ (550$ when I ordered, early access price). Providing some impressive specs for it’s price. Even when comparing to consumer grade products!
|CPU||ARM Cortex A72 x16|
|RAM||Up to 64GB @ 3200MT/s|
|Storage||SATA 3.0 x4|
NVMe PCIe 3.0×4 x1
On board 64GB EMMc
Micro SD reader
|PCIe||PCIe 3.0×8 open slot x1|
10G Fiber x4
|USB||USB 3.0 x4|
USB 2.0 x4
|Misc||USB Serial console x1|
Built in L2 Ethernet
Just to appropriate how good the specs are. The main CPU – NXP LX2160A is comparable or outperforms a Ryzen5 2400G (~150US$ as of writing) while using less power and the a 10G Fiber NIC with the same capabilities costs 330$. That makes 70$ for the rest of the stuff. Or 270$ if you buy it now. In any case is a good deal. Remember, this is an (edge) server class hardware.
I forget to also take a photo of the board itself (You’ll see the board later on, in the case). Anyway, it’s a mini-ITX form factored board with a tiny heat sink attached. So small that I’m suspecting the fan might have to work very hard to keep the CPU cool and making the classic server noise. So I got myself a sound suppression case among other parts to begin testing the board.
Delta Eletronics is a local company whom design and build very high quality PSUs for other companies. And they do sell their PSU under their own brand in Taiwan. So they are my choice to go when building computers. And the other parts are just random picks I made. The only bet I made was the RAM I used wasn’t on the tested RAM list.
Getting the OS working
Since this isn’t an x86 machine. I was expecting troubles getting a custom Linux working on it. And it was a whole lot of trouble…
SolidRun, the manufacturer of LX2K, does provide build scripts for Ubuntu and they promise Debian and switching to the mainline kernel. But, of course, I want to run Arch Linux. And their official Ubuntu build targets 3200MT/s memory while I’m using a 2600MT/s one (yes, the DDR clock is determined by the boot loader, unlike on x86). So I’ve to build my own.
The process is quite boring to write so I’m glossing over it. It involves finding out why trying to build an OS image using an Xavier board over SSHFS because I don’t want to build GCC is a bad idea, why using Docker is better than forcing some debian-only dependencies onto my Arch system and figuring out what all the magic-number means. After that, getting Arch Linux to work is quite easy. Their build script expects a directory containing the root filesystem in a certain location. I just replaced it with Arch Linux ARM‘s ARMv8 Genetic build and tune a few FS related numbers. Bam! it works!
Now time to install the image onto the NVMe Drive. SolidRun have documented how its done via UBoot.
load mmc 0:1 0xa4000000 ubuntu-core.img nvme scan nvme write 0xa4000000 0 262144 nvme write 0xac000000 262144 262144 nvme write 0xb4000000 524288 204800
However they never specify how the parameters are calculated nor UBoot’s documentation being anywhere helpful. The worst problem being that my Arch image is about 4x bigger then the minimal Ubuntu generate by the build script. – I gave up after days of trying, none of the values matched any reasonable formula and any larger value then the shown ones crash uboot. Fortunately, It seems that the uboot command are writing ~1.8GB of data into NVMe – writing 204800+262144 blocks of 4096 bites per block (typical SSD block size). I should be good in such case. Even if somehow that breaks a part of the FS. I can hope for the basic utilities (kernel, init, network and pacman) still being functional and just reinstall every package.
That legit works! Arch booted normally. Then I expend the root partition to the entire disk and run through the normal Arch Linux installation process. Setting up local, fstab, among other stuff. Oh, and remember to lock the kernel version or make a backup of the NXP OoT kernel and point uboot to use it before running system update. The build script installs the kernel at
/boot/Image, which is the same place Arch Linux ARM stores the kernel. As mainline kernel won’t boot on the board now. You need to keep NXP’s kernel.
With the system fully working. It’s time to change a few parameters here and there.
First of all, I noticed the CPU governor is set to “performance” on boot when messing with kconfig. But I never bothered to rebuild the entire image and reinstall everything. So a small script is written to set the governor to “onemand” and setting peripherals to a power saving mode (generated using powertop). Then ask systemd to execute the script right after most initialization is done.
#!/usr/bin/bash # Set CPU to only use high frequency when needed cpupower frequency-set -g ondemand # Enable power saving features for peripherals # SATA powersaves for f in /sys/class/scsi_host/host*/link_power_management_policy; do echo 'min_power' > f; done # NVMe Disk echo 'auto' > '/sys/bus/pci/devices/0000:01:00.0/power/control'; # GPU (RX 570) echo 'auto' > '/sys/bus/pci/devices/0001:01:00.0/power/control'; # VM writeback echo '1500' > '/proc/sys/vm/dirty_writeback_centisecs';
At this point, I have noticed the CPU fan spinning at max speed even though the fan controller supports PWM and
cat /sys/class/hwmon/hwmon1/fan1_fault shows
0. A few hours of head scratching later, I still have no clue why the fan speed isn’t under control. That’s when I had a look into the board. Ahh yeah… they attached a 3-pin low speed cable to the 4 pin fan header. Which, the 4th pin is the PWM signal.
This is a bad sign… Having a low speed cable means the default is too high. I can connect the fan directly to the PWM header on the board. Now the fan control works as expected. Originally the fan ran at 9000RPM constantly. It doped to ~7000RPM idle. Just that I’ve to tolerate some noise under load (11000RPM!) until I can get replacement parts.
Getting ROOT and Jupyter working
I use ROOT, a data analysis framework by CERN, a lot in work and for personal projects. However Arch Linux’s ROOT package isn’t available for ARM. Building from source it is.
git clone https://github.com/root-project/root.git mkdir build cd build cmake .. make -j16
And just figure out the dependencies on-the-fly. I also enabled C++17 support, TMVA on CPU, GSL, etc.. that I can benefit from. Then
sudo make install to install ROOT to the system.
One of the issue with ROOT is it doesn’t work with Jupyter by default. You run
root --notebook to launch a ROOT capable Jupyter notebook. It su*ks. The JupyROOT directory (I don’t know why this isn’t a repo by itself) documents how to make the ROOT C++ kernel available under Jupyter.
cd /path/to/root/source/build source bin/thisroot.sh cp -r $ROOTSYS/etc/notebook/kernels/root ~/.local/share/jupyter/kernels
Then you can launch Jupyter as usual.
Web services and reverse proxy
I need more than just Jupyter on my server. I also want a dashboard, vscode-server, NextCloud among other services to make a truly awesome server. All of these services speaks the HTTP protocol. Yet you can’t bind all of them to port 443. The solution is to use a reverse HTTP proxy that listens on port 443 then forward the request to the appropriate application.
Nginx is my default solution to build a reverse proxy. Nginx gas lower latency comparing to Apache. Installed it via
sudo pacman -S nginx. Configured it to forward anything under
/jupyter to my Jupyer server running on
locahost:xxxx (I changed the port and only binding it to localhost for security). The same for the dashboard, NextCloud, etc… all of them are hosted either on localhost or using a Unix Domain Socket. Also I’ve to get Jupyter to start at boot and loading ROOT (recall my current installation requires me a shell script beforehand). Well… you launch
sh from systemd, I guess? That’s my best solution anyway.
[Unit] Description=JupyterLab After=syslog.target network.target [Service] User=my_user WorkingDirectory=/home/my_user/ ExecStart=sh -c "source ~/Documents/root/build/bin/thisroot.sh; /usr/bin/jupyter lab --ip localhost --port xxxx" StandardOutput=syslog StandardError=syslog Restart=on-failure [Install] WantedBy=multi-user.target
I properly should find a way to get ROOT properly installed. Good enough for now.
The Pros and Cons
I experimented with other stuff that’s too boring to write. But TL;DR:
- The entire system runs <60 watt under full load
- Good performance. Outperforms x86 from time to time
- 10GBit SFP ports
- Great driver support
- Haven’t ran into any driver issue
- Most packages available in Arch Linux ARM
And the bads
- Have to build a few packages from scratch
- Don’t run distributed FS on it
- They rely on fast cores, not the core count
- I got 6MB/s using MoosFS w/ 2 HDD…
- Mainline kernel still WIP
- Have to build kernel modules from scratch if needed
Issue and bugs
With all the work I’ve done. There’s still two issues I’ve with the server. Good none of them are deal breakers.
U-Boot does not auto boot without a serial connection
As stupid as it looks. For some very strange reason. UBoot does not automatically boot Linux after a few seconds of no keyboard presses. I don’t know why nor have the ability to investigate. I stuck a 1st Gen Raspberry Pi next to the server and connect power and USB to it. Problem solved.
Fan running fast
As I found right after finishing building the OS image. The board runs very loud. So I ordered a few low speed fan cables with different resistance to find a suitable one. – I found 57Ohm resistance work well limiting the maximum speed to 8700RPM while the idle speed is still 7000PRM for reasons.
The build scripts build the Linux 4.19 kernel. Which is a bit old in the Arch standard. I’d really want to run an up to date Linux 5.6. Epically considering Purism have been upstreaming a ton for power related patches for their i.MX 8 processor (the LX2610A on board is a far cousin of it). New features like better ZFS and Btrfs would make huge differences.
That’s it. How I build myself a working ARM server. Feel free to comment or contact me if you have questions.
Hey, Thank You for a hope. I had hard time to make linux working at all, finally I ran pre-build by SolidRun U-Boot + Fedora 33 on 2666MHz but SFP ports doesn’t work.. – Others has reported the same. It would be great if You make some tutorial. My goal is to use it partly as router with pfSense.. probably I can virtualise this but I need SFP ports to work first
Thanks for commenting. The SFP+ ports doesn’t/won’t work out of the box. You’ll need to have a kernel with proper driver (SolidRun’s build script enables it) and configure the SFP+ via NXP’s `restool`. Fedora users can install it via the COPR repository. https://copr.fedorainfracloud.org/coprs/boeroboy/honeycomb/ . After that, run:
sudo ls-addni dpmac.3 # enables interface 3. For some reason the FSP+ on HoneyComb starts from 3
Then you should see a new interface in `ip a` or `ifconfig`. You can use it as usual. I believe Fedora’s restool package enables all SFP+ at boot. Otherwise you want to write yourselves a systemd script to start it on startup.
To be more certain, run `sudo restool dpmac info dpmac.3` to check the interface state. It should say `endpoint state: 0` when enabled but no module inserted.
Someone else have enabled the SFP+ on Fedora: https://fedoramagazine.org/fedora-aarch64-on-the-solidrun-honeycomb-lx2k/
Just for documentation and for people not using Fedora. You can get the source code of restool on github: https://github.com/qoriq-open-source/restool then patch out some linking errors with `find -type f -name ‘*.c’ | xargs sed -i ‘s/^enum mc_cmd_status mc_status;/static enum mc_cmd_status mc_status;/’` in order to build it.
For other Arch users, I’ve created an AUR package: https://aur.archlinux.org/packages/restool/
About the fan, did you toggle it In the UEFI?.
OR, was it in default?.