2024 Jan 28

power efficient topton motherboard for homelab storage node

Inspired by The Perfect Home Server 2023 – 48TB, 4x 2.5Gbit LAN, 18W, Quiet & Compact I decided to replace one of my cluster nodes with Topton N5105 motherboard which seemed to be a perfect fit for storage node in my homelab. It’s a mini ITX motherboard with 6x SATA ports, 4x 2.5Gbit LAN and 2x M.2 slots. It’s powered by Intel® Celeron® N5105 which is a quad-core CPU with 10W TDP. Motherboard can support up to 64GB of RAM and has 2x DDR4 SODIMM slots.

Sounds great, right? Well, there are some drawbacks:

first of all, you will deal with lack of technical support and warranty: try to get BIOS update for example or return the product if it’s faulty - sending back to China is prohibitively expensive. You can of course play with chargeback but no one is going to give you back your time;
secondly my model does not run without attached display. It’s not a big deal since one can use dummy HDMI plug, but it was annoying to get motherbaord and learn by trail and error why system is not booting;
finally, one will have to deal with stability issues - system starts and freezes randomly. I’ve tried to troubleshoot it and spent a lot of time on this issue which makes the whole idea barely worth it. Unfortunately, Wolfgang’s video only mentions issues with power management but does not point out solutions. Maybe it’s due to the fact that one can never be sure what kind of hardware is going to be shipped, just look at all of different vendors trying to position themselves under topton brand. Anyway, issues with power management are just the tip of the iceberg.

Making it all work

Setup

I have not modified hardware in any way. Some people are trying to replace cooling system, but I did not find temperature to be problematic - at the beginning at least!. I’ve installed 1 x 32GB RAM stick - which starts to be questionable since N5105 processor supports only 16GB according to spec while motherboard spec says “maximum capacity of single memory: 32GB”. I’ve added 3x 4TB HDDs for storage and 1x 256GB SSD for system. Then I’ve installed Proxmox 8.x with one LXC container and one VM.

Display issues

In order to have display working one needs to use nomodeset as kernel parameter and possibly simplefb module in initramfs (this was necessary on proxmox 8.x but not on debian 12). Not sure what is the reason for this, but it’s connected with i915 driver.

Update kernel parameters in /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="earlyprintk=vga nomodeset"

Update initramfs modules in /etc/initramfs-tools/modules:

simplefb

update-grub
update-initramfs -u

In order to use it without monitor attached I’m using HDMI dummy plug. It’s a small device that tricks motherboard into thinking that display is attached. Fortunately, it’s cheap: HDMI Dummy Plug or ADAPTER HDMI VGA 4K DUMMY

Random freezes

I’ve tried to troubleshoot this issue for a while. I’ve tried to disable C-states, disable power management in BIOS and via kernel parameters - nothing helped. Finally, I’ve updated proxmox to 8.1 and installed latest intel microcode:

$ cat /proc/cpuinfo | grep microcode
microcode	: 0x24000024
$ apt-cache madison intel-microcode
intel-microcode | 3.20231114.1~deb12u1 | http://ftp.pl.debian.org/debian bookworm/non-free-firmware amd64 Packages

Yes, you will have to add non-free-firmware to your sources.list in order to install it. I’ve also set VM CPU to host and enabled Intel specific features:

Raw qemu config /etc/pve/qemu-server/101.conf:

args: -cpu kvm64,+cx16,+lahf_lm,+popcnt,+sse3,+ssse3,+sse4.1,+sse4.2
cpu: host,flags=+ssbd;+hv-evmcs;+aes
...

Next thing to check is C-states. They are disabled in my version by default since JMB585 controller does not support it and crashes system below C3. I wanted to save as much power as I could so I’ve enabled it in BIOS but in order to make it work one needs to limit max C-state via kernel parameters:

Update kernel parameters in /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="earlyprintk=vga nomodeset intel_idle.max_cstate=3"

If this does not help then you probably have issues with cooling system. Not sure if it’s my particular motherboard, or it’s more common, but I’ve seen people stating that their motherboards are working fine with default cooling system. Possibly because a lot of them are using it to achieve the lowest possible power consumption and one way of optimizing it is to set CPU to the lowest possible frequency. For example this article is often referenced on forums if it comes to cooling. They mention cooling issues causing CPU throttling but not system freeze. TBH this is not exactly my motherboard since it’s more a router based on Topton N5105 and whole enclosure is one big radiator. Anyway, if freezes are happening then my bet would be on cooling system.

Dealing with overheating

Despite the fact that I’ve installed the newest microcode and was running kernel with nomodeset and intel_idle.max_cstate=3 I was still experiencing random freezes. At this point I was not sure what to do with it and decided to run memtest. It was running fine for a while, but then it just restarted. I’ve tried to run it again and it happen again. I’ve decided to check temperatures, and it turned out that CPU was running close to 75C which should be fine (I was being told by some hardware experts I know). So is it memory issue then? I decided to run memtest on memory sticks from topton build but on different machine, and it was running fine. In the end all evidence seems to point to topton motherboard. The next thing to do would be to improve cooling and see if it helps.

First of all I’ve replaced thermal paste between CPU and heatsink just to make sure. Then I’ve replaced cooling fan with something bigger as you can see on the picture above. Things are running fine now, and I’m not experiencing any freezes.

NIC issues

https://forum.proxmox.com/threads/intel-i226-v-pci-passthrough-failure.130632/

https://unix.stackexchange.com/questions/729508/how-to-permanently-disable-eee-energy-efficient-ethernet-on-ethernet-card

Updating BIOS

I’m not sure if this step helped with anything, but I was quite desperate to get it working without display attached. I’m adding it here just in case. From what I have understood, most of the boards available under topton name are clones of CWWK board N5105/N6005. In fact CWWK logo is displayed during BIOS startup. Through CWWK site I’ve found BIOS for my board: 1.N5105-N6005-NAS-BIOS | 畅网官方网盘.

Instead of writing ISO to USB stick one needs to mount it and simply copy files to FAT32 formatted USB stick.
dump BIOS from motherboard after booting from USB stick
flash BIOS

Optimizing power consumption and performance

Whole point of using this motherboard is to have low power consumption. Apparently, it’s possible to get it down to 15W but default BIOS settings aren’t optimal. My approach here is to squeeze as much performance as possible while having the lowest possible power consumption. Some people on the forums do not care about performance and try to get the lowest possible power consumption and so they are disabling CPU frequency scaling and setting CPU to the lowest possible frequency. If you are up to minimize power consumption then you can follow TOPTON N5095/N5105/N100 opnsense & proxmox powersave tuning.

Setting	Value	Default value
Advanced → ACPI Settings -> Enable ACPI Auto Configuration	Enabled	Disabled
Advanced → CPU Configuration -> PECI	Enabled	Enabled
Boot performance mode	Turbo Performance	Turbo Performance
Advanced → Power & performance → GT Power Management Console	200Mhz	Default max frequency

Advanced → Power & performance → CPU – Power Management Control

Setting	Value	Default value
Boot performance mode	Turbo Performance	Turbo Performance
Platform PL1 Enable	Enabled	Disabled
Platform PL1 Power	6000	N/A
Platform PL2 Enable	Enabled	Disabled
Platform PL2 Power	12000	N/A
C states	Enabled	Disabled

Enabling power save on sound card needs to be done via /etc/modprobe.d/disable-snd-hda-intel.conf and shaves around 1-1.5W. It worked better than blacklisting sound card modules which didn’t save as much power as power save option. I’ve added following line to the file:

options snd_hda_intel power_save=1

Additional 2W can be saved by enabling power saving for i915 driver. Update kernel parameters in /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="earlyprintk=vga nomodeset intel_idle.max_cstate=3 i915.enable_dc=1 i915.enable_psr=1 
consoleblank=60"

Conclusions

After optimizing power settings system draws around 38-39W with 3x 4TB HDDs and 1x 256GB SSD running ZFS RAID5 while idle that would suggest motherboard draws around 13W (assuming HDD drives consume around 8W each).
ZFS achieves read IOPS avg=53.01 and write IOPS avg=17.86 and IOPS is important factor if one wants to build NAS. Limits are set by JMB585 chipset for SATA ports. One would achieve similar results when using Odroid H3+ motherboard and PCIE M.2 to SATA3.0 based on JMB585
If I had knew how much hassle it takes to make this board work I’d not buy it. I’ve spent too much time trying to figure out why it’s not working and how to fix. It would be easier to go with Odroid H3+ at similar price point and similar power consumption. I’ve also seen people recommending ASRock N100DC-ITX and recently CM3588 CPU + CM3588 NAS Kit from FriendlyElec. The latter one is a bit more risky since it’s Chinese board based on ARM compute module from Rockchip.

References

VM freezes irregularly
TOPTON N5095/N5105/N100 opnsense & proxmox powersave tuning - windgate
IOPS benchmark made with: fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=128k --iodepth=64 --size=64G --readwrite=randrw --rwmixread=75