The history of the world is a continuous succession of contradictions. The announcement from MIPS Technologies about their decision of definitely abandoning MIPS arch in favour of RISC-V is just another example. But the truth is that things are far from trivial in this topic. Even when the end-of-life date for the MIPS architecture looks closer in time than ever, there are still infrastructures and platforms what need to keep being supported and maintained for this architecture in the meantime. To make the situation more complex, at the same time I am writing this post the Loongson Technology Ltd is announcing a new 16-Core MIPS 12nm CPU for 256-Core (Tom’s Hardware news). Loongson Technology also says that they keep a strong commitment with RISC-V for the future but they will keep their bet for MIPS64 in the meantime. So if MIPS is going to die it going be in lovely death.
In this context, here in Igalia we are hosting and maintaining the CI workers for the JavaScriptCore 32-bit (MIPS) infrastructure for the WebKit web browser engine.
No one ever said that finding end-user hardware of this kind of system is easy-peasy. The options in the market often don’t achieve the sufficient level of maturity or come with a poor set of hardware specifications. The choices are often not intended for long time-consuming CPU tasks, or they simply lack good OS support (maintenance, updates, custom kernels, open-source drivers …).
Nowadays we are using a parallelized cluster of MIPSEL CI 20 boards to move the JavaScriptCore 32-bits (MIPS) CI workers. Don’t get me wrong: the CI 20 boards are certainly not bad. These boards are really great for development and evaluation purposes, but even rare failures become commonplace when you run 30 of them 24/7 in parallel. For this reason some time ago we started looking for an alternative that would eventually replace them. And this was when we found the following candidate.
The candidate
We had a look at what Debian was using for their QA infrastructure and talked to the MIPS team – credits to Berto García who helped us with this – and we concluded that the Loongson 3B4000 MIPSel board was a promising option so we decided to explore it.
We started looking for information about this CPU model and we found references for the Loongson 3A4000 + Fuloong 2.0 Mini-PC. This computer is a kind of very interesting end-user product based on the MIPS64el architecture. In particular, this computer uses a similar but more recent and powerful evolution of the Loongson 3B4000 processor. The Fuloong 2.0 comes in a barebone format with the Loongson-3A R4 (Loongson-3A4000) @ 1500MHz, a quad-core processor, with 8GB DDR4 RAM and a 1TB NVMe of internal storage. These technical specifications are completed with a Realtek ALC662 sound card, 2x USB 3.0 ports + 1x USB Type-C + 4x USB 2.0, 2x HDMI video outputs, 2x Ethernet (WGI211AT), audio connectors, M.2 slot for WiFi module and, finally, a Vivante GL1000 GPU (OpenGL ES 2.0/1.1). This specifications are clearly far from the common constraints of the regular development MIPS boards and are technically a serious candidate for replacing the current boards used in the CI cluster.
However, the acquisition of this kind of products has some non-technical cons that is important to have in mind before taking any decision. For example, it is very difficult to find a reseller in Europe providing this kind of machines. This means that this computer needs to be directly shipped from China, which also means that the acquisition process can suffer from the common problems of this kind of orders: higher delivery time (~1 month), paperwork for customs, taxes, delivery tracking issues … Anyway, this post is intended to keep the focus on the technical details ;-). The fact is, once these issues are solved you will receive a machine similar to this one shown in the photos:
The unboxing
The machine comes with a pre-installed custom distro (“Dragon Dream F28”, based on Fedora 28). This distro is quite old but it is the one provided by the manufacturer (Lemote). Apparently it is the only one that, in theory, fully supports the machine. The installed image comes with a desktop environment on top of an X server. The distro is also synced with an RPM repository hosted by Lemote. This is really convenient to start experimenting with the computer and very useful to get information about the system before taking any action on the computer. Here is the output of some commands:
# cat /proc/cpuinfo system type : generic-loongson-machine machine : loongson,generic processor : 0 cpu model : Loongson-3 V0.4 FPU V0.1 model name : Loongson-3A R4 (Loongson-3A4000) @ 1500MHz CPU MHz : 1500.00 BogoMIPS : 2990.15 wait instruction : yes microsecond timers : yes tlb_entries : 2112 extra interrupt vector : no hardware watchpoint : no isa : mips1 mips2 mips3 mips4 mips5 mips32r1 mips32r2 mips64r1 mips64r2 ASEs implemented : vz msa loongson-mmi loongson-cam loongson-ext loongson-ext2 shadow register sets : 1 kscratch registers : 6 package : 0 core : 0 ... (x4)
… dmesg:
Mar 9 12:43:19 fuloong-01 kernel: [ 2.884260] Console: switching to colour frame buffer device 240x67 Mar 9 12:43:19 fuloong-01 kernel: [ 2.915928] loongson-drm 0000:00:06.1: fb0: loongson-drmdrm frame buffer device Mar 9 12:43:19 fuloong-01 kernel: [ 2.919792] etnaviv 0000:00:06.0: Device 14:7a15, irq 93 Mar 9 12:43:19 fuloong-01 kernel: [ 2.920249] etnaviv 0000:00:06.0: model: GC1000, revision: 5037 Mar 9 12:43:19 fuloong-01 kernel: [ 2.920378] [drm] Initialized etnaviv 1.3.0 20151214 for 0000:00:06.0 on minor 1
… lsblk:
# lsblk nvme0n1 259:0 0 477G 0 disk ├─nvme0n1p1 259:1 0 190M 0 part /boot/efi ├─nvme0n1p2 259:2 0 1,7G 0 part /boot ├─nvme0n1p3 259:3 0 7,5G 0 part [SWAP] ├─nvme0n1p4 259:4 0 46,6G 0 part / └─nvme0n1p5 259:5 0 421,1G 0 part /home
Getting Debian into the Fuloong 2.0
The WebKitGTK and WPE WebKit CI infrastructure is entirely based on Debian Stable and/or Ubuntu LTS. This is according to the WebKitGTK maintenance and development policy. For that reason we were pretty interested in getting the machine running with Debian Stable (“buster” as of this writing). So what comes next is the description of the installation process of a pure Debian base system hybridized with the Lemote Fedora Linux kernel using an external USB storage stick as the bootable disk. The process is a mix between the following two documents:
Those documents provide a good detailed explanation of the steps to follow to perform the installation. Only the installation of the kernel and the grub2-efi differs a bit but let’s come back to that later. The idea is:
- Set the EFI/BIOS to boot from the USB storage (EFI)
- Install the base Debian OS in a external microSD card connected to the USB3-SS port
- Keep using the internal nvme disk as the working dir (/home, /var/lib/lxc)
The installation process is initiated in the pre-installed Fedora image. The first action is to mount the external USB storage (sda) in the living system as follows:
# lsblk sda 8:0 1 14,9G 0 disk ├─sda1 8:1 1 200M 0 part /mnt/debinst/boot/efi └─sda2 8:2 1 10G 0 part /mnt/debinst nvme0n1 259:0 0 477G 0 disk ├─nvme0n1p1 259:1 0 190M 0 part /boot/efi ├─nvme0n1p2 259:2 0 1,7G 0 part /boot ├─nvme0n1p3 259:3 0 7,5G 0 part [SWAP] ├─nvme0n1p4 259:4 0 46,6G 0 part / └─nvme0n1p5 259:5 0 421,1G 0 part /home
As I said, the steps to install the Debian system into the SDcard are quite straightforward. The problems begins during the installation of GRUB and the Linux kernel …
The Linux Kernel
Having followed the guide we will reach the Install a Kernel step. Debian provides a Loongson Linux 4.19 kernel for the Loongson 3A/3B boards.
ii linux-image-4.19.0-14-loongson-3 4.19.171-2 mips64el Linux 4.19 for Loongson 3A/3B ii linux-image-loongson-3 4.19+105+deb10u9 mips64el Linux for Loongson 3A/3B (meta-package) ii linux-libc-dev:mips64el 4.19.171-2 mips64el Linux support headers for userspace development
It is quite old in comparison with the one that the Lemote Fedora distro contains (5.4.63-20201012-def) so I prefered to keep the one, although it should be possible to get the machine running with this kernel as well.
Grub2 EFI, first attempt trying to build it for the device
This is the main issue that I found. The first thing that I tried was to look for a GRUB package with EFI support in the mips64el Debian chroot:
root@fuloong-01:/# apt search grub | grep efi <<empty>>
The frustration came quickly when I didn’t find any GRUB candidate. It was then when I remembered that there was a grub-yeeloong package in the Debian repository that could be useful in this case. The Yeeloong is the predecessor of the Loongson so what I tried next was to rebuild the GRUB package but adding the mips64el architecture for the grub-yeeloong package. Something like the following:
- Getting the Debian sources and dependencies for the grub2 packages:
apt source grub2 apt install debhelper patchutils python flex bison po-debconf help2man texinfo xfonts-unifont libfreetype6-dev gettext libdevmapper-dev libsdl1.2-dev xorriso parted libfuse-dev ttf-dejavu-core liblzma-dev wamerican pkg-config bash-completion build-essentia
- Patching the /debian/control file using this patch
- … and then to build the Debian package:
~/debs# cd grub2-2.02+dfsg1 && dpkg-buildpackage
~/debs/grub2-2.02+dfsg1# ls ../ grub-common-dbgsym_2.02+dfsg1-20+deb10u3_mips64el.deb grub-yeeloong_2.02+dfsg1-20+deb10u3_mips64el.deb grub2_2.02+dfsg1-20+deb10u3.debian.tar.xz grub2_2.02+dfsg1.orig.tar.xz grub-common_2.02+dfsg1-20+deb10u3_mips64el.deb grub2-2.02+dfsg1 grub2_2.02+dfsg1-20+deb10u3.dsc grub-mount-udeb_2.02+dfsg1-20+deb10u3_mips64el.udeb grub2-common-dbgsym_2.02+dfsg1-20+deb10u3_mips64el.deb grub2_2.02+dfsg1-20+deb10u3_mips64el.buildinfo grub-yeeloong-bin_2.02+dfsg1-20+deb10u3_mips64el.deb grub2-common_2.02+dfsg1-20+deb10u3_mips64el.deb grub2_2.02+dfsg1-20+deb10u3_mips64el.changes
The .deb package is built correctly but the problem is the binary. It lacks EFI runtime support so it is not useful in our case:
******************************************************* GRUB2 will be compiled with following components: Platform: mipsel-none <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< With devmapper support: Yes With memory debugging: No With disk cache statistics: No With boot time statistics: No efiemu runtime: No (only available on i386) grub-mkfont: Yes grub-mount: Yes starfield theme: Yes With DejaVuSans font from /usr/share/fonts/truetype/ttf-dejavu/DejaVuSans.ttf With libzfs support: No (need zfs library) Build-time grub-mkfont: Yes With unifont from /usr/share/fonts/X11/misc/unifont.pcf.gz With liblzma from -llzma (support for XZ-compressed mips images) With quiet boot: No *******************************************************
This is what happens if you still try to install it:
root@fuloong-01:~/debs/grub2-2.02+dfsg1# dpkg -i ../grub-yeeloong-bin_2.02+dfsg1-20+deb10u3_mips64el.deb ../grub-common_2.02+dfsg1-20+deb10u3_mips64el.deb ../grub2-common_2.02+dfsg1-20+deb10u3_mips64el.deb
root@fuloong-01:~/debs/grub2-2.02+dfsg1# grub-install /dev/sda Installing for mipsel-loongson platform. ... grub-install: warning: WARNING: no platform-specific install was performed. <<<<<<<<<< Installation finished. No error reported.
There is not glue between EFI and GRUB. Files like BOOTMIPS.EFI, gcdmips64el.efi and grub.efi are missing so this is package is not useful at all:
root@fuloong-01:~/debs/grub2-2.02+dfsg1# ls /boot/ System.map-4.19.0-14-loongson-3 config-4.19.0-14-loongson-3 efi grub grub.elf initrd.img-4.19.0-14-loongson-3 vmlinux-4.19.0-14-loongson-3
root@fuloong-01:~/debs/grub2-2.02+dfsg1# ls /boot/grub fonts grubenv locale mipsel-loongson
root@fuloong-01:~/debs/grub2-2.02+dfsg1# ls /boot/efi/ <<empty>>
root@fuloong-01:~/debs/grub2-2.02+dfsg1# ls /boot/ System.map-4.19.0-14-loongson-3 config-4.19.0-14-loongson-3 efi grub grub.elf initrd.img-4.19.0-14-loongson-3 vmlinux-4.19.0-14-loongson-3
root@fuloong-01:~/debs/grub2-2.02+dfsg1# ls /boot/grub grub/ grub.elf
The grub-install command will also confirm that the mips64el-efi target is not supported:
root@fuloong-01:~/debs/grub2-2.02+dfsg1# /usr/sbin/grub-install --help Usage: grub-install [OPTION...] [OPTION] [INSTALL_DEVICE] Install GRUB on your drive. ... --target=TARGET install GRUB for TARGET platform [default=mipsel-loongson]; available targets: arm-efi, arm-uboot, arm64-efi, i386-coreboot, i386-efi, i386-ieee1275, i386-multiboot, i386-pc, i386-qemu, i386-xen, i386-xen_pvh, ia64-efi, mips-arc, mips-qemu_mips, mipsel-arc, mipsel-loongson, mipsel-qemu_mips, powerpc-ieee1275, sparc64-ieee1275, x86_64-efi, x86_64-xen
Second attempt, the loongson-community Grub2 EFI
Now that we know that we can not use an official Debian package to install and configure GRUB it is time for a bit of google-fu.
I must have a lot of practice since it only took me a short while to find that the Lemote Fedora distro provides its own GRUB package for the Loongson and, later, I found new hope reading this article. This article explains how to build the GRUB from loongson-community with EFI support so what I would do next was the obvious logical step: To try to build it and check it:
-
-
git clone https://github.com/loongson-community/grub.git cd grub bash autogen.sh ./configure --prefix=/opt/alternative/ make ; make install
- The configure output looks promising:
******************************************************* GRUB2 will be compiled with following components: Platform: mips64el-efi <<<<<<<<<<<<<<<<< Looks good. With devmapper support: Yes With memory debugging: No With disk cache statistics: No With boot time statistics: No efiemu runtime: No (not available on efi) grub-mkfont: No (need freetype2 library) grub-mount: Yes starfield theme: No (No build-time grub-mkfont) With libzfs support: No (need zfs library) Build-time grub-mkfont: No (need freetype2 library) Without unifont (no build-time grub-mkfont) With liblzma from -llzma (support for XZ-compressed mips images) *******************************************************
… but unfortunately I started to have more and more build errors in every step. Errors like these:
-
cc1: error: position-independent code requires ‘-mabicalls’
grub_script.yy.c:19:22: error: statement with no effect [-Werror=unused-value]
build-grub-module-verifier: error: unsupported relocation 0x51807.
… so after several attempts I finally gave up trying to build the loongson-community with GRUB EFI support. Here the patch with some of the modifications that I tried in the code just in case you are better at solving these build errors than me.
Third attempt, reusing the GRUB2 EFI resources from the pre-installed system
… and the last one.
My winner horse was the simpler solution: to reuse the /boot and /boot/efi directories installed in the Fedora system as base for a new Debian system:
-
- Clone the tree in the destination dir:
cp -a /boot /mnt/debinst/boot
- Replace the UUIDs patch
The /boot dir in the target installation will be look like this:
[root@fuloong-01 boot]# tree /mnt/debinst/boot/ /mnt/debinst/boot/ ├── boot -> . ├── config-5.4.60-1.fc28.lemote.mips64el ├── e8a27b4e4fcc4db9ab7a64bd81393773 │ └── 5.4.60-1.fc28.lemote.mips64el │ ├── initrd │ └── linux ├── efi │ ├── boot │ │ ├── grub.cfg │ │ └── grub.efi │ ├── EFI │ │ ├── BOOT │ │ │ ├── BOOTMIPS.EFI │ │ │ ├── fonts │ │ │ │ └── unicode.pf2 │ │ │ ├── gcdmips64el.efi │ │ │ ├── grub.cfg │ │ │ └── grubenv │ │ └── fedora │ ├── mach_kernel │ └── System │ └── Library │ └── CoreServices │ └── SystemVersion.plist ├── extlinux ├── grub2 │ ├── grubenv -> ../efi/EFI/BOOT/grubenv │ └── themes │ └── system │ ├── background.png │ └── fireworks.png ├── grub.cfg ├── grub.efi ├── initramfs-5.4.60-1.fc28.lemote.mips64el.img ├── loader │ └── entries │ └── e8a27b4e4fcc4db9ab7a64bd81393773-5.4.60-1.fc28.lemote.mips64el.conf ├── lost+found ├── System.map-5.4.60-1.fc28.lemote.mips64el ├── vmlinuz-205 └── vmlinuz-5.4.60-1.fc28.lemote.mips64el
- Clone the tree in the destination dir:
… et voilà!
Finally we have a pure Debian Buster root base system hybridized with the Lemote Fedora Linux kernel:
root@fuloong-01:~# cat /etc/debian_version 10.8
root@fuloong-01:~# uname -a Linux fuloong-01 5.4.60-1.fc28.lemote.mips64el #1 SMP PREEMPT Mon Aug 24 09:33:35 CST 2020 mips64 GNU/Linux
root@fuloong-01:~# cat /etc/apt/sources.list deb http://httpredir.debian.org/debian buster main contrib non-free deb-src http://httpredir.debian.org/debian buster main contrib non-free deb http://security.debian.org/ buster/updates main contrib non-free deb http://httpredir.debian.org/debian/ buster-updates main contrib non-free
root@fuloong-01:~# apt update Hit:1 http://httpredir.debian.org/debian buster InRelease Get:2 http://security.debian.org buster/updates InRelease [65,4 kB] Get:3 http://httpredir.debian.org/debian buster-updates InRelease [51,9 kB] Get:4 http://security.debian.org buster/updates/main mips64el Packages [242 kB] Get:5 http://security.debian.org buster/updates/main Translation-en [142 kB] Fetched 501 kB in 1s (417 kB/s) Reading package lists... Done Building dependency tree Reading state information... Done 3 packages can be upgraded. Run 'apt list --upgradable' to see them.
With this hardware we can reasonably run native GDB directly on it and have the possibility to run other tools in the host (e.g. you can run any monitoring agent on it to get stats and so). Definitely, having this hardware enabled for using it in the CI infrastructure will be a promising step towards a better QA for the project.
That is all from my side. I will probably continue dedicating some time to get buildable packages of GRUB-EFI and the Linux Kernel that we could use for this and similar machines (e.g. for tools like perf who needs to have the userspace binaries in sync with the kernel version). In the meantime, I really hope that this can be useful to someone out there who is interested in this hardware. If you have some comment or question or you simply wish to share your thoughts about this just leave a comment.
Stay safe!
503: Service Unavailable 