Running Debian in a Fuloong 2.0 Mini-PC (MIPS64el CPU Loongson 3A4000)

The history of the world is a continuous succession of  contradictions. The announcement from MIPS Technologies about their decision of definitely abandoning MIPS arch in favour of RISC-V is just another example. But the truth is that things are far from trivial in this topic. Even when the end-of-life date for the MIPS architecture looks closer in time than ever,  there are still infrastructures and platforms what need to keep being supported and maintained for this architecture in the meantime. To make the situation more complex, at the same time I am writing this post the Loongson Technology Ltd  is  announcing a new 16-Core MIPS 12nm CPU for 256-Core (Tom’s Hardware news). Loongson Technology also says that they keep a strong commitment with RISC-V for the future but they will keep their bet for MIPS64 in the meantime. So if MIPS is going to die it going be in lovely death.

In this context, here in Igalia we are hosting and maintaining the CI workers for the JavaScriptCore 32-bit (MIPS) infrastructure for the WebKit web browser engine.

Build worker for JavaScriptCore 32-bit (MIPS) host at Igalia

No one ever said that finding end-user hardware of this kind of system is easy-peasy. The options in the market often don’t achieve the sufficient level of maturity or come with a poor set of hardware specifications. The choices are often not intended for long time-consuming CPU tasks, or they simply lack good OS support (maintenance, updates, custom kernels, open-source drivers …).

Nowadays we are using a parallelized cluster of MIPSEL CI 20 boards to move the JavaScriptCore 32-bits (MIPS) CI workers. Don’t get me wrong: the CI 20 boards are certainly not bad. These boards are really great for development and evaluation purposes, but even rare failures become commonplace when you run 30 of them 24/7 in parallel. For this reason some time ago we started looking for an alternative that would eventually replace them. And this was when we found the following candidate.

The candidate

We had a look at what Debian was using for their QA infrastructure and talked to the MIPS team – credits to Berto García who helped us with this – and we concluded that the Loongson 3B4000 MIPSel board was a promising option so we decided to explore it.

We started looking for information about this CPU model and we found references for the Loongson 3A4000 + Fuloong 2.0 Mini-PC. This computer is a kind of very interesting end-user product based on the MIPS64el architecture. In particular, this computer uses a similar but more recent and powerful evolution of the Loongson 3B4000 processor. The Fuloong 2.0 comes in a barebone format with the Loongson-3A R4 (Loongson-3A4000) @ 1500MHz, a quad-core processor, with 8GB DDR4 RAM and a 1TB NVMe of internal storage. These technical specifications are completed with a Realtek ALC662 sound card, 2x USB 3.0 ports + 1x USB Type-C + 4x USB 2.0, 2x HDMI video outputs, 2x Ethernet (WGI211AT), audio connectors, M.2 slot for WiFi module and, finally, a Vivante GL1000 GPU (OpenGL ES 2.0/1.1). This specifications are clearly far from the common constraints of the regular development MIPS boards and are technically a serious candidate for replacing the current boards used in the CI cluster.

However, the acquisition of this kind of products has some non-technical cons that is important to have in mind before taking any decision. For example, it is very difficult to find a reseller in Europe providing this kind of machines. This means that this computer needs to be directly shipped from China, which also means that the acquisition process can suffer from the common problems of this kind of orders: higher delivery time (~1 month), paperwork for customs, taxes, delivery tracking issues … Anyway, this post is intended to keep the focus on the technical details ;-). The fact is, once these issues are solved you will receive a machine similar to this one shown in the photos:

The unboxing

The machine comes with a pre-installed custom distro (“Dragon Dream F28”, based on Fedora 28). This distro is quite old but it is the one provided by the manufacturer (Lemote). Apparently it is the only one that, in theory, fully supports the machine. The installed image comes with a desktop environment on top of an X server. The distro is also synced with an RPM repository hosted by Lemote. This is really convenient to start experimenting with the computer and very useful to get information about the system before taking any action on the computer. Here is the output of some commands:

# cat /proc/cpuinfo
system type : generic-loongson-machine
machine : loongson,generic
processor : 0
cpu model : Loongson-3 V0.4 FPU V0.1
model name : Loongson-3A R4 (Loongson-3A4000) @ 1500MHz
CPU MHz : 1500.00
BogoMIPS : 2990.15
wait instruction : yes
microsecond timers : yes
tlb_entries : 2112
extra interrupt vector : no
hardware watchpoint : no
isa : mips1 mips2 mips3 mips4 mips5 mips32r1 mips32r2 mips64r1 mips64r2
ASEs implemented : vz msa loongson-mmi loongson-cam loongson-ext loongson-ext2
shadow register sets : 1
kscratch registers : 6
package : 0
core : 0
... (x4)

dmesg:

Mar 9 12:43:19 fuloong-01 kernel: [ 2.884260] Console: switching to colour frame buffer device 240x67 
Mar 9 12:43:19 fuloong-01 kernel: [ 2.915928] loongson-drm 0000:00:06.1: fb0: loongson-drmdrm frame buffer device 
Mar 9 12:43:19 fuloong-01 kernel: [ 2.919792] etnaviv 0000:00:06.0: Device 14:7a15, irq 93 
Mar 9 12:43:19 fuloong-01 kernel: [ 2.920249] etnaviv 0000:00:06.0: model: GC1000, revision: 5037 
Mar 9 12:43:19 fuloong-01 kernel: [ 2.920378] [drm] Initialized etnaviv 1.3.0 20151214 for 0000:00:06.0 on minor 1

lsblk:

# lsblk
nvme0n1 259:0 0 477G 0 disk
├─nvme0n1p1 259:1 0 190M 0 part /boot/efi
├─nvme0n1p2 259:2 0 1,7G 0 part /boot
├─nvme0n1p3 259:3 0 7,5G 0 part [SWAP]
├─nvme0n1p4 259:4 0 46,6G 0 part /
└─nvme0n1p5 259:5 0 421,1G 0 part /home

Getting Debian into the Fuloong 2.0

The WebKitGTK and WPE WebKit CI infrastructure is entirely based on Debian Stable and/or Ubuntu LTS. This is according to the WebKitGTK maintenance and development policy. For that reason we were pretty interested in getting the machine running with Debian Stable (“buster” as of this writing). So what comes next is the description of the installation process of a pure Debian base system hybridized with the Lemote Fedora Linux kernel using an external USB storage stick as the bootable disk. The process is a mix between the following two documents:

Those documents provide a good detailed explanation of the steps to follow to perform the installation. Only the installation of the kernel and the grub2-efi differs a bit but let’s come back to that later. The idea is:

  • Set the EFI/BIOS to boot from the USB storage (EFI)
  • Install the base Debian OS in a external microSD card connected to the USB3-SS port
  • Keep using the internal nvme disk as the working dir (/home, /var/lib/lxc)

The installation process is initiated in the pre-installed Fedora image. The first action is to mount the external USB storage (sda) in the living system as follows:

# lsblk
sda 8:0 1 14,9G 0 disk
├─sda1 8:1 1 200M 0 part /mnt/debinst/boot/efi
└─sda2 8:2 1 10G 0 part /mnt/debinst
nvme0n1 259:0 0 477G 0 disk
├─nvme0n1p1 259:1 0 190M 0 part /boot/efi
├─nvme0n1p2 259:2 0 1,7G 0 part /boot
├─nvme0n1p3 259:3 0 7,5G 0 part [SWAP]
├─nvme0n1p4 259:4 0 46,6G 0 part /
└─nvme0n1p5 259:5 0 421,1G 0 part /home

As I said, the steps to install the Debian system into the SDcard are quite straightforward. The problems begins during the installation of GRUB and the Linux kernel …

The Linux Kernel

Having followed the guide we will reach the Install a Kernel step. Debian provides a Loongson Linux 4.19 kernel for the Loongson 3A/3B boards.

ii linux-image-4.19.0-14-loongson-3 4.19.171-2 mips64el Linux 4.19 for Loongson 3A/3B
ii linux-image-loongson-3 4.19+105+deb10u9 mips64el Linux for Loongson 3A/3B (meta-package)
ii linux-libc-dev:mips64el 4.19.171-2 mips64el Linux support headers for userspace development

It is quite old in comparison with the one that the Lemote Fedora distro contains (5.4.63-20201012-def) so I prefered to keep the one, although it should be possible to get the machine running with this kernel as well.

Grub2 EFI, first attempt trying to build it for the device

This is the main issue that I found. The first thing that I tried was to look for a GRUB package with EFI support in the mips64el Debian chroot:

root@fuloong-01:/# apt search grub | grep efi
<<empty>>

The frustration came quickly when I didn’t find any GRUB candidate. It was then when I remembered that there was a grub-yeeloong package in the Debian repository that could be useful in this case. The Yeeloong is the predecessor of the Loongson so what I tried next was to rebuild the GRUB package but adding the mips64el architecture for the grub-yeeloong package. Something like the following:

  • Getting the Debian sources and dependencies for the grub2 packages:
    apt source grub2
    apt install debhelper patchutils python flex bison po-debconf help2man texinfo xfonts-unifont libfreetype6-dev gettext libdevmapper-dev libsdl1.2-dev xorriso parted libfuse-dev ttf-dejavu-core liblzma-dev wamerican pkg-config bash-completion build-essentia
    
  • Patching the /debian/control file using this patch
  • … and then to build the Debian package:
    ~/debs# cd grub2-2.02+dfsg1 && dpkg-buildpackage
    
    ~/debs/grub2-2.02+dfsg1# ls ../
    grub-common-dbgsym_2.02+dfsg1-20+deb10u3_mips64el.deb grub-yeeloong_2.02+dfsg1-20+deb10u3_mips64el.deb grub2_2.02+dfsg1-20+deb10u3.debian.tar.xz grub2_2.02+dfsg1.orig.tar.xz
    grub-common_2.02+dfsg1-20+deb10u3_mips64el.deb grub2-2.02+dfsg1 grub2_2.02+dfsg1-20+deb10u3.dsc
    grub-mount-udeb_2.02+dfsg1-20+deb10u3_mips64el.udeb grub2-common-dbgsym_2.02+dfsg1-20+deb10u3_mips64el.deb grub2_2.02+dfsg1-20+deb10u3_mips64el.buildinfo
    grub-yeeloong-bin_2.02+dfsg1-20+deb10u3_mips64el.deb grub2-common_2.02+dfsg1-20+deb10u3_mips64el.deb grub2_2.02+dfsg1-20+deb10u3_mips64el.changes
    

The .deb package is built correctly but the problem is the binary. It lacks EFI runtime support so it is not useful in our case:

*******************************************************
GRUB2 will be compiled with following components:
Platform: mipsel-none <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
With devmapper support: Yes
With memory debugging: No
With disk cache statistics: No
With boot time statistics: No
efiemu runtime: No (only available on i386)
grub-mkfont: Yes
grub-mount: Yes
starfield theme: Yes
With DejaVuSans font from /usr/share/fonts/truetype/ttf-dejavu/DejaVuSans.ttf
With libzfs support: No (need zfs library)
Build-time grub-mkfont: Yes
With unifont from /usr/share/fonts/X11/misc/unifont.pcf.gz
With liblzma from -llzma (support for XZ-compressed mips images)
With quiet boot: No
*******************************************************

This is what happens if you still try to install it:

root@fuloong-01:~/debs/grub2-2.02+dfsg1# dpkg -i ../grub-yeeloong-bin_2.02+dfsg1-20+deb10u3_mips64el.deb ../grub-common_2.02+dfsg1-20+deb10u3_mips64el.deb ../grub2-common_2.02+dfsg1-20+deb10u3_mips64el.deb
root@fuloong-01:~/debs/grub2-2.02+dfsg1# grub-install /dev/sda
Installing for mipsel-loongson platform.
...
grub-install: warning: WARNING: no platform-specific install was performed. <<<<<<<<<<
Installation finished. No error reported.

There is not glue between EFI and GRUB. Files like BOOTMIPS.EFI, gcdmips64el.efi and grub.efi are missing so this is package is not useful at all:

root@fuloong-01:~/debs/grub2-2.02+dfsg1# ls /boot/
System.map-4.19.0-14-loongson-3 config-4.19.0-14-loongson-3 efi grub grub.elf initrd.img-4.19.0-14-loongson-3 vmlinux-4.19.0-14-loongson-3
root@fuloong-01:~/debs/grub2-2.02+dfsg1# ls /boot/grub
fonts grubenv locale mipsel-loongson
root@fuloong-01:~/debs/grub2-2.02+dfsg1# ls /boot/efi/
<<empty>>
root@fuloong-01:~/debs/grub2-2.02+dfsg1# ls /boot/
System.map-4.19.0-14-loongson-3 config-4.19.0-14-loongson-3 efi grub grub.elf initrd.img-4.19.0-14-loongson-3 vmlinux-4.19.0-14-loongson-3
root@fuloong-01:~/debs/grub2-2.02+dfsg1# ls /boot/grub
grub/ grub.elf

The grub-install command will also confirm that the mips64el-efi target is not supported:

root@fuloong-01:~/debs/grub2-2.02+dfsg1# /usr/sbin/grub-install --help
Usage: grub-install [OPTION...] [OPTION] [INSTALL_DEVICE]
Install GRUB on your drive.
...
--target=TARGET install GRUB for TARGET platform
[default=mipsel-loongson]; available targets:
arm-efi, arm-uboot, arm64-efi, i386-coreboot,
i386-efi, i386-ieee1275, i386-multiboot, i386-pc,
i386-qemu, i386-xen, i386-xen_pvh, ia64-efi,
mips-arc, mips-qemu_mips, mipsel-arc,
mipsel-loongson, mipsel-qemu_mips,
powerpc-ieee1275, sparc64-ieee1275, x86_64-efi,
x86_64-xen

Second attempt, the loongson-community Grub2 EFI

Now that we know that we can not use an official Debian package to install and configure GRUB it is time for a bit of google-fu.

I must have a lot of practice since it only took me a short while to find that the Lemote Fedora distro provides its own GRUB package for the Loongson and, later, I found new hope reading this article. This article explains how to build the GRUB from loongson-community with EFI support so what I would do next was the obvious logical step: To try to build it and check it:

    • git clone https://github.com/loongson-community/grub.git
      cd grub
      bash autogen.sh
      ./configure --prefix=/opt/alternative/
      make ; make install
    • The configure output looks promising:
      *******************************************************
      GRUB2 will be compiled with following components:
      Platform: mips64el-efi <<<<<<<<<<<<<<<<< Looks good.
      With devmapper support: Yes
      With memory debugging: No
      With disk cache statistics: No
      With boot time statistics: No
      efiemu runtime: No (not available on efi)
      grub-mkfont: No (need freetype2 library)
      grub-mount: Yes
      starfield theme: No (No build-time grub-mkfont)
      With libzfs support: No (need zfs library)
      Build-time grub-mkfont: No (need freetype2 library)
      Without unifont (no build-time grub-mkfont)
      With liblzma from -llzma (support for XZ-compressed mips images)
      *******************************************************

    … but unfortunately I started to have more and more build errors in every step. Errors like these:

cc1: error: position-independent code requires ‘-mabicalls’
grub_script.yy.c:19:22: error: statement with no effect [-Werror=unused-value]
build-grub-module-verifier: error: unsupported relocation 0x51807.

… so after several attempts I finally gave up trying to build the loongson-community with GRUB EFI support. Here the patch with some of the modifications that I tried in the code just in case you are better at solving these build errors than me.

Third attempt, reusing the GRUB2 EFI resources from the pre-installed system

… and the last one.

My winner horse was the simpler solution: to reuse the /boot and /boot/efi directories installed in the Fedora system as base for a new Debian system:

    • Clone the tree in the destination dir:
      cp -a /boot /mnt/debinst/boot
    • Replace the UUIDs patch

    The /boot dir in the target installation will be look like this:

    [root@fuloong-01 boot]# tree /mnt/debinst/boot/
    /mnt/debinst/boot/
    ├── boot -> .
    ├── config-5.4.60-1.fc28.lemote.mips64el
    ├── e8a27b4e4fcc4db9ab7a64bd81393773
    │   └── 5.4.60-1.fc28.lemote.mips64el
    │   ├── initrd
    │   └── linux
    ├── efi
    │   ├── boot
    │   │   ├── grub.cfg
    │   │   └── grub.efi
    │   ├── EFI
    │   │   ├── BOOT
    │   │   │   ├── BOOTMIPS.EFI
    │   │   │   ├── fonts
    │   │   │   │   └── unicode.pf2
    │   │   │   ├── gcdmips64el.efi
    │   │   │   ├── grub.cfg
    │   │   │   └── grubenv
    │   │   └── fedora
    │   ├── mach_kernel
    │   └── System
    │   └── Library
    │   └── CoreServices
    │   └── SystemVersion.plist
    ├── extlinux
    ├── grub2
    │   ├── grubenv -> ../efi/EFI/BOOT/grubenv
    │   └── themes
    │   └── system
    │   ├── background.png
    │   └── fireworks.png
    ├── grub.cfg
    ├── grub.efi
    ├── initramfs-5.4.60-1.fc28.lemote.mips64el.img
    ├── loader
    │   └── entries
    │   └── e8a27b4e4fcc4db9ab7a64bd81393773-5.4.60-1.fc28.lemote.mips64el.conf
    ├── lost+found
    ├── System.map-5.4.60-1.fc28.lemote.mips64el
    ├── vmlinuz-205
    └── vmlinuz-5.4.60-1.fc28.lemote.mips64el

… et voilà!

Finally we have a pure Debian Buster root base system hybridized with the Lemote Fedora Linux kernel:

root@fuloong-01:~# cat /etc/debian_version
10.8
root@fuloong-01:~# uname -a
Linux fuloong-01 5.4.60-1.fc28.lemote.mips64el #1 SMP PREEMPT Mon Aug 24 09:33:35 CST 2020 mips64 GNU/Linux
root@fuloong-01:~# cat /etc/apt/sources.list
deb http://httpredir.debian.org/debian buster main contrib non-free 
deb-src http://httpredir.debian.org/debian buster main contrib non-free 
deb http://security.debian.org/ buster/updates main contrib non-free 
deb http://httpredir.debian.org/debian/ buster-updates main contrib non-free
root@fuloong-01:~# apt update
Hit:1 http://httpredir.debian.org/debian buster InRelease
Get:2 http://security.debian.org buster/updates InRelease [65,4 kB]
Get:3 http://httpredir.debian.org/debian buster-updates InRelease [51,9 kB]
Get:4 http://security.debian.org buster/updates/main mips64el Packages [242 kB]
Get:5 http://security.debian.org buster/updates/main Translation-en [142 kB]
Fetched 501 kB in 1s (417 kB/s)                                
Reading package lists... Done
Building dependency tree       
Reading state information... Done
3 packages can be upgraded. Run 'apt list --upgradable' to see them.

With this hardware we can reasonably run native GDB directly on it and have the possibility to run other tools in the host (e.g. you can run any monitoring agent on it to get stats and so). Definitely, having this hardware enabled for using it in the CI infrastructure will be a promising step towards a better QA for the project.
That is all from my side. I will probably continue dedicating some time to get buildable packages of GRUB-EFI and the Linux Kernel that we could use for this and similar machines (e.g. for tools like perf who needs to have the userspace binaries in sync with the kernel version). In the meantime, I really hope that this can be useful to someone out there who is interested in this hardware. If you have some comment or question or you simply wish to share your thoughts about this just leave a comment.

Stay safe!

Building Chromium in MacOS with a Linux icecc cluster

Many times, during these last months, I thought to keep updated my blog writing a new post. Unfortunately, for one or another reason I always found an excuse to not do so. Well, I think that time is over because finally I found something useful and worthy the time spent time on the writing.

– That is OK but … what are you talking about?.
– Be patient Pablo, if you didn’t skip the headline of the post you already know about what I’m talking, probably :-).

Yes, I’m talking about how to setup a MacPro computer into a icecc cluster based on Linux hosts to take advantage of those to get more CPU power to build heavy software projects, like Chromium,  faster. The idea besides this is to distribute all the computational work over Linux nodes (fairly cheaper than any Mac) requested for cross-compiling tasks from the Mac host.

I’ve been working as a sysadmin at Igalia for the last couple of years. One of my duties here is to support and improve the developers building infrastructures. Recently we’ve faced long building times for heavy software projects like, for instance, Chromium. In this context, one of the main  issues that I had to solve is  how to build Chromium for MacOS in a reasonable time and avoiding to spend a lot of money in expensive bleeding edge Apple’s hardware to get CPU power.

This is what this post is about. This is an explanation about how to configure a Mac Pro to use a Linux based icecc cluster to boost the building times using cross-compilation. For simplicity, the explanation is focused in the singular case of just one single Linux host as icecc node and just one MacOS host requesting for compiling tasks but, in any case, you can extrapolate the instructions provided here to have many nodes as you need.

So let’s go with the the explanation but, first of all, a summary for those who want to go directly to the minimal and essential information …

TL;DR

On the Linux host:

# Configure the iceccd
$ sudo apt install icecc
$ sudo systemctl enable icecc-scheduler
$ edit /etc/icecc/icecc.conf
ICECC_MAX_JOBS="32"
ICECC_ALLOW_REMOTE="yes"
ICECC_SCHEDULER_HOST="192.168.1.10"
$ sudo systemctl restart icecc

# Generate the clang cross-compiling toolchain
$ sudo apt install build-essential icecc
$ git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git ~/depot_tools
$ export PATH=$PATH:~/depot_tools
$ git clone https://github.com/psaavedra/chromium_clang_darwin_on_linux ~/chromium_clang_darwin_on_linux
$ cd ~/chromium_clang_darwin_on_linux
$ export CLANG_REVISION=332838  # or CLANG_REVISION=$(./get-chromium-clang-revision)
$ ./icecc-create-darwin-env
# copy the clang_darwin_on_linux_332838.tar.gz to your MacOS host

On the Mac:

# Configure the iceccd
$ git clone https://github.com/darktears/icecream-mac.git ~/icecream-mac/
$ sudo ~/icecream-mac/install.sh 192.168.1.10
$ launchctl load /Library/LaunchDaemons/org.icecream.iceccd.plist
$ launchctl start /Library/LaunchDaemons/org.icecream.iceccd.plist

# Set the ICECC env vars
$ export ICECC_CLANG_REMOTE_CPP=1
$ export ICECC_VERSION=x86_64:~/clang_darwin_on_linux_332838.tar.gz
$ export PATH=~/icecream-mac/bin/icecc/:$PATH

# Get the depot_tools
$ cd ~
$ git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
$ export PATH=$PATH:~/depot_tools

# Download and build the Chromium sources
$ cd chromium && fetch chromium && cd src
$ gn gen out/Default --args='cc_wrapper="icecc" \
  treat_warnings_as_errors=false \
  clang_use_chrome_plugins=false \
  use_debug_fission=false \
  linux_use_bundled_binutils=false \
  use_lld=false \
  use_jumbo_build=true'
$ ninja -j 32 -C out/Default chrome

… and now the detailed explanation

Installation and setup of icecream on Linux hosts

The installation of icecream on a  Debian based Linux host is pretty simple. The latest version (1.1) for icecc is available in Debian testing and sid for a while so everything that you must to do is install it from the APT repositories. For case of stretch, there is a backport available  in the apt.igalia.com repository publically available:

sudo apt install icecc

The second important part of a icecc cluster is the icecc-scheduler. This daemon is in charge to route the requests from the icecc nodes which requiring available CPUs  for compiling to the nodes of the icecc cluster allowed to run remote build jobs.

In this setup we will activate the scheduler in the Linux node (192.168.1.10). The key here is that only one scheduler should be up at the same time in the same network to avoid errors in the cluster.

sudo systemctl enable icecc-scheduler

Once the scheduler is configured and up, it is time to add icecc hosts to the cluster. We will start adding the Linux hosts following this idea:

  • The IP of the icecc scheduler is 192.168.1.10
  • The Linux host is allowed to run remote jobs
  • The Linux host is allowed to run up to 32 concurrent jobs (this is arbitrary decision and can be adjusted per each particular host)
    # edit /etc/icecc/icecc.conf
    ICECC_NICE_LEVEL="5"
    ICECC_LOG_FILE="/var/log/iceccd.log"
    ICECC_NETNAME=""
    ICECC_MAX_JOBS="32"
    ICECC_ALLOW_REMOTE="yes"
    ICECC_BASEDIR="/var/cache/icecc"
    ICECC_SCHEDULER_LOG_FILE="/var/log/icecc_scheduler.log"
    ICECC_SCHEDULER_HOST="192.168.1.10"

We will need to restart the service to apply those changes:

sudo systemctl restart icecc

Installing and setup of icecream on MacOS hosts

The next step is to install and configure the icecc service on our Mac.  The easy way to get icecc available on Mac is icecream-mac project from darktears. We will do the installation assuming the following facts:

  • The local user account in Mac is psaavedra
  • The IP of the icecc scheduler is 192.168.1.10
  • The Mac is not allowed to accept remote jobs
  • We don’t want run use the Mac as worker.

To get the icecream-mac software we will make a git-clone of the project on Github:

git clone https://github.com/darktears/icecream-mac.git /Users/psaavedra/icecream-mac/
sudo /Users/psaavedra/icecream-mac/install.sh 192.168.1.10

We will edit a bit the /Library/LaunchDaemons/org.icecream.iceccd.plist daemon definition as follows:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>org.icecream.iceccd</string>
    <key>ProgramArguments</key>
    <array>
      <string>/Users/psaavedra/icecream-mac/bin/icecc/iceccd</string>
      <string>-s</string>
      <string>192.168.1.10</string>
      <string>-m</string>
      <string>2</string>
      <string>--no-remote</string>
    </array>
    <key>KeepAlive</key>
    <true/>
    <key>UserName</key>
    <string>root</string>
  </dict>
</plist>

Note that we are setting 2 workers in the Mac. Those workers are needed to execute threads in the host client host for things like linking … We will reload the service with this configuration:

launchctl load /Library/LaunchDaemons/org.icecream.iceccd.plist
launchctl start /Library/LaunchDaemons/org.icecream.iceccd.plist

Getting the cross-compilation toolchain for the icecream-mac

We already have the icecc cluster configured but, before to start to build Chromium on MacOS using icecc, there is still something before to do. We still need a cross-compiled clang for Darwin on Linux and, to avoid incompatibilities between versions, we need a clang based on the very same version that your Chromium code to be compiled.

You can check and get the cross-compilation clang revision that you need as follows:

cd src
CLANG_REVISION=$(cat tools/clang/scripts/update.py | grep CLANG_REVISION | head -n 1 | cut -d "'" -f 2)
echo $CLANG_REVISION
332838

In order to simplify this step.  I made some scripts which make it easy the generation of this clang cross-compiled toolchain. On a Linux host:

  • Install build depends:
    sudo apt install build-essential icecc
  • Get the Chromium project depot tools
    git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git ~/depot_tools  
    export PATH=$PATH:~/depot_tools
  • Download the psaavedra’s scripts (yes, my scripts):
    git clone https://github.com/psaavedra/chromium_clang_darwin_on_linux ~/chromium_clang_darwin_on_linux
    cd ~/chromium_clang_darwin_on_linux
  • You can use the get-chromium-clang-revision script to get the latest clang revision using in Chromium master:
    ./get-chromium-clang-revision
  • and then, to build the cross-compiled toolchain:
    ./icecc-create-darwin-env

    ; this script encapsulates the download, configure and build of the clang software.

  • A clang_darwin_on_linux_999999.tar.gz file will be generated.

Setup the icecc environment variables

Once you have the /Users/psaavedra/clang_darwin_on_linux_332838.tar.gz generated in your MacOS. You are ready to set the icecc environments variables.

export ICECC_CLANG_REMOTE_CPP=1
export ICECC_VERSION=x86_64:/Users/psaavedra/clang_darwin_on_linux_332838.tar.gz

The first variable enables the usage of the remote clang for C++. The second one establish toolchain to use by the x86_64 (Linux nodes) to build the code sent from the Mac.

Finally, remember to add the icecc binaries to the $PATH:

export PATH=/Users/psaavedra/icecream-mac/bin/icecc/:$PATH

You can check and get the cross-compiled clang revision that you need as follows:

cd src
CLANG_REVISION=$(cat tools/clang/scripts/update.py | grep CLANG_REVISION | head -n 1 | cut -d "'" -f 2)
echo $CLANG_REVISION
332838

… and building Chromium, at last

Reached this point, it’s time to build a Chromium using the icecc cluster and the cross-compiled clang toolchain previously created. These steps follows the official Chromium build procedure and only adapted to setup the icecc wrapper.

Ensure depot_tools is the path:

cd ~git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
export PATH=$PATH:~/depot_tools
ninja --version
# 1.8.2

Get the code:

git config --global core.precomposeUnicode truemkdir chromium
cd chromium
fetch chromium

Configure the build:

cd src
gn gen out/Default --args='cc_wrapper="icecc" treat_warnings_as_errors=false clang_use_chrome_plugins=false linux_use_bundled_binutils=false use_jumbo_build=true'
# or with ccache
export CCACHE_PREFIX=icecc
gn gen out/Default --args='cc_wrapper="ccache" treat_warnings_as_errors=false clang_use_chrome_plugins=false linux_use_bundled_binutils=false use_jumbo_build=true'

And build, at last:

ninja -j 32 -C out/Default chrome

icemon allows you to graphically monitoring the icecc cluster. Run it in remote from your Linux host if you don’t want install it in the MacOS:

ssh -X user@yourlinuxbox icemon

; with icemon you should see how each build task is distributed across the icecc cluster.

icemon

Let’s encrypt!

“The Let’s Encrypt project aims to make encrypted connections to World Wide Web servers ubiquitous. By getting rid of payment, web server configuration, validation emails, and dealing with expired certificates it is meant to significantly lower the complexity of setting up and maintaining TLS encryption.”Wikipedia

Let’s Encrypt is a service provided by the Internet Security Research Group (ISRG), a public benefit organization. Major sponsors are the Electronic Frontier Foundation (EFF), the Mozilla Foundation, Akamai, and Cisco Systems. Other partners include the certificate authority IdenTrust, the University of Michigan (U-M), the Stanford Law School, the Linux Foundation as well as Stephen Kent from Raytheon/BBN Technologies and Alex Polvi from CoreOS.

So, in summary, with Let’s Encrypt you will be able to request for a valid SSL certificate, free of charges, issued for a recognized CA and avoiding the mess of the mails, requests, forms typical of the webpages of the classic CA issuers.

The rest of the article shows the required steps needed for a complete setup of a Nginx server using Let’s encrypt issued certificates:

Install letsencrypt setup scripts:

cd /usr/local/sbin
sudo wget https://dl.eff.org/certbot-auto
sudo chmod a+x /usr/local/sbin/certbot-auto

Preparing the system for it:

sudo mkdir /var/www/letsencrypt
sudo chgrp www-data /var/www/letsencrypt

A special directory must be accesible for certificate issuer:

root /var/www/letsencrypt/;
location ~ /.well-known {
allow all;
}

Service restart. Let’s encrypt will be able to access to this domain and verify the athenticity of the request:

sudo service nginx restart

Note: At this point you need a valid A DNS entry pointing to the nginx server host. In my example, I will use the mydomain.com domain as an example.

Requesting for a valid certificate for my domain:

sudo certbot-auto certonly -a webroot --webroot-path=/var/www/letsencrypt -d mydomain.com

If everything is ok, you will get a valid certificate issued for your domain:

/etc/letsencrypt/live/mydomain.com/fullchain.pem

You will need a Diffie-Hellman PEM file for the crypthographic key exchange:

sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048

Setting the nginx virtual host with SSL as usual but using the Let’s encrypt issued certificate:

...
ssl on;
ssl_certificate /etc/letsencrypt/live/mydomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mydomain.com/privkey.pem;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/ssl/certs/dhparam.pem;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 5m;

add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
...

Check the new configuration and restarting the Nginx:

nginx -t
sudo service nginx restart

Renewing the issued certitificate:

cat >> EOF > /etc/cron.d/letsencrypt
30 2 * * 1 root /usr/local/sbin/certbot-auto renew >> /var/log/le-renew.log
35 2 * * 1 root /etc/init.d/nginx reload
EOF

The NFS 16 groups limit issue

The last Friday I was involved in a curious situation trying to setup a NFS server. The NFS server was mounted in UNIX server which was using UNIX users accounts assigned to many groups. These users were using files and directories stored in the NFS server.

As brief description of the situación which incites this post, I will say that the problem occurs when you are using UNIX users which are assigned in more than 16 UNIX groups. In this scenario, if you are using NFS (whatever version) with the UNIX system authentication (AUTH_SYS), quite common nowadays in spite of the security recommendations, you will get a permission denied during the access to certain arbitrary files and directories. The reason is that the list of secondary groups assigned to the user is truncated by the AUTH_SYS implementation. That is simple amazing!

Well, to be honest, this is not an unknown NFS problem. This limitation is here, around us, since the early stages of the modern computing technology. After a quick search on Internet, I found the reason why this happens and it is not a NFS limitation but it is a limit specified on AUTH_SYS:

   The client may wish to identify itself, for example, as it is
   identified on a UNIX(tm) system.  The flavor of the client credential
   is "AUTH_SYS".  The opaque data constituting the credential encodes
   the following structure:

         struct authsys_parms {
            unsigned int stamp;
            string machinename<255>;
            unsigned int uid;
            unsigned int gid;
            unsigned int gids<16>;
         };

The root cause

AUTH_SYS is the historical method which is used by client programs contacting an RPC server need. This allows the server get information about how the client should be able to access, and what functions should be allowed. Without authentication, any client on the network that can send packets to the RPC server could access any function.

AUTH_SYS has been in use for years in many systems just because it was the first authentication method available but AUTH_SYS is not a secure authentication method nowadays. In AUTH_SYS, the RPC client sends the UNIX UID and GIDs for the user, the server implicitly trusts that the user is who the user claims to be. All the this information is sent through the network without any kind of encryption and authentication, so it is high vulnerable.

In consequence, AUTH_SYS is an insecure security mode. The result is this can be used as the proverbial open lock on a door. Overall  the technical articles about these matters highly suggest the usage of other alternatives like NFSv4 (even NFSv3) and Kerberos, but  yet AUTH_SYS is commonly used within companies, so we must still deal it.

Note: This article didn’t focus in security issues. The main purpose of this article is describe a specific situation and show the possible alternatives identified during the troubleshooting of the issue.

Taking up the thread …

I was profiling a situation where the main issue was leaded by a UNIX secondary groups list truncation. Before continue, some summary of the context here: A UNIX user has a primary group, defined in the passwd database, but can also be a member of many other groups, defined in the group database. A UNIX system hardcoded  a limit of 16 groups that a user can be a member of (source). This means that clients into UNIX groups only be able to access to 16 groups. Quite poor when you deal with dozens and dozens of groups.

As we already know, the problem is focused in the NFS fulfilment with the AUTH_SYS specifications, which has an in-kernel data structure where the groups a user has access to is hardcoded as an array of 16 identifiers (gids). Even though Linux now supports 65536 groups, it is still not possible to operate on more than 16 from userland.

My scenario …

at this moment, I had identified this same situation in my case. I had users assigned to more than 16 secondary groups, I had a service using a NFS for the data storage but, in addition, I had some more extra furnitures in the room:

  • Users of the service are actual UNIX accounts. The authorization to for the file accessing is delegated to the own UINIX system
  • I hadn’t got a common LDAP server sharing the uids and gids
  • The NFS service wasn’t under my control

; this last point turned my case a little bit more miserable as we will see later.

 Getting information from Internet …

first of all, a brief analysis of the situation is always welcome:

– What is the actual problem? This problem occurs when a user, who is a member of more than 16 groups, tries to access a file or directory on an nfs mount that depends on his group rights in order to be authorized to see it.  Isn’t it?
– Yes!
– So, whatever thing that you do should be starting by asking on Google. If the issue was present for all those years, the solution should be also present.
– Good idea! – I told concluding the dialog with myself.

After a couple of minutes I had a completed list of articles, mail archives, forums and blog posts which throw up all kind of information about the problem. All of them talked about the most of the points introduced up to this point in this article. More or less interesting each one, one of them sticked out respect the others. It was the solving-the-nfs-16-group-limit-problem posted article from the xkyle.com blog.

The solving-the-nfs-16-group-limit-problem article describes a similar situation and offers it own conclusions. I must admit that I am pretty aligned with these conclusions and I would recommend this post for a deep reading.

The silver bullet

This solution is the best case. If you have the control of the NFS and you are running a Linux kernel 2.6.21 at least. This kernel or newer supports a NFS feature with allows ignore the gids sent by the RPC operations, instead of uses the local gids assigned to the uid from the local server:

-g or --manage-gids
Accept requests from the kernel to map user id numbers into lists of group id numbers for use in access control. An NFS request will normally (except when using Kerberos or other cryptographic authentication) contains a user-id and a list of group-ids. Due to a limitation in the NFS protocol, at most 16 groups ids can be listed. If you use the -g flag, then the list of group ids received from the client will be replaced by a list of group ids determined by an appropriate lookup on the server. Note that the 'primary' group id is not affected so a newgroup command on the client will still be effective. This function requires a Linux Kernel with version at least 2.6.21.

The key for this solution is get synchronized the ids between the client and the server. A common solution for this last requirement it is a common Name Service Switch (NSS) service. Therefore, the --manage-gids option allows the NFS server to ignore the information sent by the client and check the groups directly with the information stored into a LDAP or whatever using by the NSS. For this case, the NFS server and the NFS client must share the UIDs and GIDs.

That is the suggested approaching suggested in solving-the-nfs-16-group-limit-problem. Unfortunately, it was not my case :-(.

But not in my case

In my case, I had no way for synchronize the ids of the client with the ids of the NFS server. In my situation the ids in the client server was obtained from a Postgres database added in the NSS as one of the backends, there was not any chance to use these backend for the NFS server.

The solution

But this was not the end. Fortunately, the nfs-ngroups patchs developed by frankvm@frankvm.com expand the variable length list from 16-bit to 32-bit numeric supplemental group identifiers. As he says in the README file:

This patch is useful when users are member of more than 16 groups on a Linux NFS client. The patch bypasses this protocol imposed limit in a compatible manner (i.e. no server patching).

That was perfect! It was that I was looking for exactly. So I had to build a custom kernel patched with the right patch in the server under my control and voilá!:

wget https://cdn.kernel.org/pub/linux/kernel/v3.x/linux-3.10.101.tar.xz
wget http://www.frankvm.com/nfs-ngroups/3.10-nfs-ngroups-4.60.patch
tar -xf linux-3.10.101.tar.xz</code><code>
cd linux-3.10.101/
patch &lt; ../3.10-nfs-ngroups-4.60.patch
make oldconfig
make menuconfig
make rpm
rpm -i /root/rpmbuild/RPMS/x86_64/kernel-3.10.101-4.x86_64.rpm
dracut "initramfs-3.10.101.img" 3.10.101
grub2-mkconfig &gt; /boot/grub2/grub.cfg

Steps for CentOS, based on these three documents: [1] [2] [3]

Conclusions

As I said this post doesn’t make focus in the security stuffs. AUTH_SYS is a solution designed for the previous times before Internet. Nowadays, the total interconnection of the computer networks discourages the usage of kind methods like AUTH_SYS. It is an authentication method too much naive in the present.

Anyway, the NFS services are still quite common and many of them are still deployed with AUTH_SYS, not Kerberos or other intermediate solutions.  This post is about a specific situation in one of these deployments. Even if these services should be progressively replaced by other more secure solutions, a sysadmin should demand practical feedback about the particularities of these legacy systems.

Knowledge about the NFS 16 secondary groups limit and the different recognized workaround are still interesting from the point of view of the know-how. This post shows two solutions, even three if you consider the Kerberos choice, to fix this issue … just one of them fulfill with my requirements in my particular case.

Pablo says: “welcome ess-pipe-de to my life!”

Recently, some guy suggests me the usage of spiped instance of “SSH -L” to generate secure and more robust tunnels in peers under my control. The father of the creature is Alex Polvi (https://twitter.com/polvi) which doesn’t looks like as the new guy in the class: CEO in CoreOs Inc., previously General Manager on Rackspace, Product Manager and Sysadmin for mozilla.org. So, you can feel free to trust on spiped the next time you wish a protected peer-to-peer communication between a pair of servers:

 

To set up an encrypted and authenticated pipe for sending email between two
systems (in the author's case, from many systems around the internet to his
central SMTP server, which then relays email to the rest of the world), one
might run

# dd if=/dev/urandom bs=32 count=1 of=keyfile
# spiped -d -s '[0.0.0.0]:8025' -t '[127.0.0.1]:25' -k keyfile

on a server and after copying keyfile to the local system, run

# spiped -e -s '[127.0.0.1]:25' -t $SERVERNAME:8025 -k keyfile

at which point mail delivered via localhost:25 on the local system will be
securely transmitted to port 25 on the server.

 

Suggested post: http://www.daemonology.net/blog/2012-08-30-protecting-sshd-using-spiped.html

Original repository in github: https://github.com/polvi/spiped

Hide the VLC cone icon in the browser-plugin-vlc for Linux (Mozilla or Webkit) (Debian way)

vlc
VideoLAN’s fu***ng cone

The next instructions describes how to proceed to hide the VLC cone icon in the VLC plugin for Web browsers. I think this tip can be useful for another ninjas in so far as there is not a lot of information on Internet which describes this. Instructions are based on the Debian way and use the Debian/DPKG tools but I guess that the example is far enough explicit to be extrapolated to other environments.

Requirements:

  • You need to install all the build-dependences for the browser-plugin-vlc before execute dpkg-buildpackage -rfakeroot

Steps:

  • apt-get source browser-plugin-vlc
  • cd npapi-vlc-2.0.0/
  • edit npapi/vlcplugin_gtk.cp and replace the code as follows:
    --- npapi-vlc-2.0.0.orig/npapi/vlcplugin_gtk.cpp
    +++ npapi-vlc-2.0.0/npapi/vlcplugin_gtk.cpp
    @@ -46,12 +46,13 @@ VlcPluginGtk::VlcPluginGtk(NPP instance,
         vol_slider_timeout_id(0)
     {
         memset(&video_xwindow, 0, sizeof(Window));
    -    GtkIconTheme *icon_theme = gtk_icon_theme_get_default();
    -    cone_icon = gdk_pixbuf_copy(gtk_icon_theme_load_icon(
    -                    icon_theme, "vlc", 128, GTK_ICON_LOOKUP_FORCE_SIZE, NULL));
    -    if (!cone_icon) {
    -        fprintf(stderr, "WARNING: could not load VLC icon\n");
    -    }
    +    cone_icon = NULL;
     }
     
     VlcPluginGtk::~VlcPluginGtk()
    
  • dpkg-source –commit
  • dpkg-buildpackage -rfakeroot
  • cd ../
  • ls browser-plugin-vlc_2.0.0-2_amd64.deb

Installation:

  • dpkg -i browser-plugin-vlc_2.0.0-2_amd64.deb

Paramiko example

I have the pleasure of presenting a tip from the past. Today from long time ago: Paramiko.

import os
import paramiko
hostname="vps.doc.com"
username="admin"
password="password"
port=22
remotepath="/tmp/test"

ssh = paramiko.SSHClient()
ssh.load_host_keys(os.path.expanduser(os.path.join("~", ".ssh", "known_hosts")))
ssh.connect(hostname, port=port, username=str(username), password=str(password))
sftp = ssh.open_sftp()

remote_file = sftp.file(remotepath, "r")
remote_file.set_pipelined(True)
file_lines = remote_file.read()
return file_lines
file_lines = ...

sftp.open(remotepath, 'w').write(file_lines)
sftp.close()
ssh.close()

Tips about FFserver & FFmpeg

FFmpeg

Today, I want to share one tip about ffmpeg and ffserver multimedia video tools and  server. FFmpeg is a open source project that produces libraries and programs for handling multimedia data. FFserver is a HTTP and RTSP multimedia streaming server for live broadcasts. It can also time shift live broadcast.

All the settings used in this article have been tested on AMD64 Debian Squeeze OS using
FFmpeg Debian packages of the Debian-Multimedia repositories:

ffmpeg 5:0.6.1+svn20101128-0.2

You can get the Debian Multimedia repositories adding this lines to your APT sources.list
file:

deb http://www.debian-multimedia.org squeeze main
deb-src http://www.debian-multimedia.org squeeze main

Note that, with this same version, I’ve observe a problem trying to run ffserver:

Mon Apr 25 13:29:09 2011 Aspect ratio mismatch between encoder and muxer layer

To work ffserver in this version of ffmpeg is neccesary to hack the source code:

    1. Install DPKG development tools:  apt-get install dpkg-dev
    2. Get sources: apt-get source ffmpeg
    3. Go to sources directory: cd ffmpeg-dmo-0.6.1+svn20101128/
    4. Apply the patch:
      Index: libavutil/rational.h
      ===================================================================
      --- libavutil/rational.h    (revision 25549)
      +++ libavutil/rational.h    (working copy)
      @@ -29,7 +29,6 @@
      #define AVUTIL_RATIONAL_H
      
      #include <stdint.h>
      -#include <limits.h>
      #include "attributes.h"
      
      /**
      @@ -44,16 +43,13 @@
      * Compare two rationals.
      * @param a first rational
      * @param b second rational
      - * @return 0 if a==b, 1 if a>b, -1 if a<b, and INT_MIN if one of the
      - * values is of the form 0/0
      + * @return 0 if a==b, 1 if a>b and -1 if a<b
      */
      static inline int av_cmp_q(AVRational a, AVRational b){
      const int64_t tmp= a.num * (int64_t)b.den - b.num * (int64_t)a.den;
      
      if(tmp) return ((tmp ^ a.den ^ b.den)>>63)|1;
      -    else if(b.den && a.den) return 0;
      -    else if(a.num && b.num) return (a.num>>31) - (b.num>>31);
      -    else                    return INT_MIN;
      +    else    return 0;
      }
      
      /**
  • Install all the dependences neccesaries to build the package:
    apt-get install  debhelper libmp3lame-dev zlib1g-dev libvorbis-dev libsdl-dev libfaac-dev quilt texi2html libxvidcore4-dev liblzo2-dev libx264-dev  libtheora-dev libgsm1-dev ccache libbz2-dev libxvmc-dev libdc1394-22-dev libdirac-dev   libschroedinger-dev libspeex-dev yasm libopenjpeg-dev libopencore-amrwb-dev libvdpau-dev libopencore-amrnb-dev libxfixes-dev libasound-dev libva-dev libjack-dev libvpx-dev  librtmp-dev doxygen
  • Build the packages: dpkg-buildpackage -rfakeroot
  • Finally you’ll have the new *deb packages:
    # ls ../*.deb
    ffmpeg_0.6.1+svn20101128-0.2_amd64.deb         libavfilter-dev_0.6.1+svn20101128-0.2_amd64.deb
    ffmpeg-dbg_0.6.1+svn20101128-0.2_amd64.deb     libavformat52_0.6.1+svn20101128-0.2_amd64.deb
    ffmpeg-doc_0.6.1+svn20101128-0.2_all.deb     libavformat-dev_0.6.1+svn20101128-0.2_amd64.deb
    libavcodec52_0.6.1+svn20101128-0.2_amd64.deb     libavutil50_0.6.1+svn20101128-0.2_amd64.deb
    libavcodec-dev_0.6.1+svn20101128-0.2_amd64.deb     libavutil-dev_0.6.1+svn20101128-0.2_amd64.deb
    libavcore0_0.6.1+svn20101128-0.2_amd64.deb     libpostproc51_0.6.1+svn20101128-0.2_amd64.deb
    libavcore-dev_0.6.1+svn20101128-0.2_amd64.deb     libpostproc-dev_0.6.1+svn20101128-0.2_amd64.deb
    libavdevice52_0.6.1+svn20101128-0.2_amd64.deb     libswscale0_0.6.1+svn20101128-0.2_amd64.deb
    libavdevice-dev_0.6.1+svn20101128-0.2_amd64.deb  libswscale-dev_0.6.1+svn20101128-0.2_amd64.deb
    libavfilter1_0.6.1+svn20101128-0.2_amd64.deb

After to install the ffmpeg packages, you’ll can to run ffserver adjusted like you want. for this aim, you can run as follow: ffserver -f your_ffserver_settings.conf. The ffserver configuration file should have this structuration:

  • Main settings:
     # Port on which the server is listening. You must select a different
     # port from your standard HTTP web server if it is running on the same
     # computer.
     Port 8090
     # Address on which the server is bound. Only useful if you have
     # several network interfaces.
     BindAddress 0.0.0.0
     RTSPPort 554
     RTSPBindAddress 0.0.0.0
     # Number of simultaneous HTTP connections that can be handled. It has
     # to be defined *before* the MaxClients parameter, since it defines the
     # MaxClients maximum limit.
     MaxHTTPConnections 2000
     # Number of simultaneous requests that can be handled. Since FFServer
     # is very fast, it is more likely that you will want to leave this high
     # and use MaxBandwidth, below.
     MaxClients 1000
     # This the maximum amount of kbit/sec that you are prepared to
     # consume when streaming to clients.
     MaxBandwidth 1000
     # Access log file (uses standard Apache log file format)
     # '-' is the standard output.
     CustomLog -
     # Suppress that if you want to launch ffserver as a daemon.
     NoDaemon
  • Definition of the live feeds. Each live feed contains one video and/or audio sequence coming from an ffmpeg encoder or another ffserver. This sequence may be encoded simultaneously with several codecs at several resolutions. You must use ffmpeg to send a live feed to ffserver. In this example, you can type ffmpeg http://localhost:8090/feed1.ffm  or ffmpeg   -f alsa   -i hw:1   -f video4linux2 -r 25 -s 352x288  -i /dev/video0   http://localhost:8090/feed1.ffm:
     ################################################################################
     <Feed feed1.ffm>
     # ffserver can do time shifting. It means that it can stream any
     # previously recorded live stream. The request should contain:
     # "http://xxxx?date=[YYYY-MM-DDT][[HH:]MM:]SS[.m...]".You must specify
     # a path where the feed is stored on disk. You also specify the
     # maximum size of the feed, where zero means unlimited. Default:
     # File=/tmp/feed_name.ffm FileMaxSize=5M
     File /tmp/feed1.ffm
     FileMaxSize 100M
     # You could specify
     # ReadOnlyFile /saved/specialvideo.ffm
     # This marks the file as readonly and it will not be deleted or updated.
     # Specify launch in order to start ffmpeg automatically.
     # First ffmpeg must be defined with an appropriate path if needed,
     # after that options can follow, but avoid adding the http:// field
     # Launch ffmpeg
     # Only allow connections from localhost to the feed.
     ACL allow 127.0.0.1
     </Feed>
  • Setting a RTSP/RTP stream:
     ################################################################################
     # It's a lot of important the .sdp extension to allow RTP working well.
     #
     # Note that AVOptionVideo is only interesting for libx264 video codec:
     # For RTSP:
     # ffplay  rtsp://10.121.55.148:554/live.sdp
     #
     # For SDP (RTP):
     #   vlc  http://10.121.55.148:8090/live.sdp
     #
     <Stream live.sdp>
     Format rtp
     Feed feed1.ffm
     ### MulticastAddress 224.124.0.1
     ### MulticastPort 5000
     ### MulticastTTL 16
     # NoLoop
     VideoSize 352x288
     VideoFrameRate 15
     VideoBitRate 200
     # Alternative video codecs:
     # VideoCodec h263p
     # VideoCodec h263
     # VideoCodec libxvid
     # VideoQMin 10
     # VideoQMax 31
     VideoCodec libx264
     AVOptionVideo me_range 16
     AVOptionVideo i_qfactor .71
     AVOptionVideo qmin 30
     AVOptionVideo qmax 51
     AVOptionVideo qdiff 4
     # AVOptionVideo coder 0
     # AVOptionVideo flags +loop
     # AVOptionVideo cmp +chroma
     # AVOptionVideo partitions +parti8x8+parti4x4+partp8x8+partb8x8
     # AVOptionVideo me_method hex
     # AVOptionVideo subq 7
     # AVOptionVideo g 50
     # AVOptionVideo keyint_min 5
     # AVOptionVideo sc_threshold 0
     # AVOptionVideo b_strategy 1
     # AVOptionVideo qcomp 0.6
     # AVOptionVideo bf 3
     # AVOptionVideo refs 3
     # AVOptionVideo directpred 1
     # AVOptionVideo trellis 1
     # AVOptionVideo flags2 +mixed_refs+wpred+dct8x8+fastpskip
     # AVOptionVideo wpredp 2
     ## AVOptionVideo flags +global_header+loop
     # NoAudio
     AudioCodec libmp3lame
     AudioBitRate 32
     AudioChannels 1
     AudioSampleRate 24000
     ## AVOptionAudio flags +global_header
     </Stream>
  • Setting a FLV stream ouput:
     ################################################################################
     # FLV output - good for streaming
     <Stream test.flv>
     # the source feed
     Feed feed1.ffm
     # the output stream format - FLV = FLash Video
     Format flv
     VideoCodec flv
     # this must match the ffmpeg -r argument
     VideoFrameRate 15
     # generally leave this is a large number
     VideoBufferSize 80000
     # another quality tweak
     VideoBitRate 200
     # quality ranges - 1-31 (1 = best, 31 = worst)
     VideoQMin 1
     VideoQMax 5
     VideoSize 352x288
     # this sets how many seconds in past to start
     PreRoll 0
     # wecams don't have audio
     Noaudio
     </Stream>
  • Setting a ASF stream ouput:
     ################################################################################
     # ASF output - for windows media player
     <Stream test.asf>
     # the source feed
     Feed feed1.ffm
     # the output stream format - ASF
     Format asf
     VideoCodec msmpeg4
     # this must match the ffmpeg -r argument
     VideoFrameRate 15
     # transmit only intra frames (useful for low bitrates, but kills frame rate).
     # VideoIntraOnly
     # if non-intra only, an intra frame is transmitted every VideoGopSize
     # frames. Video synchronization can only begin at an intra frame.
     VideoGopSize 40
     # generally leave this is a large number
     VideoBufferSize 1000
     # another quality tweak
     VideoBitRate 200
     # quality ranges - 1-31 (1 = best, 31 = worst)
     VideoQMin 1
     VideoQMax 15
     VideoSize 352x288
     # this sets how many seconds in past to start
     PreRoll 0
     # generally, webcams don't have audio
     # Noaudio
     AudioCodec libmp3lame
     AudioBitRate 32
     AudioChannels 1
     AudioSampleRate 24000
     </Stream>
  • Other streams availables:
     # Multipart JPEG
     #<Stream test.mjpg>
     #Feed feed1.ffm
     #Format mpjpeg
     #VideoFrameRate 2
     #VideoIntraOnly
     #NoAudio
     #Strict -1
     #</Stream>
     # Single JPEG
     #<Stream test.jpg>
     #Feed feed1.ffm
     #Format jpeg
     #VideoFrameRate 2
     #VideoIntraOnly
     ##VideoSize 352x240
     #NoAudio
     #Strict -1
     #</Stream>
     # Flash
     #<Stream test.swf>
     #Feed feed1.ffm
     #Format swf
     #VideoFrameRate 2
     #VideoIntraOnly
     #NoAudio
     #</Stream>
     # MP3 audio
     #<Stream test.mp3>
     #Feed feed1.ffm
     #Format mp2
     #AudioCodec mp3
     #AudioBitRate 64
     #AudioChannels 1
     #AudioSampleRate 44100
     #NoVideo
     #</Stream>
     # Ogg Vorbis audio
     #<Stream test.ogg>
     #Feed feed1.ffm
     #Title "Stream title"
     #AudioBitRate 64
     #AudioChannels 2
     #AudioSampleRate 44100
     #NoVideo
     #</Stream>
     # Real with audio only at 32 kbits
     #<Stream test.ra>
     #Feed feed1.ffm
     #Format rm
     #AudioBitRate 32
     #NoVideo
     #NoAudio
     #</Stream>
     # Real with audio and video at 64 kbits
     #<Stream test.rm>
     #Feed feed1.ffm
     #Format rm
     #AudioBitRate 32
     #VideoBitRate 128
     #VideoFrameRate 25
     #VideoGopSize 25
     #NoAudio
     #</Stream>
  • Other special streams:
     # Server status
     <Stream stat.html>
     Format status
     # Only allow local people to get the status
     ACL allow localhost
     ACL allow 192.168.0.0 192.168.255.255
     #FaviconURL http://pond1.gladstonefamily.net:8080/favicon.ico
     </Stream>
    
     # Redirect index.html to the appropriate site
     <Redirect index.html>
     URL http://www.ffmpeg.org/
     </Redirect>

Extra references:

Deux ex virtual machine

El siguiente texto no se trata de un alegato a favor de los sistemas de virtualización a nivel OS como OpenVZ y VServer y en contra de los sistemas de virtualización completa como Xen, VMware, VBox, KVM …
No, no se trata de eso, si no de que, en principio,  para un entorno homogéneo y con pocos recursos hardware o con servicios que, a priori, sea difícil saber como van a evolucionar en el tiempo con respecto al consumo de los recursos HW, el uso de uso de estos contenedores se me antoja personalmente mucho más práctico y versátil que las otras dos alternativas (no virtualización y virtualización completa).

***

La empresa Pérez y Familia son una PYME de medio tamaño (unos 50 trabajadores). La misma, tiene un importante Departamento de Informática formado por un gurú Unix (top 9 en el
ranking mundial de freaks) y un becario a media jornada. Por supuesto, tienen una partida presupuestaria de 1200 chocomonedas para todo el año destinadas principalmente a mantener la familia numerosa del señor que mantiene fiel y cuasiquereligiosamente la máquina del café. El Dep. de Informática en su CPD, ubicado inmediatamente debajo de la mesa del becario, tiene una pequeña infraestructura de servidores:

  • (A) 2 PentiumIV con unas a cuantas tarjetas de red a modo de Router corporativo en HA
  • (B) 1 PowerEdge 1890 DualCore que compraron cuando los inicios de la empresa
  • (C) 2 X PCs clónicos 8core con 8GB de RAM comprados recientemente en MarcaMedia a petición explícita del Dep. de Informática

Al departamento de informática se le da vía libre para instalar los SO bajo su antojo, siempre y cuando las horas extra caigan de su lado. Así que como casi una excepción empresarial, se forman su propia taifa dentro del reino de los Pérez. Por otro lado, la empresa poco a poco, a lo largo de su tiempo de vida fue incorporando herramientas informáticas a sus procesos de trabajo:

  • Al principio, los de Informática montaron sus servidores que sólo sirven para consumir ciclos de procesado y algunas cosas freakies más:
    • Un servidor DNS, que si un DHCP, y un Snort en A
    • Que si un OpenVPN en A
    • Que si unos repositorios en B
  • Luego, vinieron las herramientas de apoyo al proceso:
    • Gestor documental en B
    • CRM en B
  • Tiempo más, llegaron más herramientas que se fueron incorporando:
    • La intranet
    • La extranet
    • El streamer de música
    • El repositorio de código
    • El inventario y el sistema de monitorización de servicios
    • y un largo etc …

Todo esto se fue metiendo en B y C como se pudo. A mayores, cada cierto tiempo algo o alguién pide la evaluación de un software para ver si es rompedor. Lógicamente, estas pruebas se hacen en los entornos de testing (aka, el portátil del becario, el suyo el que se compro con la subvención de estudiante matriculado). Todo esto va generando el escenario, preparando el clímax de nuestra historia, el preludio del ansiado momento que nos llevará al clímax y servirá de justificante al desenlace.

Y llegó el momento, Dirección exige la instalación del ERP UltimatumTotalNG. Este está pensado para RedHat4.0 si y sólo con versiones concretas de ciertos tipos de base de datos y demás y Dirección, que es experta en toma de decisiones técnicas como Franco lo era en la Teoría de la Relatividad de Einstein, no quiere ni oir de hablar de otras distros o versiones que no sean las que el apuesto y sofisticado consultor externo ha sugerido para el UltimatumTotalNG.

Llegados a este punto los del Dep. Informática (o sea el becario y el gurú Top10) se  encuentran de repente contra la espada y la pared. Ya que ellos decidieron en el pasado, allá por el tiempo de la patata (esta es buena, a ver quien la pilla), elegir Debian como OS y lo fueron actualizando hasta Squeeze ni más ni menos. ¿Que pasará? ¿Será el fin de nuestros héroes? ¿Como podrán salir de esta encrucijada? Veamos cómo actúan:

– ¿Que podemos hacer? dijo el Gurú Unix.

– Bien, analicemos la situación, dijo el becario. Tenemos muchos servicios instalados pero realmente estos no están consumiendo siempre los recursos del sistema (50 clientes no son muchos clientes al fin y al cabo). Con las máquinas que tenemos bien podemos aguantar todos estos servicios. No obstante, tenemos un problema con los procesos de actualización: No es la primera vez que por motivo de la actualización de un servicio concreto, se ven afectados otros servicios independientes y nos cae un bronca encima.

– Bien, podríamos usar virtualización. Movemos todos los servicios que están en C” a C’ y preparamos esta máquina para albergar máquinas KVM y así vamos migrando todo sucesivamente (empezando por la RedHat del ERP) hasta tener todo virtualizado.

– Perfecto, crack!

(tiempo que transcurre en liberar C” de servicios)

– …. Bien, ahora ¿como dimensionamos las imágenes? ummm hombre dale medio core a esta MV que no se suele usar muy a menudo.
– Nooo, estás loco!, si ahí va a ir el CRM, entre las nueve y nueve y media los de marketing se ponen a hacer consultas como locos. Ya sabes que no son muy cuidadosos a la hora de ejecutar los filtros. Se nos van a echar encima, dale por lo menos dos cores …

(después de 5 horas más)

– Venga, créame otra MV para este servicio.
– Oops!, creo que tenemos todos los recursos físicos asignados.
– Pero ¿como es posible?, si antes estábamos dando el servicio perfectamente con el mismo número de servidores!!!.
– Si, pero también es cierto que teníamos en la misma máquina el CRM y el Gestor documental y estos, a pesar de usar todos los recursos físicos, uno tenía sus cargas de trabajo por la mañana y otro por la tarde por lo que la máquina estaba bien balanceada. Con la reserva de recursos de las máquinas hemos partido los recursos a la mitad, con lo que otros servicios menores que también se corrían sobre dicha máquina se han quedado sin recursos para una nueva MV donde alojarlos.

… (todo los demás transcurrido es una sucesión de culpas, reproches, #epicfails, #workarrounds, excusas, horas extras y #fatalities) …

Finalmente, los informáticos fueron despedidos. El gurú escribió un libro y se retiró con el dinero que ganó y el Becario, frustrado por su primera experiencia laboral, diseño un sistema de jails para Linux llamado VUltimatum con un funcionamiento muy similar a lo que conocemos como VServers, Zones, Jails, VZ, Containers. Debido a esto, es altamente reconocido en su gremio profesional y, a mayores, tiene un trabajo digno como reponedor del Carrefour. Podría decir que la vida le sonríe.