frankie-tales

Playing with Vagrant, Virtualbox and Guix

2024-09-04T18:20:00Z

It is specifically convenient using Guix-the-system within a foreign distribution, such as Debian, for development and tests. The package management system can be used on top of the system, but I find it quite interesting to explore the potential of the Guix distribution in the context of virtualized environments. For personal use, that is also the ideal way to avoid breaking your own daily boxes every couple of days with daredevil approaches to personal computing.

I find it quite handy to use Vagrant to synthetically describe multiple virtual hosts within a virtual network, by means of a single Vagrantfile. Nowadays, there are multiple virtualization systems usable for the same purpose, including the libvirt ecosystem. Under the hood, those systems can generally manage multiple virtual machine engines (or even cloud systems). Vagrant under Debian can use either Libvirt+KVM or Virtualbox (the default one used by the upstream Vagrant) as primary engines.

Users can develop their own Vagrant images (called boxes) or use those made available on the Hashicorp cloud, as developed by third-party teams (which is the common case).

Unfortunately, while there are a handful of Debian boxes developed by the Debian Cloud Team, like many other ones for multiple distributions and releases, this is not the case for Guix. At the time of this post, there is only a single box created some years ago by Tom Parker-Shemilt.

Of course, the Guix team also made a qcow2 image available to run a virtualized desktop machine under a QEMU/KVM system, as explained in the manual. What is described in this post can also be easily applied to create an alternative custom implementation with QEMU. What is required is to create a virtual disk image of some kind and use it to populate a custom distribution configuration via guix. The resulting image will be runnable under a suitable virtualization system in the host foreign distribution.

Let's assume to use any recent version of Vagrant and Virtualbox, both installed on the host system, along with the Guix package manager. The following are the steps required to create a Vagrant box with a minimal configuration of Guix that is compatible (with some limitations) with Vagrant and can be used to run any number of Guix VMs. Each Guix vm can be reconfigured for specific goals using multiple configurations.

In this case, we will not need to use Packer to prepare a box because the vagrant package command directly supports Virtualbox images with the same purpose.

Create an empty virtual disk image and partition it.
Prepare a suitable configuration written in Guile for Guix.
Run an initial setup of the mounted virtual disk using guix on your host.
Create a basic Virtualbox machine and attach the virtual disk to it.
Convert the machine into a Vagrant box and register it.
Prepare a Vagrantfile snippet and run the vm(s).
(Re)configure vm(s) on the basis of your needs.
Done!

Create a virtual disk image and partition it

This is the straightforward part. Here, I'm using the vdi format, the native Virtualbox one, but the VMware vmdk format can be used instead.

 # load the Network Block Device module
 sudo modprobe nbd
 # create a thin device
 qemu-img create -f vdi guix-hd.vdi 100G
 # ... mount it
 sudo qemu-nbd -f vdi --connect=/dev/nbd0 guix-hd.vdi
 # ... partition it 
 sudo parted /dev/nbd0 mklabel msdos
 sudo parted -a cylinder /dev/nbd0 mkpart primary ext4 1 93G
 sudo parted -a cylinder /dev/nbd0 mkpart primary linux-swap 93G 100%
 sudo parted /dev/nbd0 set 1 boot on 
 # ... create a suitable fs
 sudo mkfs.ext4 /dev/nbd0p1
 sudo mkswap /dev/nbd0p2
 # ... have a look to the partions IDs
 sudo blkid /dev/nbd0p1 
 sudo blkid /dev/nbd0p2
 # ... mount the root fs
 sudo mount /dev/nbd0p1 /mnt

Prepare a suitable configuration written in Guile for Guix

Any valid configuration can be prepared, in this case this is a minimal one with a pair of changes for Vagrant. Specifically, a vagrant user in the wheel group needs to be added, and it should be able to run sudo without a password. Even, the unsecure Vagrant key needs to be authorized to access via ssh.

Note that the key is added at the system level, and the root user has no password, which should be appropriately changed after the first run or, even better, at provisioning time. That is a requirement if the box would be exposed on public network, because the default private and public keys of Vagrant are publicly distributed.

That is usually and automagically done by vagrant at first boot, but Guix is a read-only system and - as we will see - the Guix system is still not completely supported by Vagrant.

ROOTFS_UUID=$(sudo blkid -o value /dev/nbd0p1|head -1)
SWAP_UUID=$(sudo blkid -o value /dev/nbd0p2|head -1)
DEVICE=/dev/nbd0

# Download the unsecure ssh key used by Vagrant
wget -q -O vagrant.pub https://raw.githubusercontent.com/mitchellh/vagrant/master/keys/vagrant.pub

cat >guix-config.scm <<EOF
;; Indicate which modules to import to access the variables
;; used in this configuration.
(use-modules (gnu))
(use-modules (gnu system))
(use-modules (gnu services avahi))
(use-service-modules desktop networking ssh)

;; Prepare a salt function for a less silly password encryption
(set! *random-state* (random-state-from-platform))
(define str "0123456789abcdefghijklmnopqrstuvwxyz")
(define rnd-chr (lambda () (string-ref str (random (- (string-length str) 1)))))
(define salt (lambda () (string-append (string (rnd-chr)) (string (rnd-chr)) (string (rnd-chr)))))

(operating-system
  (locale "en_US.utf8")
  (timezone "Europe/Rome")
  (keyboard-layout (keyboard-layout "us" "intl"))
  (host-name "guix")

  ;; The list of user accounts ('root' is implicit).
  (users (cons* (user-account
                  (name "vagrant")
                  (comment "Vagrant user")
                  (group "users")
                  (home-directory "/home/vagrant")
                  (password (crypt "vagrant" (string-append "\$6\$" (salt))))
                  (supplementary-groups '("wheel" "netdev" "audio" "video")))
                %base-user-accounts))

  ;; Packages installed system-wide.  Users can also install packages
  ;; under their own account: use 'guix search KEYWORD' to search
  ;; for packages and 'guix install PACKAGE' to install a package.
  (packages (append (list (specification->package "nss-certs")
                          (specification->package "rsync"))
                    %base-packages))

  ;; Below is the list of system services.  To search for available
  ;; services, run 'guix system search KEYWORD' in a terminal.
  (services
   (append (list (service dhcp-client-service-type)
                 (service openssh-service-type
                    ;; here the official unsecure Vagrant ssh key is used...
                    (openssh-configuration
                      (authorized-keys \`(("vagrant" ,(local-file "vagrant.pub")))))))

           ;; This is the default list of services we
           ;; are appending to.
           %base-services))

  ;; Authorize vagrant to run sudo without password.
  (sudoers-file
    (plain-file "sudoers"
                 (string-append (plain-file-content %sudoers-specification)
                                "vagrant ALL=(ALL) NOPASSWD: ALL\\n")))

  (bootloader (bootloader-configuration
                (bootloader grub-bootloader)
                (targets (list "$DEVICE"))
                (keyboard-layout keyboard-layout)))
  (swap-devices (list (swap-space
                        (target (uuid
                                 "$SWAP_UUID")))))

  ;; The list of file systems that get "mounted".  The unique
  ;; file system identifiers there ("UUIDs") can be obtained
  ;; by running 'blkid' in a terminal.
  (file-systems (cons* (file-system
                         (mount-point "/")
                         (device (uuid
                                  "$ROOTFS_UUID"
                                  'ext4))
                         (type "ext4")) %base-file-systems)))
EOF

Run an initial setup of the mounted virtual disk using `guix` on your host

The system init can be run more than once, if required.

sudo guix pull # only if needed to update packages...
sudo guix system init guix-config.scm /mnt
sudo umount /mnt
sudo qemu-nbd --disconnect /dev/nbd0

Create a basic Virtualbox machine and attach the virtual disk to it

Now, the virtual disk should already be registered and visibile under your Virtualbox configuration.

VDI_UUID=`vboxmanage showhdinfo guix-hd.vdi|grep ^UUID|awk '{print $2}'` && echo $VDI_UUID

should show the hexadecimal code associated to the now populated disk. It is now possible to create a simple virtual machine, for instance:

vboxmanage createvm --name=Guix --default --ostype=Linux_64 --register
vboxmanage modifyvm Guix --memory=4096 --cpus=2 --ioapic=on --vram=256 --cpu-profile=host \
                --audio-enabled=off --usb-xhci=off --usb-ehci=off --usb-ohci=off --mouse=ps2

and attach the virtual disk to it, as follows:

vboxmanage storageattach Guix --storagectl=SATA --type=hdd --port=0 --device=0 --medium=$VDI_UUID
vboxmanage showvminfo Guix

Convert the machine into a Vagrant box and register it

The resulting vm can be directly used under Virtualbox, but the final touch is creating a proper Vagrant box to recycle and possibly publish on the cloud.

vagrant package --base Guix --output guix-small.box
vagrant box add guix-small.box --name=guix-small
vagrant box list

Now the new box is ready for use in multiple configurations within a Vagrantfile.

Prepare a Vagrantfile snippet and run the vm(s)

A simple Vagrantfile can be used to create an instance of the box, such as:

Vagrant.configure("2") do |config|
    config.vm.define "guix1" do |vm1|
     vm1.vm.box = "guix-small"
     vm1.vm.network "private_network", ip: "192.168.1.2"
     vm1.vm.provider "virtualbox" do |vb1|
        vb1.memory = "8192"
    	vb1.name = "guix8G"
    	vb1.cpus = 4
    end
    config.vm.provision "shell", inline: <<-SHELL
        guix pull
        guix install htop 
     SHELL
    end
end

A new machine can be created, started up and connected easily, with also an initial provisioning, by issuing:

vagant up guix1
vagrant ssh guix1

(Re)configure vm(s) on the basis of your needs

Both the box and the dependent virtual machines can be reconfigured as usual by guix system reconfigure in the guest or even remounting the original virtual disk (or any of the vmdk copies cloned by Vagrant) as previously done, then re-issuing a system init with a new Guile configuration. Note that the same reconfigure can also be run as described in the manual.

Missing features in Vagrant to support Guix

Vagrant needs to recognize the installed system in order to perform a few operations, but unfortunately that is still not the case for Guix. Specifically, that prevents the capability of halting the system, which is not exactly a nice thing. That must be done manually by running halt within an ssh session. For the same reason, Vagrant is not able to fully configure networking, so the second network interface shown in the previous Vagrantfile needs to be configured manually by adding a suitable static-networking-service-type section to the Guile configuration.

A suitable support would need to be added as a proper vagrant plugin as in the case of other operating systems and distributions. So, good but not good enough, maybe it is matter for another post, even if I'm not exactly a Ruby language fan.

In the meantime, have a good time by upping and destroying Vagrant Guix boxes.

The Guix system, take two

2024-09-02T20:00:00Z

Let's give a second look at Guix-the-system the main GNU Project distribution I dealt with in a previous post. This post is not specifically limited to the distribution, it is also of interest when using Guix in a foreign distribution, even if some configuration details change.

Substitutes and grafts

As said previously, the daily use of a store-based rolling distribution adds some overhead to the system at both upgrade and installation time. This pain (that remembers me the old times of the source-based Gentoo distribution with its emerge tool) is specifically alleviated by the use of the so-called substitutes servers, which provide pre-built binaries for the base system, as well as alternative/unofficial packages, too.

The fall-back alternative is based on regular build-from-sources on the host system, which could imply long times for both installations and distribution upgrades. The official Guix system comes with a couple of official substitutes (i.e., ci.guix.gnu.org and bordeaux.guix.gnu.org) but others can be added, including possibly any suitable user's server in the LAN.

Another trick in Guix for alleviating the need for long local rebuilds is the use of grafts that are sort of in-place replacements for binary dependencies, expressed at the source level. To explain in brief, if a package A@1 has been replaced by A@2 and they both maintain the same ABI, any reverse dependency B can avoid being rebuilt. This is called a graft in Guix, and greatly simplifies the long chains of forced rebuilds in many cases (for instance, in case of security upgrades). Specifically, the @ in a Guix package is the version separator.

A monorepo for the whole distribution

The Guix source archive is strictly based on git and distributed development by means of a monorepo, which, along with the need to represent the tree of dependencies for any package and its updates at run-time require some operations that are specific of the Guix package management approach:

pulling
garbage collection
branching
using channels

The pull phase specifically could require ages (hours on my old low-end Asus EEEpc), so it is an operation that should be applied typically in batch mode and quite often in order to minimize execution times. It is not a potentially destructive operation - similar to an apt update with steroids - so it is better to perform it quite often, every few days. At the end of the day, it does not alter the status of the system, so it is safe enough for background execution.

The gc operation works directly on the store to purge obsolete (out of the current status tree of dependencies) entries. If you jump to a past status, the purge would impact bandwidth and CPU loads, of course.

The branch specification has the same role as any sane distributed environment organization of code. It is usable to pull from separate branches of archives, instead of following the default one, latest. This feature can be paired with the Guix time-machine to jump to any past tree of packages in the chronology of the archive sources.

Finally, a channel is simply an alternative archive of sources that is prepared by third-party teams to integrate the official Guix one. For instance, a few independent packages are offered by INRIA and other institutions for the HPC community, as well as an handful of non-free packages hosted on the nonguix repository to solve the well-known users-hostages' dilemma about using closed-source firmwares and a few other proprietary stuff.

One Scheme to Rule Them All

An exciting feature of Guix-the-System is the use of Guile to describe the whole system, including the core, all services, and even users. In perspective, all installed packages could be also fully configured by using functional Guile snippets of code. This is something currently done in Debian only in a limited way, through debconf templates and dpkg selections. In practice, today, one has to use ansible or similar tools to declare configurations in some ad hoc DSL, using tons of plugins and templates, case by case. In a word, it is a mess.

This is, at least for me, the most intriguing feature and open exciting possibilities.

Currently, any developer with a reasonably decent computer can easily use Guix to rebuild and customize Guix itself by starting from a monorepo fork, changing its main configuration and adding/modifying packages in a totally independent and self-consistent way. Something that in traditional distributions is done by a plethora of tools and interfaces, written in multiple generale-purpose or specific languages, and often not wholly documented and held together with the glue.

Once added a layer of general-purpose configurators for common packages the whole distribution generation could become fully autoconsistent and complete for any host or set of boxes. Isn't that challenging?

Guix in foreign distributions

Using Guix as an additional package manager in a foreign distribution has more limitations, of course. First of all, one must deal with systemd as the typical init system. Therefore, the general configuration of the host cannot be expressed in a synthetic way as a Scheme script. That said, it is perfectly possible to use Guix as a development environment for multiple languages ecosystems, thanks to various Guile build modules. Even, it is possible to run Guix-the-system in a container (or a virtual machine), to use the host system just as a basic platform and create Guix-based services and applications on top.

But those will be the topics for other posts...

References

An initial dive into Guix

2024-08-18T19:00:00Z

In the last few days, I got familiar with Guix, which is both a modern package management system and the main GNU Project distribution for Linux and Hurd (the Guix system). As a package management system, it can be installed on most foreign distributions, including Debian and any other, as an alternative/additional packaging system.

I both installed the Guix system natively on a small ancient laptop of mine (an ASUS EEEpc 1215N), and the Guix package manager on one of my Debian stable boxes. An interesting variant could be installing the whole system under a container in a non-interactive mode, but that may be a task for the future. Indeed, the last one could be the most exciting application of Guix for reproducible deployment.

Guix, the package manager

Guix (the package management system) is the most interesting part. It is a modern system with multiple advanced features, inspired by Nix, a pre-existing system based on a JSON-based DSL. Nix and Guix are both declarative and functional package managers with similar goals for software maintenance.

Both of them declare to have the largest collection of FOSS packages in the world, but ok: both currently have hubs with tens of thousands of binary packages. Maybe not the largest, but respectful. Of course, Guix is an FSF project and therefore highly choosy about the software that can be distributed within the Guix archives. That's not specifically different from the Debian approach, but for the derogation that historically the Debian Project considered for limited proprietary-but-distributable stuff (the non-free+contrib section).

One interesting aspect of Guix (at least for me) is that it is specifically based on Guile, an extendable Scheme dialect which is the main DSL used in the GNU ecosystem. All packages are expressed as small snippets of Guile functions that declare pre-dependencies, as well as the build, installation and test phases of each software.

Anyone who has worked on software packaging in Debian from the beginning knows that the mythical debian/rules is essentially a Makefile with steroids, accompanied by a handful of declarative files, lately simplified by the use of some frameworks, such as Joey Hess's debhelper or others commonly used in the past. Maybe not the most elegant approach to packaging and configuring, let me say. Probably at the time - 30 years ago or so - the most pragmatic one, for sure. And it worked even for a lot of years.

In respect with traditional system-wide packaging, the Nix/Guix approach has several interesting features:

transitional, per-user, multi-profile capabilities
a rolling-back capability at the system and user level
an all-in-one system of packing with dependencies
a single expressive way of defining software configurations at both system-wide and per-user levels

Guix per se adds the use of Guile to all the rest so that all configurations are synthetically expressed in S-exp, without the need to learn yet another DSL to describe the software chains of dependencies.

Guix, the system

The Guix free-software-only system has some interesting characteristics, including the use of shepherd as an alternative Guile-based init system and the rolling-on distribution style. The non-FHS basic organization of the filesystem could also pose some problems to install software that are strictly dependent on that, that's for sure a good reason to use Guix-on-Debian instead of Guix-the-system only. That issue is also partially mitigated by a combination of a container technology support, and an FHS emulation layer.

In my opinion, the whole thing is quite interesting for building development environments and exploring reproducible deployment systems.

The GNU touch

But for the FSF strictly free approach to collecting software (including the missing firmware blobs for the Linux-libre software, as for Debian until version 12), the Guix system has some typical geek-only pillars for its ecosystem and community:

strictly read the reference manuals and info files (i.e., do your homework)
use mailing lists
use IRC dedicated channels
use your brain and experience to solve issues

Nothing different from the traditional Debian project community and approach to personal computing. Therefore, nowadays something for geeky folk mainly, I guess.

Issues and challenges

For sure, the user workflow to install and run software changes radically and in a very different manner. One has the need to get familiar the Guix CLI interface and mode of operations. The Guix approach to deployment and maintenance adds an evident overhead to the system (for both storage and CPUs), partially mitigated by the use of substitutes/hubs to reduce building from source requirements. Not the best thing for old boxes, I guess.

Anyway, it is possible to use local substitutes to reduce the load for average systems. As a rolling distribution and/or software hub, I found it reasonably updated, but that largely depends on applications and domains. Nobody makes miracles in those regards, DebianGis packages in testing/unstable are more up-to-date for geospatial apps, and probably even more flexible. Sorry, no silver bullet, guys. Even Guix does not have (still?) a dedicated security team, so I would recommend it currently only for personal/development use, not for servers.

An important feature of Guix is its support for reproducible software deployment, as an alternative/integration to the ubiquitous containers, an aspect to deeply explore.

References

The main resources about Guix are the Reference Manual and The Cookbook, of course. I found some interesting non-trivial articles about Guix and its internals on some indie sites here listed:

Rebuild process running

2024-07-04T19:00:00Z

A long time ago, in a galaxy far, far away... I used to have a personal home page hosted somewhere. That was one of many other side projects of my life. I discovered some years ago that having multiple failed side projects along the way is a common experience for geeks and for many non-geek people, too. For reasons that are still not completely clear to me, some months ago, I decided that restarting a personal website in 2024 could be a not so weird idea.

One of the motivations is probably connected to the general idea that abandoning corporate social networks and regaining possession of our digital identities is nowadays the way to go. The indieweb is a return to the original idea of the World Wide Web and a duty of all of us geeks of the early days of the Internet.

For the same reason, the only social network I'm now actively using is Mastodon.

That said, blogging could be a challenging activity for me, I rarely think my casual thoughts are deep and interesting enough to be good for a public exposure. I generally prefer to focalize on activities that require more time and efforts than a few minutes to write down a draft note. But anyway, let me start and see how it will go in the next months.

In the next few sections I'll dial with how I created this system and what are my goals for the next short future.

Jamstack

First of all, I decided almost immediately to run a jamstack system for generating static pages starting from simple text documents. Such kind of system is more than enough for my purpose and even for a lot of web sites out there that instead use Wordpress or other exotic content management systems with a DBMS.

Lisp and Scheme

Once decided for a jamstack the possible choices are embarrassingly numerous, but as a few people could know, I always prefer by age and inclination unconventional niche solutions for my personal stuff. So, why not taking in consideration a family of languages that last year I decided to go in detail and never touched before? That would be the perfect occasion to motivate myself into such study.

Guile and Haunt

Fortunately, the possibile choices for a jamstack application is much more restricted, and I soon focalized on the GNU Guile dialect of Scheme and the Haunt blogging application by David Thompson. In just a couple of hours I managed to bootstrap all required modules and scripts to create a working system. Great!

So what?

The following posts and work will probably be concentrated on refinements of the styling and code, but at least I finalized an initial version of this new blog to write about... what? Well, I'm a techie and researcher in Earth Observation and remote sensing of the environment, with a special interest in geospatial technologies at large, so some topics will be obvious. For all the rest, who knows? Have a look to my haunt from time to time, it could be interesting...

frankie-tales

Playing with Vagrant, Virtualbox and Guix

Create a virtual disk image and partition it

Prepare a suitable configuration written in Guile for Guix

Run an initial setup of the mounted virtual disk using guix on your host

Create a basic Virtualbox machine and attach the virtual disk to it

Convert the machine into a Vagrant box and register it

Prepare a Vagrantfile snippet and run the vm(s)

(Re)configure vm(s) on the basis of your needs

Missing features in Vagrant to support Guix

The Guix system, take two

Substitutes and grafts

A monorepo for the whole distribution

One Scheme to Rule Them All

Guix in foreign distributions

References

An initial dive into Guix

Guix, the package manager

Guix, the system

The GNU touch

Issues and challenges

References

Rebuild process running

Jamstack

Lisp and Scheme

Guile and Haunt

So what?

Run an initial setup of the mounted virtual disk using `guix` on your host