AMD Catalyst driver patch for CONFIG_USER_NS

As hinted in this post, I have been experimenting with LXC for while. It has been a fruitful journey and I ended up with an LXC-version of Fatdog sandbox which enables one to run a sandbox with full privilege separation (unlike the standard sandbox which only uses chroot), which I will write in another day.

Running lxc requires the lxc userspace tools (which is on the URL above, or here for the bleeding edge), and requires some kernel compile-only options activated. No, unlike OpenVZ or Linux-VServer, you don't need custom kernels --- all that is needed for lxc have been slowly merged into mainline kernel in the last few years, but you do need to enable these features when you built the kernel.

One of the latest feature is user namespace (CONFIG_USER_NS). This feature, in short, enables uid mapping - a "root" (uid 0) in the container can be just a regular user in the host system (e.g. uid 1000). Without this feature, a root is a root is a root - inside and outside the container and it has the same power (unless you shave its capabilities - but again that's another story). This feature was finally merged in Linux 3.8, however it wasn't really working well because a lot of other kernel components were not adjusted for it (mainly filesystems). In Linux 3.9, the situation is a bit better, only XFS can't handle it. With the hard work from the kernel and XFS team, we'd expect that XFS will be fully compatible with user namespace by the time Linux 3.10 is released.

I have been testing with this feature in Linux 3.9, and found out that the AMD Catalyst driver would not compile. That's a BIG problem. My laptop is well supported by the radeon open-source driver, but I regret to say that while radeon probably trumps out in terms of performance, it is not a winner when it comes to power saving department. My laptop runs a full 20 centigrade hotter using the radeon driver compared to the Catalyst driver. AMD definitely needs to do more work on this! (Are you listening, AMD?)

Anyway, the conclusion is that I need that Catalyst driver to work. Just like any reasonable guy, I looked for some patches on Google but I can't find any :(

So I went on to look at the source on my own. Fortunately, it is not too difficult to fix it, and I have created a patch for it (it is a one-line patch, literally). I tested this against Catalyst 12.8 but it should work for later versions too (up to 13.4).

You can get the patch from here.


Posted on 27 May 2013, 19:45 - Categories: Linux
Comments - Edit - Delete

Booting GPT disk in BIOS systems using syslinux

Recently I was asked whether it is possible to boot GPT disk under a system with BIOS firmware.

I was somehow under the impression that GPT-partitioned disk (GPT disk for short), though perfectly usable in Linux, will not boot under BIOS systems. You need EFI or UEFI systems to boot GPT disk - that was the conventional wisdom that I subscribed to; and that was the piece of wisdom I was about to dump to the inquirer.

The good thing is that I am aware that the wisdom I subscribed to is rather old; that there may be some advancement in the fields; and especially that a GPT partition usually has a MBR protective partition which in theory could contain bootable code just like a standard MBR does; so I decided to do some quick research and tests first to avoid embarassing myself by giving the wrong answer.

It was a good decision as it turns out that GPT disk is indeed bootable under BIOS too! The answer actually depends on the BIOS and the bootloader.

As long as the BIOS in your system is happy to load MBR code and execute it - you're already halfway to the happy end. Some BIOS-es only check that the MBR is valid and will happily execute it if it does (these are the good ones), some are more tricky and check whether the MBR has a valid bootable partition (this is still workable though it needs more work), but some are recalcitrant and need a specific crafted MBR to boot from it - if you happen to have one of these, your luck may have run out.

The second part is the bootloader. GRUB2 definitely supports booting from GPT disk, and so is syslinux. Legacy Grub (Grub 0.97) does not, however there seems to be a patched version which does. Grub4Dos definitely does not support it, at least at the time of writing.

Okay, so how do one configure it? Every bootloader is different, of course, but I'll set the example with syslinux, as it is pretty easy.

For the following I will assume that your GPT disk is /dev/sda and your (soon-to-be) bootable partition is partition 3, that is, /dev/sda3. Obviously change all references to /dev/sda, /dev/sda3 and the partition 3 to your own setup.

1. Use GPT partitioning tool to mark that the partition is going to be bootable.
If you use gparted, choose "Manage Flags" and tick "legacy boot".

If you use sgdisk, type
sgdisk -A 3:set:2 /dev/sda
which means set bit 2 ("legacy BIOS partition") on partition 3 of disk /dev/sda to "on".

If you use the interactive gdisk (by typing "gdisk /dev/sda"), enter the "extra functionality (experts only)" by typing "x", and then choose "set attributes" by typing "a". You will be asked for which partition to update, choose "3", and from there you will be asked which bit to enable, so choose "legacy BIOS bootable" by typing "2". Don't forget to save your changes by typing "w".

2. Next, mount the partition somewhere, say in /mnt/data
mount /dev/sda3 /mnt/data

Let's assume that the partition (for the sake of the discussion) is a Linux filesystem such as ext3 or ext4.

3. Use syslinux to install the boot loader, pretending that it is a regular partition:
extlinux -i /mnt/data
extlinux -i /mnt/data/boot
if you want to keep syslinux files under /boot directory. Then please add the usual syslinux.conf configuration to boot your system - location of your kernel, initrd, rootfs, etc.

4. The final magic is to copy syslinux's MBR code which is capable of booting GPT partition. This code is stored (usually) in /usr/share/syslinux/gptmbr.bin. All you need to do is copy this to your disk's MBR.
dd if=/usr/share/syslinux/gptmbr.bin of=/dev/sda bs=440 count=1

5. Congratulations, you're done! Do the usual cleanup (un-mount /mnt/data, etc) and then reboot.

6. Assuming however, you have done all these correctly and BIOS still won't boot the system (saying no bootable disk found etc), you may need to mark the protective MBR as bootable. You need to do this using "fdisk". Start "fdisk" on the disk (fdisk /dev/sda) and ignore the warnings. Type "a" to toggle the bootable flag, and choose "1" for the partition to enable. ("1", not "3", because now we are talking about the protective MBR, and there is only one partition in the MBR - the protective partition). Don't forget to save your work by typing "w". Note that doing this is actually discouraged, see for example: (Matt Garret is the expert on all things UEFI, he is the author of the 'shim' bootloader used by many Linux distros which provides compatibility with UEFI and Secure Boot) - but if this is what it takes for your system to boot, then so be it.

Caution 1: Installing bootloader is not for the faint of heart, and you'd better have a backup plan if this is your *primary* computer, in case you're doing it wrong and the system cannot come to life after that. Installing a bootloader also means that ALL your previously installed operating system will not boot unless you configure the new bootloader to boot them. If you don't understand the consequences of the actions described in the above steps, you'd better stay away from doing this altogether. You have been warned.

Caution 2:The method described above assumes that you use a GPT with a standard protective MBR partition. In this scheme, you only have one partition defined on the MBR - partition of type 'ee' which covers the entire disk; and this is the recommended way to do it. gdisk actually offers another type of protective MBR called "hybrid MBR", in this case, the GPT partition definitions are copied and mirrored into the MBR partition definitions - if you have 3 partitions in GPT you will have 4 partitions in MBR (3 GPT + original protective). This method is highly discouraged, and the reason why is written here: (by the author of gptfdisk itself).

Posted on 23 May 2013, 5:41 - Categories: Linux
Comments - Edit - Delete

Fatdog64 sandbox as application container

Fatdog64 has a built-in sandbox for experimenting. It is an old feature, existing since as early as Fatdog64 520. It has been a useful feature, I use it almost always when it comes to building and packaging applications. But it has one main weakness - there can only be one sandbox running at one time. In other words, you cannot run multiple sandboxes at the same time.

That restriction was largely for stability reasons (long time ago, the sandbox didn't have proper termination code - you could accidentally close the sandbox shell and the sandbox will continue to run, with no way to properly shut it down). That has been addressed for a while now so it is no longer an issue.

But what purpose would it serve to be able to run multiple sandboxes? After all, sandbox was originally invented to "test foreign applications" by installing them in a throwaway memory filesystem (tmpfs). What good use could ever come out of running it multiple times?

The key answer is rw-sandbox. Sandbox actually consist of two applications - sandbox proper and a counter-part called rw-sandbox. rw-sandbox is an under-utilised, under-appreciated application that works and behaves just like the regular sandbox except that it stores all the changes (="state" if you will) into a persistent image file instead of in a throwaway memory layer. Thus it is possible to start an rw-sandbox session, do something and then leave, to return and re-start the session later and find all the changes are still there. The difference between sandbox and rw-sandbox is just like running Fatdog without a savefile (sandbox) and with a savefile (rw-sandbox) - in fact rw-sandbox's image has identical format as Fatdog's savefile.

As it turned out, rw-sandbox is ideal for isolate applications. One can install and run an application into a rw-sandbox, and keep its state - the binaries, the configuration files, etc, from one invocation to another - just like as if the application is installed on the main system itself. In fact, with rw-sandbox, I could install the same application in multiple rw-sandboxes, each with a different configuration. Pop! Instant "multiple-profile" for applications even for those who don't support it natively. I could also install different version of the application, without conflicting with each other.

This also happens to be another way to deliver packages - instead of the regular PET package or SFS package, one can deliver a compressed rw-sandbox image. It has the benefits of containing both the application binaries *and* configuration in one place - the rw-sandbox. (Using PET, both the binaries and configs are stored in savefile. Using SFS, the binaries is in the SFS but the configs is in the savefile).

In other words, one can use rw-sandbox as an "application container". And what makes it more useful is that since the application (and all of its configuration) resides outside the savefile, outside the OS), one can upgrade the system (within limits of compatibility), the OS, etc even wiping out and re-creating savefile from scratch if need to, and the application and its data is still faithfully there for you to use.

And how about backing up? As easy as backing up a savefile: copy the rw-sandbox image, and you're good to go.

Of course, it is only useful if we can run multiple sandboxes at the same time - otherwise it is too restrictive if when I run LibreOffice from the sandbox, I can't run Sweet Home 3D stored in different sandbox.

That would be motivation enough to modify sandbox, and so I did. Multiple sandbox can now be run at the same time; and this change will be in the next release of Fatdog.

Note: The sandbox is currently implemented as chroot jail. As such, it currently only isolates the filesystem, but nothing else. Applications within sandboxes shares networks, process ids etc with the rest of the system and thus malicious applications can easily kill each other, even on those on the "host" or on another sandbox. Thus this method is only suitable for "well-behaved" applications. Obviously this method would also not be suitable for "network" or "server" applications (those that listens on sockets), unless you can configure which socket to listen/connect to so they don't try to stomp on each other's network sockets.

If you want complete isolation then consider using other methods, such as User Mode Linux (UML) discussed here (also included in Fatdog64, see this FAQ entry) or even KVM/Qemu (available from the Package Manager). In the future, I may add Linux Containers too.

Posted on 23 May 2013, 3:36 - Categories: Fatdog64 Linux
Comments - Edit - Delete

Detecting keyboard state in shell

A discussion happened recently and a need was raised to detect key states (whether a certain key is pressed) in shell, see here (second comment). So I went googling for a while, found the answer here: and tidy it up a little bit, as attached.

Usage: kbstate key [devpath]
Returns: 0 if key is pressed, 1 otherwise, 255 for errors.

key is either lshift, rshift, lalt, ralt, lctrl, or rctrl (that's left/right shifts, left/right alts, left/right controls) in English keyboard, not sure the equivalents in others. If you hack the source you can easily check for other keys too.

devpath is the path to the Linux's keyboard event device. The default value is /dev/input/by-path/platform-i8042-serio-0-event-kbd (that is, the built-in PS/2 keyboard) but most of the time you would specify /dev/input/event5 or something. Scan /proc/bus/input/devices, find a device whose handlers is "kbd", and you know which /dev/input/eventXXX you need to specify.
If you need to test multiple keyboards (PS/2, USB, etc) you will need to call kbstate multiple times, one for each devpath.

Get the source, 32-bit static binary and 64-bit static binary.

Posted on 16 May 2013, 23:20 - Categories: Linux
Comments - Edit - Delete

guess_fstype from busybox

guess_fstype is a small utility to - well, "guess" the "filesystem" that resides in a given block device. It is used extensively in Puppy Linux and its variants including Fatdog. Its job is so important that if you have a malfunctioning binary then most of desktop filesystem operations will cease to work correctly.

guess_fstype was originally cobbled from guess_fstype_* functions from mount.c in util-linux (long before util-linux had blkid), it came from the same set of functions which powers "mount --guess-fstype" (which also gives the binary its name).

Over time, more and more filesystem type detections got added, and the code becomes more and more messier. It still works, and it still works beautifully, but it is not easy to maintain. So I began to search for a replacement.

klibc has an explicit standalone "fstype" binary, which is used to do exactly what guess_fstype does (but its output is different). It is small and fast and static - it's a good candidate. Unfortunately, its repertoire of filesystem detection is less than sterling - it lacks popular filesystems such as FAT and NTFS. While it can be patched to add those two, the point of my exercise is to try to *not* to maintain the detection code myself.

I turned on to busybox, which also has filesystem detection code. As it turns out, it is relatively easy to use them for a standalone application, and to my delight they are rather complete too. So I created a patch that will create a guess_fstype-equivalent from busybox.

To use the patch:
1. Extract vanilla busybox 1.21 source
2. Apply the patch (get it from here)
3. Go to the volume-id directory
4. Type "make"
5. You will end-up with a bb_guess_fstype which can be renamed to guess_fstype for proper use.

Busybox's guess_fstype recognises less than the original Puppy's busybox, but I don't think this is a problem as what it lacks are "exotic" filesystems anyway (everyone ever user xiafs recently? ) and would be very rarely used, if ever, indeed.

You can fine-tune the filesystem to detect by editing the fake_libbb.h (removing the defines) as well as the Makefile (removing the object code from being linked in).

When new version of busybox is released, all that I need to do is to update this patch. There will be no need to separately maintain the filesystem detection code anymore. As for now, I actually included the f2fs filesystem detection code in the patch, but as soon as this is supported by busybox natively, it will be dropped.

This new guess_fstype will be in the next version of Fatdog.

Posted on 15 May 2013, 17:28 - Categories: Linux
Comments - Edit - Delete

Testing LF preloader

Okay, I have played with Linux Foundation's preloader for a short time. It currently doesn't play nice with Matt's shim - shim requires signed binaries; while preloader (in its current state) does not accept signed binaries The good thing is that they are working on it.

Posted on 15 Feb 2013, 22:30 - Categories: Linux
Comments - Edit - Delete

Linux Foundation Secure-Boot compatible UEFI boot loader released

We have been using Matt Garret's shim since last December for our Fatdog64 testing purposes. Our general feeling is that while it works (and it works well), it is not user-friendly.

Linux-Foundation's UEFI PreLoader fulfills the same function and needs are shim; and from what we can see it may be a bit simpler for those who just bypass Secure Boot (as opposed to shim, which intends to use Secure Boot to make Linux boots securely). But while it was conceived first, its release was delayed - shim got released first and that's why we use it.

But the news is in: It has been released!. This is good news, and we will probably play with it for Fatdog64. However, as Fatdog64 is so close to beta, we will have to make a decision whether to postpone beta or postpone the testing with the preloader.

We will see. Either way, Matt said that he and James Bottomley (the one who did the preloader) are planning to combine effort and merge their works. Meanwhile, let's just celebrate that we now have two alternative ways of working with Secure Boot machines. Ah, the joy of open source

Posted on 12 Feb 2013, 23:42 - Categories: Linux
Comments - Edit - Delete

Pages: ... [15] [16]