Fatdog64 630RC1 is released
Release notes here: http://distro.ibiblio.org/fatdog/web/630rc1.html.Get it from ibiblio.org http://distro.ibiblio.org/fatdog/iso/ or its mirrors.
Forum thread: http://murga-linux.com/puppy/viewtopic.php?p=730926#730926
Highlights:
- kernel 3.11.4 with dynamic power management support for radeon graphic cards (lower temperature, better battery life, better performance). Please see this post from the radeon driver developer himself to see how to control the power/performance balance.
- updated Xorg and multimedia libraries
- multiple sandboxes (as first discussed here).
- LXC-based sandbox (as first discussed here). Note that the current 630RC1 kernel doesn't support user namespaces because it still conflicts with XFS and we feel that XFS is more important across the board than user namespaces. This may change if we decide to adopt kernel 3.12 which has resolved this issue.
Comments - Edit - Delete
Fatdog64 Update
I have been busy with FatdogArm in recent times but it doesn't mean that Fatdog64 is ignored. In fact, the development of the next version of Fatdog64 is currently in the works now.Fatdog64 development usually alternates between two phases - development phase and release phase.
During development phase, kirk and I get very active and do all the usual stuff you usually attribute to "development" - and in between we will release alpha or beta releases in quick succession for others to tests so that we can squash the bugs. When the quality is good enough, we will make a release - this marks the beginning of the release phase.
During this release phase, generally all development slows down except for immediate bug fixes and/or experimental ideas for the next release; it also helps to relieve us from burnouts and focus on other things in life. After a while (usually a period of between 3 to 6 months), the development phase will begin again.
Fatdog64 621 was last released on 8 May this year, so it is due for a refresh, which is in the works. One of the thing we waited before we update Fatdog64 was for kernel 3.11, which, among others, comes with improved power management for radeon cards.
The next release of Fatdog64 will still be based on the 600-series base; this (barring unforeseen circumstances) will be the final release of the series, and expected to last until early next year - thus making the entire series to last about two years from its initial release (which is another Fatdog tradition - Fatdog64 500 series also last for about two years from its initial release).
It is exciting times; we have Fatdog Next running in our system and while we still have to tweak it, things are looking good.
Comments - Edit - Delete
Updates
I added information on how to customise FatdogArm here; and I added a sample script on how to run Slacko side-by-side with Fatdog64 here.Comments - Edit - Delete
Speeding up ARM compilation
Compiling on the ARM (Mele) is slow. I recently tried to build the latest version of Seamonkey (SM) 2.20 for FatdogArm. As I said in this post, each build takes about 15-18 hours on the Mele. Yes, it is that slow. Now in the case of SM 2.20 it is even worse. Even after 24 hours it failed to build - because apparently 512MB RAM is now no longer enough to build it (it fails at the link phase). I tried twice - each totalling about 24 hours - and both failed. That's two days wasted.I wanted to try again but I wanted to have a better way than this. I can't be spending 24hours for every build. There must be a better way than this.
Fortunately, there is - enter distcc, a distributed C compiler. distcc allows you to combine the power of several machines to simultaneously compile a single package. The machines don't even have to be identical, if they have the appropriate cross-compiler installed they can be used to join compilation-cluster of other platforms.
Using distcc, the ARM machine acts as the main controller, spreading the compilation load to other, faster machines. From the package build standpoint, however, it still looks and feels like native compilation. Thus, all the ease of native compilation with the speed of cross-compilation.
Using Fatdog64 in combination with FatdogArm, I managed to cutdown the "compilation phase" of SM 2.20 from 15 hours to 3 hours (plus another 3 hours for linking on either side). That's a 5-fold increase in speed, allowing me to perform 4 builds in 24 hours.
Do I get your attention already? This is how to do it.
The fight with SM is still on-going though. Despite the fact that I can compile much faster now, SM still refuses to build because it trashes the swapfile during libxul.so linking stage. In simple words - it is running out of memory. My Mele only has 512MB and apparently it's not good enough, and I don't have any other hardware with more memory, so I'm now building it using distcc-assisted Qemu (here for details of running FatdogArm in Qemu) - using the Vexpress emulation with 1GB RAM. This is similar to what Aboriginal Linux does. For SM build, however, it is very slow, because while the C and C++ compilation is much faster, SM has tons of python code as part of its build system - and python code can't be farmed out to other machines
. Still, without distcc, I wouldn't even consider to do it in Qemu at all.Comments - Edit - Delete
How to run Slacko Puppy side-by-side with Fatdog64
Somebody asked me recently whether it is possible to run Slacko Puppy side-by-side (or along side) Fatdog - that is, without the need for dual-booting etc.It is possible.
In my original response I provided the outline steps of how to do so, but now I thought others may be interested too - so I decided to write an article about it. The article is here.
Enjoy.
Comments - Edit - Delete
Linux Container in Fatdog
IntroductionAs noted in this and this post, I have been experimenting with Linux Containers (LXC or lxc for short) lately.
What is LXC? In short, it is a collection of technologies that allows processes to be compartmentalised. A process in one compartment (or "container", in LXC-speak) cannot interact with processes in other compartments. It's the equivalent (in terms of concept) of FreeBSD jail in Linux.
What is it good for? It's good to secure 'server' processes, as when these processes are broken, an attacker can only affect things inside the container (presumably, each server process is allocated one container to limit this kind of damage). This has been the basis for the so-called Operating System Virtualisation.
In this respect, LXC is late to the game. FreeBSD has its jail for years. Solaris as 'Solaris Container', and so is many other operating systems (except Windows, probably
). In Linux alone, it has been preceeded by Linux VServer and Virtuozzo/OpenVZ by years. They also have more features that LXC currently doesn't have. LXC however, has the benefit of being available from the vanilla kernel, not patches needed - just compile-time configuration.
More information about LXC: here (LXC manpage), here (IBM developerworks), here (Ubuntu community wiki) and many others you can find using your favorite search engine.
LXC support in Fatdog
In next version of Fatdog, the kernel will support LXC.
Many years in the making, the final component of LXC ("user namespaces") was merged with the kernel in Linux 3.8. Unfortunately, this component requires extensive changes to other kernel components that when many of them were not ready for this change; thus making 3.8 kernel less than ideal for running LXC. The situation is better in Linux 3.9 - only the XFS filesystem still doesn't work with user namespaces. I was hoping that 3.10 would fix this but as of today, at 3.10-rc4, I see don't see it happening, so if we ever need to ship Fatdog with one of these kernel, we will have to choose either to enable XFS filesystem support, or to enable user namespaces. I hope we don't have to choose by the time we release. LXC can still run without user namespaces feature, although it will be less secure.
In addition to the kernel, one needs to install LXC userspace utilities from the Package Manager (this model is similar to User Mode Linux support in Fatdog - the scripts are there, but you need to install the userspace utilities and the UML kernel first).
Now, when creating the Fatdog convenience interface to LXC, I have two choices:
a) follow the sandbox model (zero configuration)
b) follow the UML model (persistent configuration)
I have decided to follow the sandbox model (zero configuration). The scripts are called sandbox-lxc (and rw-sandbox-lxc for the one with persistent storage). The reasons I go with this model is:
a) it is slightly easier to use
b) if one needs to run an LXC container with persistent configuration, the LXC userspace tools themselves already supports this model (see lxc-create and format of lxc-template config-file, so there is no need to duplicate this excellent functionality.
c) lastly, with the sandbox model you can easily copy/transfer/backup the container root filesystem from the "host" (e.g. a tool like sb2dir.sh will work; if you need to copy files to the sandbox just copy it over to the sandbox's fakeroot).
How to use these scripts? They are identical with sandbox scripts, so instead of sandbox.sh you run sandbox-lxc.sh. Instead of rw-sandbox.sh you run rw-sandbox-lxc.sh.
When run, in addition to asking about the layers you want to use, it will also ask whether you want to be able to launch a desktop. If you said yes, the host's /tmp will be mapped inside the container - which means the processes inside the sandbox can change stuff in the host system and connect to processes in the host - but it is a requirement if you want to run Xnest from inside the container.
These scripts by default will run without user namespaces. You can tell them to run with user namespaces by putting IDMAP=yes to the environment before running them, like this: IDMAP=yes sandbox-lxc.sh. Support for kernel user namespaces is auto-detected, if the kernel doesn't support it IDMAP will have no effect.
When run without user namespaces, these capabilities are dropped: sys_module (load kernel module), sys_time (change system time), and syslog (change kernel logging parameters).
When run with user namespaces, all the UID are shifted by 10000 (ie, the container's root is actually a user with UID 10000 in the system).
In either case, the container cannot create block device nodes (only character devices) and will have its own hostname and network interface which you need to configure manually if you want to connect to the network from within the container.
These scripts, the LXC-enabled kernel will be in the next release of Fatdog (the LXC-userspace tools are already in Package Manager but they are useless without LXC-enabled kernel). Beware that they are still considered experimental, as is LXC itself.
Comments - Edit - Delete
Fun with Fatdog
One of the forum member in Puppy Linux forum, Ted Dog, asked whether Fatdog can be modified to run with uncompressed filesystem in RAM, provided that one has enough RAM to do so. His idea is that since uncompressed Fatdog stands at around 700MB, on machines with more than 4GB it should be possible to have one giant initrd that contains uncompressed filesystem that got fully loaded into RAM and run Fatdog from there. Since there is no decompression is involved other than possible once at boot-up (if the initrd is compressed), it should run like the wind. The original post was here.There is no practical purpose of doing this (that I can see), but it sounds fun :)
There are two possible way of doing it:
1. Expand the content of the basesfs (fd64-620.sfs currently) and merge it with initrd.
2. Expand the content of the basesfs and store it into uncompressed ext2 filesystem.
In either case, it should be possible to include modifications of the original Fatdog stuff like a "remaster".
Fatdog already supports booting without any basesfs. It also supports basesfs with filesystems other than squashfs. So all that is needed is to prep the initrd.
For 1, what is needed is to get a snapshot of the running system, and do minimal modification so that it is bootable, and re-create the initrd from there.
For 2, what is needed is to create a basesfs that contains uncompressed ext2 filesystem instead of squashfs, and insert this basesfs into an existing initrd.
As it turns out - Fatdog already has the tools to help to do both: the sandbox. One runs sandbox to have a "snapshot" of the running system which one can play inside without affecting the real system. When one starts sandbox, one can choose which layers to be included in the sandbox - including the modifications already in the savefile. As it turns out, the sandbox root filesystem is also visible from outside the sandbox, in "/mnt/fakeroot" for 621 and "/mnt/sb/fakeroot" for 622 onwards. By choosing the correct layers, we can get the needed view for either 1 or 2.
From here it is straightforward.
For 1, one just needs to create the initrd from the sandbox root, after undo-ing some changes done by sandbox script, and restoring deleted links required for booting, and a minor modification to the initrd init script (/sbin/system-init).
For 2, it is even simpler - just create an empty ext2 image file, and copy over the sandbox root to that image file, again after undoing some changes done by the sandbox script. Then open up an existing initrd and overwrite the basesfs (fd64-620.sfs) there with the image file you've just created; then rebuild the script.
Fatdog's initrd currently stands at around 210MB. This "giant" initrd, when gzipped, is about 260MB. It's bigger because it is compressed with gzip instead of with xz; if one choose xz compression it will have more or less the same size. But this size difference is not important, either way the initrd will be decompressed before Fatdog starts - and the memory consumed will be the decompressed size of the initrd which is around 700MB. But then, as Ted Dog said, there will be no decompression overhead for anything, so it should run like the wind :)
I have written the details and the exact steps on how to do both 1) and 2) in the forum, so I will not repeat it here.
Method 1
Method 2
Comments - Edit - Delete
Fatdog64 sandbox as application container
Fatdog64 has a built-in sandbox for experimenting. It is an old feature, existing since as early as Fatdog64 520. It has been a useful feature, I use it almost always when it comes to building and packaging applications. But it has one main weakness - there can only be one sandbox running at one time. In other words, you cannot run multiple sandboxes at the same time.That restriction was largely for stability reasons (long time ago, the sandbox didn't have proper termination code - you could accidentally close the sandbox shell and the sandbox will continue to run, with no way to properly shut it down). That has been addressed for a while now so it is no longer an issue.
But what purpose would it serve to be able to run multiple sandboxes? After all, sandbox was originally invented to "test foreign applications" by installing them in a throwaway memory filesystem (tmpfs). What good use could ever come out of running it multiple times?
The key answer is rw-sandbox. Sandbox actually consist of two applications - sandbox proper and a counter-part called rw-sandbox. rw-sandbox is an under-utilised, under-appreciated application that works and behaves just like the regular sandbox except that it stores all the changes (="state" if you will) into a persistent image file instead of in a throwaway memory layer. Thus it is possible to start an rw-sandbox session, do something and then leave, to return and re-start the session later and find all the changes are still there. The difference between sandbox and rw-sandbox is just like running Fatdog without a savefile (sandbox) and with a savefile (rw-sandbox) - in fact rw-sandbox's image has identical format as Fatdog's savefile.
As it turned out, rw-sandbox is ideal for isolate applications. One can install and run an application into a rw-sandbox, and keep its state - the binaries, the configuration files, etc, from one invocation to another - just like as if the application is installed on the main system itself. In fact, with rw-sandbox, I could install the same application in multiple rw-sandboxes, each with a different configuration. Pop! Instant "multiple-profile" for applications even for those who don't support it natively. I could also install different version of the application, without conflicting with each other.
This also happens to be another way to deliver packages - instead of the regular PET package or SFS package, one can deliver a compressed rw-sandbox image. It has the benefits of containing both the application binaries *and* configuration in one place - the rw-sandbox. (Using PET, both the binaries and configs are stored in savefile. Using SFS, the binaries is in the SFS but the configs is in the savefile).
In other words, one can use rw-sandbox as an "application container". And what makes it more useful is that since the application (and all of its configuration) resides outside the savefile, outside the OS), one can upgrade the system (within limits of compatibility), the OS, etc even wiping out and re-creating savefile from scratch if need to, and the application and its data is still faithfully there for you to use.
And how about backing up? As easy as backing up a savefile: copy the rw-sandbox image, and you're good to go.
Of course, it is only useful if we can run multiple sandboxes at the same time - otherwise it is too restrictive if when I run LibreOffice from the sandbox, I can't run Sweet Home 3D stored in different sandbox.
That would be motivation enough to modify sandbox, and so I did. Multiple sandbox can now be run at the same time; and this change will be in the next release of Fatdog.
Note: The sandbox is currently implemented as chroot jail. As such, it currently only isolates the filesystem, but nothing else. Applications within sandboxes shares networks, process ids etc with the rest of the system and thus malicious applications can easily kill each other, even on those on the "host" or on another sandbox. Thus this method is only suitable for "well-behaved" applications. Obviously this method would also not be suitable for "network" or "server" applications (those that listens on sockets), unless you can configure which socket to listen/connect to so they don't try to stomp on each other's network sockets.
If you want complete isolation then consider using other methods, such as User Mode Linux (UML) discussed here (also included in Fatdog64, see this FAQ entry) or even KVM/Qemu (available from the Package Manager). In the future, I may add Linux Containers too.
Comments - Edit - Delete
Fatdog64 621 is released
Belated announcement (my DSL was down), but anyway ...Forum Announcement: http://murga-linux.com/puppy/viewtopic.php?p=698707#698707
Release Notes: http://distro.ibiblio.org/fatdog/web/621.html
Get it from: http://distro.ibiblio.org/fatdog/iso
Fatdog64 621 fixes the most annoying bugs from 620 - the "eject problem". Mostly a problem when you use a desktop with an optical drive that can be commanded to close its tray, as well as enabling panning to work properly.
It also has a new on-shutdown filesystem cleanup code which in theory should work identically if not better than before.
Other than that it is identical to 620 release - same kernel, same browser, same devx.
Comments - Edit - Delete
Fatdog64 620 Final is released
I'm running out of time, so I will short and sweet. Fatdog 620 Final has been released with a lot of improvements. Check it out!Announcement: http://murga-linux.com/puppy/viewtopic.php?t=85684
Release Notes: http://distro.ibiblio.org/fatdog/web/620.html
Get it from: http://distro.ibiblio.org/fatdog/iso
Comments - Edit - Delete
