Fatdog64 811 is released
Fatdog64 811 was released on 10 September 2020.Release notes here, announcement here.
Get it from the usual mirrors: ibiblio.org, aarnet, uoc.gr, or nluug.nl.
Comments - Edit - Delete
New forum for Fatdog
Fatdog has, for years, since its inception, always been piggybacking on Puppy Linux forum. Firstly because of its obvious roots; also because we simply didn't have enough man power to run and police an independent forum ourselves.As the previous blog entry indicates, however, John de Murga, the owner of the Puppy Linux forum, sadly passed away, and the original Puppy Linux forum went down (for a time).
Behind the scenes people are frantically working to bring it up, but there are also efforts to setup a replacement forums, including one by rockedge.
By now, both efforts have been fruitful: the old forum has been resurrected and the rockedge forum has flourished with old and new members (re-)joining it.
The stewards of Puppy Linux has decided that the old forum will continue in read-only mode as archives, while rockedge forum will take over as the new Puppy Linux forum.
Rockedge has graciously offered a section of the forum specifically for Fatdog, for which we are very grateful.
After some internal discussions, we have decided to take the offer and continue the tradition of piggybacking on Puppy Linux forum. All of Fatdog team members have now re-joined there as well.
For your reference:
---------
The new forum address: https://forum.puppylinux.com (Fatdog posts will be in the Fatdog section)
The old (archived) forum (in case you need to review/check old postings for older version of Fatdogs): http://oldforum.puppylinux.com (Fatdog posts are in the Puppy Projects section)
We'll see you there in the new forum!
Comments - Edit - Delete
Goodbye to a friend - John de Murga
John de Murga (aka John Murga) was the owner of the official Puppy Linux forum, formerly located in http://murga-linux.com/puppy/index.phpIf you use Fatdog64, you will know that we hosted our support threads on the same forum.
But my involvement with the Puppy Linux forum dates back well before then. I started browsing the forum when I was still a Windows user back in 2006; and joined in 2007 when I was converted into Puppy Linux user - and had been a regular on the forum ever since. Until the forum went down in early July 2020.
---
But in all my years being John's forum, I don't really know John Murga. Not personally. Never met him. Never spoke with him. Not even knowing how he looked like - well, not until a few days before I wrote this. Not through my fault though, I only know bits and pieces that he himself chose to share, for he himself claimed to be 'the man of mystery'.
He and his work, however, left an indelible mark on my life. It was such a simple work. Setting up an online forum. What was so difficult about it? Get a web-hosting site. Click a button, and you've got a forum and up running. Easy. Everyone can do it, right?
No. Not every one can do what John did.
The forum he set up was the forum where I spent lots of my past life on. Where I got help when I started and when I stumbled. Where I, eventually, helped others as I became proficient enough. Where I, unexpectedly, met and became friends with people in the real world.
All of these because he had the tenacity to keep the forum going for 15+ years; and let the forum to govern itself instead of strict policing found in many others. He even had the magnanimity to allow people to post links of competing projects and forums. This was one of the factors why so many people stayed on and the forum grew to become the melting pot of folks from all walks of life, sharing the same interest - Puppy Linux, and Linux in general.
So it was great shock and sadness that I learn of his passing recently in May 2020. I was even more stunned to know that he had left a young family - a daughter and twins who were born shortly after his passing. I have young children myself, I could not even begin to contemplate how it must have felt, to be left so soon, and so suddenly.
Nevertheless, I would like offer a prayer of hope for whom that John's family, that John had left a legacy that not many people could manage to do: to change the life of thousands, if not then ten thousands. John had left many friends that he himself probably didn't know. I was one of them, one among the thousands. For us, he is a hero. In every sense of the word.
Goodbye, John, and all blessings in your next journey.
You will be missed, but not forgotten.
---
Eulogy from 01micko, the current steward of Puppy Linux
Eulogy from Barry Kauler, the creator of Puppy Linux
Comments - Edit - Delete
Fatdog64 updates - behind the scenes
Fatdog64 was last released on 23 Jan. And although we have been quiet, it doesn't mean that things stopped. Things are still going in the background, bug fixing, adding features, etc.Here are a few major things that has been going on, in no particular order.
1. Puppy Linux forum user ICPUG has indicated possible issues with Fatdog savefile residing on an NTFS partition, especially partition shared with Windows. This is due to the way Fatdog uses ntfs-3g: it runs with full permission control enabled. It makes NTFS behaves like POSIX filesystem and we can use it for save directory (not only save file), but on the other hand it makes Windows complain about not-granted access each time the partition needs to be accessed from Windows.
Based on this feedback, we have added "ntfsnoperm" boot parameter to disable that permission control. When this parameter is used, ntfs-3g is run without permission: it would behave just like FAT filesystem (all files and directories are owned by a given uid/gid specified at mount time). It would not be possible to use a save directory on NTFS, but at least the permission wouldn't be touched and Windows will stop complaining.
2. The same ICPUG also found an old bug related to the above: where the user "spot" would be unable to access a Downloads folder that has been relocated to a save partition if the save partition is NTFS. This has been fixed.
3. Fatdog has long supported btrfs. The kernel is built to support btrfs and we have the complete btrfs-progs included; and nominally we support having save directory on btrfs. But this does not actually work; because "aufs" - the unifying filesystem layer that we use to make the magic happen, does not support btrfs the way it supports other filesystem. For the techie: the problem is that aufs cannot have its xino file inside btrfs. It has to be somewhere else.
Thanks for our team member SFR (his Puppy Forum name), this was brought to attention and he shared a fix too. The fix was tested, worked on, and finally merged: now, save directory will work seamlessly on btrfs-formatted partition.
4. The original motivation for doing (3) above was actually to attempt to use a compressed filesystem. This has been a particular interest for me for years. The last time I explored this (a couple of years ago), btrfs wasn't mature enough and there was no other native writable filesystem with compression support. Sure, there were native filesystem with compression support - but they were all readonly (squashfs being the most popular one). Sure, a couple of FUSE filesystems support compression too (in fact it's their number one feature), but they were not actively maintained and being done in userspace, it was slow. I never got enough motivation to implement them properly with Fatdog.
Btrfs finally changed this equation. By now it should be considered mature enough (although probably not as hyped as before), and it supports compression as well. In fact, it supports three different compression algorithms: lzo, lz4 and zstd.
So as we fixed btrfs save directory support, we grafted compression support too. If the kernel parameter "btrfscompress" is specified, the compression support for btrfs will be enabled, using the algorithm specified on that parameter. This has been tested to work wonderfully and using zstd, the compression rate is about 30% on average.
5. We have an update on the in-house "screencaster" application (a program record the video of the display). It now has the ability to take repeated screen capture, better quality video, among other things.
6. Step, another member of Fatdog team, has also revamped the "Samba Shares" application. It has been re-factored and is heavily tested to work across different Samba servers (different Windows versions, etc).
7. We have also fixed a long-standing bug when running Fatdog with RAM layer operation, which can cause inconsistencies if the system is heavily used (a lot of filesystem access) when the "merge down" process happens. SFR found this to be irritating enough to find a solution that works better; and this got incorporated into Fatdog.
An additional feature was also added: the ability to remove whiteouts if they don't hide any files on the lower SFS layers, but not activated by default due to possible conflict with multisession operation (we haven't tested the interaction yet).
8. We also include the "updater script" which re-enable VLC's ability to play youtube video directly, without having to update VLC itself. The script is called update-vlc-playlist-luac.sh. SFR found this script.
And of course, many other smaller bugfixes and feature addition, as well as adding and updating packages on the repo as well.
So when will we have a new release? Well, it will be released when it is released.
Comments - Edit - Delete
Fatdog64 810 Final is released
Maintenance release, basically a bug-fixed version of 810 Beta.Release Notes and Forum announcements
Can't believe that Fatdog is coming to its twelfth year.
Download locations: ibiblio.org, aarnet, uoc.gr, nluug.nl.
Comments - Edit - Delete
Fatdog64 810 Beta is Released
Maintenance release.Release Notes and Forum announcements
Download locations: ibiblio.org, aarnet, uoc.gr, nluug.nl.
Comments - Edit - Delete
Fatdog64 801 is Released
It is a rolled-up updates, containing bugfixes and minor feature updates.Release Notes and Forum announcements
Get it from the usual locations:
Primary site - ibiblio.org (US)
nluug.nl - European mirror
aarnet.edu - Australian mirror
uoc.gr - European mirror
ISO Builder: Get it from here: http://distro.ibiblio.org/fatdog/iso/builder/ and choose the builder dated 2019.05 and the package list for 801.
Enjoy.
Comments - Edit - Delete
Fatdog64 800 Final released
Just two weeks ago we released 800RC. Based on the feedback and our own day-to-day usage, we feel that it is now stable enough and can used to replace the last stable version of Fatdog 721, hence the final release.The list of changes from 800RC isn't many, and you can check them out in the Release Notes. Forum Announcement is here.
You can get it from ibiblio and the usual mirrors:
Get it from the usual locations:
Primary site - ibiblio.org (US)
nluug.nl - European mirror
aarnet.edu - Australian mirror
uoc.gr - European mirror
In this release we also publish the ISO builder suitable for making your own custom versions of Fatdog64 800. Get it from here: http://distro.ibiblio.org/fatdog/iso/builder/ and choose the builder dated 2019.02 and the package list for 800.
Enjoy.
Comments - Edit - Delete
Fatdog64 800RC Release
About two and half months after the initial 800 Alpha release, we finally release the first Release Candidate (RC). There is one beta release in between on 20 December which I didn't get to announce here (Christmas time - busy days).As usual it's package updates and bug fixes - but mainly bug fixes. Not only regressions caused by new packages, but also long-standing bug fixes from earlier version. Hence the recommendation to update. On my last test, I could still run this release on a 1GB Intel Atom N450 Acer eMachines netbook from ca. 2012; so if your machine is similar or more powerful than that, you can run it too.
If things are going to plan and there is no embarassing bugs, this release will become final.
This is the Release Notes; and this is the Forum Announcement; but if you're not familiar with the Alpha or Beta release I would suggest you read a little bit on both. We will probably copy and consolidate all the changes for the Final release, but not until then.
You can get it from ibiblio and the usual mirrors:
Get it from the usual locations:
Primary site - ibiblio.org (US)
nluug.nl - European mirror
aarnet.edu - Australian mirror
uoc.gr - European mirror
In this release we also publish the ISO builder suitable for making your own custom versions of Fatdog. Get it from here: http://distro.ibiblio.org/fatdog/iso/builder/ and choose the builder dated 2019.02 and the package list for 800rc.
Enjoy.
Comments - Edit - Delete
The Road to Fatdog64 800 Alpha
Today, the first of Fatdog64 800 series is released to the wild - the Alpha release. Despite being labelled "alpha", this release has been tested for a few months and has been used day-to-day in real production machines (in varying degrees) for about two months by all of us, in the team.It has not been a smooth ride all along. Living in the "bleeding edge" means that you really need to prepare to bleed (that's why we don't update the base on every release - it would be downright impossible). Latest packages don't always build, and when they do, they don't always run, and they do, they don't always run stably, and when they do, they don't always work, and when they do, they don't always work correctly, and when they do, they don't always provide good performance, and so on, and so on - you get the picture.
But we have finally arrived. It may not be perfect, and it never will, but for us - it is good enough for day-to-day usage; so the decision to release.
This blog post documents a few of the stumbling blocks that we passed through on our way to the Alpha release. By sharing the knowledge, I hope that others on the same journey can avoid them.
Warning: the information that comes after this is going to be very technical. Don't worry if you don't understand it - just use whatever that you do.
The particular bug that took as considerable time (weeks) to solve was this: it's a bug that causes the desktop to unpredictable, involutary exit to the console.
And it turns out, this problem has __multiple__ underlying causes. Each cause ends up with the X server exiting (sometimes gracefully and sometimes not) so that we're dropped back the console.
Sometimes there crash messages in Xorg.0.log, sometimes don't. Sometimes the exit is immediate (X goes down and we're back in the console), sometimes it's gradual (applications start to fail one by one, before X itself finally gives up its ghost and dies).
Bug #1: fontconfig doesn't like its cache to be tampered with.
This bug happens only when running with the RAM layer (savefile=ram:device:xxx, in Puppy's parlance this is pupmode=13). When anything that uses fontconfig starts (e.g X, gtk2, etc), fontconfig will be initialised and its first action is to scan all the font directories and makes its cache.
When we run using the RAM layer, these caches are stored in the RAM layer, and eventually will be "merged-down" to the actual savelayer (copying the files from the RAM layer to the savelayer, and then removing the copy on the RAM layer, and then refreshing the aufs layered filesystem so that the copy on the RAM layer gets re-surfaced on the root of the filesystem).
This has worked for the longest time, but we found out that in Fatdog 800 this isn't the case. Even the process of copying the files from RAM layer to savelayer (not even deleting them) triggered a cascade of failure in fontconfig, which eventually resulted a crash in all higher-level libraries and applications that uses this.
Fixes #1: We still don't really know what changed - this could be a change in the kernel, fontconfig itself, glibc, or others (remember, in 800, with a new base, **all things were updated** so we can't easily isolate one component from another), but once we know what triggered the collapse, we worked around it by making sure that fontconfig caches are not touched during the merging process.
Bug #2: Xorg server crash on radeon-based systems when DRI3 and glamor is enabled (the default settings).
This has been a long running bug due to Mesa (open-source OpenGL 3D library) changing its infrastructure to support newer, more powerful radeon cards, changing both the acceleration API (DRI2 to DRI3), acceleration methods (EXA to glamor), memory management (GEM, GBM, etc).
But the bug isn't in Mesa alone. Eventually Mesa needs to interface with the video driver, so co-operation with xf86-video-ati driver is needed.
And then there is the kernel too, the radeon DRM driver from the kernel.
There are multiple components and every component is a potential source of failure (which in this case they all contributed to the problem one way or another).
To make it worse, the problem is completely unpredictable. The system can run hours without a glitch before the desktop crashed, or it can crash in next hour after booting. It brought a completely un-reliable experience.
There is no good solution for this because every component keeps changing, so updating one component could very well break another. All we can do is watch bugzillas, forums, mailing lists, and listen to possible solutions.
Fixes #2: I think in this case, we got lucky. Most of the bugs were fixed in mesa 18.2.3, and the final bug was fixed in xf86-video-ati git-master, one commit after 18.1.0. We're going to stick with this combination for a while!
Bug #3: After an unpredictable amount of time, Xorg server will crash (due to failed assert), giving up messages similar to this:
[xcb] Unknown sequence number while processing queue
[xcb] Most likely this is a multi-threaded client and XInitThreads has not been called
[xcb] Aborting, sorry about that.
pidgin: xcb_io.c:259: poll_for_event: Assertion `!xcb_xlib_threads_sequence_lost' failed.
There is one unanswered report here: https://bugzilla.redhat.com/show_bug.cgi?id=1582478
This initially happened on ROX-Filer when it was worked heavily, so naturally we thought the problem was with ROX-Filer. But then it started to happen elsewhere (other GTK applications), so it could be anywhere in gtk2, glib, glibc, aufs kernel module, or even the kernel itself.
We scoured for solutions for similar problems, and we got some solution like this: https://forum.freecadweb.org/viewtopic.php?t=28054
But they don't work and they don't make sense. FreeCAD is a 3D-heavy application, so disabling DRI3 and (in some others links, disabling 3D hardware-acceleration) doesn't make sense for applications like ROX-Filer which is purely 2D and doesn't make use of any acceleration at all.
This was especially difficult to pinpoint because it happened randomly; and it was one of the thing I would have filed into "unsolved" files, were it not for SFR. SFR found a way to reproduce this problem reliably (by clicking the "refresh" button on ROX-Filer a few hundred times - and he even provided a automation script so we don't have to buy a new mouse after every experiment
).
Once we can reproduce this, re-building the libraries with debug symbols and running them under "gdb" quickly pointed out that the problem is within libxcb - a library at the bottom of the Xorg stack.
Bisecting on libxcb, we found that the problem is caused by a particular code commit that tries to "fix" another bug when dealing with Vulkan drivers:
https://cgit.freedesktop.org/xcb/libxcb/commit/?id=fad81b63422105f9345215ab2716c4b804ec7986
But the way it was fix is, in my opinion, is incorrect (it was reading stuff when it shouldn't - this kind of thing should be protected with a mutex or we'll end up with a race).
Fixes #3: So we reverted this commit and poof! - the problem disappears. I tested clicking the button to about 16,000 times and no more crash.
This was the last bug we squashed.
So there. One symptom, three underlying problems, that can be triggered at different semi-random times, resulting in a totally different error messages and behaviour, confusing all of us.
We falsely declared victory after the first and the second were squashed - only to be humiliated when the crash happened again, in a slightly different way. By the time we squashed the last bug, we were wary enough __not__ to declare it fixed until a few days later and it was finally confirmed that we've finally made it through.
All in all, we spent more than a month to solve all of them. Now that we've past through them, I hope others can avoid the same mistake.
Meanwhile, enjoy Fatdog64 800 Alpha!
Forum announcement here: http://www.murga-linux.com/puppy/viewtopic.php?t=114719
Release Notes here: http://distro.ibiblio.org/fatdog/web/800a.html
Comments - Edit - Delete
