Turbovnc now plays well with xrdp
I've recently toyed with xrdp to see how far it has progressed. Quite to happy to see that it finally has x11rdp back on the table in the latest git-master (0.7.0). During my experiment, though, I noticed that I can't login to any of sesman-Xvnc session. There was no error message from xrdp-sesman or from Xvnc (which was launched by sesman) itself, and there was no message from the rdesktop screen either. Surprisingly, I can *connect* to the underlying Xvnc server, but xrdp can't seem to connect to it, no matter what I did. It just seemed to hang during early protocol negotiation phase. For the record, my Xvnc is actually turbovnc.Naturally, I did a google search and I only found one occurrence of someone complaining that turbovnc doesn't play well with xrdp, with no solution (the only solution was to use tightvnc instead); this was from 2011. Hmmm. There was another somewhat unrelated 2010 post that they managed to use turbovnc with xrdp, but were having problems with x11rdp. I wasn't using x11rdp, but the interesting part is that it seemed that they could get turbovnc play well with xrdp - but no clue on exactly how. Mmmm, okay. Time to roll-up the sleeves again.
Initially I thought the problem was with xrdp, so I spent some time debugging it. I came to the conclusion that the hanging situation happened because xrdp (or rather, its vnc plugin) was waiting for authentication acknowledgement from the Xvnc but never got it. Upon closer examination and comparison with RFB protocol specification, I was quite certain that xrdp implemented the protocol correctly. But why did it hang, while my regular vncviewer (which is tightvnc's, not turbovnc's), didn't?
As I went on for a little while, I noticed that (tight)vncviewer connected to the (turbovnc) Xvnc using RFB protocol version 3.8. This protocol is quite different from the one used by xrdp; it used an earlier version, version 3.3. So I got tightvnc source, and hacked it a little to force it to use RFB 3.3 instead of the latest supported (in this case 3.8). Look and behold - this time the vncviewer hung, too! There must be something weird in turbovnc's implementation of the version 3.3 of the RFB protocol.
A little more debugging later, I came to the conclusion that there was indeed a bug in turbovnc's implementation of RFB 3.3 and 3.7 protocol (basically anything other than 3.8). In addition it also has a bug that render "no authentication" (ie, password-less) connection useless. Once found, it was easy to fix both of them.
It is to be noted that most vncviewers these days implement RFB 3.8 protocol; and most people will run Xvnc with authentication, so it is unlikely that you will encounter either one of these bugs. But if you do, and you need a solution, hit into my patches page and get the patch from there. Oh and by the way my patch also changes the root of X11 to be in /usr instead of /usr/X11R6 (that was long ago), and the location of the font directory (FontDir), as well as disabling PAM by default (you can re-enable it by removing the patch that disable it - it should be obvious which one). The patch is meant for turbovnc 1.1 but it will also apply cleanly to turbovnc 1.2 (which still has the bug).
As a happy ending, with the patch xrdp now works beautifully with turbovnc.
Edit - Delete
No comments posted yet.
