Pseudo-terminals and Xterms

Ok, so I've been trying to work out a nice way to get input and output from our simulator separate to the input and output to the simulated program. This is particularly important if we want to have user control of the simulator while the simulation is running.

This is exactly what pseudo-terminals are good for. On Solaris, see ptm(7D) and pts(7D) (or go to Sun's online web-pages: ptm(7D).

Essentially a pseudo-terminal is like a bidirectional pipe(2), but also with terminal semantics, so there are special characters so the user can do things like backspace, line erase, etc.

A pseudo-terminal has a master end and a slave end. The master end is where the user is (expecting output and reading user input), the slave end is where the program is (expecting user input and writing output). The slave end will respond to the usual terminal ioctl(2)'s that the program wants to use (for example, TCGETS, in order to get a copy of the termios structure from the terminal). It turns out (see below) that this ordering (master=user, slave=program) is crucial.

Ok, so I could use a pseudo-terminal, but I still needed a way for the user to input into that pseudo-terminal: I wanted to use the actual terminal for simulator input and control. It turns out that xterm(1) has a switch exactly for this purpose: -Sccn. The manpage says:

     -Sccn   This option specifies the last two  letters  of  the
           name  of a pseudoterminal to use in slave mode, plus
           the number of the inherited  file  descriptor.   The
           option  is  parsed ``%c%c%d''.  This allows xterm to
           be used as an input and output channel for an exist-
           ing  program  and  is  sometimes used in specialized
           applications.

However, modern pseudo-terminals are created using the /dev/ptmx and ptsname(3C) interface (rather than using the old, BSD-style search through /dev/ptyXY where X is [pqrs...] and Y is [0-9a-f]. The -Sccn switch was designed for ye-olde BSD type of pseudo-terminal days.

It turns out that newer versions of xterm know about this, and have extended the switch to allow -Sccc.../d where the ccc...'s are the trailing part of the slave device name (the basename) and the d is the device descriptor (with a / between them to separate them). However, Solaris's xterm doesn't support that.

But it also turns out that xterm ignores the cc characters these days anyway. How did I find this out? I read the source. Yuck, but true. Although I didn't read the source for the Solaris version of xterm it seemed likely that it ignored it too. Especially since Solaris's CDE terminal dtterm(1) also had a similar -Sccn switch and an extension switch:

        -Sc.n
            Equivalent to -Sccn, but provided for systems with
            a larger pseudo-terminal device name space.  The c
            argument specifies the last component of the
            pseudo-terminal device slave name.  The terminal
            emulator ignores this value and the value may be
            empty.  The n argument specifies the number of the
            file descriptor that corresponds to the pseudo-
            terminal device's already opened master side.

So I wrote a little program to test this out. See pseudo.c.

When I first tried this I was getting really weird results: the xterm wouldn't show anything the user typed, but the characters would flow one-by-one to the parent program (i.e., not line-based as I expected). Also, anything I wrote back from the parent program would not only get printed on the xterm window, but would also end up back in the input of the parent program's end of the pseudo-terminal.

After much fiddling and thinking, I realised that what I wanted to do with the pseudo-terminal is different to most examples of it: I wanted to give the master to the child program, not the slave: I wanted the child to be the user input and the parent to simply be a proxy between the user and the simulated program (going back to the original simulation IO problem). So after simply switching slave and master, the program worked as expected.

However, I also got an additional line of input (both appearing in the xterm and in the parent program's pseudo-terminal input). This line contained a hexadecimal number. After reading the xterm source, it turns out it writes the X-window id of itself to the terminal so the parent program can control it if it wants to. That also appears in the xterm itself since by default user input is echoed (terminal control local flag ECHO).

I'm not interested in the xterm's X-window id, and I figure it could be confusing to see an xterm pop up with 0xc0002 or somesuch in it. So there's some fudging going on that ensures that we don't see it: Firstly, before forking the child I get the pseudo-terminal (default) settings, and then I clear ECHO. So when the xterm pops up, it will write its X-window id, but it won't appear in the xterm. Then post-fork, the parent reads and discards a line from the xterm: this should be the xterm's window id. The parent then restores the original terminal settings, so ECHO is set if it was already, which appears to be the default.

To get this working I did a lot of reading: Stevens' "Advanced Programming in the UNIX Environment", the Solaris man-pages, and the sources to xterm (CVS) and expect (CVS).