Hans Hübner has been at it again, and has developed code to allow a Teensy AVR development board to adapt a Symbolics keyboard to USB.

I ordered a Teensy board of my own, and was hoping to develop a small adapter to avoid cutting up my keyboard cable, but Teensy does not work with my old Mac OS X 10.4 system.

I wanted to post a link referring to recent progress on adapting Symbolics keyboards to modern PCs. Hans Hübner announced that the kbdbabel project (apparently based on 8051 microcontrollers) was able to be adapted to support the slim-line (Rev. C) Symbolics keyboard, adapting it to PS/2 protocol.

My own work in this area ground to a halt pretty soon after I received the ADB adapter from DKS. I’m happy to see that others have been able to make progress!

UPDATE: Rainer Joswig has commented on his experience using the kbdbabel adapter to connect a Symbolics keyboard to a Macbook running Mac OS X.

UPDATE 2: Hans has developed a USB adapter solution.

As I alluded to in an earlier post, Symbolics Genera takes a slightly different approach to packages than most current environments (such as SLIME under Emacs).

Part of that is because, unlike GNU Emacs today, the editor and all its intelligence is running inside the Lisp environment; another part is due to Genera supporting multiple dialects of Lisp, each slightly compatible with the other.

One consequence is that Zmacs pays close attention to the “file attributes line” in your Lisp source files. (That’s the line at the beginning of the file which contains the marker “-*-”). Other Lisp environments use a heuristic, like looking for the first IN-PACKAGE form.

A chicken-and-egg problem arises when loading a file that expects to be read in a particular package. What if the package doesn’t exist? Well, it isn’t possible to use that package to read the file. But the package itself is defined in a file.

For me this caused a bit of confusion, because if you try to load a file with a non-existent package defined in the attributes, and you have not given Genera permission to create the package, it complains that “<package name> is not meaningful as a package name in <Lisp dialect>”.

You can press Resume to allow it to be created, and then the problem goes away until you try to bootstrap again without the package existing.

The answer is found in the documentation section “Specifying Packages in Programs.” If the package attribute is enclosed in parentheses, it is automatically created if it does not exist. Furthermore, you can include one package name, or a list of package names, as the second element of the list in order to specify packages which this package uses.

The other way to approach this is to create a system definition file which defines the packages up front.

(Another aspect that made me more confused is that the package name was “6502″, which if in the package attribute without quotes will read as an integer, which cannot designate a package. I had to muck around a bit trying double-quotes and vertical-bar escapes, in an effort to make the problem go away, before I realized the “not meaningful” was not referring to violating some non-existent rule about package names. From time to time, I would create the package and not realize that made the message go away.)

Having hacked on the COMFY code for a while in Emacs Lisp, I got the itch to move it over to my MacIvory, which has been waiting for real work to do. I’ve crafted some tests in the Emacs Lisp using elk-test (trying to add heirarchical suites-of-suites along the way), and wanted to port them to one of the Common Lisp test frameworks (I’m thinking of checking out Stefil, LIFT, and rt, which I’ve used briefly). While copying the files, I ran into a few issues.

1.  I couldn’t figure out an automatic way to handle line-ending issues in the text files. I ended up locating the file on the Mac OS side in the file browser, Mouse-R clicking to Edit the file, then M-%, C-q [LINE key] [Return] C-q [Return] [Return] ! in Zmacs, then saving to a file on the Ivory file system. (I like the offer to create non-existent directories on a save.) There is probably a Copy File(s) command I could have used to move them en masse, and I should have been able to code up a quick conversion routine or Zmacs keyboard macro, but it worked.
2. Trying to compile the RT framework, I think I had been misled by my experience in the Genera Development Tutorial. The Common-Lisp mode apparently is Common Lisp the Language, 2d ed. What I was looking for is ANSI-Common-Lisp. The confusion cause all sorts of problems with DEFPACKAGE, and the presence of FUTURE-COMMON-LISP made me even more confused. An e-mail on the SLUG list apparently explains the various Common-Lisp strategies on Genera.

Working with ResEdit and DeRez, I believe I have traced out the keyboard protocol used by the MacIvory ADB adapter. My adapter is on the blink, probably because of the small screw I found rattling about inside. I hope DKS can bring it back to life, or the PIC-based project I had begun will make its way back to the front burner.

The ADB adapter has not just given me some help uncovering the behavior of the Rev. C Symbolics keyboard protocol, but also works when attached to my MacIvory. Along the way (and with the help of David K. Schmidt) I encountered some glitches, mostly stemming from my original accidental trashing of the System file DKS installed.

There are a few software components the ADB adapter box needs to convert the Symbolics keyboard to something the Mac OS 7.6.1 (including the “FEP” window which communicates to the Ivory board on a low level) will accept as a keyboard with the right keymapping, and furthermore to something the Genera application will recognize with all the Symbolics-specific keys. Without these components, you will see one of the following two stages of malfunction.

Stage 1: The mouse connected to the ADB adapter works fine. In the Mac OS (including the FEP window), there is total keyboard confusion, with the keys all scrambled. On the Genera side, things are less scrambled, but some keys are missing, such as “e”, “t”, “u”, and “o”; the “o” in fact causes the Mac menu bar to flash. Modifier keys seem absent. If you activate the keyboard control application, Genera seems to crash easily. Peter Paine reported this at the time System 7.5.3 came out.

If you do a custom install of the “Symbolics Keyboard for System 7″ selection in the MacIvory software installer, you get the Symbolics Keyboard extension icon at Mac OS startup, and a Symbolics keyboard layout choice in the Keyboard control panel. The situation improves, reaching

Stage 2: The alphanumeric keys work for both the Mac OS and for Symbolics. However, Genera treats a press of the “Page” key as a press of the “Line” key, treats “Escape” as “Select”, and ignores the \{ and |} keys, the Hyper modifier key, and most of the large function keys (Function, Refresh, Square, Circle, Triangle, Suspend, Resume, Abort, Network, Local, Select, Line). Kent Pitman reported this after an upgrade to System 7.5.

The solution is to do a full install of the MacIvory support software. I believe this is necessary to get all of the resources installed in the Mac’s System file.

I’ll try to gather more details on the resources; it would be nice to understand how to make a box that takes, for instance, a PS/2 keyboard & mouse and connects it to the MacIvory, although the Symbolics keyboards and mice may be more plentiful than than the Ivory boards at this point, so it might be mere archaeology. More practical, if ADB adapters are scarcer than Ivory boards, would be a combination of the Apple keyboard (which works well enough for MacIvory use) and a three-button PS/2 mouse with a “partial ADB adapter” allowing it to provide multi-button mouse function to Genera.

UPDATE: I examined the resources, and was able to decode the Symbolics ADB adapter encoding.

The ADB adapter has fulfilled its main purpose: providing an example of hardware which can speak to my Rev. C Symbolics keyboard.

The apparent trick is that the clock pulses must be narrow, without being too rapid (i.e., a relatively low duty cycle.) A 10 microsecond long low pulse on /CLR, followed 30 microseconds after the beginning of that pulse by the clock going low for roughly 9 microseconds, continuing with a clock period of approximately 50 microseconds does the trick. The ADB adapter pauses somewhat between groups of 11 clock pulses, corresponding to the natural “section breaks” of the keyboard mapping; I don’t know if that is strictly necessary. My first crude approximation, using bit-bashing for the PIC with timing loops resulted in Caps Lock and Mode Lock that felt a bit sluggish—needing to be held down relatively long to “lock” or “unlock.” The overall cycle time may be important in that regard.

In any event, I reconstructed the keyboard mapping from the LMKBD project, although I chose to count pulses from zero, rather than n=1 in that listing.

Unfortunately, this seems to make the PIC EUSART not fit the application: its clock pulses have a symmetric duty cycle. I had hoped to use the serial port intelligence to offload enough of the bit-handling to allow for rapid response that the ADB protocol seems to require. Perhaps I need to look at some of the other peripherals; can I use some pulse-width modulation output to drive the keyboard and the EUSART in synchronous slave mode together?

David K. Schmidt evidently scrounged up another ADB converter box which he just sold to me. Assuming I haven’t blown up my Symbolics keyboard, I may finally have what I wanted: a piece of hardware that knows how to talk to my Rev. C keyboard! I hope that will shorten the time to coerce my PIC circuit to do the same.

Step 1 in the Symbolics keyboard to PIC project was not as quick as I had hoped.

It was pretty easy to generate the reset pulse, and get the serial port to send a clock signal to the keyboard. I then was using an oscilloscope to look at the keyboard response, but various attempts to get the PIC code to light up LEDs in response didn’t work at all. In fact, I was being bitten by two bugs, which I used the PICkit 2 in-circuit debug to discover.

1. An array overrun bug (it seems the default radix in MPASM is hex, so reading 16 bytes into a sixteen element array doesn’t work! Unfortunately, my byte counter was located after the array, was stomped directly from 6 to zero, missing the “decrement and skip if zero” termination test until the array covered well into register bank 1.) Safety tip of the day: put your counter before the array!
2. Something not addressed in the early lessons: ANSEL. The RX/DT pin of the 16F690 is shared with AN11, and the low bits of PORT C driving the LEDs are shared with AN4 through AN7. That means, unless one clears the corresponding bit in the ANSEL registers, they read as digital 0, no matter what the RX data is, and no matter what the state of my PORT C output. Toggling RC5 with a read-modify-write operation was clearing my LEDs before I could see them. Read your data sheets carefully!

A few tips/gripes on the PICkit 2 in-circuit debugging experience

1. At least for newbies like me, and probably for more complex applications, any ICD is a great puzzle solver. At US$45 plus shipping, it is a great deal.
2. The white triangle on the PICkit 2 programmer goes to pin 6 (near the “ICSP” silkscreen) of the AC164110—mark a triangle there to make it more foolproof.
3. The green execution arrow stops after the line marked with the red B. I.e., the instruction with the breakpoint on it gets executed before control comes back to the user. This is counter to my habits built up using the similar looking VBA debugger in Excel.
4. With the 16F690, you get only one active ICD breakpoint. That’s much better than zero, of course.
5. The W register indicator in the MPLAB IDE seems not to work (stuck at zero); look at the Special Function Register window instead.
6. Another MPLAB IDE gripe: hovering over constants in code (such as a bit number) shows the value of the File Register at that constant address. Not so helpful when I want to check which bit position I am trying to flip.
7. Modifying special function registers in the File Register window is evidently forbidden; modify them in the Special Function Register window.

Modifying special function registers directly is a real time-saver. Instead of downloading different code to set up the baud rate registers, I edit them by hand to experiment. (And, unfortunately, my Rev. C keyboard behaves differently at different clock rates! More to come on a separate page.)

The first step is to connect the Symbolics keyboard to the PIC. I am using the 16F690 which comes with the PICkit 2 starter kit on the low pin count demo board (Microchip DM164120-1). The keyboard cable mates to a 6-pin modular jack (such as Digikey part number 609-1061-ND), which I tacked onto the side of the board. Unfortunately, this mod jack has pin spacings of 50 mil “horizontally” between successive pins, so I bent up the odd pins and soldered the even pins to the demo board.

My first task will be to power up the keyboard, code a scanning loop, and try to detect keystrokes for a single key, such as the Left Control Key, displaying the count of key-down events in binary to the LEDs connected to RC0-RC3. If the keyboard is not debounced by the internal microcontroller, the count will not increment cleanly, but I hope to at least prove that I can talk to the keyboard and make sense of what comes back. I’m not sure how much current the keyboard will need, so I might have to power the demo board with an external supply.

[UPDATE: It turns out the standard numbering of modular jack pins is the reverse of the pin numbering of the P1 header inside the keyboard. I've corrected the table above to reflect the standard modular jack scheme. Looking into the opening of the jack, with the locking tab down, pin 1 is on the left side.]

I made a little exercise out of compacting the example “switch debounce” code that the PIC tutorial examples uses. My next exercise will be to improve the debouncing in the “reverse variable speed LED rotation by pressing switch” by interleaving the debouncing check into the delay loop. As it stands, when the rotation is slow, the example can miss brief switch pushes.

My glorious savings of three instructions is documented on a separate page.

One nifty trick I found through Google that might be very helpful if I have to debounce 88 keys on a Symbolics keyboard (naturally, the same number as on a piano) is called “vertical counters.” This consists basically of replacing counter increments with boolean operations, which can be easily performed in parallel, if the corresponding bits for multiple counters are grouped in register-sized words.

This page shows how to use a four-count two-bit vertical counting scheme to debounce eight switches in parallel.

A few comments about the PICkit 2 learning experience.

There’s a slight bit of polish that could be applied to the example code.

1. A few redundant instructions, such as
   • Extraneous operations setting register pages
   • In A-to-D examples, the data sheet seems to say the 5 microsecond settling time is needed only when changing the input source, not for every conversion, as in the examples
   • For the 5 microsecond interval, a few of those microseconds are covered by the non-NOP instructions before the conversions
2. In one of the variable-speed rotating bits examples, the carry flag is not explicitly cleared when it needs to be clear. When I modified the example to change the sign of the pot setting-to-delay conversion, I would sometimes set the carry, putting extra bits into the display. Took me a good fifteen minutes to debug that.
3. I think a few extra instructions are wasted using the W register and an explicit move to the file register when the file register destination could be used instead

I also got bitten by the issue mentioned briefly in the PICkit 2 release note: if you program the examples from the IDE, the IDE forces the line controlled by the push-button low, meaning the push-button examples fail to work, unless you program the device from the dedicated programming application and the .HEX file produced by the IDE.

I am slightly disappointed that there doesn’t seem to be any visible progress getting the PICkit 2 with version 2 firmware to work under Mac OS X.

Also, I find a real lack of documentation of the PICkit 2 “Debug Express” support for in-circuit debugging. This may be because Microchip doesn’t want to undercut sales of the MPIDE 2 hardware. I think, however, one can use a combination of the AC162061 (containing an ICD version of the 16F690) and AC164110 (to convert the ICSP six-pin interface to the modular plug used by the usual ICD interface) to do in-circuit debugging of 16F690 applications with the PICkit 2, at an additional cost of about $50.

I’ve started sketching out some ideas for interfacing the Symbolics keyboard on a dedicated page.

[UPDATE: Microchip does, after all, document the PICkit 2 Debug Express hardware requirements (scroll down a bit to see what I missed before.) I ordered the combination even before I found the document, and when the backorder is filled, I will be giving it a try myself.]

[UPDATE: Jeff Post announced that he releasing alpha code supporting v2 firmware for the PICkit 2 on Linux/Mac OS X]

My latest diversion is a Microchip PICkit2. The idea is to make something that can understand the Symbolics keyboard I have, and then connect that something to other things that can speak Apple Desktop Bus (ADB), PS/2 keyboard protocol, or USB, in roughly that order. If I can make something that takes a multi-button PS/2 mouse and a Symbolics keyboard and connects both to a MacIvory, I could be set for beautiful three-button mousing and classic keyboarding on my Lisp Machine.

I’ve mostly traced out the schematic of my Rev. C Symbolics slim-line keyboard (thank god for simple two-layer boards!). Rev. C has LED’s in the Caps Lock and Mode Lock keys, in contrast to this picture of the similar Symbolics Rev. B keyboard. Compared to the Symbolics 3600 keyboard schematic, mine is roughly similar, but unfortunately has a 40-pin microcontroller (an Intel 8749H, a member of the MCS-48 family) soldered in. I am certain I could make something work with the simple protocol the 3600 keyboard apparently implements, but if the microcontroller does anything beyond driving the LEDs on and off based on simply counting the corresponding keystrokes, I might be sunk. I’ve fired off a few questions to people who can hopefully tell me that all of these are plug- and wire-compatible.

As for the 3600, it is pretty obviously a matter of pulling the reset line low to clear the counters, and clocking through the other 127 counter states to read off the switch matrix closures on the 16 X by 8 Y (I surmise that high on the output would mean the corresponding key is down.) I’m not sure how frequently I have to collect the bits to avoid missing keypresses, and what debouncing I might have to do.

The strange thing about my Rev C is that the ICs driving what I guess is the X direction are a pair of 74145 devices with what the datasheet claims are open-collector outputs, and the IC connected to what I guess is the the Y direction is a 7442 with conventional TTL outputs. That seems to mean that scanning an X column with a pressed key causes an open-collector pulling low to compete with the TTL high output on the Y row, except when the Y scan tests that line with a TTL low. What good that does is a mystery. That, and I haven’t finished tracing out the part of the circuit with an LM319 comparator. The 3600 schematic drives the X columns (one low, the rest high) and muxes the Y rows (driving ordinary inputs through any pressed keys) to the readout.

[UPDATE: Seems that the keyboard protocol should be identical, so I'm going to forge ahead with the plan, and assume the circuit knows how to work itself.]

[UPDATE 2: fixed the link to the online 3600 schematic.]

[UPDATE 3: I probably should have given more explicit acknowledgment for the inspiration of the project. If I had not seen asciilifeform's schematic revealing the simplicity of the protocol, I'm sure I would have been far less tempted to start this project.]

I’ve only had a bit of time to play with my MacIvory, but already things are getting interesting.

One is getting used to the system administration style: typical UNIX system configuration resides in text files, and the boot process reads them. Genera is different. The state of the system is contained in the memory image (“world”); you change it through various operations—some interactive, some by loading Lisp files—and preserve it by saving a new world as a file, which can be defined incrementally based on an existing world. The boot process involves loading that saved file back into the living Lisp machine. I haven’t yet developed the sense to complete a block of tasks before saving the world, and how much change pushes one from incremental to a complete world.

Another was getting the network to speak, and part of that is the timewarp of working in Mac OS 7, an era in which Mac networking was going through the introduction of Open Transport, and an AppleTalk era which Mac OS X refuses to acknowledge. Luckily, I’ve got a PowerBook G3 that runs Mac OS 9. My first ethernet card didn’t work, either because I screwed up some software setting or because the hardware was bad. (More embarrassing disclosure later.) DKS sent me a replacement, which worked immediately. I’ve only got ChaosNet configured on the MacIvory side at this point, and my ChaosNet work isn’t up-to-date on my Powerbook, so I’ve only seen raw ethernet packets in tcpdump, but my archaeology is already progressing—documentation to follow soon, as well as getting some Python software (with libpcap or Mac OS X kernel packet filters?) to respond to Chaos-formatted Ethernet packets.

As for the embarrasing disclosure, one of the first things I did was try to set the video card to a mode not supported by my Sony Multisync 15sf monitor. Instead of waiting for the change to timeout and revert, I hit Esc or Command-. or some other key which confirmed the change, leaving me with a blank screen. I tried various reboot, PRAM zapping, or magic key sequences to force the Radius Precision Color 24X to a good mode, but none of that worked, and at some point I screwed up the system. Once I hooked up the Dell 2007fp that understood 1152×870, I saw the floppy-with-blinking-question-mark. Recovery without a working network card involved the ancient hassles of multiple-floppy installs.

I have a Kensington ADB multi-button mouse, but all the drivers I have downloaded are either for Mac OS 8 or cause serious bugginess: system freezes or error -192 when launching the configuration application. I hope I don’t have to roll my own software to get multi-button goodness to work with the Ivory. (At this date, few vendors pay much attention to 68k Macintosh support.)

UPDATE: I’ve captured a bit of the tcpdump results in my ChaosNET information page.

Just sent off the Paypal payment for a MacIvory II system with 2.6 MW of memory. I’m hoping also to get a Symbolics keyboard, although Symbolics is apparently out of the ADB box that allows the keyboard to be used with the vintage Macs.

It was a tough call whether to splurge for this or, for about twice the money, a new MacBook Pro. I’m hoping the Symbolics machine will be much more entertaining.

My main idea of a project is to use the Lisp Machine as a Chaos FILE host, in order to allow development on the CADR simulator with a versioned file system, without having to simulate and learn ITS or TOPS-20.  To get that to work, I probably will need to do a little bit of investigation to understand how the Symbolics box gets configured to talk on the network, what Chaos packets look like on the Ethernet, how to get those to my Python/Lisp code on the modern machine, and, for data security what would be needed to talk to a CD-ROM drive on a remote computer, or to use a remote computer to do backups (to a simulated tape drive?).  In order to get 2.6 MW of NuBus memory, I had to choose a Mac config without a CD-ROM drive, and I’m not too excited about trying to get an external SCSI drive for this old machine. Backing up to floppies is another thing I don’t want to relive.

Some other things I’m looking forward to are Zmail, Concordia (to generate HTML output, anyone?) and Document Examiner,  using CLIM, and grokking the Symbolics compiler(s).

I have been kicking around a few ideas in the Lisp OS direction, and wanted to capture this one.

The problem I see with a classic Lisp OS is that the architecture is either completely new, and therefore will take a lot of work to get to near parity with Lisp-on-current-OS, possibly revealing serious deficiencies before it gets there (e.g., I believe Genera would require substantial clean-up work to port to a typical SMP system), or is roughly similar to a current OS, and therefore, what real difference does it make to write it in Lisp, except fun and NIH?

Instead, I think it makes sense to try to address any performance issues a current OS kernel causes for Lisp, and possibly to incorporate the basic “Lisp core” as a kernel personality, meaning simple Lisp executables could be lightweight. Meanwhile, you get all the driver work the usual kernel guys provide, with SMP, optimized virtual memory and file systems, robust networking stacks, etc. Some people have worked on kernel patches that help Java virtual machines give hints to the Linux virtual memory system.

Lisp guys get a set of kernel patches to make their life somewhat nicer if they own their box, with 100% Linux compatibility for all of their other needs, and in the real world, as opposed to dreams of Lisp paradise that never materialize. And if the Lisp personality is really nice, one could look at more kernel extensions (some kind of very Lispy inter-process communication, for instance).

I really just want a way for the core Lisp library and system Lisp globals to be provided by the kernel, protected against modification by the user process, except through syscalls, try to figure out a simple but nice memory map to support this, looking at linking GC to the scheduler and/or virtual memory in some way if it would help performance, possibly handling the memory protection barriers without a context switch, or providing a “shadow” GC thread for every Lisp process.

Most of the Lisp stuff would come from an implementation like SBCL (or at least compatible calling conventions and memory layout, so that the compiler work could eventually be used), perhaps with some surgery being done on the process memory map and “purification” to separate the Lisp into chunks such as

• Basic Lisp library (car, cdr, NIL, aref, …, basically the CL package), non-volatile, shared between Kernel Lisp and user Lisp processes.
• “Lispux” package that wraps the OS networking stacks, multithreading support, etc., non-volatile, provided to user Lisp processes. Kernel Lisp wouldn’t use these.
• A memory space holding Lisp system globals (glimpse of the underlying OS environment; maybe some interprocess communication stuff)
• A memory space holding Lisp-specific kernel structures; for instance, GC bookkeeping. The Kernel Lisp would presumably include the code to handle garbage collection events. The user process would not be allowed to muck with these. There would also probably be a Kernel Lisp heap unless it could be written to be totally non-consing.
• The Kernel Lisp FASL; this is the portion of the Kernel Lisp implementation written in Lisp.
• A memory space holding the user process’s “purified” executable image. Possibly partitioned into non-volatile portions and usual dynamic portions.
• Usual process heap, process stack
• Some kind of support for shared pages of tagged data, that can only be usefully shared with other Lisp-aware processes.

One of the main issues I am trying to get my head around is what Lisp tagged data looks like when the context switch is made to kernel space. In order for GC to work, the pointer values which make sense under the user process memory map must still make sense in the GC context. This makes me think that the Kernel Lisp needs to have basically the same memory layout, with the current Lisp process mapped into it at the standard location, but with different permissions, and with certain parts visible that are invisible in the user context. I believe this is quite different from Linux; from what I can tell, the Linux kernel simply sees the skeleton of the process (the kernel memory map structure, and which pages of physical memory the process has in core), but has to do special work to follow a pointer, for instance. It seems to me doing GC like this would require more overhead than it could save.
I’m wondering whether there is a better alternative to Linux; the main criteria are source hackability, acceptance in the Lisp community, acceptable driver support both current and in the future (including SMP, because that is where the future is going), and relatively good existing applications support. The only alternatives I see is are the BSD variants. I don’t know which of these makes the most sense. I’m guessing SBCL/x86/Linux folks represent the bulk of people who would be interested in something like this. It would be nice to have an explicit mechanism for different “personalities” so that launching an executable FASL would invoke this personaity, but launching a native binary would not see anything much different (except perhaps for a few new syscalls that provide hooks to some of the Lispy goodness, or maybe just an ability to inspect and debug Kernel Lisp.)

I wanted to store a link to Alastair Bridgewater’s (a.k.a. nyef on IRC) SBCL-based LispOS notes.

Nyef’s most apparent distinguishing characteristic is an amazing willingness to hack low-level internals on Intel x86-based Lisp implementations. Given the claim that “The classic problem with the ‘LispOS Project’ is that a large portion of the early effort would involve low level hardware hacking“, nyef seems to be the ideal candidate for overcoming that problem.

Also, just wanted to record a link to a vendor selling the remnants of the Interlisp environment. Seems like for something under US$3000, one can purchase a x86-Linux-compatible version of the Interlisp programming environment.

A while back I had some idle thoughts about making Lisp machines relevant in 2006. I updated it when people clicked there from a Google search result on Lisp Machine keyboard.

Any comments on that page, you can leave them here.

My blogging has been much less frequent of late, mostly because of an unfortunate abundance of non-computer issues. Despite those, I believe I have made some progress in my Chaosnet-on-OpenMCL work, but I’ve switched most of my hobby-development to Python, mostly to get in touch with trends outside the Lisp Universe.

My current main project circulates around ChaosNET emulation; the main thing going for Python here is a socket interface that is roughly compatible between UNIX-like and Windows operating systems. Lisp can support that as well, but not for my favorite and familiar implementations. (One thing I hope to try before I get through this is to get an asyncore-like framework set up in CLISP.)

So far, I have implemented a `chaosd’ process in Python that simulates multiple ChaosNET `ethers’ in the form of localhost TCP streams (on Windows or UNIX-like) or UNIX-domain sockets (on UNIX-like); each stream corresponds to a subnet. The idea has manifold goals:

• Allow the usim CADR simulator to communicate on Windows with minimal alterations using Winsock in place of BSD sockets (Done) (Brad Parker’s server, however, works with forking and other UNIXisms that would require more work)
• Allow separation between local activity and activity exposed or connected to the wider Internet
• Provide ‘masquerade’ and ‘network translation’ bridges, to flexibly map external resources, such as an ITS or TOPS-20 simulator into something that appears to be a local network. (The translation here is necessary because ChaosNET is a LAN architecture; outside resources might have their own incompatible scheme of subnetting and addressing. These translating gateways would allow me to place external UDP-responding resources into arbitrary Chaos addresses in my local scheme, without requiring any change on the external resource.)
• Eventually, provide small modular `gateway’ simulators that appear to be ChaosNET resources on a local subnet, but translate to external services such as AOL Instant Messenger or IRC.
• Recreate `historically accurate’ network organizations for amusement
• If I ever break down and get a real Symbolics machine, I expect it to be able to speak Chaos over IP or raw Ethernet; I would like some way to connect that to multiple simulators on a single Mac or PC

As a step in this direction, I have implemented a stream-to-UDP masquerade, that automatically takes traffic on a local `ether’ for a particular address and feeds it over UDP using Bjorn Victor’s protocol. Running this on two machines, I am able (mostly) to allow a Lisp machine simulator on my Linux box and another on my Windows box to both believe they are on a single Chaosnet LAN, and to communicate with one another. The next step is to get something like a local ITS or TOPS-20 simulator running in KLH-10 on Linux supporting Bjorn’s UDP protocol (or possibly Bjorn’s machine) to provide FILE service and e-mail service to my Lisp machine simulators on either machine. I have been learning to recompile ITS, with mixed success. My initial impressions on Python in another post.

Just wanted to capture a few more thoughts; over the weekend, I experienced what I think was my first case of deadlock. My chaosnet-on-OpenMCL implementation had worked just fine when the two sides of a connection were careful to exchange only one packet at a time, taking turns. When I tried to listen to a SEND (the Lisp machine analogue to an instant message), the CADR emulator sent two packets in quick succession, and my OpenMCL process would freeze trying to receive it.

As an aside, I get the feeling that I don’t really know how to debug this kind of thing in OpenMCL: looking at backtraces does not always make it completely obvious which semaphore or lock is being waited for in the various threads, and I don’t think it is possible to look at semaphores to see which thread(s) are waiting.

In any case, I concluded that my somewhat haphazard use of a “count lock” to protect access to packet number information in a connection was either being used recursively, or violating a lock heirarchy. So, after some time trying to apply my powers of deduction, I decided to look at the ChaosNET problem from a high level to determine what locking is actually necessary.

The main complexity has to do with the transmission window. In theory, it seems like an ideal case for a counting semaphore: the send window is a limited resource, and sending processes must wait until a spot within the window is available. (Generally, one would wish to preserve FIFO order, but strictly speaking, multiple threads trying to transmit on the same channel might not deserve any guarantees that their packets will happen in any particular order.)

However, there are several things unusual about the window: the thread handling incoming packets can get both acknowledgements in ordinary packets, as the receiving user process consumes packets from the window, and in STS packets that could, at any time resize the window. In the usual semaphore model, it is not easy to deterministically reduce the maximum level of the semaphore. I coded an unsatisfactory approach which does a trial-wait-on-semaphore for the number of window packets that have been taken away. But these might not succeed, if the window is filled above the new capacity. In that case, the window must presumably be allowed to drain below the new level before any waiting process gets allowed to put packets in. It might be possible to have a separate integer variable tracking the “excess window consumption,” and for processes to loop back and wait again for the semaphore if the window has been “tightened” since they first waited. I don’t know if it is possible for threads to maintain their original FIFO order in this case, or whether it matters. (I’m not sure it is reasonable for Chaos processes to reduce the window, once it is declared in the initial STS negotiating the connection, as opposed to widening it to allow for better throughput. The AIM memo is silent on this.)

Interestingly, the draft chaosnet memo does discuss enlarging and shrinking the window, and mentions WIN and WTS packets.

In terms of avoiding deadlock, there are many participants: the user reading process looks at the window to see if the connection might be getting “stymied”, and needs an STS to be transmitted. When STS or acknowledgements come in, the retransmission of the receipted or acknowledged packets should be suppressed, and the acknowledgements (but not receipts) open up space in the window. In the Lisp Machine design, the receive-sts et al. use the RECEIPT function to trim the send-pkts list. I felt that contending with the sending/retransmitting process was asking for trouble. Therefore, I chose to have the retransmission process look at the receipt and acknowledgement status to clean up the send-pkts while it needs exclusive access to the send-pkts list anyway. The retransmission process needs access to the send-pkts list to retransmit, and to the counters if it is required to do clean up. Furthermore, updates of the acknowledgement and receipt states should rachet only forward, i.e., a stale packet coming through the pipe should not erase the acknowledgement or receipting indicated by the logically latest packet received up to this point. That requires locking to protect the read-modify-write update of those status numbers.

One interesting gap in the ChaosNET documentation is whether retransmitted packets can or should update their acknowledgement and receipt information to reflect what has happened to the connection in the meantime. The Lisp Machine implementation seems to retransmit the packet byte-for-byte. An alternative would be to update those packets. In practice, I suspect retransmission should be conservative; it is necessary when the receiving process has fallen behind, or the channel is experiencing loss, and the bias should be toward more retransmission than strictly necessary, instead of hoping to “make up for lost time” by advancing the state of the return channel.

I have created enough of a system to work with the SEND transaction, except for one detail: in a Chaos “stream”, my functions indicate an EOF condition only when the request for a byte has gone “off the end.” I felt the EOF detection should predict whether the next byte request should succeed; but I could not remember how this works elsewhere.