Compiler Tools, grammars, lexers, and the Syntax Editor in Genera

Conclusion

Genera's Compiler Tools, context-free grammar, LALR(1), LL(1), lexer, and Syntax Editor systems are a coherent language-toolchain substrate, not six unrelated end-user programs. Together they let a language product describe tokens and grammar, build compact parsing machinery, retain an incremental syntax model of a Zmacs buffer, and share compiler and debugger abstractions across Symbolics C, Fortran, and Pascal.

The stack has four broad layers:

1. Compiler Tools (CTS) supplies language-neutral compiler representations, control-flow graphs, data-flow and temporary/register allocation machinery, source-location records, include-file caching, machine-dependent stack accounting, and a debugger bridge for non-Lisp values and mixed-language frames.
2. Context Free Grammar, LALR 1, and LL-1 represent terminals, nonterminals, productions, dialects, FIRST/FOLLOW and nullability data, then build or execute table-driven bottom-up and predictive parsers.
3. Lexer Runtime executes a compact table-driven finite-state lexer with actions, regular-expression descriptions, lookahead, skipped tokens, and serialized lexer objects. Its editor support preserves tokenization state across Zmacs lines.
4. Syntax Editor combines the LL(1) parser and editor lexer with Zmacs. It maintains tokens, parse trees, error positions, section boundaries, templates, construct-aware movement and deletion, indentation, and electric face/case transformations without forbidding temporarily invalid text.

“Syntax Editor” is therefore not a separate screen editor competing with Zmacs. It is a set of ZWEI mixins, buffer structures, commands, and language hooks used by C and Pascal modes. The manuals expose it through those modes; the source media exposes the reusable machinery beneath them.

All thirteen named systems are present on the inspected licensed Open Genera release media, but a fresh isolated Genera 8.5 world reported every one as unloaded. The CTS, GRAMMAR, and LEXER packages were likewise absent while the ordinary ZWEI package was present. No optional system was loaded, no grammar was compiled, and no file was written. This page consequently describes the installed product from declarations, compiled-object inspection, and public language manuals, while keeping live operation as an explicit follow-up.

What this dossier does and does not cover

This page documents the common implementation layer. The language products that consume it have their own user interfaces, compiler options, listeners, debugger commands, and foreign-function conventions; see Symbolics C, Fortran, and Pascal when that dossier is available. The ordinary Lisp compiler and Zmacs editor are covered separately in Lisp runtime, compiler, and development environment and Lisp-machine text editors.

The names also need a boundary:

Evidence classes

• System declarations are readable licensed files in the inspected Open Genera media. They establish names, dependencies, distribution flags, and module order. This page publishes analysis and short identifiers, not the proprietary source.
• Compiled-object inspection means inert string and symbol census of the shipped VBIN files. It establishes that named structures and operations occur in those objects. It is not decompilation and does not prove every control-flow detail.
• System-directory records establish release versions and the source-world context. Adjacent generated records sometimes disagree about whether another subsystem was marked Experimental; those adjectives are reported as record-local state rather than flattened into a universal product status.
• Manual evidence comes from the public archival User's Guide to Symbolics C and User's Guide to the Pascal Tool Kit. Their C and Pascal editor-mode chapters document the user-facing behavior of the shared Syntax Editor.
• Runtime observation means the fresh, read-only 2026-07-18 Xvfb/VLM session described below. It establishes only the load state of that exact Genera 8.5 world.
• Public CADR comparison uses the pinned public MIT CADR source tree. An absence claim is limited to the searched revision and named facilities.

Release-media identity

The inspected system-directory record for Syntax Editor Support was written on 1998-09-11 in a source world reporting System 452.18. It records the following released versions:

Every declaration uses an empty or nil advertised-in value. That is strong evidence that these are dependency systems rather than ordinary selectable applications. It does not mean they were unused: the C and Pascal manuals expose their behavior through language modes, and the language system declarations depend on them.

Some generated system-directory headers call a neighboring component Experimental, while the component's own later record lists it without that word. For example, the LL-1 record presents Syntax Editor Support as released but still calls LL-1 itself experimental in the source-world inventory. The stable claims here are the numeric released versions and each component's own declaration flags. A precise history of when each experimental designation was added or removed remains a TODO.

How the pieces fit together

The declared dependency direction is:

Compiler Tools Runtime
        |
        +--> Compiler Tools Development --> Compiler Tools Debugger clients
        |                     |
        |                     +------------------------------+
        |                                                    |
Context Free Grammar --> LALR(1)                             |
        |                                                    |
        +--------------> LL(1) --> LL1 Editor Support -------+
                                      |                      |
Lexer Package --> Minimal Lexer Runtime                       |
        |              |                                     |
        +----------> Lexer Runtime --> Lexer Editor Support --+
                                                             |
Syntax Editor Runtime ---------------------------------------+
                                                             |
                         Syntax Editor Support <--------------+
                                      |
                             C and Pascal editor modes

The graph is not merely a packaging curiosity. It explains the product's division of labor:

• a batch parser can use CFG, a lexer, and either parser without Zmacs;
• a small delivered program can use Minimal Lexer Runtime instead of the full regular-expression and serialization layer;
• the Syntax Editor Runtime supplies buffer sectionization behavior without pulling in the complete compiler-development stack;
• full Syntax Editor Support deliberately depends on Compiler Tools Development because language units, source definitions, compilation, and editor sectionization share compiler representations and source-location services.

Compiler Tools (CTS)

Context Free Grammar

LALR(1) support

LALR 1 is the bottom-up parser branch. Its system depends on Context Free Grammar and loads definitions, parser, a test/try module, and FASD support. The compiled objects name:

• states, start and goal states, transition and action tables;
• a parse stack, production actions, and production queues;
• token enumeration and lexer linkage;
• conflicts and error states;
• next-legal-token queries, error-context printing, syntax-error repair, and spelling-error search;
• serialized parser state including compact tables and self-mapping tables.

The presence of repair and next-legal-token machinery matters: this was not just an accept-or-reject batch parser. It could explain or recover from an invalid token stream and provide completion-oriented information to a client.

The shipped system declares binaries but no source. Accordingly, “LALR(1)” here is the product's own name and intended parser class; this page does not claim to have independently proved the lookahead-state-merging algorithm from machine code.

LL(1) support

LL-1 is the predictive-parser branch and the parser directly incorporated into the Syntax Editor support graph. Its compiled objects name:

• parse and token stacks, parse rows, productions, and grammar goals;
• full and compact parse tables plus a feasibility table;
• next-legal-token and token-insertion operations;
• error nonterminals, error recovery, error reporting, and parse-error counts;
• start/finish hooks for a parse and for individual productions;
• editor parser modules layered on the ordinary parser.

The feasibility table and next-legal-token interface fit an interactive editor: the client needs to know which continuations remain possible before the program is complete. The editor-specific ed and ed-parser binaries are packaged in the LL1 Editor Support subsystem rather than in the core LL-1 system, preserving a batch/runtime boundary.

The name does not prove that an entire C or Pascal grammar is strictly LL(1) in the simplest textbook sense. The binaries include repair, inserted-token, error- nonterminal, and extended-syntax machinery, and a language may partition or augment its grammar for editing. “LL-1 support system” is the warranted claim.

Lexer Runtime

Syntax Editor architecture

Complete common Syntax Editor interaction surface

The following tables combine the standard common command summary printed in the C manual with the matching Pascal descriptions. Language-specific compiler and mode commands belong in the language dossier. Key names retain Symbolics notation: c- is Control, m- is Meta, and sh- is Shift.

Source-visible commands beyond the compact manual table

An inert symbol census of ed-commands and its companion modules found the following additional command families. Some are development diagnostics or tests, not advertised everyday commands. Compiled string fragments sometimes append encoding artifacts to symbol names, so the table reports normalized names only when corroborated across modules or by a readable identifier.

The raw ed-commands binary yielded 50 unique COM--like strings, but that is not a trustworthy command count: several include compiled-string suffixes, some generic inherited commands appear in the object, and additional common commands are defined in sectionization or templates modules. The warranted deliverable is the complete public common interaction table above plus this source-only family inventory, not a false exact total.

Serialization and generated artifacts

CFG, both parser systems, the lexer, and the lexer editor support each include a module named fasd. The compiled symbols name FASD-FORM, block writing, load-time flavor lookup, and construction of a fasload pathname. Serialized state includes:

• grammar and lexer links;
• compact COMB arrays and self-mapping tables;
• LALR state, transition, action, error, and repair tables;
• LL(1) parse and feasibility tables;
• lexer start states, transition data, keyword maps, skipped-token behavior, and state-to-regular-expression descriptions;
• editor-lexer begin/middle/end terminal information.

These are generated Lisp-machine compiled-data artifacts intended to be loaded back into the Genera environment. They are meaningfully inspectable and in principle recoverable at an object/symbol level, but they are not plain portable grammar files and not virtual-machine snapshots. A FASD or VBIN can preserve tables and executable initialization without preserving the original comments, macro structure, grammar-source formatting, or generator implementation.

This distinction parallels the larger compiled-object and UNFASL boundary: extraction can recover behaviorally relevant objects and names, but does not recreate original source authorship or layout.

Manual/source findings that are easy to miss

The manuals are strongest on user workflow; the media reveals several additional architectural points:

1. The editor tracks lexical state across lines rather than tokenizing lines in isolation.
2. Parse trees can be partial, broken, or fragmented while still supporting editing.
3. Buffer and sectionization ticks permit incremental rebuilds of changed units.
4. Templates are syntactic tokens in ordinary text, not an external form model.
5. Grammar tables include approximation, feasibility, error-repair, and next- legal-token information useful before a program is valid.
6. Compiler Tools includes an include-file cache with interactive inspection and clearing commands.
7. The debugger is explicitly hybrid: it has hooks for language expressions, foreign conceptual types, Lisp frames, and mixed stack navigation.
8. Dedicated randomized tree-edit tests and representation-invariant checks were shipped with the implementation.
9. Syntax Editor Runtime is separable from full Support, allowing a language runtime or buffer to carry lighter sectionization behavior.
10. The product stores generated grammar, parser, and lexer objects through FASD rather than rebuilding every table from source on each load.

Runtime study

Fresh isolated session d38-language-tools-20260718, generation 1, ran from 09:48:11 through 09:50:17 EDT on 2026-07-18. It used the repository's Genera Xvfb computer-use harness and the identified Genera 8.5 world, VLM, and debugger listed below.

The session performed two read-only evaluations:

1. it asked SCT:GET-SYSTEM-VERSION about all thirteen systems in the release- media table;
2. it looked up packages CTS, GRAMMAR, LEXER, and ZWEI.

Every system query returned NIL. The first three package lookups returned NIL; ZWEI returned the resident editor package. This proves the language toolchain layers were not loaded in that base world even though Zmacs itself was present. It does not prove the media is unloadable or that another saved world would omit them.

No optional system was loaded, no parser or editor mode was initialized, no file was opened or written, and no world was saved. The base and private world SHA-256 both remained a8ee5e86cc7e322f7385af3e0cd579d7650d4dcfc3ce328acbf8b25515dd0672. The action log contains four intent/outcome records, is 2,500 bytes, and has SHA-256 10f5c10b8bba2becfd7ab327c561f6ab9e750d1bbab53ea8649a28a0d393d523.

The VLM accepted its shutdown confirmation and began cleanup, then reached the known bounded cleanup stall. The harness forced termination and recorded forced_after_confirmed_shutdown_stall: true; this was not orderly VLM process exit. No saved guest state was expected or retained.

Public CADR comparison

The pinned public MIT CADR source revision 8e978d7 contains the Lisp compiler, ZWEI's Lisp syntax scanner, indentation and structural editing, reader finite-state machinery, and language-specific editing modes. A case-insensitive search of that revision did not find systems or definitions named Compiler Tools, Context Free Grammar, LALR, LL-1, Lexer Runtime, Syntax Editor, or DEF-LBE.

That bounded absence supports a careful conclusion: the specific layered product described here belongs to the later Symbolics Genera environment and should not be retroactively assigned to System 46. CADR still had syntax-aware Lisp editing and finite-state readers; those are functional ancestors or parallels at a broad problem-domain level, not proof of direct subsystem lineage.

The LM-3 fossil should be checked separately for intermediate names or designs. Until that search is complete, this page does not claim the exact first release or lineage of CTS, CFG, or the Syntax Editor.

Preservation and recovery implications

• Keep all recovered licensed VBIN, FASD, source, grammar, parser, and lexer data untracked. Only original analysis, tools, and evidence-only catalogs belong in the repository.
• Preserve the four media directories together. Syntax Editor Support cannot be understood fully from syntax-editor alone because it depends on CTS, CFG, LL(1), and lexer editor support.
• Preserve system declarations and system-directory files alongside compiled binaries. They retain dependency order, version history, aliases, source- distribution boundaries, and release status that symbol extraction cannot reconstruct reliably.
• Treat a symbol/string census as evidence of vocabulary, not recovered source. Compiled string artifacts, uninterned symbols, and generic inherited code can distort exact command counts.
• A future extractor could safely emit an evidence-only JSON inventory of systems, modules, exported names, and hashes without redistributing executable or grammar data. It must not emit reconstructed proprietary tables into tracked output.
• The most useful behavioral recovery target is a disposable loaded C or Pascal session. It can exercise template insertion, incremental errors, cross-line lexing, sectionized compilation, include-cache invalidation, and hybrid debugging without trying to treat the whole VLOD as a simple archive.

Open questions

• TODO: load the licensed language systems in a throwaway world only after a reproducible, read-only guest media mount is available, then exercise the common commands in both C and Pascal modes.
• TODO: determine the exact source syntax accepted by the lexer regular-expression compiler and CFG/LBE declarations without reconstructing or publishing licensed grammar source.
• TODO: establish the precise include-cache invalidation rule on local and remote file systems.
• TODO: test how partial edits delimit incremental re-lexing, re-parsing, and sectionization, including multi-line comments and strings.
• TODO: record which common commands are exported for ordinary users and which are internal diagnostics in a loaded Syntax Editor Support world.
• TODO: inspect LM-3 fossils for an intermediate CTS, CFG, lexer, or syntax-editor lineage before making a first-release claim.
• TODO: capture and review a small, substantive syntax-template or error-navigation screenshot if a loaded language world becomes available.

Artifact provenance

The purchased Open Genera archive is 206,213,430 bytes with SHA-256 89fb3e76b91d612834f565834dea950b603acf8f9dbacacdd0b1c3c284a2d36e. The runtime study used:

Representative readable declaration hashes are:

The public C manual copy inspected locally is 543,959 bytes with SHA-256 549baa34a6428feeff6c225492db64e6342e9684589c9b983c33ceac1b382db2. The Pascal Tool Kit guide is 607,569 bytes with SHA-256 f9c2a855fc75eac23afd66d55dd1247fb828edfd8b361c14ee790cb2eacf8895.

Public sources

• Symbolics, User's Guide to Symbolics C, especially “C Editor Mode,” “C Mode Completion and Templates,” “Electric C Mode,” and “Summary of Standard Editor Mode Commands.”
• Symbolics, User's Guide to the Pascal Tool Kit, especially “Pascal Editor Mode,” “Template and Completion Commands,” and “Electric Pascal Mode.”
• MIT CADR System Software, public source revision 8e978d7, used only for the bounded comparison above.

Last verified: 2026-07-18.