Zmail filters, universes, and Profile semantics

Filter definition

A named filter has five independently observable parts (S452-F):

FilterDefinition {
  name-symbol
  optional-documentation
  editable-definition-form
  executable FILTER-FUNCTION property
  recoverable EXPR-FILTER-FUNCTION representation
  menu-entry in USER-FILTER-ALIST
}

The menu entry and the two function properties are mutable runtime state. The profile source section is a separate persistence representation. Editing a form does not change the runtime definition until Done or Sample evaluates it; evaluating it does not by itself save the profile.

Universe definition and universe object

A named universe has separate definition and expansion state (S452-U):

UniverseDefinition {
  name-symbol
  serialized-description
  expanded-universe-object
  membership in UNIVERSE-LIST
}

The serialized description is stored as one symbol property. The expanded object is stored as another. The ordered name list is a third object. These three writes are not one transaction.

An expanded universe implements a sequence-like mapping protocol. A universe may be a direct sequence, a lazy mail-file indirection, a dynamic composite, a variable indirection, or a temporary resource. SIMPLE-P distinguishes direct sequences from the composite mapping path; it is not a claim that a direct sequence is immutable.

Profile document

The Profile is executable Lisp source, not a passive preference serialization (S452-P, S452-D). Its editor interval owns:

• a source pathname and file identity;
• a modified/save tick and a separate option-variable tick;
• an automatically generated block of non-default LOGIN-SETQ forms;
• ordinary user-authored Lisp outside that block;
• named filter and universe definition sections; and
• an optional compiled file whose source identity and creation date are checked.

The generated block and named-definition sections are edited by different code paths. A conforming implementation MUST NOT imply that saving option values alone captures a newly defined filter or universe unless its definition form is also in the profile source.

Outer form and execution model

The persisted source form has this grammar (S452-F):

filter-definition := (DEFINE-FILTER name-symbol (message-variable)
                       [documentation-string]
                       body-form*)

body-form* is general Lisp, not a sealed query language. The form may call other functions, refer to global state, signal conditions, or have side effects. The macro ensures that an empty body is false. It also scans the body for free references to the conventional STATUS and KEYWORDS names. When required, it injects parsing of the message status and retrieval of its keyword list before the user body. A locally bound name is not treated as one of those implicit references.

This has two required consequences:

1. A strict compatibility profile MUST evaluate trusted profile/filter Lisp with the historical power and risks of executable initialization code.
2. A safe declarative clone MAY accept only the constructor subset below, but MUST advertise that restriction and MUST NOT claim historical Profile compatibility.

Constructor-generated subset

The Filter pane's buttons and menus synthesize this subset. These are exact semantic shapes, written with descriptive operators where the original uses ordinary Lisp symbols (S452-F):

The constructor composes ordinary truth values. AND and OR preserve Lisp short-circuit order. NOT negates any non-null result; it does not require a canonical Boolean input.

Reading, evaluation, compilation, and completion order

Done and Sample share the same definition-completion operation (S452-F):

1. Position the editor stream at the start and read one Lisp object using the interval's syntax, defaulting to Common Lisp syntax when no syntax is recorded. Trailing objects are not checked by this operation.
2. Evaluate that object.
3. Require the value to identify a function property whose property name is FILTER-FUNCTION; otherwise signal the task's “not a filter definition” failure.
4. Compile that function-property specification.
5. If compilation added warnings, ask whether to accept them. Declining aborts the current command.
6. Return the filter name. Done exits; Sample then surveys the selected universe with the just-defined filter.

Evaluation precedes the result-shape check and compilation. Evaluating an arbitrary non-filter form can therefore have effects before rejection. The definition macro also updates or appends the menu entry before the generated executable definitions finish installing. A compilation warning is offered after those mutations. A declined warning, compilation failure, or later Sample failure does not roll them back. A transactional clone is a safety-corrected profile, not strict S452 behavior.

End-of-file while reading is reported as unbalanced parentheses. The general read error branch does not manufacture a valid form. No subsequent mutation may be claimed from the reader alone, although evaluation of a successfully read malicious form remains unrestricted.

Naming and redefinition

Filter names are package symbols. The task runs with the ZWEI package selected; typing a new name is read and interned there. A left click on Name rewrites only the name in the editable form. Middle, right, or keyboard entry chooses an existing definition; if none exist, the operation fails without creating one. This executable branch contradicts the pane's broader left/middle wording, so strict conformance uses the executable branch (S452-F).

Defining a new symbol appends a menu entry, then optionally sorts the list using the configured predicate. Redefining a symbol replaces its menu metadata without moving it first. Because the sort predicate is itself evaluated configuration, a strict profile preserves its ability to signal or execute arbitrary trusted code.

Calling a named filter resolves the symbol's current FILTER-FUNCTION property at call time. A missing property signals that the symbol is not a filter. Thus a parent filter sees a later successful redefinition of a referenced child without recompiling the parent.

Built-in predicate families

For every source-defined “predicate and opposite” pair, the opposite evaluates the same body and applies Lisp NOT to its result (S452-F). It is not an independently optimized relation. Errors, parsing, and other side effects occur in the same order before negation.

Before and After are standalone functions, not generated opposite pairs. Before uses strict <; After uses >=. In the filter chooser, negating Before is normalized to After, and negating After is normalized to Before. A message with no date makes both positive predicates false; an explicit Lisp NOT around Before would instead be true, so the chooser's normalization is significant.

Search-key coercion and case

The macro-level search-key grammar has three branches (S452-F):

• an ordinary string selects the ordinary search function;
• a 16-bit extended-search string, or a constructor form that creates one, is parsed into literal, finite-state, or finite-state-expression search state; and
• every other key is rejected during macro expansion.

Ordinary STRING-SEARCH and STRING-EQUAL in this source environment compare characters without case distinction. S452-CASE establishes that the ordinary search calls CHAR-EQUAL/%STRING-EQUAL; a separately named Exact operation uses CHAR=/%STRING=. Therefore literal header, subject, name-fragment, user-at-host, and routed-name matches in this contract are case-insensitive.

Finite-state and finite-state-expression keys preserve the extended-search parser's own expression semantics. This page does not reduce that expression language to a literal string or claim that every expression has one global case rule. A conforming implementation MUST pass the parsed matcher and key through unchanged.

General header search recursively descends cons-valued header fields and succeeds if any leaf string matches. An exact header-type list uses identity membership; a header supertype request uses the header-type lattice.

Address matching

Address matching first classifies the parsed search key (S452-F):

1. A finite-state or finite-state-expression key searches each recipient's preserved original string or source interval.
2. A key surrounded by the two internal partial-match delimiters searches for a case-insensitive substring of the parsed address name.
3. A leading delimiter requires a case-insensitive suffix; a trailing delimiter requires a case-insensitive prefix.
4. A literal containing @ is split at the first @; both parsed name and printable host name must match case-insensitively.
5. A literal containing ! or % but no @ compares the complete parsed name case-insensitively.
6. Every other literal compares a cached true name case-insensitively.

True-name derivation strips routing between the last ! and first % in the normal route shape. If neither delimiter exists, the whole name is the true name. If the two delimiters occur in the source's exceptional crossing/adjacent arrangement, the whole name is retained. The derived value is cached in the parsed address property list; an explicitly present null true-name property means “use the original name.”

The partial-match delimiters are single-character symbols emitted by the historical extended-search UI. A file-compatible implementation must preserve their character identity. A semantic-only implementation may expose readable aliases, but must map them to the four behaviors above without swapping prefix and suffix.

Date and expiration rules

The constructor parses a date assuming universal-time zone argument zero and distinguishes absolute from relative input (S452-F):

• If the parser says the input is relative, the user may retain an expression whose offset is recomputed from the time at which the filter runs, or freeze the parsed instant into a normalized absolute date string.
• Absolute On filters compile one midnight and use the half-open interval [midnight, midnight + 86400).
• Absolute Before is < midnight; absolute After is >= instant.
• Relative forms compute an offset from a normalized record of the definition-time date and apply that offset to the execution-time clock.
• A missing message date makes On, Before, and After false.
• An expiration scalar expires only when now > expiration, not at equality. If the expiration property is a list, its last element is the comparison instant. A missing expiration is not expired.

The fixed 86,400-second On interval is historical behavior. A civil-calendar clone that silently substitutes daylight-saving-aware next-midnight arithmetic is not a strict implementation.

Serialized grammar

The definition form is:

(DEFINE-UNIVERSE name-symbol () description)

The ignored empty list is nevertheless required by the macro call grammar. The description grammar is (S452-U):

description := file-name-string
             | DEFAULT | CURRENT | LOADED
             | defined-universe-symbol
             | (any-symbol)
             | (NOT-GLYPH description)
             | (INTERSECTION-GLYPH description*)
             | (UNION-GLYPH description*)

The preserved text source encodes the single-character operator print names with octets 0x05 (Not), 0x12 (Intersection), and 0x13 (Union). These are historical character-set glyph tokens, not ASCII control operations. A representation-compatible writer must emit symbols with those print names in the selected source encoding. A semantic implementation may additionally accept ASCII aliases.

NIL, an arbitrary non-string atom, an unknown multi-element operator, and a Not form with other than one operand are invalid. Source-visible edge cases are:

• any one-element list whose element is a symbol passes both checker invocations, even when that symbol is not a defined universe; an unknown symbol then fails later during expansion;
• an empty Union expands to an empty set;
• an empty Intersection is initialized from the current complete sequence list, representing the source's “everything” convention; and
• a zero-operand Not passes the shallow one-element-list check but fails during expansion when its missing component is expanded.

These edge cases distinguish the strict profile from a mathematically tidier parser.

Definition and expansion order

Macro expansion checks the grammar while allowing forward symbol names. Evaluation then performs these effects in order (S452-U):

1. Recheck the quoted description and normally require referenced named universes to be defined at that time. The one-element-list escape above is the exception.
2. Record source-definition metadata.
3. Append the name to UNIVERSE-LIST if it is not already present by identity.
4. Store the serialized description property.
5. Expand the description to an object and store that object property.
6. Return the name symbol.

Step 1 precedes registry mutation, so an ordinary missing dependency changes nothing. A missing name hidden inside the accepted one-element-list shape fails only in step 5. That failure, and any other failure in steps 4–5, can leave the name registered and/or the new description installed without a new expanded object. The UI catches an evaluation error, beeps, reports it, and remains in the task; it does not restore the prior properties.

Definition order in a profile is therefore normative: referenced names must already be active when the generated runtime check executes. Cycles introduced through redefinition can pass name-existence checks and fail or recurse during expansion.

Expansion semantics

A parent named universe captures the child objects returned during its own definition. Redefining a child replaces the child's stored object but does not walk and rewrite already expanded parents. The parent must be redefined or reloaded to capture the replacement. This differs from named filters, which resolve a child's function property at call time.

Preparation, mapping, and caching

A direct sequence prepares before mapping unless NO-PREPARE was explicitly set. For an ordinary sequence, preparation expunges when its expunge tick is absent or older than the global buffer-expunge tick. A message buffer first ensures that the file is fully loaded and applies its distinct expunge rule (S452-U).

Mapping over a direct sequence visits the half-open index range [START, END) in ascending order, or descending from END - 1 when FROM-END is true. The callback is invoked once for each visited array member. A caller-supplied bad range or a callback failure is not normalized or rolled back by this layer.

Dynamic set universes retain an array, a composite modification tick, and a child-to- last-seen-tick association list. Before recomputing they prepare children and update the remembered child ticks. If any child is newer than the composite, the composite sets its result fill pointer to zero, rebuilds, and only then advances its own tick. Failure during rebuild may therefore expose an emptied or partially rebuilt array, while the stale composite tick causes a later call to retry.

Set operations have these exact meanings:

• Union: visit children in listed order, append each child's messages, then destructively eliminate duplicate message identities with the source's from-end policy.
• Intersection: if the child list is empty, replace it with a copy of the current sequence list; otherwise sort children in place by increasing message count, then emit messages in the smallest child's order when every other child contains the same message.
• Not: iterate every currently loaded mail-buffer sequence and emit messages not present in the one source universe. Collections are not independently traversed, because they duplicate mail-buffer messages.

Neither set construction nor generic multiple-universe mapping is a snapshot transaction across files. Child preparation, lazy loading, callback effects, and a later failure may already have occurred.

Selecting the active definition set

The Filters and Universes Profile buttons edit which runtime definitions are copied into the Profile source (S452-P). The operation:

1. Sectionizes the current editor interval and determines which registered names already have a definition section of the matching type.
2. Lets the user choose a new active set, possibly defining a new item.
3. If the active set is unchanged, returns without rewriting.
4. Resolves and validates the recoverable definition of every newly active name before deleting any definition text.
5. For each removed filter or universe, checks configured associations and asks permission before removing those associations.
6. Deletes every old matching definition section from the Profile interval.
7. Appends the recoverable definition of each newly active name, in selected order.
8. Redisplays and sectionizes the changed interval.

Step 4 prevents a missing or unrecognizable newly active definition from destroying old definition text. The whole operation is still not atomic. Confirming cleanup for one removed name mutates association alists immediately; declining a later prompt can leave that earlier cleanup in memory even though definition rewriting is aborted. A failure after step 6 can leave definition text deleted or only partly reinserted.

Removing a definition section does not unregister its runtime filter function or universe properties. It controls future persistence, not immediate runtime lifetime.

Recovering a filter definition

Definition recovery first consults the executable function property, then its digest representation, then asks the Profile editor for a matching source definition. It undigests the lambda, verifies recognizable generated provenance, removes injected status/keyword setup and compiler declarations, merges available documentation, and reconstructs a DEFINE-FILTER form (S452-F). An atom, an unrecognized lambda shape, or missing evidence fails rather than inventing source.

The reconstruction is behaviorally equivalent source for the recognized generated shape; it is not proof of the author's original formatting or macro body.

Source file and compiled cache

On entry, Profile probes the source file. A different external file version can trigger a prompt to revert; a missing source creates a new Profile buffer and inserts current settings; an unavailable host records a warning state rather than pretending the file was read (S452-P).

The optional compiled Profile is a cache, not the canonical editable document. Profile compares its recorded source pathname identity and creation date with the source. A missing compiled file, a character file where a compiled file is expected, a different source identity, or an older compiled creation date clears the “compiled matches source” tick. Compile writes the binary file, reloads the Profile with reset enabled, and only then records the new compile tick.

Generated option block and Save modes

Before inserting option settings, Profile recomputes the site-relative Dont-Reply-To deltas. It finds and replaces the prior generated block of recognized LOGIN-SETQ forms, writes current non-default Zmail and selected hardcopy options, updates the editor-variable tick, and restores temporary configuration-name normalization in an unwind cleanup (S452-P). Arbitrary Lisp outside the marked block remains user-owned text.

Reset reloads Profile values and may offer to revert edited source text. Defaults loads system defaults without loading the user profile, refreshes the choices, and then may insert changed variables. Neither path is described as an all-or-nothing transaction.

How to read the inventory

The next table is the exact source-form denominator requested for this profile. It does not include hardcopy options or the separately declared site-alist *MAIL-SENDING-MODE*, because neither is a textual DEFINE-ZMAIL-USER-OPTION form in the selected files.

The Type column is a choose-variable-values editor descriptor, not a Common Lisp declaration. The Default column preserves source spelling and gives the evaluated value when quoting matters. Exposure codes are:

• CVV: present in the ordinary Profile choose-variable-values pane;
• SPECIAL: active and persistent but hidden by source-local :NEVER restriction;
• DYNAMIC: visibility is changed by another option; and
• INACTIVE: text is inside a block comment and does not register an option.

Every ordinary CVV edit first delegates value parsing to the generic chooser, then advances the Profile variable tick, requests an explicit command-documentation update, and reselects Zmail. The Update column lists only additional source-visible effects:

• none: no bespoke Profile update beyond that generic sequence;
• rename: mail-file rename rewrites this option, marks Profile changed, and refreshes the chooser when Profile is open;
• mail-doc / reply-doc: the named command's mouse documentation is associated with the option and is recomputed by the generic update request;
• calendar-exit: leaving Profile recomputes calendar windows if the value changed;
• dont-diff: recompute hidden site-add/site-delete lists under a lock;
• danger-visibility: reconstruct or show/hide dependent dangerous-command choices;
• danger-register: new dangerous commands update this value/default/menu and may reconstruct the pane; and
• special-menu: mutation is owned by a dedicated Profile menu rather than CVV.

Complete 81-form inventory

The two commented GMSGS restrictions do not restrict rows 61–62 in this source profile. The inactive row 81 also contains an inactive COMSAT-only restriction; it must not be counted among the 80 registered Zmail user options.

Dynamic option applicability

The clean source Profile pane applies these exact visibility rules (S452-P):

• Confirmation policy :ALL: hide Selected Dangerous Commands; show the mapping threshold.
• :SELECTIVE: show Selected Dangerous Commands; show the threshold only when the selected set contains the Map Over command.
• :NONE: hide both dependent choices.

Changing either controlling choice reconstructs the CVV pane only when its visible set changed. Registration of a new dangerous command sorts the command menu, replaces the selected-option menu metadata, preserves prior selected identities, selects the newly registered command, and reconstructs an existing Profile pane.

Source-local :NEVER restrictions hide rows 1–4 and 32–33 from CVV. They remain active user options for serialization and are managed by special Profile operations or derived-list logic. An external site can add further restrictions; none are inferred here.

Derived update contracts

*DONT-REPLY-TO* is a merged effective value. A site initialization resets its default from the site option, then combines it with hidden addition and deletion lists under one lock. Editing the visible merged list or inserting changed Profile variables recomputes those hidden deltas under the same lock. The profile persists the deltas so a later site-default change can be merged; it does not treat the visible list as a simple independent default (S452-D, S452-P).

When a mail file is renamed, Profile checks pathname equality after merging both names with Zmail pathname defaults. It rewrites the three scalar pathname options, the three pathname-list options, and the two filter-association alists marked rename above. The first change advances the variable tick and reports that Profile must be saved; an open Profile chooser is refreshed after all rewrites. A failure partway through this loop is not rolled back (S452-P).

Changing calendar week-start takes effect in the variables immediately, but calendar window recomputation is delayed to the unwind cleanup when leaving the Profile command. This cleanup also runs after an abnormal exit from the editor. Mail and Reply documentation associations are refreshed during ordinary variable-choice handling; the association is not a general callback attached to the variable cell.

Filter tests

1. FILT-READ-ONE: place two forms in the editor; prove only the first is read by definition completion.
2. FILT-WRONG-SHAPE: evaluate a harmless effectful form returning a non-filter value; prove the effect occurs before rejection.
3. FILT-WARNING-ABORT: install a definition which produces a compiler warning, decline it, and prove the runtime menu/definition mutation is not rolled back.
4. FILT-IMPLICIT-STATE: define bodies with free Status, free Keywords, locally bound names, and neither; prove parsing/keyword setup is injected only for the first two free-reference cases.
5. FILT-OPPOSITE-EFFECT: use a predicate body that increments a counter and returns non-null; prove the opposite increments once and returns false.
6. FILT-KEYWORD-IDENTITY: use distinct same-print-name symbols and prove keyword membership uses identity.
7. FILT-SUBJECT-CASE: compare mixed-case canonical and real subjects; prove exact and ordinary substring paths are case-insensitive while retaining their distinct canonical/real-subject inputs.
8. FILT-HEADER-RECURSION: search a nested cons-valued header and prove any matching leaf succeeds.
9. FILT-ADDRESS-MATRIX: test plain, @, !, %, prefix, suffix, substring, original-string FSM, and interval FSM keys with mixed case.
10. FILT-DATE-BOUNDARIES: test one second before, exactly at, and one second after Before/After thresholds and both ends of the On half-open interval.
11. FILT-EXPIRATION-BOUNDARIES: test missing, scalar, list, equality, and one-second- later expiration values.
12. FILT-CHILD-REDEFINE: redefine a named child filter and prove an existing parent sees the new function without parent recompilation.

Universe tests

13. UNI-GRAMMAR-TABLE: enumerate strings, three built-ins, defined/undefined symbols, singleton lists, every operator arity, arbitrary atoms, and unknown operators against the exact accept/reject table.
14. UNI-OPERATOR-OCTETS: round-trip the three one-character operator print names in the selected source encoding and verify octets 05, 12, and 13.
15. UNI-EMPTY-SETS: prove empty Union is empty and empty Intersection derives from the complete current sequence list.
16. UNI-ZERO-NOT: prove the shallow check accepts the singleton operator form and expansion subsequently fails without claiming atomic restoration.
17. UNI-DEFINITION-ORDER: reference an undefined child, then define it first and retry; prove registry mutation begins only after the runtime dependency check.
18. UNI-PARTIAL-INSTALL: inject a failure after name/description installation but before expanded-object installation; observe the partial properties.
19. UNI-CHILD-REDEFINE: define parent from child, redefine child, and prove the old parent retains its captured object until parent redefinition.
20. UNI-LAZY-FILE: define from a synthetic file-name string and prove no load occurs until preparation.
21. UNI-UNION-ORDER: use overlapping child sequences and prove source child order plus from-end duplicate elimination.
22. UNI-INTERSECTION-ORDER: use unequal child sizes and prove the child list is reordered smallest-first and output follows that child.
23. UNI-NOT-DOMAIN: include a collection and loaded buffers; prove complement scans loaded mail buffers rather than treating the collection as an independent domain.
24. UNI-REBUILD-FAILURE: fail after one element is appended; prove partial cache, stale composite tick, and retry on the next prepare.
25. UNI-MAP-RANGE: exercise forward/reverse start/end boundaries and a failing callback; prove visit order and lack of rollback.

Profile and option tests

26. PROFILE-OPTION-DENOMINATOR: parse the five selected files without evaluating them; require 81 textual forms, 80 active, one block-commented, with names, types, and defaults identical to the inventory above.
27. PROFILE-RESTRICTIONS: require exactly six active source-local :NEVER options; prove the two GMSGS and inactive COMSAT restrictions do not execute.
28. PROFILE-DANGER-VISIBILITY: exhaust :ALL, :SELECTIVE with/without Map Over, and :NONE; compare the exact dependent-pane visibility.
29. PROFILE-DANGER-REGISTER: add a disposable dangerous command and prove menu sort, selected/default update, and pane reconstruction order.
30. PROFILE-DONT-DIFF: vary site default and visible list under concurrent site update; prove lock-serialized additions, deletions, and merged value.
31. PROFILE-RENAME: rename a synthetic mail file and compare exactly the eight marked option/association surfaces and one-time changed notification.
32. PROFILE-CALENDAR-EXIT: change week-start and leave normally and abnormally; prove calendar recomputation occurs during cleanup in both cases.
33. PROFILE-DEFINITION-PREFLIGHT: include one unrecoverable newly active definition; prove no old definition sections are deleted.
34. PROFILE-LATE-CANCEL: approve cleanup for one removed definition and reject a later dependency prompt; prove the first cleanup can remain.
35. PROFILE-SAVE-MATRIX: execute Insert, Save, Compile, Reap, and save-all with all prompt branches; compare generated text, disk state, load calls, and ticks.
36. PROFILE-CACHE-IDENTITY: test missing, character, wrong-source, older, and matching compiled Profile files against compile-tick state.