Statice persistent object and database environment

Template schema and real schema

A define-schema form creates a template schema in Lisp virtual memory. define-entity-type adds persistent classes, inheritance, attributes, storage hints, and generated access functions to that template. make-database copies the template into a new database as its real schema. Thereafter the template and real schema are distinct objects: administrators can add physical indexes to one database without changing every database or the Lisp definition, and a new program definition does not mutate existing storage by itself.

with-database associates a pathname with a schema and establishes a current database for its body. open-database, with-current-database, current-database, and terminate-database expose the longer-lived connection boundary. A database's stored schema name must name a template schema present in the Lisp world; the Browser likewise needs that package and schema loaded before it can interpret persistent records.

Entity types, attributes, and relationships

Entity types support multiple inherited attributes, although the 2.1 notes forbid duplicate inherited attribute names. An attribute can be scalar or set-valued, nullable or :no-nulls, entity-valued or ordinary, unique, read only, cached, initialized by a form, placed in an area or attribute set, and given generated accessors. Entity-valued attributes can declare an inverse, so a relationship is navigable in both directions. Type options choose areas, type sets, clustering, constructors, initialization keywords, instance variables, documentation, and multiple-attribute indexes.

An entity handle is a Lisp proxy for a persistent identity. The handle stores enough identity and database context to find the record; it is not the entire entity serialized into the heap. Unique IDs let references remain meaningful across pages and sessions. The server keeps its ID source in FEPn:>UNIQUE-ID.FEP.1, a two-block FEP file. The guide says the system can recreate it if lost, but explicitly advises leaving it alone.

Records and physical layout

The source separates entity records from relationship records:

• one entity record holds a persistent entity's scalar values and inherited slots;
• a set-valued relationship normally has a relationship record per member;
• a special unique one-to-many entity relationship can store the back pointer in the target entity record instead;
• areas group pages, while type sets and attribute sets identify the pages that hold particular kinds of records;
• clustering hints place entities or relationships that are likely to be read together near one another.

The 2.0 guide gives a Statice page as 1,152 bytes on 36xx machines and 1,280 bytes on Ivory machines. One directory-property description also calls a block 1,152 bytes while parenthetically noting 1,280-byte Ivory FEP blocks. Page size is therefore a server/hardware format property, not a portable constant. Database copy detects incompatible page sizes and directs the user to textual Dump/Load instead.

The page cache has a configurable buffer-replacement limit. Least-recently used pages can be reclaimed only when they are not held by a transaction; pages above the configured limit are treated as most-recently used. A database can thus be larger than Lisp virtual memory, but long transactions can pin enough pages to defeat replacement.

Indexes are physical, not semantic

Statice indexes are B*-tree structures in the Statice-Index layer. They sort keys and accelerate access without changing which entities satisfy a query. The product supports normal attribute indexes, inverse indexes for relationship directions, exact and case-sensitive variants where applicable, and indexes on several attributes. A multiple index can service predicates that constrain a leading subsequence of its fields; arbitrary trailing-only conditions do not have the same ordered access path.

The query planner can intersect several applicable indexes. Without one it scans the smallest suitable physical population: a type set, attribute set, or area. Adding an index therefore trades storage and update/lock work for query speed. The 2.1 notes additionally report that indexes can improve deletion speed, not only lookup.

Query language

for-each is the compiled query form. It binds an entity variable, compiles predicates against the known schema, selects access paths, and evaluates a body for each result. The dynamic for-each* and count-entities* variants accept type and condition data at run time. The 2.1 notes place two important limits on for-each*: one entity variable per call, and no iteration over set-valued attributes. The later source still contains those entry points, but this study did not load the system to test whether release 466 retains the limits.

The Browser source's exact comparison vocabulary is:

The Browser asks for attribute/value/operator triples in an Accept Values form. Program code can express richer compiled conditions and ordering. Index selection is an optimization underneath the truth conditions; a query still has the same intended result when it falls back to a scan.

Transaction contract

with-transaction establishes the supported atomic update boundary. A normal return commits; a nonlocal exit or process death aborts. Nested transactions do not independently commit: the outermost transaction owns the eventual commit or abort. The manual promises atomicity, isolation, and persistence of committed changes across failures.

The page layer uses two-phase, page-granularity locking:

• an unlocked page can acquire a shared read lock or exclusive write lock;
• a transaction retains acquired locks until commit or abort;
• an upgrade from read to write can participate in a deadlock;
• the detector handles two-way, n-way, and upgrade cycles and aborts the youngest transaction in a detected cycle;
• a restartable abort normally retries, with a documented default retry limit of 100.

This retry behavior is semantically visible. Any non-database side effect in a transaction body can occur more than once. The 2.1 notes therefore advise small transactions, avoiding irreversible effects, avoiding unnecessary nesting, and never overriding the internal without-aborts protection. The debugging variable *restart-testing* deliberately forces retry paths.

A client or server failure aborts unfinished work. A server failure aborts the transactions using that server across clients. The 2.1 notes also say not to warm boot while Statice locks are held: world recovery is not a substitute for the database transaction protocol.

Schema change

Changing a template schema does not rewrite a real schema. Update Database Schema compares them and plans actions. Compatible changes include adding or deleting entity types and attributes, and adding or deleting indexes, type sets, and function/attribute sets. New attributes begin null or empty. Deleting a type deletes its entities, and deleting an attribute does not necessarily recover its occupied space.

Changes to inheritance, an entity type's area, or an attribute's stored type, cardinality, uniqueness, null policy, or area can require rebuilding rather than in-place update. Dumping and loading through source form is the general cross-format reconstruction path.

Two independent warnings make Update Database Schema unsafe to recommend for preservation work:

1. the Statice 2.1 release notes explicitly say not to use it because of reliability problems and say to back up first if it must be used;
2. the later inspected update-schema.lisp.~29~ contains a source comment that warnings when data will be lost are still to be done, immediately noting cascading entity and tuple deletion.

The second point is source-visible behavior not disclosed by the 2.0 command description. This museum performed no schema update.

Development workflow

A normal development sequence is:

1. load the development system and define a schema plus entity types;
2. create a DBFS database from the template, or open an existing database whose schema name matches a loaded template;
3. create entities and relationships inside small transactions;
4. add physical indexes after measuring real access patterns;
5. query with compiled for-each, or use dynamic operators and the Browser for exploration;
6. dump source-form data or take a complete server backup before risky schema or storage work;
7. terminate an exact-copy source before opening its clone, because the copy preserves unique identities.

The shipped example system includes book, bank, university, image, extensible type, presentation-type, Joshua, and finger-oriented examples. They establish integration surfaces but are not separate Statice products. Exhaustive example walkthroughs are deferred to the repository's D60 examples dossier.

DBFS as a Genera file-system type

The Statice File System is registered in a namespace as a File System object of type DBFS. It lives on a Symbolics server host, but clients address its namespace name rather than embedding that host in every database pathname. A site can therefore move the file-system service while preserving logical names. More than one Statice file system can reside on a host.

A DBFS pathname has a host and directory components, for example IRIS:>DIRECTORY>DATABASE, but no ordinary file type or version. Lookup is case-insensitive while names preserve case. Relative pathnames, directory wildcards, completion, and directory commands are supported. The exposed file properties are author, creation date, comment, directory status, and length in blocks.

This namespace resemblance does not make a database an ordinary byte stream:

The DBFS root directory is represented in a directory database, rooted from the FEP .UFD. The server stores partition information, a log descriptor, and the directory root identity as file-system metadata. Create Statice File System constructs that managed structure; it is not synonymous with making one user database.

Local and remote clients

A local client reaches the server's disk layer directly. A remote client asks the host for a server process, then uses pooled connections for page requests; idle connections can be scavenged. Two services divide the work:

• DBFS-PAGE, documented at TCP port 569, carries synchronous page requests;
• ASYNCH-DBFS-PAGE, documented at TCP port 568, sends reverse notifications such as cached-page invalidation.

The asynchronous channel improves cache behavior but is not the sole correctness mechanism. The 2.0 manual describes TCP and Chaos service registrations. The 2.1 notes say to use TCP because the Chaos path is unreliable. The later Open Genera command source only constructs TCP service triplets. This is a specific release difference, not evidence that the older manual invented its Chaos configuration.

The commands' keyword is visibly named TCP Not Present. In the later source, setting it makes the procedure treat the TCP tuple as already present and return without updating the host object. That matches its documentation—do not add the tuple—although the resulting “already has” status message is potentially misleading.

Log and failure recovery

The server's recovery log grows automatically and records enough page activity to abort unfinished work and restore a consistent committed state after a failure. It is part of the server layer, not a user-level audit trail. A complete backup is supported. The manual also describes a continuous archive design but says that facility is currently unsupported; this page does not turn that design into a shipped capability.

Complete backup and restore

The Statice File System Operations program manages volume sets and volume sequences. A complete backup can run without disabling the file system. A complete restore is destructive: it first removes the databases currently in the selected Statice file system, after confirmation, then restores the backup. Compare Backup Volume Set compares saved and current pages.

The volume library is itself a Statice database. It tracks volume identities, backup runs, files, and notes, and supports queries such as backup history and finding the volume that contains a named database or file ID. The manual names cartridge and industry-standard tape choices and local or remote tape drives; its recommendation is remote tape over TCP. The media also includes fuzzy backup compiled modules. Their source is not present, so their algorithm and supported user contract remain a TODO; their names alone do not justify a claim of online incremental backup.

Selective restore

Selective Restore exposes the following decisions:

• target Statice File System;
• whether to disable it for the restore;
• one or more database path patterns;
• whether to repatriate recovered files, with Yes, No, or Query policy;
• name-conflict action: leave, rename existing, replace existing, load under a unique name, or query;
• automatic or manual volume selection;
• tape device and volume host;
• detailed-progress display.

The 2.0 manual advises application programmers to leave repatriation off by default. Selective recovery is still a server administration operation; it is not a transaction-level undo command.

Database dump/load source form

Dump Database writes a textual reconstruction file to an ordinary file system. Its control records include domain, relation, committed-domain, and index forms; its data records describe entities and relationship data. Dump holds one long consistent transaction, whereas Load commits many smaller transactions. Load therefore reconstructs logical contents but is not a byte-for-byte image restore. Clustering is re-established during the load.

The manual warns that this source form can be larger than the original binary database and can run into the historical LMFS file-size limit of approximately 15 MB. It is nevertheless the portable path between servers with different Statice page sizes. The later Open Genera source adds a Data Only option to Load Database; that option is absent from the 2.0 command dictionary and loads records against an existing schema rather than rebuilding schema/index definitions.

An exact Copy Statice Database preserves unique IDs. The source and its copy must not both remain open: terminate the first before opening the duplicate. This is an identity collision constraint, not just a filename rule.

Persistence artifacts at a glance

Schema reconstruction is intentionally lossy

Show Database Schema can reconstruct entity area/type-set/multiple-index options and attribute unique/index/inverse-index/area/attribute-set/no-nulls options. The source does not reconstruct all source-level choices. It omits at least attribute caching, initialization forms, inverse forms, clustering, accessor/reader/writer names, and read-only declarations, plus entity constructor, conc-name, default-init plist, documentation, initialization-keyword, instance-variable, and own-cluster choices. Its output is useful evidence about the real schema, not guaranteed recovery of the author's original declarations.

Two source-only command findings

The later Copy Statice Database source offers :comment in the accepted property list but tests for :copy-comment when binding the internal copy flag. On the inspected source this mismatch appears to prevent the requested comment property from being copied. The matching VBIN has the same source version, but runtime confirmation remains a TODO because the system was not loaded.

The later Load Database source exposes Data Only; the 2.0 command dictionary does not. This is a later user-facing addition rather than an undocumented option inferred from an internal helper.

Window structure

The source defines six panes:

1. a title pane naming the open database and state;
2. a viewer/typeout pane for the selected entity and explanatory output;
3. Current Candidates, holding the active query's entity handles;
4. Queries, retaining result-set objects that can be reselected;
5. a Commands interactor;
6. a Command Menu pane.

The fixed top-level menu contains exactly Open Database, Query, Begin Edits, End Edits, Abort Edits, and Help. Viewer output and the two history panes are separate: selecting a query changes the candidate source; selecting a candidate changes the viewed entity.

Complete Browser-defined command surface

The inspected Browser source defines 13 program commands. Inherited Help and scrolling commands remain available but are not relabeled as Browser-defined.

No other fixed Browser keybindings were found in this source. Standard Dynamic Windows editing, scrolling, completion, abort, and Help behavior comes from the inherited command tables and can be site- or release-sensitive; this article does not invent editor-style bindings for them.

Presentation gestures

The Browser displays only the first three members of a set-valued attribute, then an ellipsis and remaining count. That limit is a source constant, not a manual approximation. Attribute values are presented as editable forms unless the schema marks the attribute read-only, exposing the inherited Dynamic Windows structure-slot modification action.

Immediate and deferred editing

Outside an edit session, changing one displayed attribute opens its own transaction and writes it. The source explicitly does not recheck that value against the value originally displayed, so a concurrent change can be overwritten.

Begin Edits instead records proposed values. At End Edits, the Browser opens a transaction, compares current storage with the old displayed values according to the chosen policy, and writes the queued updates. Under Query, each conflict presents old, current, and proposed values and asks how to proceed. A source-only qualification is important: the comparison loops filter through the storage handler's indexable-p. Thus the displayed All or Changed policy does not visibly prove comparison of every non-indexable extensible value. Runtime confirmation of that edge case remains a TODO.

Another source-visible detail is that Query counts entities in one transaction and retrieves them in a later transaction. The confirmation estimate and the final result therefore need not describe one atomic snapshot if another client commits between those phases. The result container accommodates the eventual fetch; the displayed count should be read as an estimate at confirmation time.

Screenshot status

Screenshot TODO: the base licensed world did not have the Statice packages or Browser system loaded, and the safe probe deliberately did not load optional software or create a database. A meaningful Browser capture additionally needs a disposable, rights-cleared test database and loaded matching template schema. Without those prerequisites, a blank or error window would not verify the query, presentation, or edit behavior described here. No screenshot is published in place of that missing runtime evidence.

Complete top-menu inventory

The compiled maintenance module contains exactly these 19 top-menu labels in the inspected release:

The lower pane additionally installs ordinary or Statice command-table entries for Show Directory, Create Statice File System, Delete File, Rename File, Show File, Expunge Directory, Copy Statice Database, Delete Statice File System, Show All Statice File Systems, Add Statice Partition, and Show Statice Partitions. Show DBFS Meters also exists as an internal command. Their presence does not override DBFS's restrictions: for example, ordinary Show File cannot display database pages as a text file.

Find Backup Volumes appears in the later compiled program and prompts for a file name or file ID. It is absent from the public 2.0 command dictionary, so it is a source/binary-visible later addition. No more elaborate search options are asserted.

Maintenance option surfaces

Complete Backup, Compare Backup Volume Set, and Complete Restore share a core Accept Values surface: Statice File System, volume-set name, volume sequence number, device, volume host, and detailed-progress display. Complete Restore adds the destructive replacement confirmation.

Selective Restore supplies the complete policy set listed in the recovery section. Describe Backup Volume asks for the tape type/device and volume host; Describe Statice File System asks for the file-system name; Enable/Disable File System and Show Backup History select a file system; Find Backup Volumes accepts the filename-or-ID search key. Initialization and device-specific prompts can vary with installed tape support, so this article does not claim an exact hardware menu without a loaded runtime.

Maintenance screenshot status

Screenshot TODO: DBFS-Utilities was not loaded in the base world and this study did not load it. A publishable maintenance screenshot should be made only in a disposable server configuration, because even opening Accept Values beside live DBFS presentations risks inviting a destructive Complete Restore, Delete, partition, or enable/disable action. The future capture must stop before End, record the exact menu and options, and undergo the repository's screenshot rights review.