genieacs/ARCHITECTURE.md

# Architecture

This document describes the high-level architecture of GenieACS. If you want to
familiarize yourself with the codebase, you are in the right place.

## Bird's Eye View

GenieACS is a TR-069 Auto Configuration Server (ACS). It manages CPE devices
(routers, modems, gateways) using the CWMP protocol (TR-069). The system
consists of four network-facing services that share a MongoDB database:

```
                  +------------+
  CPE Devices --->| CWMP (7547)|-+
                  +------------+ |
                  +-----------+  |     +---------+
  CPE Devices --->| FS (7567) |--+---->| MongoDB |
                  +-----------+  |     +---------+
                  +-----------+  |
  OSS / Scripts ->| NBI (7557)|--+
                  +-----------+  |
                  +-----------+  |
  Administrators->| UI (3000) |--+
                  +-----------+
```

- **CWMP** -- The core TR-069 protocol server. CPE devices connect here to
  report their state and receive configuration instructions via SOAP/XML over
  HTTP.
- **NBI** -- Northbound Interface. A REST API for external systems (OSS/BSS,
  scripts, automation) to manage devices, tasks, presets, and configuration
  programmatically.
- **FS** -- File Server. Serves firmware images and configuration files to CPE
  devices during download operations.
- **UI** -- Web interface. A Koa-based backend serving a Mithril.js single-page
  application for administrators to browse devices, manage configuration, and
  trigger operations.

All four services follow the same process model: a primary process forks
configurable worker processes via Node.js `cluster` (see `cluster.ts`). Workers
connect to MongoDB and start an HTTP(S) server.

## Code Map

```
bin/                        Service entry points (5 executables)
lib/                        All backend logic
  common/                   Shared utilities (expression engine, Path, errors)
    expression/             Expression parser, evaluator, normalizer, minimizer
  cwmp/                     CWMP-service-specific DB and caching
  db/                       Database layer (MongoDB collections, query synthesis)
  ui/                       UI-service-specific API, DB, and caching
  types/                    (empty, reserved)
ui/                         Frontend SPA (Mithril.js)
  components/               Reusable UI components (parameter, tags, ping, etc.)
  css/                      Stylesheets (vanilla CSS)
  icons/                    SVG icons (compiled into a sprite)
test/                       Unit tests (Node.js native test runner)
build/                      Build scripts (esbuild-based)
public/                     Static assets (logo, favicon)
```

### Entry Points (`bin/`)

Each file in `bin/` bootstraps one service. They are structurally identical:
initialize logging, read config, fork workers in the primary process, connect to
MongoDB and start the HTTP server in each worker.

- `genieacs-cwmp.ts` -- CWMP service. Unique in that it disables HTTP keep-alive
  (`keepAliveTimeout: 0`) and provides custom `onConnection` / `onClientError`
  hooks for TR-069 session lifecycle management.
- `genieacs-nbi.ts` -- NBI service. Straightforward REST API server.
- `genieacs-fs.ts` -- File server. The leanest service; does not use the
  extensions subsystem.
- `genieacs-ui.ts` -- UI service. Wraps the Koa application as the HTTP
  listener.
- `genieacs-ext.ts` -- **Not a service.** This is a child process worker spawned
  by the extensions subsystem. It communicates with its parent via IPC,
  executing user-defined extension scripts in isolation.

### The Expression System (`lib/common/expression/`)

The `Expression` type (`string | number | boolean | null | any[]`) is the most
important abstraction in the codebase. It is a recursive, Lisp-like S-expression
where scalars are literal values and arrays are operations (the operator at
index 0, operands following). For example:

```
["AND", ["=", ["PARAM", "Device.ModelName"], "BrandX"], [">", ["PARAM", "Events.Inform"], 1000]]
```

Expressions are used pervasively:

- **Query/filter language** -- Database queries for devices, faults, presets,
  etc. are all represented as expressions and compiled to MongoDB filters.
- **Configuration values** -- Config entries and preset preconditions are
  expressions, enabling dynamic evaluation.
- **Authorization** -- Permission filters and validators are expressions.
- **Pagination cursors** -- Keyset pagination boundaries are expressed as filter
  expressions.

The expression pipeline flows through four modules:

1. `parser.ts` -- Parses a SQL-like text syntax into the AST (uses Parsimmon).
   Also provides `map()` / `mapAsync()` tree-walking primitives and
   `stringify()` for serialization.
2. `normalize.ts` -- Algebraic normalization using exact rational polynomial
   arithmetic over bigints. Ensures equivalent expressions have the same
   canonical form (e.g., `a + 2 > b` and `a > b - 2` normalize identically).
3. `synth.ts` -- Boolean logic minimization via the Espresso algorithm
   (espresso-iisojs). Converts expressions to minimal sum-of-products form with
   three-valued logic (true/false/null). Handles domain-specific constraints
   like comparison ordering and LIKE pattern relationships.
4. `util.ts` -- Runtime evaluation. Resolves `PARAM` references against concrete
   data, evaluates operators, and supports partial evaluation (returns a reduced
   expression if some values are unknown).

`bigint.ts` provides a named-function wrapper around native bigint operations
for readability. `pagination.ts` implements cursor-based pagination by
generating filter expressions from sort-key bookmarks.

### The Path System (`lib/common/path.ts`, `lib/common/path-set.ts`)

`Path` represents a TR-069 parameter path (e.g., `Device.WiFi.SSID.1.Name`).
Paths can contain wildcards (`*`) and alias expressions (`[key:value]`) for
query-based addressing.

Key design decisions:

- **Interned/flyweight** -- All `Path` instances are cached. `Path.parse()` is
  the only way to create them; the constructor is protected. Two-generation LRU
  cache rotated every 120 seconds.
- **Bitmask encoding** -- The `wildcard` and `alias` fields are bitfields for
  O(1) segment-type checking. This limits paths to 32 segments.
- **Immutable** -- Segment arrays are `Object.freeze()`-d.

`PathSet` is a multi-indexed collection of paths supporting pattern-matching
queries. It indexes paths by length, by segment-at-position, and by string
representation. The `find()` method uses set intersection to efficiently match
wildcard patterns.

### CWMP Protocol Layer (`lib/cwmp.ts`, `lib/soap.ts`, `lib/xml-parser.ts`)

`xml-parser.ts` is a custom single-pass XML parser (no DOM). It scans
character-by-character with bitwise state flags, building a tree of elements
with namespace support. Does not support CDATA.

`soap.ts` handles both parsing CPE SOAP messages and generating ACS SOAP
responses. It dispatches on the SOAP body's method name to type-specific parsers
for Inform, TransferComplete, GetParameterNamesResponse, etc. Supports CWMP
versions 1.0 through 1.4.

`cwmp.ts` is the HTTP-level CWMP request handler. It manages the session state
machine:

1. **State 0** -- Expects an Inform. Authenticates the device (Basic or Digest
   auth, configurable via expression). Acquires a distributed lock. Loads device
   data from MongoDB. Sends InformResponse.
2. **State 1** -- Waits for the CPE to send an empty POST (ready for ACS RPCs).
   Processes any TransferComplete messages.
3. **State 2** -- The ACS drives RPCs (GetParameterNames, GetParameterValues,
   SetParameterValues, AddObject, DeleteObject, Download, Reboot, FactoryReset).
   The CPE responds to each.

Session persistence across TCP disconnects: when a socket closes mid-session,
the entire `SessionContext` is serialized to Redis (the MongoDB `cache`
collection) and restored when the CPE reconnects (identified by a session
cookie).

### Session Engine (`lib/session.ts`)

The session engine implements the **declaration-driven data fetching** pattern.
Rather than issuing RPCs imperatively, provisions create `Declaration` objects
stating what paths and attributes they need to read or write. The engine then:

1. Processes all declarations into a `SyncState` -- a structured plan of which
   parameters to refresh, which values to set, which instances to create/delete,
   etc.
2. Generates the minimal set of CWMP RPCs needed to fulfill the plan.
3. After each RPC response, updates `DeviceData` and re-evaluates.
4. Iterates until all declarations are satisfied.

The preset system (`applyPresets` in `cwmp.ts`) implements a policy engine:
presets are rules with precondition expressions that, when matched, contribute
provisions to the session. After provisions execute, if device data changed,
presets are re-evaluated (up to 4 cycles to prevent infinite loops).

### Provisions and Virtual Parameters

**Provisions** are the unit of configuration intent. Built-in provisions
(`default-provisions.ts`) include `refresh`, `value`, `tag`, `reboot`, `reset`,
`download`, and `instances`. Custom provisions are user-defined JavaScript
scripts stored in the `provisions` MongoDB collection.

**Virtual parameters** are scripts that present computed/derived values as if
they were real device parameters under the `VirtualParameters.*` namespace. They
run in two phases: a "get" phase reads real parameters and returns a computed
value; a "set" phase translates a desired value into real parameter changes.
Virtual parameters can reference other virtual parameters (up to depth 8).

### Sandbox (`lib/sandbox.ts`)

The sandbox provides a secure execution environment for provision and virtual
parameter scripts using `vm.Script` with a 50ms timeout. It uses a
**replay-based execution model**:

1. A script runs and calls `declare()` to request data.
2. When `commit()` is called, the script throws a sentinel symbol and exits.
3. The engine fetches the requested data via CWMP RPCs.
4. The script is **re-run from the beginning** with the fetched data available.
5. Earlier `declare()` calls return cached results; the script progresses
   further.
6. This repeats until the script completes without throwing.

The sandbox API: `declare(path, timestamps, values)` returns a
`ParameterWrapper` proxy; `clear(path, timestamp, attributes)` invalidates
cached data; `ext(...args)` calls external extensions (results are cached per
revision to survive replays); `commit()` explicitly triggers a fetch cycle.
`Math.random()` is replaced with a seeded PRNG for determinism.

### Device Data Model (`lib/types.ts`, `lib/device.ts`)

`DeviceData` is the in-memory working copy of a device's parameter tree during a
session:

- `paths: PathSet` -- All known parameter paths.
- `timestamps: VersionedMap<Path, number>` -- When each path was last refreshed.
- `attributes: VersionedMap<Path, Attributes>` -- Per-path attributes (object,
  writable, value, notification, accessList), each paired with a timestamp.
- `trackers` / `changes` -- Change tracking for re-evaluation.

`VersionedMap` (in `versioned-map.ts`) provides multi-revision snapshots,
enabling the sandbox replay model where scripts may be re-run at different
revision levels.

`device.ts` handles setting and clearing parameter data with invariant
enforcement (e.g., if `value` is set, `object` is forced to 0; parent paths are
ensured to exist). The `unpack()` function resolves wildcards and alias paths
against concrete device data.

### NBI (`lib/nbi.ts`)

A raw Node.js HTTP listener (no framework) with regex-based URL routing.
Endpoints include CRUD for presets, provisions, virtual parameters, objects, and
files; device task management (with optional synchronous execution via
connection request); fault management; generic collection querying; and ping.

The NBI uses MongoDB-style JSON queries directly. For the `devices` collection,
`query.ts` expands user-friendly queries by auto-appending `._value` to
parameter paths and generating multi-type interpretations (string, number, date,
regex) for filter values.

### File Server (`lib/fs.ts`)

A minimal HTTP file server reading from MongoDB GridFS. Supports GET/HEAD with
full HTTP caching (ETag, Last-Modified, If-None-Match, If-Modified-Since) and
Range requests (HTTP 206) for partial content downloads. Files are cached
in-memory via memoization.

### UI Backend (`lib/ui.ts`, `lib/ui/`)

A Koa application with JWT authentication, role-based authorization, and a rich
CRUD API under `/api/`. The root route serves an HTML shell that bootstraps the
Mithril.js SPA with injected config, user info, and hashed asset filenames.

`lib/ui/api.ts` defines generic CRUD endpoints for all resource types (devices,
presets, provisions, files, config, users, permissions, faults, tasks) with
authorization checks at every level. Specialized endpoints handle file
upload/download, synchronous task execution, tag management, password changes,
and CSV export.

`lib/ui/db.ts` translates between the UI's flat parameter representation and
MongoDB's nested document structure. The `flattenDevice()` function is the key
transformation: it recursively walks the nested device document and produces a
flat key-value map (e.g.,
`Device.WiFi.SSID.1.Name -> {value: [...], writable: true}`).

`lib/ui/local-cache.ts` caches permissions, users, and config in-process with
hash-based revision tracking. The `getConfig()` function resolves config
expressions with dynamic context and backward compatibility for legacy config
keys.

### Database Layer (`lib/db/`)

`db/db.ts` is the single entry point to MongoDB. It manages 14 collections:

| Collection          | Purpose                          |
| ------------------- | -------------------------------- |
| `devices`           | CPE device parameter trees       |
| `presets`           | Configuration rules              |
| `provisions`        | Provision scripts                |
| `virtualParameters` | Virtual parameter scripts        |
| `objects`           | Generic objects                  |
| `tasks`             | Queued device management tasks   |
| `faults`            | Error records with retry state   |
| `operations`        | In-flight async operations       |
| `files` (GridFS)    | Firmware images and config files |
| `permissions`       | RBAC rules                       |
| `users`             | User accounts                    |
| `config`            | Key-value configuration          |
| `cache`             | Distributed cache (TTL index)    |
| `locks`             | Distributed locks (TTL index)    |

`db/synth.ts` is the sophisticated expression-to-MongoDB query compiler. It
normalizes expressions, converts them to a Boolean satisfiability representation
using the `Clause` hierarchy, minimizes via Espresso, and emits MongoDB
`$and`/`$or`/`$not` filter objects. This is used by the UI backend.

`cwmp/db.ts` handles CWMP-specific persistence: loading the nested device
document into `DeviceData` (`fetchDevice`), diffing changes back into MongoDB
update operations (`saveDevice`), and managing faults, tasks, and operations.

### Query Systems

There are two separate query paths:

1. **NBI queries** (`lib/query.ts`) -- Processes MongoDB-style JSON queries from
   external API consumers. Expands parameter names, generates multi-type value
   interpretations, and passes through to MongoDB directly.

2. **UI/Expression queries** (`lib/db/synth.ts`) -- Compiles the internal
   expression language into optimized MongoDB filters via Boolean minimization.
   This is the more sophisticated path, used by the UI backend.

### Frontend (`ui/`)

A Mithril.js SPA with hash-based routing (`#!/`). Key modules:

- `app.ts` -- Route definitions. The `pagify()` function wraps each page into a
  `RouteResolver` that handles initialization, error boundaries, and data
  fulfillment.
- `layout.ts` -- Top-level layout (header, navigation, content, overlay).
- `store.ts` -- Centralized data store. Implements a query-based reactive cache
  with deduplication and incremental fetching. The `fulfill()` method (called
  after every render) batches pending queries, computes filter diffs via
  `unionDiff()`, and fetches only missing data. Connection monitoring polls
  every 3 seconds for server health, clock skew, and config changes.
- `components.ts` -- Component registry with a context propagation system. The
  proxied `m()` function resolves string component names and the `m.context()`
  API passes data (like the current device object) down the component tree
  without explicit prop threading.
- `smart-query.ts` -- Translates user-friendly `Label: value` searches into
  filter expressions with type-aware matching (string, number, timestamp, MAC
  address, tag).
- `task-queue.ts` -- Two-stage task pipeline: staging (user configures tasks)
  then queue (tasks are committed and executed via the backend).
- `dynamic-loader.ts` -- Lazy loading of heavy libraries (CodeMirror, YAML) via
  dynamic `import()`.

### Build System (`build/`)

The build is a self-bootstrapping esbuild pipeline (`npm run build` pipes
`build/build.ts` through esbuild then node). It produces:

- **Backend binaries** -- 5 entry points bundled for Node.js 12+ with shebang
  banners, executable permissions, and `.js` extension stripped.
- **Frontend bundle** -- `ui/app.ts` bundled for browsers (ESM, code-split) with
  content-hashed filenames.
- **CSS** -- `ui/css/app.css` bundled and minified with content-hashed output.
- **SVG sprite** -- All icons in `ui/icons/` optimized via SVGO and combined
  into a single sprite.

`build/assets.ts` is a compile-time bridge: at rest it contains placeholder
filenames; during build, the `assetsPlugin` replaces them with actual
content-hashed names so both backend and frontend reference the correct assets.

Build metadata includes the date and a hash derived from the git state, appended
to the package version.

## Cross-Cutting Concerns

### Multi-Level Caching

```
memoize.ts         In-process function cache (2-4 min, two-generation rotation)
     |
local-cache.ts     In-process snapshot cache (5s refresh, hash-based revisions)
     |
cache.ts           MongoDB-backed distributed cache (configurable TTL)
     |
MongoDB            TTL index auto-expiration
```

Each service has its own `local-cache` that periodically checks the distributed
cache for staleness. Distributed locks (`lock.ts`) coordinate rebuilds so only
one worker does the expensive computation.

### Configuration (`lib/config.ts`)

Three-tier priority: CLI args > environment variables (`GENIEACS_*`) > config
file (`config.json`) > defaults. Supports per-device overrides by appending
`-OUI-ProductClass-SerialNumber` to option names.

### Distributed Locking (`lib/lock.ts`)

MongoDB-based mutual exclusion using upsert + duplicate key detection. TTL
indexes prevent deadlocks from crashed processes. Clock skew tolerance of 30
seconds.

### Authentication

- **CWMP devices** -- HTTP Basic or Digest auth, configurable per-device via
  expressions (`auth.ts`).
- **UI users** -- JWT tokens in cookies. Passwords hashed with PBKDF2-SHA512
  (10000 iterations). Role-based authorization via `Authorizer` with
  expression-based filters and validators.

### Connection Requests (`lib/connection-request.ts`)

Three methods to ask a CPE device to initiate a CWMP session:

- **HTTP** -- Standard TR-069 connection request with Digest/Basic auth.
- **UDP** -- For NAT traversal (STUN-based) with HMAC-SHA1 signed messages.
- **XMPP** -- TR-069 Annex K via a full XMPP client (`xmpp-client.ts`).

### Extensions (`lib/extensions.ts`)

User-defined scripts executed in long-lived child processes (`genieacs-ext.ts`)
communicating via IPC. Processes are lazily spawned per script name and reused.
Each request gets a unique ID; responses are matched by ID with a configurable
timeout.

### Logging (`lib/logger.ts`)

Dual-stream structured logging (application + access logs). Supports simple and
JSON formats, systemd journal integration, and automatic log rotation detection.
Protocol traces can be written to a debug file in YAML or JSON format
(`debug.ts`).

### Three-Valued Logic

The system consistently implements SQL-style three-valued logic
(true/false/null). This is visible in expression evaluation, the `Clause`
hierarchy's separate true/false/null methods, and the 2-bit minterm encoding in
the Boolean minimizer. NULL means "unknown" and propagates through operations
following SQL semantics.

## Architectural Invariants

- **The expression system does not depend on any service-specific code.** The
  `lib/common/expression/` modules are pure and shared across all services.

- **The sandbox is deterministic across replays.** `Math.random()` is seeded
  from the device ID, `Date.now()` is controlled, and extension results are
  cached. A script re-run with the same inputs produces the same outputs.

- **Services share no in-process state.** All cross-process coordination goes
  through MongoDB (the `cache` and `locks` collections). Each worker process is
  independent.

- **Device data is never mutated in place during a session.** `VersionedMap`
  provides revision-based snapshots. The sandbox writes to new revisions;
  earlier revisions remain readable for re-evaluation.

- **CWMP session exclusivity.** A distributed lock (`cwmp_session_<deviceId>`)
  ensures only one session exists per device at a time. The lock is refreshed
  periodically and released at session end.

- **The UI backend never queries MongoDB with raw user input.** All queries go
  through the expression-to-MongoDB compiler (`db/synth.ts`), which validates
  and normalizes expressions before generating filters.