diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..316f0e9 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,82 @@ +# AGENTS.md — GenieACS + +GenieACS is a TR-069 Auto Configuration Server for remote management of CPE +devices (routers, modems, gateways). TypeScript codebase compiled with esbuild, +backed by MongoDB. + +## Architecture Overview + +Four services share a single MongoDB instance: + +- **CWMP** (port 7547) — TR-069 protocol handler; manages device sessions +- **NBI** (port 7557) — Northbound REST API for external consumers +- **FS** (port 7567) — File server for firmware/config (GridFS-backed) +- **UI** (port 3000) — Web interface (Koa backend + Mithril.js SPA frontend) + +Key subsystems: expression engine (`lib/common/expression/`) compiles a +Lisp-like DSL used for queries, config, and authorization; session engine +(`lib/session.ts`) drives CWMP interactions via declarations rather than +imperative RPCs; sandbox (`lib/sandbox.ts`) runs user-defined provision scripts +in `vm.Script` with deterministic replay. + +Read `ARCHITECTURE.md` for a full map of the codebase when working on unfamiliar +areas. It covers service boundaries, the expression pipeline, the path system, +the CWMP session state machine, the database layer, and architectural +invariants. + +## Project Structure + +- `lib/` — Core server-side logic +- `lib/common/` — Shared code (runs in both Node.js and browser) +- `lib/db/` — MongoDB database layer +- `lib/ui/` — UI backend helpers +- `ui/` — Frontend SPA (Mithril.js) +- `bin/` — Service entry points (5 executables) +- `build/` — Build scripts (esbuild pipeline) +- `test/` — Unit tests (node:test) +- `docs/` — User docs (Sphinx/reStructuredText) +- `public/` — Static assets (favicon, logo) + +## Build / Lint / Test Commands + +```bash +npm run build # Production build (esbuild pipeline -> dist/) +NODE_ENV=development npm run build # Dev build (no minification) +npm run lint # Prettier + ESLint + tsc --noEmit in parallel +npm test # Compile tests with esbuild, run with node --test +``` + +### Running a Single Test File + +```bash +esbuild --log-level=warning --bundle --platform=node --target=node18 \ + --packages=external --sourcemap=inline --outdir=test test/path.ts \ + && node --test --enable-source-maps test/path.js \ + && rm test/path.js +``` + +### Running a Single Test Case + +```bash +esbuild --log-level=warning --bundle --platform=node --target=node18 \ + --packages=external --sourcemap=inline --outdir=test test/path.ts \ + && node --test --enable-source-maps --test-name-pattern="^parse$" test/path.js \ + && rm test/path.js +``` + +### Lint Sub-commands + +```bash +prettier --prose-wrap always --write . +eslint 'bin/*.ts' 'lib/**/*.ts' 'ui/**/*.ts' 'test/**/*.ts' 'build/**/*.ts' +tsc --noEmit +``` + +## Before Committing + +Read `CONTRIBUTING.md` and ensure your changes comply with it. In particular: + +- Run `npm run lint` and `npm test` and fix any failures. +- Follow the code style, naming, import, and comment conventions documented + there. +- Use the Conventional Commits format for commit messages. diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..633b0fb --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,461 @@ +# Architecture + +This document describes the high-level architecture of GenieACS. If you want to +familiarize yourself with the codebase, you are in the right place. + +## Bird's Eye View + +GenieACS is a TR-069 Auto Configuration Server (ACS). It manages CPE devices +(routers, modems, gateways) using the CWMP protocol (TR-069). The system +consists of four network-facing services that share a MongoDB database: + +``` + +------------+ + CPE Devices --->| CWMP (7547)|-+ + +------------+ | + +-----------+ | +---------+ + CPE Devices --->| FS (7567) |--+---->| MongoDB | + +-----------+ | +---------+ + +-----------+ | + OSS / Scripts ->| NBI (7557)|--+ + +-----------+ | + +-----------+ | + Administrators->| UI (3000) |--+ + +-----------+ +``` + +- **CWMP** -- The core TR-069 protocol server. CPE devices connect here to + report their state and receive configuration instructions via SOAP/XML over + HTTP. +- **NBI** -- Northbound Interface. A REST API for external systems (OSS/BSS, + scripts, automation) to manage devices, tasks, presets, and configuration + programmatically. +- **FS** -- File Server. Serves firmware images and configuration files to CPE + devices during download operations. +- **UI** -- Web interface. A Koa-based backend serving a Mithril.js single-page + application for administrators to browse devices, manage configuration, and + trigger operations. + +All four services follow the same process model: a primary process forks +configurable worker processes via Node.js `cluster` (see `cluster.ts`). Workers +connect to MongoDB and start an HTTP(S) server. + +## Code Map + +``` +bin/ Service entry points (5 executables) +lib/ All backend logic + common/ Shared utilities (expression engine, Path, errors) + expression/ Expression parser, evaluator, normalizer, minimizer + cwmp/ CWMP-service-specific DB and caching + db/ Database layer (MongoDB collections, query synthesis) + ui/ UI-service-specific API, DB, and caching + types/ (empty, reserved) +ui/ Frontend SPA (Mithril.js) + components/ Reusable UI components (parameter, tags, ping, etc.) + css/ Stylesheets (vanilla CSS) + icons/ SVG icons (compiled into a sprite) +test/ Unit tests (Node.js native test runner) +build/ Build scripts (esbuild-based) +public/ Static assets (logo, favicon) +``` + +### Entry Points (`bin/`) + +Each file in `bin/` bootstraps one service. They are structurally identical: +initialize logging, read config, fork workers in the primary process, connect to +MongoDB and start the HTTP server in each worker. + +- `genieacs-cwmp.ts` -- CWMP service. Unique in that it disables HTTP keep-alive + (`keepAliveTimeout: 0`) and provides custom `onConnection` / `onClientError` + hooks for TR-069 session lifecycle management. +- `genieacs-nbi.ts` -- NBI service. Straightforward REST API server. +- `genieacs-fs.ts` -- File server. The leanest service; does not use the + extensions subsystem. +- `genieacs-ui.ts` -- UI service. Wraps the Koa application as the HTTP + listener. +- `genieacs-ext.ts` -- **Not a service.** This is a child process worker spawned + by the extensions subsystem. It communicates with its parent via IPC, + executing user-defined extension scripts in isolation. + +### The Expression System (`lib/common/expression/`) + +The `Expression` type (`string | number | boolean | null | any[]`) is the most +important abstraction in the codebase. It is a recursive, Lisp-like S-expression +where scalars are literal values and arrays are operations (the operator at +index 0, operands following). For example: + +``` +["AND", ["=", ["PARAM", "Device.ModelName"], "BrandX"], [">", ["PARAM", "Events.Inform"], 1000]] +``` + +Expressions are used pervasively: + +- **Query/filter language** -- Database queries for devices, faults, presets, + etc. are all represented as expressions and compiled to MongoDB filters. +- **Configuration values** -- Config entries and preset preconditions are + expressions, enabling dynamic evaluation. +- **Authorization** -- Permission filters and validators are expressions. +- **Pagination cursors** -- Keyset pagination boundaries are expressed as filter + expressions. + +The expression pipeline flows through four modules: + +1. `parser.ts` -- Parses a SQL-like text syntax into the AST (uses Parsimmon). + Also provides `map()` / `mapAsync()` tree-walking primitives and + `stringify()` for serialization. +2. `normalize.ts` -- Algebraic normalization using exact rational polynomial + arithmetic over bigints. Ensures equivalent expressions have the same + canonical form (e.g., `a + 2 > b` and `a > b - 2` normalize identically). +3. `synth.ts` -- Boolean logic minimization via the Espresso algorithm + (espresso-iisojs). Converts expressions to minimal sum-of-products form with + three-valued logic (true/false/null). Handles domain-specific constraints + like comparison ordering and LIKE pattern relationships. +4. `util.ts` -- Runtime evaluation. Resolves `PARAM` references against concrete + data, evaluates operators, and supports partial evaluation (returns a reduced + expression if some values are unknown). + +`bigint.ts` provides a named-function wrapper around native bigint operations +for readability. `pagination.ts` implements cursor-based pagination by +generating filter expressions from sort-key bookmarks. + +### The Path System (`lib/common/path.ts`, `lib/common/path-set.ts`) + +`Path` represents a TR-069 parameter path (e.g., `Device.WiFi.SSID.1.Name`). +Paths can contain wildcards (`*`) and alias expressions (`[key:value]`) for +query-based addressing. + +Key design decisions: + +- **Interned/flyweight** -- All `Path` instances are cached. `Path.parse()` is + the only way to create them; the constructor is protected. Two-generation LRU + cache rotated every 120 seconds. +- **Bitmask encoding** -- The `wildcard` and `alias` fields are bitfields for + O(1) segment-type checking. This limits paths to 32 segments. +- **Immutable** -- Segment arrays are `Object.freeze()`-d. + +`PathSet` is a multi-indexed collection of paths supporting pattern-matching +queries. It indexes paths by length, by segment-at-position, and by string +representation. The `find()` method uses set intersection to efficiently match +wildcard patterns. + +### CWMP Protocol Layer (`lib/cwmp.ts`, `lib/soap.ts`, `lib/xml-parser.ts`) + +`xml-parser.ts` is a custom single-pass XML parser (no DOM). It scans +character-by-character with bitwise state flags, building a tree of elements +with namespace support. Does not support CDATA. + +`soap.ts` handles both parsing CPE SOAP messages and generating ACS SOAP +responses. It dispatches on the SOAP body's method name to type-specific parsers +for Inform, TransferComplete, GetParameterNamesResponse, etc. Supports CWMP +versions 1.0 through 1.4. + +`cwmp.ts` is the HTTP-level CWMP request handler. It manages the session state +machine: + +1. **State 0** -- Expects an Inform. Authenticates the device (Basic or Digest + auth, configurable via expression). Acquires a distributed lock. Loads device + data from MongoDB. Sends InformResponse. +2. **State 1** -- Waits for the CPE to send an empty POST (ready for ACS RPCs). + Processes any TransferComplete messages. +3. **State 2** -- The ACS drives RPCs (GetParameterNames, GetParameterValues, + SetParameterValues, AddObject, DeleteObject, Download, Reboot, FactoryReset). + The CPE responds to each. + +Session persistence across TCP disconnects: when a socket closes mid-session, +the entire `SessionContext` is serialized to Redis (the MongoDB `cache` +collection) and restored when the CPE reconnects (identified by a session +cookie). + +### Session Engine (`lib/session.ts`) + +The session engine implements the **declaration-driven data fetching** pattern. +Rather than issuing RPCs imperatively, provisions create `Declaration` objects +stating what paths and attributes they need to read or write. The engine then: + +1. Processes all declarations into a `SyncState` -- a structured plan of which + parameters to refresh, which values to set, which instances to create/delete, + etc. +2. Generates the minimal set of CWMP RPCs needed to fulfill the plan. +3. After each RPC response, updates `DeviceData` and re-evaluates. +4. Iterates until all declarations are satisfied. + +The preset system (`applyPresets` in `cwmp.ts`) implements a policy engine: +presets are rules with precondition expressions that, when matched, contribute +provisions to the session. After provisions execute, if device data changed, +presets are re-evaluated (up to 4 cycles to prevent infinite loops). + +### Provisions and Virtual Parameters + +**Provisions** are the unit of configuration intent. Built-in provisions +(`default-provisions.ts`) include `refresh`, `value`, `tag`, `reboot`, `reset`, +`download`, and `instances`. Custom provisions are user-defined JavaScript +scripts stored in the `provisions` MongoDB collection. + +**Virtual parameters** are scripts that present computed/derived values as if +they were real device parameters under the `VirtualParameters.*` namespace. They +run in two phases: a "get" phase reads real parameters and returns a computed +value; a "set" phase translates a desired value into real parameter changes. +Virtual parameters can reference other virtual parameters (up to depth 8). + +### Sandbox (`lib/sandbox.ts`) + +The sandbox provides a secure execution environment for provision and virtual +parameter scripts using `vm.Script` with a 50ms timeout. It uses a +**replay-based execution model**: + +1. A script runs and calls `declare()` to request data. +2. When `commit()` is called, the script throws a sentinel symbol and exits. +3. The engine fetches the requested data via CWMP RPCs. +4. The script is **re-run from the beginning** with the fetched data available. +5. Earlier `declare()` calls return cached results; the script progresses + further. +6. This repeats until the script completes without throwing. + +The sandbox API: `declare(path, timestamps, values)` returns a +`ParameterWrapper` proxy; `clear(path, timestamp, attributes)` invalidates +cached data; `ext(...args)` calls external extensions (results are cached per +revision to survive replays); `commit()` explicitly triggers a fetch cycle. +`Math.random()` is replaced with a seeded PRNG for determinism. + +### Device Data Model (`lib/types.ts`, `lib/device.ts`) + +`DeviceData` is the in-memory working copy of a device's parameter tree during a +session: + +- `paths: PathSet` -- All known parameter paths. +- `timestamps: VersionedMap` -- When each path was last refreshed. +- `attributes: VersionedMap` -- Per-path attributes (object, + writable, value, notification, accessList), each paired with a timestamp. +- `trackers` / `changes` -- Change tracking for re-evaluation. + +`VersionedMap` (in `versioned-map.ts`) provides multi-revision snapshots, +enabling the sandbox replay model where scripts may be re-run at different +revision levels. + +`device.ts` handles setting and clearing parameter data with invariant +enforcement (e.g., if `value` is set, `object` is forced to 0; parent paths are +ensured to exist). The `unpack()` function resolves wildcards and alias paths +against concrete device data. + +### NBI (`lib/nbi.ts`) + +A raw Node.js HTTP listener (no framework) with regex-based URL routing. +Endpoints include CRUD for presets, provisions, virtual parameters, objects, and +files; device task management (with optional synchronous execution via +connection request); fault management; generic collection querying; and ping. + +The NBI uses MongoDB-style JSON queries directly. For the `devices` collection, +`query.ts` expands user-friendly queries by auto-appending `._value` to +parameter paths and generating multi-type interpretations (string, number, date, +regex) for filter values. + +### File Server (`lib/fs.ts`) + +A minimal HTTP file server reading from MongoDB GridFS. Supports GET/HEAD with +full HTTP caching (ETag, Last-Modified, If-None-Match, If-Modified-Since) and +Range requests (HTTP 206) for partial content downloads. Files are cached +in-memory via memoization. + +### UI Backend (`lib/ui.ts`, `lib/ui/`) + +A Koa application with JWT authentication, role-based authorization, and a rich +CRUD API under `/api/`. The root route serves an HTML shell that bootstraps the +Mithril.js SPA with injected config, user info, and hashed asset filenames. + +`lib/ui/api.ts` defines generic CRUD endpoints for all resource types (devices, +presets, provisions, files, config, users, permissions, faults, tasks) with +authorization checks at every level. Specialized endpoints handle file +upload/download, synchronous task execution, tag management, password changes, +and CSV export. + +`lib/ui/db.ts` translates between the UI's flat parameter representation and +MongoDB's nested document structure. The `flattenDevice()` function is the key +transformation: it recursively walks the nested device document and produces a +flat key-value map (e.g., +`Device.WiFi.SSID.1.Name -> {value: [...], writable: true}`). + +`lib/ui/local-cache.ts` caches permissions, users, and config in-process with +hash-based revision tracking. The `getConfig()` function resolves config +expressions with dynamic context and backward compatibility for legacy config +keys. + +### Database Layer (`lib/db/`) + +`db/db.ts` is the single entry point to MongoDB. It manages 14 collections: + +| Collection | Purpose | +| ------------------- | -------------------------------- | +| `devices` | CPE device parameter trees | +| `presets` | Configuration rules | +| `provisions` | Provision scripts | +| `virtualParameters` | Virtual parameter scripts | +| `objects` | Generic objects | +| `tasks` | Queued device management tasks | +| `faults` | Error records with retry state | +| `operations` | In-flight async operations | +| `files` (GridFS) | Firmware images and config files | +| `permissions` | RBAC rules | +| `users` | User accounts | +| `config` | Key-value configuration | +| `cache` | Distributed cache (TTL index) | +| `locks` | Distributed locks (TTL index) | + +`db/synth.ts` is the sophisticated expression-to-MongoDB query compiler. It +normalizes expressions, converts them to a Boolean satisfiability representation +using the `Clause` hierarchy, minimizes via Espresso, and emits MongoDB +`$and`/`$or`/`$not` filter objects. This is used by the UI backend. + +`cwmp/db.ts` handles CWMP-specific persistence: loading the nested device +document into `DeviceData` (`fetchDevice`), diffing changes back into MongoDB +update operations (`saveDevice`), and managing faults, tasks, and operations. + +### Query Systems + +There are two separate query paths: + +1. **NBI queries** (`lib/query.ts`) -- Processes MongoDB-style JSON queries from + external API consumers. Expands parameter names, generates multi-type value + interpretations, and passes through to MongoDB directly. + +2. **UI/Expression queries** (`lib/db/synth.ts`) -- Compiles the internal + expression language into optimized MongoDB filters via Boolean minimization. + This is the more sophisticated path, used by the UI backend. + +### Frontend (`ui/`) + +A Mithril.js SPA with hash-based routing (`#!/`). Key modules: + +- `app.ts` -- Route definitions. The `pagify()` function wraps each page into a + `RouteResolver` that handles initialization, error boundaries, and data + fulfillment. +- `layout.ts` -- Top-level layout (header, navigation, content, overlay). +- `store.ts` -- Centralized data store. Implements a query-based reactive cache + with deduplication and incremental fetching. The `fulfill()` method (called + after every render) batches pending queries, computes filter diffs via + `unionDiff()`, and fetches only missing data. Connection monitoring polls + every 3 seconds for server health, clock skew, and config changes. +- `components.ts` -- Component registry with a context propagation system. The + proxied `m()` function resolves string component names and the `m.context()` + API passes data (like the current device object) down the component tree + without explicit prop threading. +- `smart-query.ts` -- Translates user-friendly `Label: value` searches into + filter expressions with type-aware matching (string, number, timestamp, MAC + address, tag). +- `task-queue.ts` -- Two-stage task pipeline: staging (user configures tasks) + then queue (tasks are committed and executed via the backend). +- `dynamic-loader.ts` -- Lazy loading of heavy libraries (CodeMirror, YAML) via + dynamic `import()`. + +### Build System (`build/`) + +The build is a self-bootstrapping esbuild pipeline (`npm run build` pipes +`build/build.ts` through esbuild then node). It produces: + +- **Backend binaries** -- 5 entry points bundled for Node.js 12+ with shebang + banners, executable permissions, and `.js` extension stripped. +- **Frontend bundle** -- `ui/app.ts` bundled for browsers (ESM, code-split) with + content-hashed filenames. +- **CSS** -- `ui/css/app.css` bundled and minified with content-hashed output. +- **SVG sprite** -- All icons in `ui/icons/` optimized via SVGO and combined + into a single sprite. + +`build/assets.ts` is a compile-time bridge: at rest it contains placeholder +filenames; during build, the `assetsPlugin` replaces them with actual +content-hashed names so both backend and frontend reference the correct assets. + +Build metadata includes the date and a hash derived from the git state, appended +to the package version. + +## Cross-Cutting Concerns + +### Multi-Level Caching + +``` +memoize.ts In-process function cache (2-4 min, two-generation rotation) + | +local-cache.ts In-process snapshot cache (5s refresh, hash-based revisions) + | +cache.ts MongoDB-backed distributed cache (configurable TTL) + | +MongoDB TTL index auto-expiration +``` + +Each service has its own `local-cache` that periodically checks the distributed +cache for staleness. Distributed locks (`lock.ts`) coordinate rebuilds so only +one worker does the expensive computation. + +### Configuration (`lib/config.ts`) + +Three-tier priority: CLI args > environment variables (`GENIEACS_*`) > config +file (`config.json`) > defaults. Supports per-device overrides by appending +`-OUI-ProductClass-SerialNumber` to option names. + +### Distributed Locking (`lib/lock.ts`) + +MongoDB-based mutual exclusion using upsert + duplicate key detection. TTL +indexes prevent deadlocks from crashed processes. Clock skew tolerance of 30 +seconds. + +### Authentication + +- **CWMP devices** -- HTTP Basic or Digest auth, configurable per-device via + expressions (`auth.ts`). +- **UI users** -- JWT tokens in cookies. Passwords hashed with PBKDF2-SHA512 + (10000 iterations). Role-based authorization via `Authorizer` with + expression-based filters and validators. + +### Connection Requests (`lib/connection-request.ts`) + +Three methods to ask a CPE device to initiate a CWMP session: + +- **HTTP** -- Standard TR-069 connection request with Digest/Basic auth. +- **UDP** -- For NAT traversal (STUN-based) with HMAC-SHA1 signed messages. +- **XMPP** -- TR-069 Annex K via a full XMPP client (`xmpp-client.ts`). + +### Extensions (`lib/extensions.ts`) + +User-defined scripts executed in long-lived child processes (`genieacs-ext.ts`) +communicating via IPC. Processes are lazily spawned per script name and reused. +Each request gets a unique ID; responses are matched by ID with a configurable +timeout. + +### Logging (`lib/logger.ts`) + +Dual-stream structured logging (application + access logs). Supports simple and +JSON formats, systemd journal integration, and automatic log rotation detection. +Protocol traces can be written to a debug file in YAML or JSON format +(`debug.ts`). + +### Three-Valued Logic + +The system consistently implements SQL-style three-valued logic +(true/false/null). This is visible in expression evaluation, the `Clause` +hierarchy's separate true/false/null methods, and the 2-bit minterm encoding in +the Boolean minimizer. NULL means "unknown" and propagates through operations +following SQL semantics. + +## Architectural Invariants + +- **The expression system does not depend on any service-specific code.** The + `lib/common/expression/` modules are pure and shared across all services. + +- **The sandbox is deterministic across replays.** `Math.random()` is seeded + from the device ID, `Date.now()` is controlled, and extension results are + cached. A script re-run with the same inputs produces the same outputs. + +- **Services share no in-process state.** All cross-process coordination goes + through MongoDB (the `cache` and `locks` collections). Each worker process is + independent. + +- **Device data is never mutated in place during a session.** `VersionedMap` + provides revision-based snapshots. The sandbox writes to new revisions; + earlier revisions remain readable for re-evaluation. + +- **CWMP session exclusivity.** A distributed lock (`cwmp_session_`) + ensures only one session exists per device at a time. The lock is refreshed + periodically and released at session end. + +- **The UI backend never queries MongoDB with raw user input.** All queries go + through the expression-to-MongoDB compiler (`db/synth.ts`), which validates + and normalizes expressions before generating filters. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7161d13..da580c6 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,31 +1,348 @@ -# Have a question? +# Contributing to GenieACS -Please ask in the [forum] instead of opening a Github Issue. The Github issue -tracker is strictly for bug reports. +## Questions and Support -# Issues +Please use the [forum](https://forum.genieacs.com) for questions, help requests, +and general discussion. GitHub Issues are reserved for confirmed bug reports. -We use GitHub issues to track bugs. Please ensure your bug description is clear -and has sufficient instructions to be able to reproduce the issue. +## Issues -If you face interoperability issues with your CPE, more often than not it's a -device related issue. Please consult the [forum] before submitting an Issue. +We use GitHub Issues to track bugs. When filing a bug report: -# Pull Requests +- Provide a clear description of the problem. +- Include steps to reproduce the issue. +- Note the GenieACS version, Node.js version, and MongoDB version. +- Include relevant log output if applicable. -Pull requests are welcome. If you have a fix for a bug, go ahead and create a -pull request. For other kinds of enhancements, please discuss in the [forum] -first. +If you face interoperability issues with a CPE, it is more often than not a +device-specific issue. Please consult the [forum](https://forum.genieacs.com) +before opening an Issue. -Please follow the guidelines [here][commit-guidelines] when committing your -changes. +## Pull Requests + +Pull requests are welcome. For bug fixes, go ahead and open a PR directly. For +new features or significant changes, please discuss in the +[forum](https://forum.genieacs.com) first to ensure alignment with the project's +direction. + +When submitting a PR: + +- Keep changes focused. One PR should address one concern. +- Run `npm run lint` and `npm test` before submitting. +- Update documentation in `docs/` if your change affects user-facing behavior. +- Add tests where appropriate (see [Testing](#testing)). +- Write a clear PR description explaining what the change does and why. + +## Code Style + +### Naming Conventions + +- **Variables and functions**: `camelCase` +- **Classes**: `PascalCase` +- **Interfaces and type aliases**: `PascalCase`, without an `I` or `T` prefix +- **Module-level constants**: `UPPER_SNAKE_CASE` +- **Private class members**: prefixed with underscore (`_name`, `_cache`) + +Choose descriptive, meaningful names. A longer clear name is better than a short +ambiguous one. + +### Imports + +Organize imports in three groups, in this order: + +1. Node.js built-in modules, using the `node:` prefix +2. External packages +3. Local project modules + +Use `.ts` extensions for all local imports. + +```typescript +import { readFileSync } from "node:fs"; +import * as http from "node:http"; + +import { Collection } from "mongodb"; + +import { connect, disconnect } from "./db.ts"; +import Path from "./common/path.ts"; +``` + +### Functions and Exports + +Use the `function` keyword for all named and exported function declarations. +Arrow functions should only be used for callbacks and short inline expressions. + +```typescript +// Named/exported functions use function keyword +export function processRequest(req: Request): Response { + // ... +} + +// Arrow functions for callbacks +items.filter((item) => item.active); +promise.then((result) => { + // ... +}); +``` + +### TypeScript + +- All function declarations must have explicit return types. This is enforced by + ESLint. +- Using `any` is permitted where full typing would be impractical, but prefer + more specific types when reasonable. +- Prefer `Record` over `{ [key: string]: T }` for mapped types. +- Use type assertions (`as`) sparingly and only when you have stronger type + knowledge than the compiler. + +### Comments + +Code should be self-documenting. Use comments sparingly and only when the "why" +is not obvious from the code itself. When you do comment, use inline `//` style. + +Do not use JSDoc (`/** */`) or block comments (`/* */`). + +```typescript +// Good: explains a non-obvious reason +// Escapes everything except alphanumerics and underscore +function escapeRegExp(str: string): string { + /* ... */ +} + +// Good: links to an external reference +// Source: http://stackoverflow.com/a/6969486 + +// Good: flags a known limitation +// TODO support "MD5-sess" algorithm directive + +// Bad: restates what the code does +// Loop through each item and increment the counter +for (const item of items) { + counter++; +} +``` + +### Error Handling + +Use `== null` (not `=== null || === undefined`) for null/undefined checks. +ESLint is configured to allow this. + +## Testing + +### When to Write Tests + +Tests are most valuable for pure logic: parsers, data transformations, +algorithms, and utility functions where inputs and outputs are well defined. A +regression test accompanying a bug fix is also worthwhile to prevent the same +issue from resurfacing. + +Not every change needs a test. Use judgment and consider the likelihood and cost +of breakage. Avoid writing tests that duplicate coverage already provided by +existing tests, and resist the urge to test trivial code just for the sake of +coverage numbers. + +### Conventions + +Tests use the Node.js built-in test runner (`node:test`) and assertion module +(`node:assert`). No external test libraries or mocking frameworks are used. + +```typescript +import test from "node:test"; +import assert from "node:assert"; + +void test("parseValue returns correct type for integer strings", () => { + const result = parseValue("42"); + assert.strictEqual(result, 42); +}); + +void test("parseValue throws on invalid input", () => { + assert.throws(() => parseValue(""), new Error("empty value")); +}); +``` + +Key conventions: + +- Prefix `test()` calls with `void` to satisfy the `no-floating-promises` lint + rule. +- Keep tests flat. Do not nest `describe` blocks. +- Use descriptive test names that state what is being tested and the expected + outcome. +- Use `assert.strictEqual()` for value comparisons and + `assert.deepStrictEqual()` for objects and arrays. + +Test files live in the `test/` directory and are named after the module they +test (e.g. `test/path.ts` tests `lib/common/path.ts`). + +## Dependencies + +This project deliberately keeps its dependency footprint small. Before adding a +new dependency: + +- Prefer Node.js built-in APIs when they can do the job. +- Consider whether the functionality is simple enough to implement directly. +- Justify the addition in your PR description. + +Do not add development tool dependencies (linter plugins, editor integrations, +etc.) without prior discussion. + +## File Organization + +| Directory | Contents | +| ------------- | -------------------------------------------- | +| `lib/` | Core server-side application code | +| `lib/common/` | Code shared between server and browser | +| `lib/db/` | Database layer (MongoDB) | +| `lib/ui/` | UI backend helpers | +| `ui/` | Frontend code (Mithril.js SPA) | +| `bin/` | Service entry points | +| `build/` | Build tooling | +| `test/` | Test files | +| `docs/` | User documentation (Sphinx/reStructuredText) | +| `public/` | Static assets (favicon, logo) | + +Place new code in the directory that matches its purpose. Server-side logic +belongs in `lib/`, code that must run in both Node.js and the browser belongs in +`lib/common/`, and frontend-only code belongs in `ui/`. + +## Documentation + +User documentation lives in the `docs/` directory as reStructuredText files +built with Sphinx and published to +[docs.genieacs.com](https://docs.genieacs.com). + +When your change affects user-facing behavior: + +- Update the relevant documentation in `docs/`. +- If adding a new feature, consider whether it warrants a new page or a section + in an existing page. +- Keep documentation concise and practical. Match the existing tone: direct, + factual, no filler. + +Documentation changes should be included in the same PR as the code change, not +submitted separately. + +### ARCHITECTURE.md + +`ARCHITECTURE.md` describes the high-level architecture of the project: the +service boundaries, major subsystems, data flow, and key invariants. It is aimed +at contributors who need a mental map of the codebase. + +This file has a different update cadence than `docs/`. It should be revisited a +few times a year rather than kept in lockstep with every code change. When you +do update it, follow these principles: + +- **Only describe things that are unlikely to change frequently.** Module + responsibilities, service boundaries, key data structures, and architectural + invariants belong here. Implementation details, function signatures, and + config option lists do not. +- **Name important files, modules, and types but do not link them.** Links go + stale. Encourage the reader to use symbol search to find named entities; this + also helps them discover related, similarly named things. +- **Keep it short.** Every recurring contributor will read it. A shorter + document is less likely to become stale and more likely to be maintained. +- **Describe the "what" and "where", not the "how".** This is a map of the + country, not an atlas of its states. Pull detailed explanations into inline + code comments or separate documents. +- **Call out architectural invariants explicitly.** Important invariants are + often expressed as the _absence_ of something (e.g., "the expression system + does not depend on service-specific code") and are hard to discover by reading + code alone. + +## Git Workflow + +### Commit Messages + +This project follows the +[Conventional Commits](https://www.conventionalcommits.org/) format: + + (): + + [optional body] + +- Use the imperative mood, present tense: "Fix bug", not "Fixed bug" or "Fixes + bug". +- Do not capitalize the first letter of the subject (the type prefix handles + visual structure). +- Do not end the subject line with a period. +- Keep the subject line under 72 characters. +- When more context is helpful, add a body separated from the subject by a blank + line, wrapped at 72 characters. The goal is to provide enough information for + someone scanning the commit history to find a specific change (e.g. for + troubleshooting or rebasing) or to draft changelog entries for a release. + Don't be verbose, but don't be cryptic either. + +#### Types + +| Type | When to use | +| ---------- | --------------------------------------------------------------------------- | +| `fix` | Bug fixes | +| `feat` | New features or capabilities | +| `refactor` | Code restructuring with no behavior change | +| `test` | Adding or updating tests | +| `docs` | Documentation changes (`docs/`, `CONTRIBUTING.md`, `README.md`) | +| `build` | Build system, dependencies, esbuild config, `package.json` scripts | +| `chore` | Maintenance tasks that don't fit above (`.gitignore`, tooling config, etc.) | + +#### Scopes + +Scope is optional. Use it when it adds useful context; omit it when the change +is cross-cutting or the subject already makes it obvious. + +| Scope | Covers | +| ------ | ----------------------------------- | +| `cwmp` | CWMP service (including extensions) | +| `nbi` | Northbound REST API service | +| `fs` | File service | +| `ui` | Web UI (frontend and backend) | +| `db` | Database layer | + +If a change touches multiple scopes, either pick the primary one or omit the +scope entirely. + +#### Examples + + feat(nbi): add bulk device delete endpoint + fix(cwmp): handle missing ParameterKey in InformResponse + refactor(db): replace raw queries with parameterized calls + fix(ui): correct parameter table sort order + test: add XML parser edge case coverage + docs: update provisioning guide for new API + build: upgrade esbuild to v0.20 + + fix(cwmp): increase server timeout to 2 mins + + To allow enough time for running unindexed queries in large + deployments. + +### Branches + +- `master` is the main development branch. +- Create a feature or fix branch for your work and open a PR against `master`. +- Use concise, descriptive branch names in lowercase with hyphens: + `fix-race-condition`, `support-xmpp-requests`. + +## Changelog + +The changelog (`CHANGELOG.md`) is maintained for each release and is written for +users and system administrators, not developers. + +- Write entries as clear, user-facing prose. Do not copy commit messages + verbatim. +- Each entry should describe what changed and, when helpful, why it matters. +- Group entries under a version heading with the release date: + `## 1.2.13 (2024-06-06)`. +- Start each entry with a verb: "Fix", "Add", "Improve", "Remove", etc. +- Include enough context that a user can understand the impact without reading + the code. + +You do not need to update the changelog in your PR. The maintainer will add +changelog entries during the release process. ## Contributor License Agreement By submitting a pull request to this repository, you acknowledge that, while -maintaining copyright, you grant Zaid Abdulla a perpetual, worldwide, non- -exclusive, no-charge, royalty-free, irrevocable license to reproduce, prepare -derivative works of, publicly display, publicly perform, sublicense, and +maintaining copyright, you grant GenieACS Inc. a perpetual, worldwide, +non-exclusive, no-charge, royalty-free, irrevocable license to reproduce, +prepare derivative works of, publicly display, publicly perform, sublicense, and distribute your contributions and such derivative works under the AGPLv3 license or any other license terms, including, but not limited to, proprietary or commercial license terms. @@ -33,7 +350,3 @@ commercial license terms. You confirm that you own or have rights to distribute and sublicense the source code contained therein, and that your content does not infringe upon the intellectual property rights of a third party. - -[commit-guidelines]: - http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html -[forum]: https://forum.genieacs.com