linuxrouter/docker/routlin-dash/GUIDE.md

# Developer Guide

This guide covers how to add features, write action handlers, and follow the conventions used throughout the codebase.

---

## Architecture Overview

This app is the web UI for Routlin, a Linux-based router software package. It runs as a Docker container and manages the same configuration files that Routlin's core scripts read and apply to the system. The app does not manage the system directly; it edits `config.json` and queues commands that Routlin's `core.py` executes.

The app is a Flask application in a Docker image.

```
app/                  Python source (baked into image)
  factory.py          Converts content.json item trees into HTML strings
  view_page.py        Routes, data loaders, token assembly, layout rendering
  navbar.json         Navigation structure
  pages/              One subdirectory per page
    <pagename>/
      content.json    Declarative page layout
      action.py       Flask Blueprint for POST actions on this page
  sanitize.py         Input sanitization (strips dangerous characters)
  config_utils.py     Config I/O, snapshot system, command queue
  authorized_accounts.json  Web UI user accounts (separate from Routlin users)

data/                 Live-mounted at runtime (./data:/data)
  styles.css          Application stylesheet
  common.js           Client-side interaction logic
  validation.js       Client-side field validation

                      # host directory mounted into container: $HOME/routlin -> /routlin_location
routlin_location/    Routlin install dir
  config.json         Main Routlin configuration (read on every request, written on save)
  validation.py       Shared validation module (imported via PYTHONPATH)
  core.py             Apply script invoked to push config changes to the system
  ddns.log            DDNS update log
  blocklists/         Downloaded blocklist files (*.con)
  .snapshots/         Config snapshot JSON files (one per saved change)
  .health             Health status JSON written by Routlin
  .dashboard-queue    Pending commands waiting to be applied
  .dashboard-done     Record of completed commands
  .dashboard-pending  Commands queued but not yet run
  .dashboard-last-run Timestamp of the last apply run
  .dashboard-lock     Lock file indicating an apply is in progress
  .ddns-last-ip-*     Cached public IP files, one per DDNS provider
  .ddns-last-service  Timestamp of the last DDNS service check
  .<iface>.pub        WireGuard public key files (e.g. .wg0.pub)
```

---

## Shared Resources with Routlin

The Routlin install directory (`$HOME/routlin` on the host) is mounted into the container at `/routlin_location` and added to `PYTHONPATH`. This is the primary integration point between the dashboard and the router software.

Files and directories under `/routlin_location` that the app reads or writes:

| Path | Access | Description |
|------|--------|-------------|
| `config.json` | read/write | Main Routlin configuration. The app reads this on every request and writes it on every save action. |
| `validation.py` | import | Shared validation module imported as `import validation as validate`. Contains field validators and `validate_config()` used by all action handlers. |
| `core.py` | exec | Routlin's apply script. The app invokes it as `python3 core.py --apply` or `--update-blocklists` to push config changes to the system. |
| `ddns.log` | read | DDNS update log shown in the DDNS page. |
| `blocklists/` | read | Directory of downloaded blocklist files (`*.con`). The app reads them to count entries and report last-updated timestamps. |
| `.snapshots/` | read/write | Directory of config snapshot JSON files. One file per saved change, used for the change history and revert feature. |
| `.health` | read | JSON file written by Routlin describing service and configuration health status. The dashboard reads it to display the health panel and auto-queue fixes. |
| `.dashboard-queue` | read/write | Pending commands waiting to be applied. |
| `.dashboard-done` | read/write | Record of completed commands. |
| `.dashboard-pending` | read/write | Commands that have been queued but not yet run. |
| `.dashboard-last-run` | read | Timestamp of the last dashboard apply run. |
| `.dashboard-lock` | read | Lock file checked to determine if an apply is currently in progress. |
| `.ddns-last-ip-*` | read | One file per DDNS provider, written by Routlin with the last-known public IP. The app reads all matching files to display the current IP. |
| `.ddns-last-service` | read | Timestamp written by Routlin each time the DDNS service runs. Displayed as "Last checked" on the DDNS page. |
| `.<iface>.pub` | read | WireGuard public key files (e.g. `.wg0.pub`). Read by the VPN page to display the server public key. |

The container also mounts `/sys/class/net` and `/sys/devices` read-only to read live network interface state (link status, speed, MAC address, MTU), and `/etc/localtime` read-only for correct timezone display.

---

## Code Style

- **ASCII only in all files.** No em dashes, en dashes, curly quotes, or any non-ASCII character. If a sentence needs a pause that would normally use a dash, restructure it.
- **No comments unless the WHY is non-obvious.** Well-named functions and variables explain the what. Comments belong on hidden constraints, subtle invariants, or workarounds for specific external bugs.
- **No docstrings on obvious functions.** Reserve them for functions with non-obvious behavior or important invariants to document.
- **`import validation as validate`** - the module is aliased because `validate` is a common local variable name and shadowing it causes confusion.
- **Imports at top, no inline imports** except in rare cases where a circular import cannot otherwise be avoided (e.g. the `from flask import abort` inside `serve_view`).

---

## Naming Conventions

### Underscore prefix

Use `_` prefix only for tiny helpers used in exactly one place. Do not use it for:
- Module-level constants (`NAVBAR_FILE`, not `_NAVBAR_FILE`)
- Functions reused by more than one caller
- Imports (`import factory`, not `import factory as _factory`)

Good uses of `_`:
- A two-line inner function defined inside a larger function and called once
- A short closure that is never referenced outside its enclosing scope

### File path constants

Define all file paths as named constants at the top of the module, using `os.path.join`:

```python
PAGES_DIR    = os.path.join(APP_DIR,     'pages')
NAVBAR_FILE  = os.path.join(APP_DIR,     'navbar.json')
CSS_FILE     = os.path.join(DATA_DIR,    'styles.css')
```

Never hardcode paths inline.

### Section headers in Python files

Use the `# Label ===...` style for major section breaks:

```python
# File loaders ======================================================

# Config data loaders ===============================================

# Routes ============================================================
```

---

## HTML Building

All HTML in `factory.py` and `view_page.py` is built by string concatenation using f-strings, not a template engine. Every value that comes from user data or config must be passed through `e()` before interpolation:

```python
from factory import e

html = f'<td class="table-cell">{e(row["description"])}</td>'
```

`e()` wraps `html.escape()`. Never skip it for user-controlled or config-sourced values.

When building multi-line HTML inside view_page.py for a token, follow the same `e()` discipline and assign the result to `tokens['MY_TOKEN']`. Mark values as `Markup()` only when they are already-safe assembled HTML that must not be double-escaped at the outer render stage.

Key principle: `factory.py` is a pure HTML builder. It has no routing, no data loading, and no Flask context other than reading the session for access level. All data loading and token assembly happens in `view_page.py`. The two modules are kept separate to avoid circular imports; `view_page.py` injects `factory.load_datasource = load_datasource` at startup.

---

## Adding a New Page

### 1. Create the page directory

```
app/pages/<pagename>/
  __init__.py         Empty file (makes it a Python package)
  content.json        Page layout
  action.py           POST action handlers
```

`<pagename>` becomes the URL: `/pagename`.

### 2. Write content.json

Every `content.json` starts with a `client_requirement` and an `items` array:

```json
{
  "client_requirement": "client_is_viewer+",
  "items": [
    {
      "type": "header_page_title",
      "items": [
        { "type": "h1", "text": "My Page" },
        { "type": "p",  "text": "Description here." }
      ]
    }
  ]
}
```

See `TYPES.md` for the full set of item types and their fields. See `TYPES.md#access-control` for the full set of `client_requirement` values.

### 3. Write action.py

```python
from pathlib import Path
from flask import Blueprint, request, redirect, flash
from auth import require_level
from config_utils import load_config, save_config_with_snapshot, verify_config_hash
import sanitize
import validation as validate

_PAGE = Path(__file__).parent.name
bp = Blueprint(_PAGE, __name__)
```

`_PAGE` is the canonical page name and redirect target. Using `Path(__file__).parent.name` means it stays correct even if the directory is renamed.

### 4. Register the blueprint

In `app/main.py` (or wherever blueprints are registered), add the new action blueprint alongside the existing ones.

---

## Token System

Tokens are `%LIKE_THIS%` placeholders in `content.json` string fields. They are substituted before the page is rendered. Tokens carry dynamic data (config values, computed HTML, JSON arrays) from Python into the declarative layout without embedding Python logic in the JSON.

Tokens are assembled in `view_page.py`'s `collect_tokens()` function:

```python
tokens['VLAN_SUBNET'] = vlan.get('subnet', '')
tokens['EXISTING_VLAN_IDS_JSON'] = json.dumps([v['vlan_id'] for v in vlans])
```

String tokens that resolve to a JSON array or object are automatically parsed back into Python structures by `factory.expand_fields()`, so they can be serialized correctly into `data-fields` attributes.

Token values are not HTML-escaped by default. When a token is used in an HTML attribute or text node, `factory.py` calls `e()` on it. When a token resolves to raw HTML meant for injection (e.g. `PENDING_ACTIONS_HTML`), it is used as-is and must be safe before it enters the token map.

---

## Action Handler Pattern

Every POST action follows the same sequence: parse, validate input, check config hash, mutate config, validate config, save.

```python
@bp.route('/action/bannedips/addip_add', methods=['POST'])
@require_level('administrator')
def addip_add():
    # 1. Parse and validate individual fields first, before touching config.
    raw = request.form.get('ip', '').strip()
    ip = validate.banned_ip(raw)
    if not ip:
        flash('Invalid IP address.', 'error')
        return redirect(f'/{_PAGE}')

    description = sanitize.text(request.form.get('description', ''))

    # 2. Check config hash (stale-config detection / CSRF guard).
    if not verify_config_hash(request.form.get('config_hash', '')):
        flash('Configuration was modified by another session. Please refresh and try again.', 'error')
        return redirect(f'/{_PAGE}')

    # 3. Load, mutate, validate the full config.
    cfg = load_config()
    entry = {'ip': ip, 'description': description, 'enabled': True}
    cfg.setdefault('banned_ips', []).append(entry)

    errors = validate.validate_config(cfg)
    if errors:
        for msg in errors:
            flash(msg, 'error')
        return redirect(f'/{_PAGE}')

    # 4. Save with a snapshot entry.
    flash(save_config_with_snapshot(
        cfg,
        path='banned_ips', key=ip, operation='add',
        before=None, after=entry,
        description=f'Added banned IP: {ip}',
    ), 'success')
    return redirect(f'/{_PAGE}')
```

Rules:
- Always redirect back to `/{_PAGE}` after a POST, never render HTML directly.
- `flash()` is the only feedback mechanism. Use `'error'` or `'success'` as the category.
- Parse all user input before checking the config hash. If input is invalid you want to bail out cheaply before acquiring anything.
- Always call `validate.validate_config(cfg)` on the mutated config before saving, even for simple operations.
- `save_config_with_snapshot` returns a human-readable success message string suitable for passing directly to `flash()`.

### Row index actions

For table row operations, extract the row index and bounds-check it:

```python
def _row_index():
    try:
        return int(request.form.get('row_index', ''))
    except (ValueError, TypeError):
        return None

idx = _row_index()
if idx is None:
    flash('Invalid request.', 'error')
    return redirect(f'/{_PAGE}')

items = cfg.get('my_list', [])
if idx < 0 or idx >= len(items):
    flash('Entry not found.', 'error')
    return redirect(f'/{_PAGE}')
```

---

## Input Handling

### Sanitize vs. validate

- `sanitize.*` functions strip or normalize raw input so it is safe to store. They do not return errors; they return a cleaned value.
- `validate.*` functions check whether a value is semantically correct for a specific field. They return a cleaned/normalized value on success or a falsy value (empty string, `None`) on failure.

```python
description = sanitize.text(request.form.get('description', ''))  # always safe
ip = validate.banned_ip(raw)                                       # returns None on failure
```

### Checkbox fields

Checkboxes are absent from the form body when unchecked. Always compare to `'on'`:

```python
enabled = request.form.get('enabled') == 'on'
```

### JSON fields from record_editor

`record_editor` submits its data as a JSON string in a hidden input. Parse it defensively:

```python
import json
raw = request.form.get('server_identities', '[]')
try:
    identities = json.loads(raw)
    if not isinstance(identities, list):
        raise ValueError
except (ValueError, TypeError):
    flash('Invalid identity data.', 'error')
    return redirect(f'/{_PAGE}')
```

---

## Access Control

### In content.json

Any item can carry `"client_requirement": "client_is_administrator+"`. Items that fail the check are omitted from the rendered output entirely. See `TYPES.md#access-control` for the full table.

The page-level `client_requirement` is inherited by all child items that do not declare their own. This means you can set `client_is_viewer+` at the page level and only gate specific cards or buttons at a higher level.

### In action.py

```python
@require_level('administrator')
def my_action():
    ...
```

Valid levels: `nothing`, `viewer`, `administrator`, `manager`. The decorator enforces the minimum rank and handles the redirect automatically.

---

## Config Persistence

`load_config()` reads `config.json` fresh on every call. Do not cache it across a request.

`save_config_with_snapshot(cfg, path, key, operation, before, after, description)` writes the config and records a snapshot entry for the change history. Always pass `before` and `after` as the specific sub-object that changed, not the whole config.

`verify_config_hash(hash)` confirms the config has not changed since the form was rendered. Always check it before mutating the config.

---

## Client-Side Validation

Attach a `validate` field to any `field` item in `content.json` to enable live validation:

```json
{ "type": "field", "name": "subnet", "validate": "ipv4" }
```

See `TYPES.md#validation` for the full table of validate values and what each accepts.

For `record_editor` fields that need subnet-aware validation, use `valtype` instead of `validate` and wire `data-dep-subnet` / `data-dep-mask` attributes via the `attrs` field.

For data that must be passed to a JS validator (such as existing VLAN IDs for uniqueness checking), use a `data-*` HTML attribute on the input rather than a global JS variable. The `existing_ids` field on a `field` item emits `data-existing-ids` on the rendered input.

---

## JS Table Workers

Non-standard `input_type` values in `inline_edit` row actions require a table worker. `factory.py` detects any `input_type` not in `STANDARD_INPUT_TYPES = {'text', 'password', 'number', 'checkbox', 'select', 'textarea'}` and emits a `<script>` block via `build_table_worker_script()` that registers the worker using `registerTableWorker(id, impl)`.

Workers implement:
- `renderCell(fDef, td, val, row)` - returns `true` if handled, `false` to fall through to default text input
- `afterRowOpen(tr, row)` - optional; called after all cells are rendered; use it to wire cross-field listeners

The worker ID is derived from the table's `datasource` field by stripping the `config:` or `live:` prefix. The only current non-standard type is `credentials` on the DDNS accounts table.