From 7e16dded0bae03d74aac3148cdf1fac40ad207f3 Mon Sep 17 00:00:00 2001 From: Eric Callahan Date: Sat, 18 May 2024 07:34:18 -0400 Subject: [PATCH] docs: add sqlite documentation Signed-off-by: Eric Callahan --- docs/changelog.md | 9 +- docs/installation.md | 34 +++-- docs/web_api.md | 332 ++++++++++++++++++++++++++++++++++++++++--- 3 files changed, 344 insertions(+), 31 deletions(-) diff --git a/docs/changelog.md b/docs/changelog.md index b6e9c3e..0c09f3f 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -61,15 +61,15 @@ The format is based on [Keep a Changelog]. ### Changed -- **build**: Bumped apprise to version `1.7.0`. +- **build**: Bumped apprise to version `1.8.0`. - **build**: Bumped lmdb to version `1.4.1` - **build**: Bumped tornado to version `6.4.0` -- **build**: Bumped jinja2 to version `3.1.3` +- **build**: Bumped jinja2 to version `3.1.4` - **build**: Bumped zeroconf to version `0.131.0` - **build**: Bumped libnacl to version `2.1.0` - **build**: Bumped distro to version `1.9.0` - **build**: Bumped pillow to version `10.3.0` -- **build**: Bumped streaming-form-data to version `1.13.0` +- **build**: Bumped streaming-form-data to version `1.15.0` - **machine**: Added `ratos-configurator` to list of default allowed services - **update_manager**: It is now required that an application be "allowed" for Moonraker to restart it after an update. @@ -91,6 +91,9 @@ The format is based on [Keep a Changelog]. - **gpio**: Migrate from libgpiod to python-periphery - **authorization**: The authorization module is now loaded as part of Moonraker's core. +- **database**: Migrated the underlying database from LMDB to Sqlite. +- **history**: Use dedicated SQL tables to store job history and job totals. +- **authorization**: Use a dedicated SQL table to store user data. ## [0.8.0] - 2023-02-23 diff --git a/docs/installation.md b/docs/installation.md index f1b4824..a47f13b 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -202,8 +202,7 @@ structure using the default data path of `$HOME/printer_data`. │   ├── moonraker.conf │   └── printer.cfg ├── database -│   ├── data.mdb -│   └── lock.mdb +│   └── moonraker-sql.db ├── gcodes │   ├── test_gcode_one.gcode │   └── test_gcode_two.gcode @@ -216,7 +215,7 @@ structure using the default data path of `$HOME/printer_data`. └── moonraker.asvc ``` -If it is not desirible for the files and folders to exist in these specific +If it is not desirable for the files and folders to exist in these specific locations it is acceptable to use symbolic links. For example, it is common for the gcode folder to be located at `$HOME/gcode_files`. Rather than reconfigure Klipper's `virtual_sdcard` it may be desirable to create a @@ -566,14 +565,29 @@ Retrieve the API Key via the browser from a trusted client: {"result": "8ce6ae5d354a4365812b83140ed62e4b"} -### LMDB Database Backup and Restore +### Database Backup and Restore -Moonraker uses a [LMDB Database](http://www.lmdb.tech/doc/) for persistent -storage of procedurally generated data. LMDB database files are platform -dependent, and thus cannot be easily transferred between different machines. -A file generated on a Raspberry Pi cannot be directly transferred to an x86 -machine. Likewise, a file generated on a 32-bit version of Linux cannot -be transferred to a 64-bit machine. +Moonraker stores persistent data using an Sqlite database. By default +the database file is located at `/database/moonraker-sql.db`. +API Endpoints are available to backup and restore the database. All +backups are stored at `/backup/database/` and +restored from the same location. Database files may contain sensitive +information, therefore they are not served by Moonraker. Another protocol +such as SCP, SMB, etc is required to transfer a backup off of the host. + +Alternatively it is possible to perform a manual backup by copying the +existing database file when the Moonraker service has been stopped. +Restoration can be performed by stopping the Moonraker service and +overwriting the existing database with the backup. + +#### LDMB Database (deprecated) + +Previous versions of Moonraker used a [LMDB Database](http://www.lmdb.tech/doc/) +for persistent storage of procedurally generated data. LMDB database files are +platform dependent, and thus cannot be easily transferred between different +machines. A file generated on a Raspberry Pi cannot be directly transferred +to an x86 machine. Likewise, a file generated on a 32-bit version of Linux +cannot be transferred to a 64-bit machine. Moonraker includes two scripts, `backup-database.sh` and `restore-database.sh` to help facilitate database backups and transfers. diff --git a/docs/web_api.md b/docs/web_api.md index 666bb00..fff5b98 100644 --- a/docs/web_api.md +++ b/docs/web_api.md @@ -3555,8 +3555,10 @@ the request. The entire settings object could be accessed by providing may be read by omitting the `key` argument, however as explained below it is not possible to modify a namespace without specifying a key. -#### List namespaces -Lists all available namespaces. +#### List Database Info + +Lists all namespaces with read and/or write access. Also lists database +backup files. HTTP request: ```http @@ -3574,14 +3576,21 @@ JSON-RPC request: Returns: -An object containing an array of namespaces in the following format: +An object containing an array of namespaces and an array of backup files. ```json { "namespaces": [ "gcode_metadata", - "history", - "moonraker", - "test_namespace" + "webcams", + "update_manager", + "announcements", + "database", + "moonraker" + ], + "backups": [ + "sqldb-backup-20240513-134542.db", + "testbackup.db", + "testbackup2.db" ] } ``` @@ -3701,6 +3710,180 @@ deleted item. } ``` +#### Compact Database + +Compacts and defragments the the sqlite database using the `VACUUM` command. +This API cannot be requested when Klipper is printing. + +HTTP request: +```http +POST /server/database/compact +``` + +JSON-RPC request: +```json +{ + "jsonrpc": "2.0", + "method": "server.database.compact", + "id": 4654 +} +``` +Returns: +An object containing the size of the database on disk before and after +the database is compacted. +```json +{ + "previous_size": 139264, + "new_size": 122880 +} +``` + +#### Backup Database + +Creates a backup of the current database. The backup will be +created in the `/backup/database/`. + +This API cannot be requested when Klipper is printing. + +HTTP request: +```http +POST /server/database/backup +Content-Type: application/json + +{ + "filename": "sql-db-backup.db" +} +``` + +JSON-RPC request: +```json +{ + "jsonrpc": "2.0", + "method": "server.database.post_backup", + "params": { + "filename": "sql-db-backup.db" + }, + "id": 4654 +} +``` + +Parameters: + +- `filename`: An optional file name for the backup file. The default + is `sqldb-backup--`. + + +Returns: +An object containing the path on disk to the backup. +```json +{ + "backup_path": "/home/test/printer_data/backup/database/sql-db-backup.db" +} +``` + +#### Delete a backup + +Deletes a previously backed up database. + +HTTP request: +```http +DELETE /server/database/backup +Content-Type: application/json + +{ + "filename": "sql-db-backup.db" +} +``` + +JSON-RPC request: +```json +{ + "jsonrpc": "2.0", + "method": "server.database.delete_backup", + "params": { + "filename": "sql-db-backup.db" + }, + "id": 4654 +} +``` + +Parameters: + +- `filename`: The name of the backup file to delete. Must be a valid + filename reported in by the [database list](#list-database-info) API. + This parameter must be provided. + +Returns: +An object containing the path on disk to the backup file that was removed. +```json +{ + "backup_path": "/home/test/printer_data/backup/database/sql-db-backup.db" +} +``` + +#### Restore Database + +Restores a previously backed up sqlite database file. The backup +must be located at `/backup/database/`. The +`` must be a valid filename reported in by the +[database list](#list-database-info) API. + +This API cannot be requested when Klipper is printing. + +!!! Note + Moonraker will restart immediately after this request is processed. + +HTTP request: +```http +POST /server/database/restore +Content-Type: application/json + +{ + "filename": "sql-db-backup.db" +} +``` + +JSON-RPC request: +```json +{ + "jsonrpc": "2.0", + "method": "server.database.restore", + "params": { + "filename": "sql-db-backup.db" + }, + "id": 4654 +} +``` + +Parameters: + +- `filename`: The name of the database file to restore. Must be a valid + filename reported in by the [database list](#list-database-info) API. + This parameter must be provided. + +Returns: +An object containing a list of restored namespaces and restored tables. +```json +{ + "restored_tables": [ + "table_registry", + "namespace_store", + "authorized_users", + "job_history", + "job_totals" + ], + "restored_namespaces": [ + "database", + "fluidd", + "gcode_metadata", + "mainsail", + "moonraker", + "update_manager", + "webcams" + ] +} +``` + ### Job Queue APIs The following endpoints may be used to manage Moonraker's job queue. @@ -7123,7 +7306,7 @@ receive the following: ### Debug APIs The APIs in this section are available when Moonraker the debug argument -(`-g`) has been supplied via the command line. Some API may also depend +(`-g`) has been supplied via the command line. Some APIs may also depend on Moonraker's configuration, ie: an optional component may choose to register a debug API. @@ -7131,11 +7314,11 @@ register a debug API. Debug APIs may expose security vulnerabilities. They should only be enabled by developers on secured machines. -#### List Database Namespaces (debug) +#### List Database Info (debug) -Debug version of [List Namespaces](#list-namespaces). Return value includes -namespaces exlusively reserved for Moonraker. Only availble when Moonraker's -debug features are enabled. +Debug version of [List Database Info](#list-database-info). Returns +all namespaces, including those exlusively reserved for Moonraker. +In addition, registered SQL tables are reported. HTTP request: @@ -7152,14 +7335,42 @@ JSON-RPC request: } ``` +Returns: + +An object containing an array of namespaces, an array of tables, and +an array of backup files. +```json +{ + "namespaces": [ + "gcode_metadata", + "webcams", + "update_manager", + "announcements", + "database", + "moonraker" + ], + "backups": [ + "sqldb-backup-20240513-134542.db", + "testbackup.db", + "testbackup2.db" + ], + "tables": [ + "job_history", + "job_totals", + "namespace_store", + "table_registry", + "authorized_users" + ] +} +``` + #### Get Database Item (debug) Debug version of [Get Database Item](#get-database-item). Keys within -protected and forbidden namespaces are accessible. Only availble when -Moonraker's debug features are enabled. +protected and forbidden namespaces are accessible. !!! Warning - Moonraker's forbidden namespaces include items such as user credentials. + Moonraker's forbidden namespaces may include items such as user credentials. This endpoint should NOT be implemented in front ends directly. HTTP request: @@ -7182,8 +7393,7 @@ JSON-RPC request: #### Add Database Item (debug) Debug version of [Add Database Item](#add-database-item). Keys within -protected and forbidden namespaces may be added. Only availble when -Moonraker's debug features are enabled. +protected and forbidden namespaces may be added. !!! Warning This endpoint should be used for testing/debugging purposes only. @@ -7219,8 +7429,7 @@ JSON-RPC request: #### Delete Database Item (debug) Debug version of [Delete Database Item](#delete-database-item). Keys within -protected and forbidden namespaces may be removed. Only availble when -Moonraker's debug features are enabled. +protected and forbidden namespaces may be removed. !!! Warning This endpoint should be used for testing/debugging purposes only. @@ -7247,6 +7456,93 @@ JSON-RPC request: } ``` +#### Get Database Table + +Requests all the contents of a specified table. + +HTTP request: +```http +GET /debug/database/table?table=job_history +``` + +JSON-RPC request: +```json +{ + "jsonrpc": "2.0", + "method": "debug.database.table", + "params": { + "table": "job_history" + }, + "id": 4654 +} +``` + +Parameters: + +- `table`: The name of the table to request. This parameter must + be provided. + +Returns: + +An object with the table's name and a list of all rows contained +within the table. The `rowid` will always be included for each +row, however it may be represented by an alias. In the example +below the alias for `rowid` is `job_id`. + +```json +{ + "table_name": "job_history", + "rows": [ + { + "job_id": 1, + "user": "No User", + "filename": "active_test.gcode", + "status": "completed", + "start_time": 1690749153.2661753, + "end_time": 1690749173.076986, + "print_duration": 0.0, + "total_duration": 19.975574419135228, + "filament_used": 0.0, + "metadata": { + "size": 211, + "modified": 1635771217.0, + "uuid": "627371e0-faa5-4ced-8bb4-7017d29226fa", + "slicer": "Unknown", + "gcode_start_byte": 8, + "gcode_end_byte": 211 + }, + "auxiliary_data": [], + "instance_id": "default" + }, + { + "job_id": 2, + "user": "No User", + "filename": "active_test.gcode", + "status": "completed", + "start_time": 1701262034.9242446, + "end_time": 1701262054.7332363, + "print_duration": 0.0, + "total_duration": 19.990913168992847, + "filament_used": 0.0, + "metadata": { + "size": 211, + "modified": 1635771217.0, + "uuid": "627371e0-faa5-4ced-8bb4-7017d29226fa", + "slicer": "Unknown", + "gcode_start_byte": 8, + "gcode_end_byte": 211 + }, + "auxiliary_data": { + "spool_ids": [ + 2 + ] + }, + "instance_id": "default" + } + ] +} +``` + #### Test a notifier (debug) You can trigger a notifier manually using this endpoint.