41 Commits

Author SHA1 Message Date
Eric Callahan
f745c2ce17
database: remove legacy database warning
Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-10-18 14:13:25 -04:00
Eric Callahan
fb679aa056
database: minor fallback fix
Fallback if not using the default path and the database has not been created.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-10-14 19:25:47 -04:00
Eric Callahan
1e8c7f0915 database: register debug endpoints
Allow full database access to registered debug endpoints.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-10-14 15:15:41 -04:00
Eric Callahan
41ea45a486 moonraker: remove alias option
Differentiate instances based on the data path provided.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-10-14 15:15:41 -04:00
Eric Callahan
524552eb84 database: add support for the data folder
Deprecate the "database_path" option.  If the database
does not exist, however the "database_path" does, it
will be used as a fallback.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-10-14 15:15:41 -04:00
Eric Callahan
4be8e2e72f
database: fix decode exception handler
Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-04-19 10:59:38 -04:00
Eric Callahan
3a384e62dd
file_manager: add registered directory validation
Make sure that paths registered with full access do not overlap one
another, nor that they overlap sensitive folders such as the database,
Moonraker's source, or Klipper's source.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-26 10:48:48 -05:00
Eric Callahan
79f867ba83
database: remove debug option
This option was intended for Moonraker developers to live test
changes to write protected namespaces.   This can be accomplished
locally with other methods, thus this option has been removed to
prevent users from compromising sections of the database.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-25 12:38:36 -05:00
Eric Callahan
f7f487ac18 database: sort protected namespaces before storing them
This makes the order in which they are reported consistent,
as the namespaces are stored in a set.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-22 07:47:38 -05:00
Eric Callahan
888fb58c4f database: sync namespace bugfix
Use the transaction object to overwrite keys so that we don't
move the cursor.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-22 07:47:38 -05:00
Eric Callahan
5f96822c90 database: don't overwrite "get" server errors
Make sure the correct exception is raised when "get_item" fails.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-22 07:47:38 -05:00
Eric Callahan
2d28a3eeff database: fix typo in http exception
Signed-off-by: Eric Callahan <arksine.code@gmail.com>
2022-02-22 07:47:38 -05:00
Eric Callahan
96b00549cf database: validate that namespace exists in ns_contains
Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-22 07:47:38 -05:00
Eric Callahan
0bd12c8dec database: don't allow sync against an empty dict
Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-22 07:47:38 -05:00
Eric Callahan
f3cd177a02 database: move_batch() fix
Raise an exception if the key lengths for the source and destination
keys do not match.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-22 07:47:38 -05:00
Eric Callahan
11e299558d database: fix insert_batch()
If a namespace does not exist insert batch must create a new one rather
than raise an exception.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-22 07:47:38 -05:00
Eric Callahan
bb39a3c1b2 database: handle invalid reduce results
It is possible to reduce to a value that is not a dict in "insert_item()"
and "update_item()".  Raise a ServerError with a clear description of
what went wrong in these situations.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-22 07:47:38 -05:00
Eric Callahan
89e78b9a00 database: fix type reported in overwrite warning
Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-22 07:47:38 -05:00
Eric Callahan
c1104617b6 database: fix typo in sync implementation
Signed-off-by: Eric Callahan <arksine.code@gmail.com>
2022-02-22 07:47:38 -05:00
Eric Callahan
3c92f42d49 database: allow synchronous calls before the server starts
It is possible that Moonraker can be instantiated while the eventloop
is running during tests.  To avoid errors allow synchronous calls until
the server starts.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-22 07:47:38 -05:00
Eric Callahan
46f74329d3 database: reduce duplicate code
Add a _get_db() method that perform the check for existance.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-17 11:19:57 -05:00
Eric Callahan
e2a62f80d4 database: introduce record based operations
Now that all transactions run in a thread it is possible for them to
interleave.  The record based operations allow for batch operations
within a single transaction.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-17 11:19:57 -05:00
Eric Callahan
e029b6c582 database: refactor to remove duplicate code
Wrap command implementations in with a _run_command() method.  All
database commands now return a Future object.  If the command was
run before the eventloop starts its possible to immediately query
the Future's result.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-17 11:19:57 -05:00
Eric Callahan
65d1f23352 database: add a sync method
Similar to the update method, however sync will remove any
keys in the database not in the new value.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-17 11:19:57 -05:00
Eric Callahan
781e3e6250 database: combine namespace wrappers
Rather than creating two wrappers, use a single wrapper whose methods
always return a future or awaitable.  If the operation occurs during
the __init__() method of a component it will be syncrhonous, and the
result from the future can be immediately queried.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-17 11:19:57 -05:00
Eric Callahan
4d0a43cb25 database: refactor initial check for empty keys
Rather than reading out the entire database, attempt to pop off
empty keys in each database.  This should should speed up
init on platforms with slow disk I/O and prevents reading
large namespaces in memory.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-17 11:19:57 -05:00
Eric Callahan
e754d123dd database: improve ns_contains method
It isn't necessary to read out the entire namespace, just attempt
to fetch the record and reduce the key list.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-17 11:19:57 -05:00
Eric Callahan
43c1b998b0 database: add methods to clear and update a namespace
Since transactions are now threaded we need updating
and clearing a namespace needs to be performed within
one transaction.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-17 11:19:57 -05:00
Eric Callahan
e895b10ac3 database: run transactions in a thread after the eventloop starts
Database transactions can perform disk I/O tasks, which at times will
block the eventloop.  Run all transactions in threads to prevent this
from occuring.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-17 11:19:57 -05:00
Eric Callahan
ac55b95c1d database: Add unsafe shutdowns to the log rollover
Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2022-02-17 11:19:57 -05:00
Eric Callahan
d23f72d232 database: increase max namespaces to 100
Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2021-12-05 06:25:24 -05:00
Eric Callahan
eda5623b2e database: fix invalid key handling
Signed-off-by: Eric Callahan <arksine.code@gmail.com>
2021-11-28 11:10:42 -05:00
Eric Callahan
42bdb42573 database: read out all namespaces on init
If a namespace contains an invalid key reading out the namespace should correct the issue.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2021-06-13 11:04:44 -04:00
Eric Callahan
c9a958f861 database: track unsafe shutdowns
Keep track of the number of unsafe shutdowns a machine experiences.  This data is useful in diagnosing potential file system issues and issues with the datbase itself.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2021-06-13 09:15:09 -04:00
Eric Callahan
9a316fe84a database: check for invalid keys
It is invalid for a namespace to contain a top level key represented as an empty bytestring.  If this is detected, log the result and drop the invalid key.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2021-06-13 08:18:12 -04:00
Arksine
410db750c6 database: add annotations
Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2021-05-18 19:16:49 -04:00
Arksine
2c078cbc7c database: don't return forbidden namespaces in the list request
Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2021-04-20 06:34:46 -04:00
Arksine
dee9055347 database: introduce forbidden namespaces
A Moonraker component may now register a namespace as "forbidden".   API requests have no read/write access to forbidden namespaces and this cannot be overridden by the database debug option.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2021-04-18 07:25:12 -04:00
Arksine
8724d5e9ed database: add "enable_database_debug" option
This allows protected namespaces to be modified externally.  This is for dev use only, modifying protected namespaces can potentially corrupt the database.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2021-04-02 11:37:21 -04:00
Arksine
da9401dd1a database: make protected namespaces persistent
This fixes a bug where it is protected namespace can be modified in an optional component if that component is disabled.

Signed-off-by:  Eric Callahan <arksine.code@gmail.com>
2021-04-02 08:25:40 -04:00
Arksine
1dc920019c moonraker: rename the "plugins" package to "components"
Signed-off-by: Eric Callahan <arksine.code@gmail.com>
2021-03-18 08:29:31 -04:00