Start writing documentation for changes

docs/README.md

@@ -0,0 +1,485 @@
+# Decman documentation
+
+This contains the core documentation for decman. Each plugin has its own documentation.
+
+- [pacman](/docs/pacman.md)
+- [systemd](/docs/systemd.md)
+- [aur](/docs/aur.md)
+- [flatpak](/docs/flatpak.md)
+
+## Decman Store
+
+Decman stores data in the file `/var/lib/decman/store.json`. This file should not be modified manually. However, if encountering bugs with decman, manual modification may be desirable. The file is JSON so editing it should be easy enough.
+
+Using the store:
+
+```py
+# The store is always given as a parameter to a method call.
+# You don't need to create new instances.
+store["key"] = value
+
+# To ensure that a key exists (with a default value if it doesn't)
+store.ensure("my_dict", {})
+store["my_dict"]["dict_key"] = 3
+```
+
+This store is also available to plugins and modules. The following keys are used by decman:
+
+- `allow_running_source_without_prompt`
+- `source_file`
+- `enabled_modules`
+- `module_on_disable_scripts`
+- `all_files`
+
+Details about the keys used by each plugin are provided in the plugin’s documentation.
+
+## Configuring decman
+
+Decman has a small number of configuration options. They are set in your source file with python. These values are prioritized over command line options.
+
+Import the config to modify it.
+
+```py
+import decman.config
+```
+
+Enable debug messages
+
+```py
+decman.config.debug_output = False
+```
+
+Disable info messages
+
+```py
+decman.config.quiet_output = False
+```
+
+Set colored output. This setting should not be used. It should be passed as a command line argument or an environment variable instead.
+
+- Command line argument: `--no-color`
+- Environment variables:
+  - `NO_COLOR`: disables color
+  - `FORCE_COLOR`: enables color
+
+```py
+  decman.config.color_output = True
+```
+
+Directory for scripts containing Modules' on_disable code
+
+```py
+decman.config.module_on_disable_scripts_dir = "/var/lib/decman/scripts/"
+```
+
+Cache directory. Plugins like the AUR plugin use this directory as their own cache.
+
+```py
+decman.config.cache_dir = "/var/cache/decman"
+```
+
+The architecture of the computer's CPU. Currently, this is only used by the AUR plugin, but it may be useful for some other plugins.
+
+```py
+decman.config.arch = "x86_64"
+```
+
+## Files and directories
+
+Decman functions as a dotfile manager. It will install the defined files and directories to their destinations. You can set file permissions, owners as well as define variables that will be substituted in the installed files. Decman keeps track of all files it creates and when a file is no longer present in your source, it will be also removed from its destination. This helps with keeping your system clean. However, decman won't remove directories as they might contain files that weren't created by decman.
+
+Variables can only be defined for files within modules. See the module example for using file variables.
+
+Files and directories are updated during the `files` execution order step.
+
+### File
+
+Declarative file specification describing how a file should be materialized at a target path.
+
+```py
+from decman import File
+import decman
+
+# To declare a file, add it's target path and create a File object
+decman.files["/home/me/.config/nvim/init.lua"] = File(
+    source_file="./dotfiles/nvim/init.lua",
+    bin_file=False,
+    encoding="utf-8",
+    owner="me",
+    group="users",
+    permissions=0o700,
+)
+```
+
+Exactly one of `source_file` or `content` must be provided.
+
+The file can be created by copying an existing source file or by writing provided content. For text files, optional variable substitution is applied at copy time. Binary files are copied or written verbatim and never undergo substitution.
+
+Ownership, permissions, and parent directories are enforced on creation. Missing parent directories are created recursively and assigned the same ownership as the file when specified.
+
+#### Parameters:
+
+- `source_file: str`: Path to an existing file to copy from. Mutually exclusive with `content`.
+- `content: str`: In-memory file contents to write. Mutually exclusive with `source_file`.
+- `bin_file: bool`: If `True`, treat the file as binary. Disables variable substitution and writes bytes verbatim.
+- `encoding: str`: Text encoding used when reading or writing non-binary files.
+- `owner: str`: System user name to own the file and created parent directories.
+- `group: str`: System group name to own the file and created parent directories. By default the `owner`'s group is used.
+- `permissions: int`: File mode applied to the target file (e.g. `0o644`).
+
+Note: Variable substitution is a simple string replacement where each key in variables is replaced by its corresponding value. No escaping or templating semantics are applied.
+
+### Directory
+
+Declarative specification for copying the contents of a source directory into a target directory.
+
+```py
+from decman import Directory
+import decman
+
+# To declare a directory, add it's target path and create a Directory object
+decman.directories["/home/me/.config/nvim"] = File(
+    source_directory="./dotfiles/nvim",
+    bin_files=False,
+    encoding="utf-8",
+    owner="me",
+    group="users",
+    permissions=0o600,
+)
+```
+
+For text files in the directory, optional variable substitution is applied at copy time. Binary files are copied or written verbatim and never undergo substitution.
+
+Ownership, permissions, and parent directories are enforced on creation. Missing parent directories are created recursively and assigned the same ownership as the target directory when specified.
+
+#### Parameters:
+
+- `source_directory: str`: Path to the directory whose contents will be mirrored into the target.
+- `bin_files: bool`: If `True`, treat all files as binary. Disables variable substitution and copies bytes verbatim.
+- `encoding: str`: Text encoding used when reading or writing non-binary files.
+- `owner: str`: System user name to own the files and directories.
+- `group: str`: System group name to own the files and directories. By default the `owner`'s group is used.
+- `permissions: int`: File mode applied to the created or updated files (e.g. `0o644`).
+
+## Modules
+
+Modules allow grouping related functionality together.
+
+A **Module** is the primary unit for grouping related files, directories, packages, and executable logic in decman. Create your own modules by subclassing `Module`. Then override the methods documented below.
+
+Each module is uniquely identified by its `name`.
+
+### Basic Structure
+
+```python
+from decman import Module
+
+class MyModule(Module):
+    def __init__(self) -> None:
+        super().__init__("my-module")
+```
+
+### Lifecycle Hooks
+
+Modules can hook into specific phases of a decman run by overriding methods.
+
+#### Before update
+
+Executed **before** any updates are applied.
+
+```python
+def before_update(self, store):
+    ...
+```
+
+#### After update
+
+Executed **after** all updates are applied.
+
+```python
+def after_update(self, store):
+    ...
+```
+
+#### On enable
+
+Executed **once**, when the module transitions from disabled to enabled.
+
+```python
+def on_enable(self, store):
+    ...
+```
+
+#### On change
+
+Executed when the module’s **content changes** between runs. Module's content is deemed changed if:
+
+- If files or directories defined within the module have their content updated
+- A plugin marks the module as changed
+  - For example, the pacman plugin marks a module as changed if the packages defined within that module change
+
+```python
+def on_change(self, store):
+    ...
+```
+
+#### On disable
+
+Executed when the module is disabled. A module is disabled when it's removed from the modules set.
+
+**Must be declared as `@staticmethod`.**
+Validated at class creation time.
+
+```python
+@staticmethod
+def on_disable():
+    import os
+    os.remove("/some/file")
+```
+
+**Important constraints:**
+
+- Code is copied verbatim into a temporary file
+- No external variables
+- Imports must be inside the function
+- Signature must be exactly `on_disable()`
+
+### Filesystem Declarations
+
+Modules can declaratively define files and directories to be installed.
+
+#### Files
+
+Returns a mapping of target paths to `File` objects.
+
+```python
+def files(self) -> dict[str, File]:
+    return {
+        "/etc/myapp/config.conf": File(source_file="./dotfiles/config.conf"),
+    }
+```
+
+#### Directories
+
+Returns a mapping of target paths to `Directory` objects.
+
+```python
+def directories(self) -> dict[str, Directory]:
+    return {
+        "/var/lib/myapp": Directory(source_directory="./dotfiles/myapp"),
+    }
+```
+
+#### File Variable Substitution
+
+Defines variables that are substituted inside **text files** belonging to the module.
+
+```python
+def file_variables(self) -> dict[str, str]:
+    return {
+        "HOSTNAME": "example.com",
+        "PORT": "8080",
+    }
+```
+
+### Extending with plugins
+
+To include plugin functionality inside a module, create a new method and mark it with the plugin's decorator. During the execution of decman, the plugin will call the marked method and use its result. Here is an example with the pacman plugin.
+
+```py
+from decman.plugins import pacman
+
+@pacman.packages
+def pacman_packages(self) -> set[str]:
+    return {"wget", "zip"}
+```
+
+## Plugins
+
+Plugins are used to manage a single aspect of a system declaratively. Decman ships with some default plugins useful with Arch Linux but it is possible to add custom plugins.
+
+To manage the execution order of plugins set `decman.execution_order`.
+
+```py
+import decman
+decman.execution_order = [
+    "files", # not a plugin but included here
+    "pacman",
+    "aur",
+    "flatpak",
+    "systemd",
+]
+```
+
+Available plugins are found in `decman.plugins`. You can add your own plugins to that dictionary.
+
+```py
+import decman
+my_plugin = MyPlugin()
+decman.plugins["my-plugin"] = my_plugin
+
+# Remember to include your plugin in the execution order
+decman.execution_order += ["my-plugin"]
+```
+
+For conveniance, decman provides some plugins with quick access.
+
+```py
+import decman
+
+decman.pacman = decman.plugins.get("pacman")
+decman.aur = decman.plugins.get("aur")
+decman.systemd = decman.plugins.get("systemd")
+decman.flatpak = decman.plugins.get("flatpak")
+```
+
+### Creating custom plugins
+
+Create your own modules by subclassing `Plugin`. Then override the methods documented below.
+
+#### Basic Structure
+
+```python
+from decman.plugins import Plugin
+
+class MyPlugin(Plugin):
+    # Plugins should be singletons.
+    # This name should be the same as the key used in decman.plugins dict.
+    NAME = "my-plugin"
+```
+
+#### Availability check
+
+Checks if this plugin can be enabled. For example, this could check if a required command is available. Returns `True` if this plugin can be enabled.
+
+This is not useful if the plugin is directly added to `decman.plugins`. However, if using the Python package method for installing plugins, this check is used before adding the plugin automatically to `decman.plugins`.
+
+```py
+def available(self) -> bool:
+    return True
+```
+
+#### Process modules
+
+This method gathers state information from modules. If the module's state has changed since the last time running this plugin, set the module to changed. For example, the pacman plugin uses this method to find which modules have methods marked with `@pacman.packages` and calls them.
+
+This method only gathers information. It doesn't apply it.
+
+```py
+from decman import Store, Module
+
+def process_modules(self, store: Store, modules: set[Module]):
+    ...
+
+    # Toy example for setting modules as changed
+    for module in modules:
+        module._changed = True
+```
+
+#### Apply
+
+Ensures that the state managed by this plugin is present on the system.
+
+`dry_run` indicates that changes should only be printed, not yet applied.
+
+`params` is a list of strings passed as command line arguments. For example running `decman --params abc def` would cause `params = ["abc", "def"]`.
+
+This method must not raise exceptions. Instead it should return `False` to indicate a
+failure. The method should handle it's exceptions and print them to the user.
+
+```py
+from decman import Store
+
+def apply(
+    self, store: Store, dry_run: bool = False, params: list[str] | None = None
+) -> bool:
+    return True
+```
+
+### Installing plugins as Python packages
+
+You can have decman automatically detect plugins by creating a Python package with entry points in `decman.plugins`. Decman also does this with its own plugins.
+
+In `pyproject.toml` set:
+
+```toml
+[project.entry-points."decman.plugins"]
+systemd = "decman.plugins.systemd:Systemd"
+pacman = "decman.plugins.pacman:Pacman"
+aur = "decman.plugins.aur:AUR"
+```
+
+## Useful utilities
+
+Decman ships with some useful utilites that can help with modules and plugins.
+
+### Run commands
+
+Runs a command and returns its output.
+
+```py
+import decman
+decman.prg(
+    ["nvim", "--headless", "+Lazy! sync", "+qa"],
+    user = "user",
+    env_overrides = {"EXAMPLE": "value"},
+    mimic_login = True,
+    pty = True,
+    check = True,
+)
+```
+
+#### Parameters
+
+- `cmd: list[str]`: Command to execute.
+- `user: str`: User name to run the command as. If set, the command is executed after dropping privileges to this user.
+- `env_overrides dict[str, str]`: Environment variables to override or add for the command execution. These values are merged on top of the current process environment.
+- `mimic_login: bool`: If mimic_login is True, will set the following environment variables according to the given user's passwd file details. This only happens when user is set.
+  - `HOME`
+  - `USER`
+  - `LOGNAME`
+  - `SHELL`
+- `pty: bool`: If `True`, run the command inside a pseudo-terminal (PTY). This enables interactive behavior and terminal-dependent programs. If `False`, run the command without a PTY using standard subprocess execution.
+- `check`: If `True`, raise `decman.core.error.CommandFailedError` when the command exits with a non-zero status. If `False`, print a warning when encountering a non-zero exit code.
+
+### Run a command in a shell
+
+Runs a command in a shell and returns its output. Almost same as `decman.prg` but takes a string argument instead of a list and for example shell redirects are allowed.
+
+```py
+import decman
+decman.sh(
+    "echo $EXAMPLE | less",
+    user = "user",
+    env_overrides = {"EXAMPLE": "value"},
+    mimic_login = True,
+    pty = True,
+    check = True,
+)
+```
+
+#### Parameters
+
+- `sh_cmd: str`: Shell command to execute.
+- `user: str`: User name to run the command as. If set, the command is executed after dropping privileges to this user.
+- `env_overrides dict[str, str]`: Environment variables to override or add for the command execution. These values are merged on top of the current process environment.
+- `mimic_login: bool`: If mimic_login is True, will set the following environment variables according to the given user's passwd file details. This only happens when user is set.
+  - `HOME`
+  - `USER`
+  - `LOGNAME`
+  - `SHELL`
+- `pty: bool`: If `True`, run the command inside a pseudo-terminal (PTY). This enables interactive behavior and terminal-dependent programs. If `False`, run the command without a PTY using standard subprocess execution.
+- `check`: If `True`, raise `decman.core.error.CommandFailedError` when the command exits with a non-zero status. If `False`, print a warning when encountering a non-zero exit code.
+
+### Errors
+
+When your source needs to raise an error, decman provides `SourceError`s. These are the errors that should be raised when decman runs your `source.py` file.
+
+```py
+import decman
+raise decman.SourceError("boom")
+```
+
+#### Decman Core
+
+Additionally, you can import the modules used by decman. They should be relatively stable and not change too much between decman versions. The module `decman.core.output` is probably the most relevant one, as it provides methods for printing output that decman uses.

docs/aur.md

@@ -0,0 +1,29 @@
+# AUR
+
+> [!NOTE]
+> Building of foreign packages is not the primary function of decman. There are some issues that I may or may not fix.
+> If you can't build a package using decman, consider adding it to `ignored_packages` and building it yourself.
+
+Decman can install AUR packages as well as user defined packages. Foreign packages are AUR and user packages combined.
+
+Here is an example of a user package. Managing user packages is somewhat cumbersome as you have to declare their versions, dependencies and make dependencies manually. However, you probably won't install many user packages anyway.
+
+```py
+# Note, decman now has a aur package, I recommend using that instead.
+# Also, this example may be out of date
+decman.user_packages.append(
+    decman.UserPackage(
+        pkgname="decman-git",
+        provides=["decman"],
+        version="0.4.1",
+        dependencies=["python", "python-requests", "devtools", "pacman", "systemd", "git"],
+        make_dependencies=[
+            "python-setuptools", "python-build", "python-installer", "python-wheel"
+        ],
+        git_url="https://github.com/kiviktnm/decman-pkgbuild.git",
+    ))
+```
+
+Building of foreign packages happens in a chroot. This creates some overhead, but ensures clean builds. By default the chroot is created to `/tmp/decman/build`. I recommend to use a tmpfs for the `/tmp/` directory to speed up builds. Also make sure that the tmpfs-partition is large enough. I recommend at least 6 GB.
+
+Build packages are stored in a cache `/var/cache/decman`. By default decman keeps 3 most recent versions of all packages.

docs/pacman.md

@@ -0,0 +1,15 @@
+# Pacman
+
+Decman can be used to install pacman packages. Decman will install all packages defined in the source and **remove** all explicitly installed packages not defined in the source. You don't need to list dependencies as those will be handeled by pacman. You can set packages to be ignored by decman, so that it won't install them nor remove them.
+
+Please keep in mind that decman doesn't play well with package groups, since all packages part of that group will be installed explicitly. After the initial run decman will now try to remove those packages since it only knows that the group itself should be explicitly installed. Instead of package groups, use meta packages.
+
+```py
+# Include only pacman packages found in the pacman repositories in here.
+decman.pacman.ignored_packages |= {"opendoas"}
+
+# Decman will highlight text from pacman commands according to these keywords,
+# if the feauture is enabled. These are the defaults.
+decman.pacman.print_highlights = True
+decman.pacman.keywords = {"pacsave", "pacnew" }
+```