19 Commits
Author SHA1 Message Date
Kivi Kaitaniemi f3910a6bc9 Release v0.3.4 2025-05-01 17:13:02 +03:00
Kivi Kaitaniemi bce9be5ebc Use pty for capturing and echoing cmd output 2025-05-01 17:10:16 +03:00
Kivi Kaitaniemi ec19b21244 Mention package groups in README.md 2025-05-01 16:49:06 +03:00
Kivi Kaitaniemi 47ea816d3f Update README.md & examples 2025-03-08 19:07:04 +02:00
Kivi Kaitaniemi 23620e86f5 Release v0.3.3 2025-03-08 17:30:45 +02:00
Kivi Kaitaniemi ef8423d128 Update README.md 2025-03-08 17:27:47 +02:00
Kivi Kaitaniemi db46ac339c Merge pull request #24 from kiviktnm/cache-aur-providers
Cache AUR package providers #23
2025-03-08 17:20:14 +02:00
Kivi Kaitaniemi 8e3646a5f0 Cache AUR package providers #23 2025-03-08 17:19:39 +02:00
Kivi Kaitaniemi cad1c2682c Merge pull request #22 from kiviktnm/fix-remove-chroot-virtual-pkgs
Fix removing virtual packages from chroots #7
2025-03-08 15:11:32 +02:00
Kivi Kaitaniemi 5ede2a355a Fix removing virtual packages from chroots #7 2025-03-08 15:09:39 +02:00
Kivi Kaitaniemi c6f180fb94 Merge pull request #15 from oatmealraisin/formatters
Add ruff and isort
2025-01-25 12:42:14 +02:00
Kivi Kaitaniemi 82a1fbdcb9 Merge pull request #16 from oatmealraisin/user_services
Fix bug in module user systemd unit handling
2025-01-25 12:21:03 +02:00
user e153b8c400 Add test for multiple module user services 2025-01-23 18:37:38 -05:00
user bde9d9c70d Fix bug in module user systemd unit handling 2025-01-23 18:10:36 -05:00
user f7567890da Add ruff and isort 2025-01-23 15:29:12 -05:00
Kivi Kaitaniemi 53b227859c Remind the user that systemd services won't be started/stopped 2025-01-19 11:23:06 +02:00
Kivi Kaitaniemi c091d18ee1 Prevent BlockingOIError from being raised when echonig large amounts of output 2024-07-28 16:01:46 +03:00
Kivi Kaitaniemi 7dacad6be2 Release v0.3.1 2024-07-28 11:19:28 +03:00
Kivi Kaitaniemi 5fe7df23db Show full PKGBUILD if a commit id is not found
If a git commit id is not found in a PKGBUILD repository, show the full
PKGBUILD instead of trying to show a git diff.

Fixes #5
2024-07-28 11:10:46 +03:00
10 changed files with 682 additions and 444 deletions
+32 -8
View File
@@ -1,5 +1,15 @@
# Decman
> 🎉 Decman now has a AUR package! 🎉
>
> To start using the AUR package simply add `decman` to `decman.aur_packages`. To ensure a smooth change, add decman-git to ignored_packages during the conversion.
>
> ```py
> import decman
> decman.aur_packages += ["decman"]
> decman.ignored_packages += ["decman-git"]
> ```
Decman is a declarative package & configuration manager for Arch Linux. It allows you to manage installed packages, your dotfiles, enabled systemd units, and run commands automatically. Your system is configured using python so your configuration can be very adaptive.
If you want, you can also use decman with other configuration languages. See the [example with TOML later in this README](#decman-with-other-configuration-languages).
@@ -80,6 +90,8 @@ Then import your module in your main source file.
import decman
from syncthing import Syncthing
# NOTE! Removing a enabled module from decman.module means that on_disable will not run.
# Instead disable the module.
decman.modules += [Syncthing()]
```
@@ -109,17 +121,22 @@ decman --help
Clone the decman PKGBUILD:
```sh
git clone https://github.com/kiviktnm/decman-pkgbuild.git
git clone https://aur.archlinux.org/decman.git
```
Review the PKGBUILD and install it.
```sh
cd decman-pkgbuild
cd decman
makepkg -si
```
So far I have not created an AUR package for decman, because I'm not sure if other people would find decman useful.
Remember to add decman to its own configuration.
```py
import decman
decman.aur_packages += ["decman"]
```
## What decman manages?
@@ -127,6 +144,8 @@ So far I have not created an AUR package for decman, because I'm not sure if oth
Decman can be used to install pacman packages. Decman will install all packages defined in the source and **remove** all explicitly installed packages not defined in the source. You don't need to list dependencies as those will be handeled by pacman. You can set packages to be ignored by decman, so that it won't install them nor remove them.
Please keep in mind that decman doesn't play well with package groups, since all packages part of that group will be installed explicitly. After the initial run decman will now try to remove those packages since it only knows that the group itself should be explicitly installed. Instead of package groups, use meta packages.
```py
# Include both foreign and pacman packages here.
decman.ignored_packages += ["yay", "opendoas"]
@@ -143,12 +162,13 @@ Decman can install AUR packages as well as user defined packages. Foreign packag
Here is an example of a user package. Managing user packages is somewhat cumbersome as you have to declare their versions, dependencies and make dependencies manually. However, you probably won't install many user packages anyway.
```py
# Note, decman now has a aur package, I recommend using that instead.
# Also, this example may be out of date
decman.user_packages.append(
decman.UserPackage(
pkgname="decman-git",
# Note, this example may not be up to date
provides=["decman"],
version="0.3.0",
version="0.3.4",
dependencies=["python", "python-requests", "devtools", "pacman", "systemd", "git"],
make_dependencies=[
"python-setuptools", "python-build", "python-installer", "python-wheel"
@@ -157,12 +177,15 @@ decman.user_packages.append(
))
```
Building of foreign packages happens in a chroot. This creates some overhead, but ensures clean builds. By default the chroot is created to `/tmp/decman/build`. I recommend to use a tempfs for the `/tmp/` directory to speed up builds. Also make sure that the tempfs-partition is large enough. I recommend at least 6 GB.
Building of foreign packages happens in a chroot. This creates some overhead, but ensures clean builds. By default the chroot is created to `/tmp/decman/build`. I recommend to use a tmpfs for the `/tmp/` directory to speed up builds. Also make sure that the tmpfs-partition is large enough. I recommend at least 6 GB.
Build packages are stored in a cache `/var/cache/decman`. By default decman keeps 3 most recent versions of all packages.
### Systemd units
> [!NOTE]
> Decman will only enable and disable systemd services. It will not start or stop them.
Decman can enable systemd services, system wide or for a specific user. Decman will enable all units defined in the source, and disable them when they are removed from the source. If a unit is not defined in the source, decman will not touch it.
### Files
@@ -273,10 +296,11 @@ aur_packages = ["protonvpn"]
enabled_systemd_units = ["NetworkManager.service"]
ignored_packages = ["yay"]
user_packages = [{
# Note, decman now has a aur package, I recommend using that instead.
# Also, this example may be out of date
pkgname="decman-git",
# Note, this example may not be up to date
provides=["decman"],
version="0.3.0",
version="0.3.4",
dependencies=["python", "python-requests", "devtools", "pacman", "systemd", "git"],
make_dependencies=[
"python-setuptools", "python-build", "python-installer", "python-wheel"
+14 -8
View File
@@ -4,7 +4,6 @@ from decman import Module, File, Directory, UserPackage, sh, prg
class MyModule(Module):
def __init__(self):
self.pkgs = ["rust"]
self.update_rustup = False
@@ -28,6 +27,8 @@ class MyModule(Module):
# or run a program with arguments.
prg(["usermod", "--append", "--groups", "mygroup", "kk"])
# NOTE! Removing a enabled module from decman.module means that on_disable will not run.
# Instead disable the module.
def on_disable(self):
# You can run commands as any user
sh("whoami", user="kk")
@@ -58,18 +59,21 @@ class MyModule(Module):
def files(self) -> dict[str, File]:
# Variables are substituted in text files automatically.
return {
"/usr/local/bin/say-hello":
File(content="#!/usr/bin/env bash\necho %msg%", permissions=0o755),
"/usr/local/bin/say-hello": File(
content="#!/usr/bin/env bash\necho %msg%", permissions=0o755
),
# Variables are not substituted in binary files.
"/usr/local/share/say-hello/image.png":
File(source_file="files/i-dont-exist.png", bin_file=True),
"/usr/local/share/say-hello/image.png": File(
source_file="files/i-dont-exist.png", bin_file=True
),
}
def directories(self) -> dict[str, Directory]:
# Directories are handeled the same way. Variables are substituted in text files.
return {
"/home/kk/.config/mod-app/":
Directory(source_directory="files/app-config", owner="kk")
"/home/kk/.config/mod-app/": Directory(
source_directory="files/app-config", owner="kk"
)
}
# Packages and systemd units are basically the same with modules as without modules.
@@ -79,10 +83,12 @@ class MyModule(Module):
return self.pkgs
def user_packages(self) -> list[UserPackage]:
# Note, decman now has a aur package, I recommend using that instead.
# Also, this example may be out of date
return [
UserPackage(
pkgname="decman-git",
version="0.3.0",
version="0.3.4",
provides=["decman"],
dependencies=[
"python",
+43 -26
View File
@@ -26,7 +26,7 @@ decman.packages += ["python", "python", "devtools", "git", "networkmanager"]
decman.ignored_packages += ["rustup", "yay"]
# Installing AUR packages is easy.
decman.aur_packages += ["protonvpn"]
decman.aur_packages += ["decman", "protonvpn"]
# To import GPG keys, set the GNUPGHOME environment variable.
# It can easily be done with python as well.
@@ -36,11 +36,13 @@ decman.config.makepkg_user = "kk"
# You can also install packages from anywhere, but then you must include some
# information about the package. The git_url is the url to the PKGBUILD,
# This example may not be up to date, but you should keep these up to date with the PKGBUILD.
#
# Note, decman now has a aur package, I recommend using that instead.
# Also, this example may be out of date
decman.user_packages.append(
UserPackage(
pkgname="decman-git",
version="0.3.0",
version="0.3.4",
provides=["decman"],
dependencies=[
"python",
@@ -58,7 +60,8 @@ decman.user_packages.append(
"python-wheel",
],
git_url="https://github.com/kiviktnm/decman-pkgbuild.git",
))
)
)
# Managing only packages with decman is not that interesting.
# Decman also has really powerful ways of managing config files, scripts etc.
@@ -68,23 +71,25 @@ decman.user_packages.append(
# Define file content inline.
# Default text file encoding is utf-8 but it can be changed.
decman.files["/etc/vconsole.conf"] = File(content="KEYMAP=us",
encoding="utf-8")
decman.files["/etc/vconsole.conf"] = File(content="KEYMAP=us", encoding="utf-8")
# Include file content from another file, set the file owner and permissions.
# The source_file is relative to the directory where the main decman source.py is located.
# By default, the file group is set to the group of the owner, but it can be overridden with the group argument.
decman.files["/home/kk/.bin/user-script.sh"] = File(
source_file="files/user-script.sh", owner="kk", permissions=0o744)
source_file="files/user-script.sh", owner="kk", permissions=0o744
)
# Non-text files such as images can also be managed.
decman.files["/home/kk/.background.png"] = File(
source_file="files/i-dont-actually-exist.png", bin_file=True, owner="kk")
source_file="files/i-dont-actually-exist.png", bin_file=True, owner="kk"
)
# If you need to install multiple files at once, use directories.
# All files from the source directory will be copied recursively to the target.
decman.directories["/home/kk/.config/app/"] = Directory(
source_directory="files/app-config", owner="kk")
source_directory="files/app-config", owner="kk"
)
# Decman has built in support for managing systemd units as well.
# Decman will enable services declared here, and disable services removed from here.
@@ -95,8 +100,9 @@ decman.enabled_systemd_units += ["NetworkManager.service"]
# You can manage units for users as well.
# Ensure that previous user unit declarations aren't overwritten and they are initialized.
decman.enabled_systemd_user_units[
"kk"] = decman.enabled_systemd_user_units.get("kk", [])
decman.enabled_systemd_user_units["kk"] = decman.enabled_systemd_user_units.get(
"kk", []
)
# Add user unit.
decman.enabled_systemd_user_units["kk"].append("syncthing.service")
@@ -173,7 +179,6 @@ decman.config.number_of_packages_stored_in_cache = 3
# Create a child class of the decman.config.Commands class and override methods.
# These are the defaults.
class MyCommands(decman.config.Commands):
def list_pkgs(self) -> list[str]:
return ["pacman", "-Qeq", "--color=never"]
@@ -193,8 +198,7 @@ class MyCommands(decman.config.Commands):
return ["pacman", "-D", "--asexplicit"] + pkgs
def install_deps(self, deps: list[str]) -> list[str]:
return ["pacman", "-S", "--color=always", "--needed", "--asdeps"
] + deps
return ["pacman", "-S", "--color=always", "--needed", "--asdeps"] + deps
def is_installable(self, pkg: str) -> list[str]:
return ["pacman", "-Sddp", pkg]
@@ -217,8 +221,7 @@ class MyCommands(decman.config.Commands):
def disable_user_units(self, units: list[str], user: str) -> list[str]:
return ["systemctl", "--user", "-M", f"{user}@", "disable"] + units
def compare_versions(self, installed_version: str,
new_version: str) -> list[str]:
def compare_versions(self, installed_version: str, new_version: str) -> list[str]:
return ["vercmp", installed_version, new_version]
def git_clone(self, repo: str, dest: str) -> list[str]:
@@ -230,6 +233,9 @@ class MyCommands(decman.config.Commands):
def git_get_commit_id(self) -> list[str]:
return ["git", "rev-parse", "HEAD"]
def git_log_commit_ids(self) -> list[str]:
return ["git", "log", "--format=format:%H"]
def review_file(self, file: str) -> list[str]:
return ["less", file]
@@ -238,19 +244,31 @@ class MyCommands(decman.config.Commands):
def install_chroot_packages(self, chroot_dir: str, packages: list[str]):
return [
"arch-nspawn", chroot_dir, "pacman", "-S", "--needed",
"--noconfirm"
"arch-nspawn",
chroot_dir,
"pacman",
"-S",
"--needed",
"--noconfirm",
] + packages
def resolve_real_name(self, chroot_dir: str, pkg: str) -> list[str]:
return [
"arch-nspawn",
chroot_dir,
"pacman",
"-Sddp",
"--print-format=%n",
pkg,
]
def remove_chroot_packages(self, chroot_dir: str, packages: list[str]):
return ["arch-nspawn", chroot_dir, "pacman", "-Rsu", "--noconfirm"
] + packages
return ["arch-nspawn", chroot_dir, "pacman", "-Rsu", "--noconfirm"] + packages
def make_chroot_pkg(self, chroot_wd_dir: str, user: str,
pkgfiles_to_install: list[str]) -> list[str]:
makechrootpkg_cmd = [
"makechrootpkg", "-c", "-r", chroot_wd_dir, "-U", user
]
def make_chroot_pkg(
self, chroot_wd_dir: str, user: str, pkgfiles_to_install: list[str]
) -> list[str]:
makechrootpkg_cmd = ["makechrootpkg", "-c", "-r", chroot_wd_dir, "-U", user]
for pkgfile in pkgfiles_to_install:
makechrootpkg_cmd += ["-I", pkgfile]
@@ -267,7 +285,6 @@ decman.config.commands = MyCommands()
class PikaurWrapperCommands(decman.config.Commands):
def list_pkgs(self) -> list[str]:
return ["pikaur", "-Qeq"]
+8 -2
View File
@@ -4,15 +4,21 @@ build-backend = "setuptools.build_meta"
[project]
name = "decman"
version = "0.3.0"
version = "0.3.4"
description = "Declarative package & configuration manager for Arch Linux."
license = {file = "LICENSE"}
authors = [
{name = "Kivi Kaitaniemi"}
]
dependencies = [
"requests"
"requests",
]
[project.scripts]
decman = "decman.app:main"
[dependency-groups]
dev = [
"isort>=5.13.2",
"ruff>=0.9.3",
]
+39 -28
View File
@@ -2,12 +2,13 @@
Module for writing system configurations for decman.
"""
import typing
import pwd
import grp
import shutil
import os
import pwd
import shutil
import subprocess
import typing
import decman.error
@@ -20,9 +21,11 @@ class UserRaisedError(Exception):
super().__init__(message)
def sh(sh_cmd: str,
def sh(
sh_cmd: str,
user: typing.Optional[str] = None,
env_overrides: typing.Optional[dict[str, str]] = None):
env_overrides: typing.Optional[dict[str, str]] = None,
):
"""
Shortcut for running a shell command.
"""
@@ -49,16 +52,20 @@ def sh(sh_cmd: str,
f"Running user defined shell command failed because the user {user} doesn't exist."
) from e
with subprocess.Popen(sh_cmd, shell=True, group=gid, user=uid,
env=env) as process:
with subprocess.Popen(
sh_cmd, shell=True, group=gid, user=uid, env=env
) as process:
if process.wait() != 0:
raise decman.error.UserFacingError(
f"Running user shell command '{sh_cmd}' as {user} failed.")
f"Running user shell command '{sh_cmd}' as {user} failed."
)
def prg(command: list[str],
def prg(
command: list[str],
user: typing.Optional[str] = None,
env_overrides: typing.Optional[dict[str, str]] = None):
env_overrides: typing.Optional[dict[str, str]] = None,
):
"""
Shortcut for running a program.
"""
@@ -74,7 +81,8 @@ def prg(command: list[str],
subprocess.run(command, check=True, env=env)
except subprocess.CalledProcessError as e:
raise decman.error.UserFacingError(
f"Running user defined program '{command}' failed.") from e
f"Running user defined program '{command}' failed."
) from e
else:
try:
uid = pwd.getpwnam(user).pw_uid
@@ -84,11 +92,11 @@ def prg(command: list[str],
f"Running user defined program failed because the user {user} doesn't exist."
) from e
with subprocess.Popen(command, group=gid, user=uid,
env=env) as process:
with subprocess.Popen(command, group=gid, user=uid, env=env) as process:
if process.wait() != 0:
raise decman.error.UserFacingError(
f"Running user program '{command}' as {user} failed.")
f"Running user program '{command}' as {user} failed."
)
class File:
@@ -127,9 +135,7 @@ class File:
if group is not None:
self.gid = grp.getgrnam(group).gr_gid
def copy_to(self,
target: str,
variables: typing.Optional[dict[str, str]] = None):
def copy_to(self, target: str, variables: typing.Optional[dict[str, str]] = None):
"""
Copies the contents of this file to the target file.
"""
@@ -138,8 +144,9 @@ class File:
target_directory = os.path.dirname(target)
def create_missing_dirs(dirct: str, uid: typing.Optional[int],
gid: typing.Optional[int]):
def create_missing_dirs(
dirct: str, uid: typing.Optional[int], gid: typing.Optional[int]
):
if not os.path.isdir(dirct):
parent_dir = os.path.dirname(dirct)
if not os.path.isdir(parent_dir):
@@ -161,8 +168,7 @@ class File:
os.chmod(target, self.permissions)
def _write_content(self, target: str, variables: dict[str, str]):
if self.source_file is not None and (self.bin_file
or len(variables) == 0):
if self.source_file is not None and (self.bin_file or len(variables) == 0):
shutil.copy(self.source_file, target)
elif self.bin_file and self.content is not None:
with open(target, "wb") as file:
@@ -177,7 +183,9 @@ class File:
with open(target, "wt", encoding=self.encoding) as file:
file.write(content)
else:
assert self.content is not None, "Content should be set since source_file was not set."
assert self.content is not None, (
"Content should be set since source_file was not set."
)
content = self.content
for var, value in variables.items():
content = content.replace(var, value)
@@ -217,10 +225,12 @@ class Directory:
if group is not None:
self.gid = grp.getgrnam(group).gr_gid
def copy_to(self,
def copy_to(
self,
target_directory: str,
variables: typing.Optional[dict[str, str]] = None,
only_print: bool = False) -> list[str]:
only_print: bool = False,
) -> list[str]:
"""
Copies the files in this directory to the target directory.
@@ -233,14 +243,15 @@ class Directory:
for src_dir, _, src_files in os.walk("."):
for src_file in src_files:
src_path = os.path.join(src_dir, src_file)
file = File(source_file=src_path,
file = File(
source_file=src_path,
bin_file=self.bin_files,
encoding=self.encoding,
owner=self.owner,
group=self.group,
permissions=self.permissions)
target = os.path.normpath(
os.path.join(target_directory, src_path))
permissions=self.permissions,
)
target = os.path.normpath(os.path.join(target_directory, src_path))
created.append(target)
if not only_print:
+71 -43
View File
@@ -8,9 +8,9 @@ import sys
import traceback
import decman
import decman.config as conf
import decman.error as err
import decman.lib as l
import decman.config as conf
from decman.lib import fpm
@@ -23,53 +23,62 @@ def main():
parser = argparse.ArgumentParser(
prog="decman",
description=
"Declarative package & configuration manager for Arch Linux",
epilog="See more help at: https://github.com/kiviktnm/decman")
description="Declarative package & configuration manager for Arch Linux",
epilog="See more help at: https://github.com/kiviktnm/decman",
)
parser.add_argument("--source",
action="store",
help="python file containing configuration")
parser.add_argument(
"--source", action="store", help="python file containing configuration"
)
parser.add_argument(
"--print",
"--dry-run",
action="store_true",
default=False,
help="print what would happen as a result of running decman")
parser.add_argument("--debug",
action="store_true",
default=False,
help="show debug output")
help="print what would happen as a result of running decman",
)
parser.add_argument(
"--debug", action="store_true", default=False, help="show debug output"
)
parser.add_argument(
"--no-packages",
action="store_true",
default=False,
help="don't upgrade any packages (including foreign packages)")
parser.add_argument("--no-foreign-packages",
help="don't upgrade any packages (including foreign packages)",
)
parser.add_argument(
"--no-foreign-packages",
action="store_true",
default=False,
help="don't upgrade foreign packages")
parser.add_argument("--no-files",
help="don't upgrade foreign packages",
)
parser.add_argument(
"--no-files", action="store_true", default=False, help="don't install any files"
)
parser.add_argument(
"--no-systemd-units",
action="store_true",
default=False,
help="don't install any files")
parser.add_argument("--no-systemd-units",
help="don't enable/disable systemd units",
)
parser.add_argument(
"--no-commands",
action="store_true",
default=False,
help="don't enable/disable systemd units")
parser.add_argument("--no-commands",
help="don't run user specified commands",
)
parser.add_argument(
"--upgrade-devel",
action="store_true",
default=False,
help="don't run user specified commands")
parser.add_argument("--upgrade-devel",
action="store_true",
default=False,
help="upgrade devel packages")
help="upgrade devel packages",
)
parser.add_argument(
"--force-build",
action="store_true",
default=False,
help="force building of packages that are already cached")
help="force building of packages that are already cached",
)
args = parser.parse_args()
@@ -105,8 +114,7 @@ def main():
l.print_debug(line)
errored = True
except decman.UserRaisedError as user_error:
l.print_error(
f"Error encountered while running the source: {user_error}")
l.print_error(f"Error encountered while running the source: {user_error}")
errored = True
# Save even when an error has occurred, since this avoids repeating steps like building pkgs.
@@ -158,13 +166,23 @@ def _set_up(store: l.Store, args):
content = file.read()
except OSError as e:
raise err.UserFacingError(
f"Failed to read source file '{store.source_file}'.") from e
f"Failed to read source file '{store.source_file}'."
) from e
os.chdir(source_dir)
sys.path.append(".")
exec(content)
return args.print, not args.no_packages, not args.no_foreign_packages, not args.no_files, not args.no_systemd_units, not args.no_commands, args.upgrade_devel, args.force_build
return (
args.print,
not args.no_packages,
not args.no_foreign_packages,
not args.no_files,
not args.no_systemd_units,
not args.no_commands,
args.upgrade_devel,
args.force_build,
)
class Core:
@@ -173,7 +191,16 @@ class Core:
"""
def __init__(self, store: l.Store, opts):
self.only_print, self.update_packages, self.update_foreign_packages, self.update_files, self.update_units, self.run_commands, self.upgrade_devel, self.force_build = opts
(
self.only_print,
self.update_packages,
self.update_foreign_packages,
self.update_files,
self.update_units,
self.run_commands,
self.upgrade_devel,
self.force_build,
) = opts
self.store = store
self.source = _resolve_source()
@@ -183,10 +210,10 @@ class Core:
for upkg in self.source.all_user_pkgs():
self.fpkg_search.add_user_pkg(
fpm.PackageInfo.from_user_package(upkg, self.pacman))
fpm.PackageInfo.from_user_package(upkg, self.pacman)
)
self.fpm = fpm.ForeignPackageManager(store, self.pacman,
self.fpkg_search)
self.fpm = fpm.ForeignPackageManager(store, self.pacman, self.fpkg_search)
def run(self):
"""
@@ -218,6 +245,8 @@ class Core:
def _disable_units(self):
to_disable = self.source.units_to_disable(self.store)
l.print_list("Disabling systemd units:", to_disable)
if to_disable:
l.print_info("Disabled systemd units won't be stopped automatically.")
if not self.only_print:
self.systemctl.disable_units(to_disable)
@@ -239,15 +268,14 @@ class Core:
if not self.only_print:
self.pacman.upgrade()
if conf.enable_fpm and self.update_foreign_packages:
self.fpm.upgrade(self.upgrade_devel, self.force_build,
self.source.ignored_packages)
self.fpm.upgrade(
self.upgrade_devel, self.force_build, self.source.ignored_packages
)
def _install_pkgs(self):
currently_installed = self.pacman.get_installed()
to_install_pacman = self.source.pacman_packages_to_install(
currently_installed)
to_install_fpm = self.source.foreign_packages_to_install(
currently_installed)
to_install_pacman = self.source.pacman_packages_to_install(currently_installed)
to_install_fpm = self.source.foreign_packages_to_install(currently_installed)
l.print_list("Installing pacman packages:", to_install_pacman)
@@ -266,9 +294,7 @@ class Core:
all_created = self.source.create_all_files(self.only_print)
to_remove = self.source.files_to_remove(self.store, all_created)
l.print_list("Ensured files are up to date:",
all_created,
elements_per_line=1)
l.print_list("Ensured files are up to date:", all_created, elements_per_line=1)
l.print_list("Removing files:", to_remove, elements_per_line=1)
if self.only_print:
@@ -286,6 +312,8 @@ class Core:
def _enable_units(self):
to_enable = self.source.units_to_enable(self.store)
l.print_list("Enabling systemd units:", to_enable)
if to_enable:
l.print_info("Enabled systemd units won't be started automatically.")
if not self.only_print:
self.systemctl.enable_units(to_enable)
+32 -13
View File
@@ -64,8 +64,7 @@ class Commands:
Running this command installs the given packages from pacman repositories.
The packages are installed as dependencies.
"""
return ["pacman", "-S", "--color=always", "--needed", "--asdeps"
] + deps
return ["pacman", "-S", "--color=always", "--needed", "--asdeps"] + deps
def is_installable(self, pkg: str) -> list[str]:
"""
@@ -110,8 +109,7 @@ class Commands:
"""
return ["systemctl", "--user", "-M", f"{user}@", "disable"] + units
def compare_versions(self, installed_version: str,
new_version: str) -> list[str]:
def compare_versions(self, installed_version: str, new_version: str) -> list[str]:
"""
Running this command outputs -1 when the installed version is older than the new version.
"""
@@ -136,6 +134,12 @@ class Commands:
"""
return ["git", "rev-parse", "HEAD"]
def git_log_commit_ids(self) -> list[str]:
"""
Running this command outputs commit hashes of the repository.
"""
return ["git", "log", "--format=format:%H"]
def review_file(self, file: str) -> list[str]:
"""
Running this command outputs a file for the user to see.
@@ -154,27 +158,42 @@ class Commands:
Running this command installs the given packages to the given chroot.
"""
return [
"arch-nspawn", chroot_dir, "pacman", "-S", "--needed",
"--noconfirm"
"arch-nspawn",
chroot_dir,
"pacman",
"-S",
"--needed",
"--noconfirm",
] + packages
def resolve_real_name(self, chroot_dir: str, pkg: str) -> list[str]:
"""
This command prints a real name of a package. For example, it prints the package which provides a virtual package.
"""
return [
"arch-nspawn",
chroot_dir,
"pacman",
"-Sddp",
"--print-format=%n",
pkg,
]
def remove_chroot_packages(self, chroot_dir: str, packages: list[str]):
"""
Running this command removes the given packages from the given chroot.
"""
return ["arch-nspawn", chroot_dir, "pacman", "-Rsu", "--noconfirm"
] + packages
return ["arch-nspawn", chroot_dir, "pacman", "-Rsu", "--noconfirm"] + packages
def make_chroot_pkg(self, chroot_wd_dir: str, user: str,
pkgfiles_to_install: list[str]) -> list[str]:
def make_chroot_pkg(
self, chroot_wd_dir: str, user: str, pkgfiles_to_install: list[str]
) -> list[str]:
"""
Running this command creates a package file using the given chroot.
The package is created as the user and the pkg_files_to_install are installed
in the chroot before the package is created.
"""
makechrootpkg_cmd = [
"makechrootpkg", "-c", "-r", chroot_wd_dir, "-U", user
]
makechrootpkg_cmd = ["makechrootpkg", "-c", "-r", chroot_wd_dir, "-U", user]
for pkgfile in pkgfiles_to_install:
makechrootpkg_cmd += ["-I", pkgfile]
+121 -119
View File
@@ -2,17 +2,19 @@
Library module for decman.
"""
import threading
import sys
import shutil
import subprocess
import json
import os
import typing
import pty
import shutil
import subprocess
import sys
import threading
import time
import typing
import decman
import decman.config as conf
import decman.error as err
import decman
_DECMAN_MSG_TAG = "[\033[1;35mDECMAN\033[m]"
_RED_PREFIX = "\033[91m"
@@ -60,12 +62,14 @@ def print_summary(msg: str):
print(f"{_DECMAN_MSG_TAG} {_CYAN_PREFIX}SUMMARY{_RESET_SUFFIX}: {msg}")
def print_list(msg: str,
def print_list(
msg: str,
l: list[str],
elements_per_line: typing.Optional[int] = None,
max_line_width: typing.Optional[int] = None,
limit_to_term_size: bool = True,
level: int = SUMMARY):
level: int = SUMMARY,
):
"""
Prints a summary message to the user along with a list of elements.
@@ -89,8 +93,11 @@ def print_list(msg: str,
max_line_width = 2**32 # Big enough to basically be unlimited
if limit_to_term_size:
max_line_width = shutil.get_terminal_size().columns - len(
_SPACING) - len(_CONTINUATION_PREFIX)
max_line_width = (
shutil.get_terminal_size().columns
- len(_SPACING)
- len(_CONTINUATION_PREFIX)
)
lines = [f"{l.pop(0)}"]
index = 0
@@ -131,10 +138,9 @@ def print_debug(msg: str):
print(f"{_DECMAN_MSG_TAG} {_GRAY_PREFIX}DEBUG{_RESET_SUFFIX}: {msg}")
def prompt_number(msg: str,
min_num: int,
max_num: int,
default: typing.Optional[int] = None) -> int:
def prompt_number(
msg: str, min_num: int, max_num: int, default: typing.Optional[int] = None
) -> int:
"""
Prompts the user for a integer.
"""
@@ -263,11 +269,12 @@ class Store:
if latest_path is None:
return None
assert latest_version is not None, "If latest_path is set, then latest_version is set."
assert latest_version is not None, (
"If latest_path is set, then latest_version is set."
)
return (latest_version, latest_path)
def add_package_to_cache(self, package: str, version: str,
path_to_built_pkg: str):
def add_package_to_cache(self, package: str, version: str, path_to_built_pkg: str):
"""
Adds a built package to the package file cache. Tries to remove excess cached packages.
"""
@@ -332,14 +339,13 @@ class Store:
d = {
"source_file": self.source_file,
"allow_running_source_without_prompt":
self.allow_running_source_without_prompt,
"allow_running_source_without_prompt": self.allow_running_source_without_prompt,
"enabled_systemd_units": self.enabled_systemd_units,
"enabled_user_systemd_units": self._enabled_user_systemd_units,
"enabled_modules": self.enabled_modules,
"created_files": self.created_files,
"package_file_cache": self._package_file_cache,
"pkgbuild_git_commits": self.pkgbuild_latest_reviewed_commits
"pkgbuild_git_commits": self.pkgbuild_latest_reviewed_commits,
}
try:
@@ -370,7 +376,8 @@ class Store:
store.source_file = d.get("source_file", None)
store.allow_running_source_without_prompt = d.get(
"allow_running_source_without_prompt", False)
"allow_running_source_without_prompt", False
)
store.enabled_systemd_units = d.get(
"enabled_systemd_units",
[],
@@ -390,12 +397,10 @@ class Store:
return store
except json.JSONDecodeError as e:
print_error(f"{e}")
raise err.UserFacingError(
"Failed to parse decman store json.") from e
raise err.UserFacingError("Failed to parse decman store json.") from e
except OSError as e:
print_error(f"{e}")
raise err.UserFacingError(
"Failed to read saved decman store.") from e
raise err.UserFacingError("Failed to read saved decman store.") from e
class Source:
@@ -455,7 +460,8 @@ class Source:
"""
for module in self.modules:
if module.enabled and module.version != store.enabled_modules.get(
module.name, module.version):
module.name, module.version
):
module.after_version_change()
elif module.enabled and module.name not in store.enabled_modules:
module.after_version_change()
@@ -467,8 +473,10 @@ class Source:
"""
created_files = []
def install_files(files: dict[str, decman.File],
variables: typing.Optional[dict[str, str]] = None):
def install_files(
files: dict[str, decman.File],
variables: typing.Optional[dict[str, str]] = None,
):
for target, file in files.items():
created_files.append(target)
@@ -481,19 +489,24 @@ class Source:
except OSError as e:
print_error(f"{e}")
raise err.UserFacingError(
f"Failed to install file to {target}.") from e
f"Failed to install file to {target}."
) from e
def install_dirs(dirs: dict[str, decman.Directory],
variables: typing.Optional[dict[str, str]] = None):
def install_dirs(
dirs: dict[str, decman.Directory],
variables: typing.Optional[dict[str, str]] = None,
):
for target, directory in dirs.items():
try:
print_debug(f"Installing directory to {target}.")
created_files.extend(
directory.copy_to(target, variables, only_print))
directory.copy_to(target, variables, only_print)
)
except OSError as e:
print_error(f"{e}")
raise err.UserFacingError(
f"Failed to install directory to {target}.") from e
f"Failed to install directory to {target}."
) from e
install_files(self.files)
install_dirs(self.directories)
@@ -531,8 +544,7 @@ class Source:
return all_dirs
def files_to_remove(self, store: Store,
created_files: list[str]) -> list[str]:
def files_to_remove(self, store: Store, created_files: list[str]) -> list[str]:
"""
Returns all files that should be removed.
"""
@@ -587,8 +599,7 @@ class Source:
result[user] = entry
return result
def packages_to_remove(
self, currently_installed_packages: list[str]) -> list[str]:
def packages_to_remove(self, currently_installed_packages: list[str]) -> list[str]:
"""
Returns all packages that should be removed. This includes pacman, aur and user packages.
"""
@@ -601,7 +612,8 @@ class Source:
return result
def pacman_packages_to_install(
self, currently_installed_packages: list[str]) -> list[str]:
self, currently_installed_packages: list[str]
) -> list[str]:
"""
Returns all pacman packages that should be installed.
"""
@@ -614,7 +626,8 @@ class Source:
return result
def foreign_packages_to_install(
self, currently_installed_packages: list[str]) -> list[str]:
self, currently_installed_packages: list[str]
) -> list[str]:
"""
Returns all aur and user packages that should be installed.
"""
@@ -680,11 +693,13 @@ class Source:
return result
def _all_user_units(self) -> dict[str, set[str]]:
result = {}
result.update(self.systemd_user_units)
for module in self.modules:
if module.enabled:
result.update(module.systemd_user_units())
result = self.systemd_user_units
for module in [m for m in self.modules if m.enabled]:
module_user_units: dict[str, list[str]] = module.systemd_user_units()
for user in module_user_units.keys():
if user not in result:
result[user] = set()
result[user].update(module_user_units[user])
return result
@@ -702,11 +717,16 @@ class Pacman:
"""
try:
packages = subprocess.run(
packages = (
subprocess.run(
conf.commands.list_pkgs(),
check=True,
stdout=subprocess.PIPE,
).stdout.decode().strip().split('\n')
)
.stdout.decode()
.strip()
.split("\n")
)
return packages
except subprocess.CalledProcessError as error:
raise err.UserFacingError(
@@ -720,9 +740,12 @@ class Pacman:
if dep in self._installable:
return self._installable[dep]
result = subprocess.run(conf.commands.is_installable(dep),
check=False,
capture_output=True).returncode == 0
result = (
subprocess.run(
conf.commands.is_installable(dep), check=False, capture_output=True
).returncode
== 0
)
self._installable[dep] = result
return result
@@ -732,18 +755,23 @@ class Pacman:
basically AUR packages.
"""
try:
output = subprocess.run(
output = (
subprocess.run(
conf.commands.list_foreign_pkgs_versioned(),
check=True,
stdout=subprocess.PIPE).stdout.decode().strip().split('\n')
stdout=subprocess.PIPE,
)
.stdout.decode()
.strip()
.split("\n")
)
except subprocess.CalledProcessError as error:
raise err.UserFacingError(
f"Failed to get foreign packages using '{error.cmd}'. Output: {error.stdout}."
) from error
try:
return [(line.split(" ")[0], line.split(" ")[1])
for line in output]
return [(line.split(" ")[0], line.split(" ")[1]) for line in output]
except IndexError as error:
raise err.UserFacingError(
f"Failed to parse foreign packages from pacman output. Output: {output}"
@@ -757,7 +785,8 @@ class Pacman:
return
returncode, output = echo_and_capture_command(
conf.commands.install_pkgs(packages))
conf.commands.install_pkgs(packages)
)
if returncode != 0:
raise err.UserFacingError(
f"Failed to install packages using pacman. Process exited with code {returncode}."
@@ -766,9 +795,11 @@ class Pacman:
print_highlighted_pacman_messages(output)
try:
subprocess.run(conf.commands.set_as_explicitly_installed(packages),
subprocess.run(
conf.commands.set_as_explicitly_installed(packages),
check=True,
capture_output=conf.suppress_command_output)
capture_output=conf.suppress_command_output,
)
except subprocess.CalledProcessError as error:
raise err.UserFacingError(
"Failed to set packages as explicitly installed using pacman."
@@ -781,8 +812,7 @@ class Pacman:
if not deps:
return
returncode, output = echo_and_capture_command(
conf.commands.install_deps(deps))
returncode, output = echo_and_capture_command(conf.commands.install_deps(deps))
if returncode != 0:
raise err.UserFacingError(
f"Failed to install packages as dependencies using pacman. Process exited with code {returncode}."
@@ -799,7 +829,8 @@ class Pacman:
return
returncode, output = echo_and_capture_command(
conf.commands.install_files(files))
conf.commands.install_files(files)
)
if returncode != 0:
raise err.UserFacingError(
f"Failed to install package files using pacman. Process exited with code {returncode}."
@@ -812,7 +843,8 @@ class Pacman:
subprocess.run(
conf.commands.set_as_explicitly_installed(as_explicit),
check=True,
capture_output=conf.suppress_command_output)
capture_output=conf.suppress_command_output,
)
except subprocess.CalledProcessError as error:
if conf.suppress_command_output:
print_error("Output:")
@@ -840,8 +872,7 @@ class Pacman:
if not packages:
return
returncode, output = echo_and_capture_command(
conf.commands.remove(packages))
returncode, output = echo_and_capture_command(conf.commands.remove(packages))
if returncode != 0:
raise err.UserFacingError(
f"Failed to remove packages using pacman. Process exited with code {returncode}."
@@ -877,57 +908,18 @@ def echo_and_capture_command(program: list[str]) -> tuple[int, str]:
Returns a tuple containing the return code of the program as well as all output of the program.
"""
with subprocess.Popen(program,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT) as process:
os.set_blocking(process.stdout.fileno(), False)
output_thread = _OutputCapturingThread(process.stdout)
output_thread.start()
output = ""
os.set_blocking(sys.stdin.fileno(), False)
def read(fd):
nonlocal output
buffer = os.read(fd, 1024)
output += buffer.decode(encoding="utf-8")
return buffer
# Capture stdin and forward it to the process in a non-blocking manner
while process.poll() is None:
inp = sys.stdin.readline()
returncode = os.waitstatus_to_exitcode(pty.spawn(program, read))
if inp:
process.stdin.write(inp.encode())
process.stdin.flush()
time.sleep(0.1)
os.set_blocking(sys.stdin.fileno(), True)
output_thread.done = True
output_thread.join()
# Capture any output that may not have been yet captured
output = output_thread.output
missing_output = process.stdout.read()
if missing_output:
output += missing_output.decode()
return (process.returncode, output)
class _OutputCapturingThread(threading.Thread):
def __init__(self, stream):
super().__init__()
self._stream = stream
self.output = ""
self.done = False
def run(self):
while not self.done and not self._stream.closed:
output = self._stream.read()
if output:
output = output.decode()
self.output += output
print(output, end="")
time.sleep(0.1)
return (returncode, output)
class Systemd:
@@ -946,12 +938,15 @@ class Systemd:
return
try:
subprocess.run(conf.commands.enable_units(units),
subprocess.run(
conf.commands.enable_units(units),
check=True,
capture_output=conf.suppress_command_output)
capture_output=conf.suppress_command_output,
)
except subprocess.CalledProcessError as error:
raise err.UserFacingError(
f"Failed to enable systemd units: {units}") from error
f"Failed to enable systemd units: {units}"
) from error
self.state.enabled_systemd_units += units
def disable_units(self, units: list[str]):
@@ -962,12 +957,15 @@ class Systemd:
return
try:
subprocess.run(conf.commands.disable_units(units),
subprocess.run(
conf.commands.disable_units(units),
check=True,
capture_output=conf.suppress_command_output)
capture_output=conf.suppress_command_output,
)
except subprocess.CalledProcessError as error:
raise err.UserFacingError(
f"Failed to disable systemd units: {units}") from error
f"Failed to disable systemd units: {units}"
) from error
for unit in units:
try:
self.state.enabled_systemd_units.remove(unit)
@@ -982,9 +980,11 @@ class Systemd:
return
try:
subprocess.run(conf.commands.enable_user_units(units, user),
subprocess.run(
conf.commands.enable_user_units(units, user),
check=True,
capture_output=conf.suppress_command_output)
capture_output=conf.suppress_command_output,
)
except subprocess.CalledProcessError as error:
raise err.UserFacingError(
f"Failed to enable systemd units: {units} for {user}."
@@ -1001,9 +1001,11 @@ class Systemd:
return
try:
subprocess.run(conf.commands.disable_user_units(units, user),
subprocess.run(
conf.commands.disable_user_units(units, user),
check=True,
capture_output=conf.suppress_command_output)
capture_output=conf.suppress_command_output,
)
except subprocess.CalledProcessError as error:
raise err.UserFacingError(
f"Failed to disable systemd units: {units} for {user}."
+220 -141
View File
@@ -11,18 +11,18 @@ Terminology:
- all dependencies: normal dependencies and build dependencies combined
"""
import shutil
import subprocess
import os
import re
import shutil
import subprocess
import typing
import requests
import decman
import decman.config as conf
import decman.lib as l
import decman.error as err
import decman.lib as l
def strip_dependency(dep: str) -> str:
@@ -58,10 +58,18 @@ class PackageInfo:
In case of AUR packages, these are fetched from AUR RPC.
"""
def __init__(self, pkgname: str, pkgbase: str, version: str,
provides: list[str], dependencies: list[str],
make_dependencies: list[str], check_dependencies: list[str],
git_url: str, pacman: l.Pacman):
def __init__(
self,
pkgname: str,
pkgbase: str,
version: str,
provides: list[str],
dependencies: list[str],
make_dependencies: list[str],
check_dependencies: list[str],
git_url: str,
pacman: l.Pacman,
):
self.pkgname = pkgname
self.pkgbase = pkgbase
self.version = version
@@ -79,22 +87,23 @@ class PackageInfo:
if pacman.is_installable(dep):
self.pacman_dependencies.append(dep)
else:
self.foreign_dependencies_stripped.append(
strip_dependency(dep))
self.foreign_dependencies_stripped.append(strip_dependency(dep))
for make_dep in make_dependencies:
if pacman.is_installable(make_dep):
self.pacman_make_dependencies.append(make_dep)
else:
self.foreign_make_dependencies_stripped.append(
strip_dependency(make_dep))
strip_dependency(make_dep)
)
for check_dep in check_dependencies:
if pacman.is_installable(check_dep):
self.pacman_check_dependencies.append(check_dep)
else:
self.foreign_check_dependencies_stripped.append(
strip_dependency(check_dep))
strip_dependency(check_dep)
)
def pkg_file_prefix(self) -> str:
"""
@@ -103,8 +112,9 @@ class PackageInfo:
return f"{self.pkgname}-{self.version}"
@staticmethod
def from_user_package(user_package: decman.UserPackage,
pacman: l.Pacman) -> "PackageInfo":
def from_user_package(
user_package: decman.UserPackage, pacman: l.Pacman
) -> "PackageInfo":
"""
Converts a UserPackage to PackageInfo
"""
@@ -132,8 +142,11 @@ class ForeignPackage:
def __eq__(self, value: object, /) -> bool:
if isinstance(value, self.__class__):
return self.name == value.name \
and self._all_recursive_foreign_deps == value._all_recursive_foreign_deps
return (
self.name == value.name
and self._all_recursive_foreign_deps
== value._all_recursive_foreign_deps
)
return False
def __hash__(self) -> int:
@@ -145,8 +158,7 @@ class ForeignPackage:
def __str__(self) -> str:
return f"{self.name}"
def add_foreign_dependency_packages(self,
package_names: typing.Iterable[str]):
def add_foreign_dependency_packages(self, package_names: typing.Iterable[str]):
"""
Adds dependencies to the package.
"""
@@ -174,8 +186,7 @@ class DepNode:
Returns True if the given package name is in the parents of this DepNode.
"""
for name, parent in self.parents.items():
if name == pkgname or parent.is_pkgname_in_parents_recursive(
pkgname):
if name == pkgname or parent.is_pkgname_in_parents_recursive(pkgname):
return True
return False
@@ -189,15 +200,15 @@ class DepGraph:
self.package_nodes: dict[str, DepNode] = {}
self._childless_node_names = set()
def add_requirement(self, child_pkgname: str,
parent_pkgname: typing.Optional[str]):
def add_requirement(self, child_pkgname: str, parent_pkgname: typing.Optional[str]):
"""
Adds a connection between two packages, creating the child package if it doesn't exist.
The parent is the package that requires the child package.
"""
child_node = self.package_nodes.get(
child_pkgname, DepNode(ForeignPackage(child_pkgname)))
child_pkgname, DepNode(ForeignPackage(child_pkgname))
)
self.package_nodes[child_pkgname] = child_node
if len(child_node.children) == 0:
@@ -212,7 +223,8 @@ class DepGraph:
raise err.UserFacingError(
f"Foreign package dependency cycle detected involving '{child_pkgname}' \
and '{parent_pkgname}'. Foreign package dependencies are also required \
during package building and therefore dependency cycles cannot be handled.")
during package building and therefore dependency cycles cannot be handled."
)
parent_node.children[child_pkgname] = child_node
child_node.parents[parent_pkgname] = parent_node
@@ -230,8 +242,7 @@ during package building and therefore dependency cycles cannot be handled.")
childless_node = self.package_nodes[childless_node_name]
for parent in childless_node.parents.values():
new_deps = childless_node.pkg.get_all_recursive_foreign_dep_pkgs(
)
new_deps = childless_node.pkg.get_all_recursive_foreign_dep_pkgs()
new_deps.add(childless_node.pkg.name)
parent.pkg.add_foreign_dependency_packages(new_deps)
del parent.children[childless_node_name]
@@ -254,6 +265,7 @@ class ExtendedPackageSearch:
self._pacman = pacman
self._package_info_cache: dict[str, PackageInfo] = {}
self._dep_provider_cache: dict[str, PackageInfo] = {}
self._known_providers_cache: dict[str, list[str]] = {}
self._user_packages: list[PackageInfo] = []
def add_user_pkg(self, user_pkg: PackageInfo):
@@ -261,6 +273,15 @@ class ExtendedPackageSearch:
Adds the given package to user packages.
"""
self._user_packages.append(user_pkg)
self._cache_pkg(user_pkg)
def _cache_pkg(self, pkg: PackageInfo):
for provided_pkg in pkg.provides:
self._known_providers_cache[provided_pkg] = self._known_providers_cache.get(
provided_pkg, []
)
self._known_providers_cache[provided_pkg].append(pkg.pkgname)
self._package_info_cache[pkg.pkgname] = pkg
def try_caching_packages(self, packages: list[str]):
"""
@@ -270,8 +291,7 @@ class ExtendedPackageSearch:
times, because then those methods don't have to make new AUR RPC requests.
"""
packages = list(
filter(lambda p: p not in self._package_info_cache, packages))
packages = list(filter(lambda p: p not in self._package_info_cache, packages))
if len(packages) == 0:
return
@@ -281,8 +301,7 @@ class ExtendedPackageSearch:
max_pkgs_per_request = 200
while packages:
to_request = map(lambda p: f"arg[]={p}",
packages[:max_pkgs_per_request])
to_request = map(lambda p: f"arg[]={p}", packages[:max_pkgs_per_request])
packages = packages[max_pkgs_per_request:]
url = f"https://aur.archlinux.org/rpc/v5/info?{'&'.join(to_request)}"
@@ -293,8 +312,7 @@ class ExtendedPackageSearch:
d = request.json()
if d["type"] == "error":
raise err.UserFacingError(
f"AUR RPC returned error: {d['error']}")
raise err.UserFacingError(f"AUR RPC returned error: {d['error']}")
for result in d["results"]:
pkgname = result["Name"]
@@ -304,9 +322,8 @@ class ExtendedPackageSearch:
for user_package in self._user_packages:
if user_package.pkgname == pkgname:
l.print_debug(
f"'{pkgname}' found in user packages.")
self._package_info_cache[pkgname] = user_package
l.print_debug(f"'{pkgname}' found in user packages.")
self._cache_pkg(user_package)
break
else: # if not in user_packages then:
info = PackageInfo(
@@ -317,10 +334,10 @@ class ExtendedPackageSearch:
make_dependencies=result.get("MakeDepends", []),
check_dependencies=result.get("CheckDepends", []),
provides=result.get("Provides", []),
git_url=
f"https://aur.archlinux.org/{result['PackageBase']}.git",
pacman=self._pacman)
self._package_info_cache[pkgname] = info
git_url=f"https://aur.archlinux.org/{result['PackageBase']}.git",
pacman=self._pacman,
)
self._cache_pkg(info)
l.print_debug("Request completed.")
except (requests.RequestException, KeyError) as e:
@@ -345,7 +362,7 @@ class ExtendedPackageSearch:
for user_package in self._user_packages:
if user_package.pkgname == package:
l.print_debug(f"'{package}' found in user packages.")
self._package_info_cache[package] = user_package
self._cache_pkg(user_package)
return user_package
url = f"https://aur.archlinux.org/rpc/v5/info/{package}"
@@ -355,8 +372,7 @@ class ExtendedPackageSearch:
d = request.json()
if d["type"] == "error":
raise err.UserFacingError(
f"AUR RPC returned error: {d['error']}")
raise err.UserFacingError(f"AUR RPC returned error: {d['error']}")
if d["resultcount"] == 0:
l.print_debug(f"'{package}' not found.")
@@ -373,11 +389,11 @@ class ExtendedPackageSearch:
make_dependencies=result.get("MakeDepends", []),
check_dependencies=result.get("CheckDepends", []),
provides=result.get("Provides", []),
git_url=
f"https://aur.archlinux.org/{result['PackageBase']}.git",
pacman=self._pacman)
git_url=f"https://aur.archlinux.org/{result['PackageBase']}.git",
pacman=self._pacman,
)
self._package_info_cache[package] = info
self._cache_pkg(info)
return info
except (requests.RequestException, KeyError) as e:
@@ -386,8 +402,7 @@ class ExtendedPackageSearch:
f"Failed to fetch package information for {package} from AUR RPC."
) from e
def find_provider(
self, stripped_dependency: str) -> typing.Optional[PackageInfo]:
def find_provider(self, stripped_dependency: str) -> typing.Optional[PackageInfo]:
"""
Finds a provider for a dependency.
@@ -410,25 +425,31 @@ class ExtendedPackageSearch:
l.print_debug("No exact name matches found. Finding providers.")
user_pkg_results = []
known_pkg_results = self._known_providers_cache.get(stripped_dependency, [])
for user_package in self._user_packages:
if stripped_dependency in user_package.provides:
user_pkg_results.append(user_package.pkgname)
if (
stripped_dependency in user_package.provides
and stripped_dependency not in known_pkg_results
):
known_pkg_results.append(user_package.pkgname)
if len(user_pkg_results) == 1:
pkg = self.get_package_info(user_pkg_results[0])
if len(known_pkg_results) == 1:
pkg = self.get_package_info(known_pkg_results[0])
assert pkg is not None
l.print_debug(
f"Single provider for '{stripped_dependency}' found in user packages: '{pkg}'."
f"Single provider for '{stripped_dependency}' found in known packages: '{pkg}'."
)
self._dep_provider_cache[stripped_dependency] = pkg
return pkg
if len(user_pkg_results) > 1:
return self._choose_provider(stripped_dependency, user_pkg_results,
"user packages")
if len(known_pkg_results) > 1:
return self._choose_provider(
stripped_dependency, known_pkg_results, "user packages"
)
url = f"https://aur.archlinux.org/rpc/v5/search/{stripped_dependency}?by=provides"
url = (
f"https://aur.archlinux.org/rpc/v5/search/{stripped_dependency}?by=provides"
)
l.print_debug(
f"Requesting providers for '{stripped_dependency}' from AUR. URL = {url}"
)
@@ -437,8 +458,7 @@ class ExtendedPackageSearch:
d = request.json()
if d["type"] == "error":
raise err.UserFacingError(
f"AUR RPC returned error: {d['error']}")
raise err.UserFacingError(f"AUR RPC returned error: {d['error']}")
if d["resultcount"] == 0:
l.print_debug(f"'{stripped_dependency}' not found.")
@@ -461,8 +481,9 @@ class ExtendedPackageSearch:
f"Failed to search for {stripped_dependency} from AUR RPC."
) from e
def _choose_provider(self, dep: str, possible_providers: list[str],
where: str) -> typing.Optional[PackageInfo]:
def _choose_provider(
self, dep: str, possible_providers: list[str], where: str
) -> typing.Optional[PackageInfo]:
min_selection = 1
max_selection = len(possible_providers)
l.print_summary(
@@ -478,7 +499,8 @@ class ExtendedPackageSearch:
f"Select a provider [{min_selection}-{max_selection}] (default: {min_selection}): ",
min_selection,
max_selection,
default=min_selection)
default=min_selection,
)
info = self.get_package_info(possible_providers[selection - 1])
if info is not None:
@@ -541,16 +563,17 @@ class ForeignPackageManager:
Class for dealing with foreign packages.
"""
def __init__(self, store: l.Store, pacman: l.Pacman,
search: ExtendedPackageSearch):
def __init__(self, store: l.Store, pacman: l.Pacman, search: ExtendedPackageSearch):
self._store = store
self._pacman = pacman
self._search = search
def upgrade(self,
def upgrade(
self,
upgrade_devel: bool = False,
force: bool = False,
ignored_pkgs: typing.Optional[set[str]] = None):
ignored_pkgs: typing.Optional[set[str]] = None,
):
"""
Upgrades all foreign packages.
"""
@@ -561,11 +584,9 @@ class ForeignPackageManager:
all_foreign_pkgs = self._pacman.get_versioned_foreign_packages()
all_explicit_pkgs = set(self._pacman.get_installed())
l.print_debug(
f"Foreign packages to check for upgrades: {all_foreign_pkgs}")
l.print_debug(f"Foreign packages to check for upgrades: {all_foreign_pkgs}")
self._search.try_caching_packages(
list(map(lambda p: p[0], all_foreign_pkgs)))
self._search.try_caching_packages(list(map(lambda p: p[0], all_foreign_pkgs)))
as_explicit = []
as_deps = []
@@ -579,8 +600,7 @@ class ForeignPackageManager:
f"Failed to find '{pkg}' from AUR or user provided packages."
)
if self.should_upgrade_package(pkg, ver, info.version,
upgrade_devel):
if self.should_upgrade_package(pkg, ver, info.version, upgrade_devel):
if pkg in all_explicit_pkgs:
as_explicit.append(pkg)
else:
@@ -592,10 +612,12 @@ class ForeignPackageManager:
self.install(as_explicit, as_deps, force)
def install(self,
def install(
self,
foreign_pkgs: list[str],
foreign_dep_pkgs: typing.Optional[list[str]] = None,
force: bool = False):
force: bool = False,
):
"""
Installs the given foreign packages and their dependencies (both pacman/AUR).
"""
@@ -607,39 +629,44 @@ class ForeignPackageManager:
return
resolved_dependencies = self.resolve_dependencies(
foreign_pkgs, foreign_dep_pkgs)
foreign_pkgs, foreign_dep_pkgs
)
l.print_list(
"The following foreign packages will be installed explicitly:",
list(resolved_dependencies.foreign_pkgs),
level=l.SUMMARY)
level=l.SUMMARY,
)
l.print_list(
"The following foreign packages will be installed as dependencies:",
list(resolved_dependencies.foreign_dep_pkgs),
level=l.SUMMARY)
level=l.SUMMARY,
)
l.print_list(
"The following foreign packages will be built in order to install other packages. They will not be installed:",
list(resolved_dependencies.foreign_build_dep_pkgs),
level=l.SUMMARY)
level=l.SUMMARY,
)
if not l.prompt_confirm("Proceed?", default=True):
raise err.UserFacingError("Installing aborted.")
l.print_summary("Installing foreign package dependencies from pacman.")
self._pacman.install_dependencies(
list(resolved_dependencies.pacman_deps))
self._pacman.install_dependencies(list(resolved_dependencies.pacman_deps))
try:
with PackageBuilder(self._search, self._store,
resolved_dependencies) as builder:
with PackageBuilder(
self._search, self._store, resolved_dependencies
) as builder:
while resolved_dependencies.build_order:
to_build = resolved_dependencies.build_order.pop(0)
pkgbase = resolved_dependencies.get_pkgbase(to_build)
package_names = resolved_dependencies.get_pkgs_with_common_pkgbase(
to_build)
to_build
)
packages = [
resolved_dependencies.packages[pkgname]
@@ -663,16 +690,17 @@ class ForeignPackageManager:
if package_files_to_install or force:
l.print_summary("Installing foreign packages.")
self._pacman.install_files(package_files_to_install,
as_explicit=list(
resolved_dependencies.foreign_pkgs))
self._pacman.install_files(
package_files_to_install,
as_explicit=list(resolved_dependencies.foreign_pkgs),
)
else:
l.print_summary("No packages to install.")
def resolve_dependencies(
self,
foreign_pkgs: list[str],
foreign_dep_pkgs: typing.Optional[list[str]] = None
foreign_dep_pkgs: typing.Optional[list[str]] = None,
) -> ResolvedDependencies:
"""
Resolves foreign dependencies of foreign packages.
@@ -690,7 +718,7 @@ class ForeignPackageManager:
graph = DepGraph()
for name in (foreign_pkgs + foreign_dep_pkgs):
for name in foreign_pkgs + foreign_dep_pkgs:
graph.add_requirement(name, None)
seen_packages = set(foreign_pkgs + foreign_dep_pkgs)
@@ -709,8 +737,7 @@ class ForeignPackageManager:
add_to.add(dep_info.pkgname)
l.print_debug(
f"Adding dependency {dep_info.pkgname} to package {pkgname}.")
l.print_debug(f"Adding dependency {dep_info.pkgname} to package {pkgname}.")
graph.add_requirement(dep_info.pkgname, pkgname)
if dep_info.pkgname not in seen_packages:
to_process.append(dep_info.pkgname)
@@ -728,10 +755,14 @@ class ForeignPackageManager:
result.pacman_deps.update(info.pacman_dependencies)
result.add_pkgbase_info(pkgname, info.pkgbase)
build_deps = info.foreign_make_dependencies_stripped + info.foreign_check_dependencies_stripped
build_deps = (
info.foreign_make_dependencies_stripped
+ info.foreign_check_dependencies_stripped
)
self._search.try_caching_packages(
info.foreign_dependencies_stripped + build_deps)
info.foreign_dependencies_stripped + build_deps
)
for depname in info.foreign_dependencies_stripped:
process_dep(pkgname, depname, result.foreign_dep_pkgs)
@@ -758,26 +789,29 @@ class ForeignPackageManager:
return result
def should_upgrade_package(self,
def should_upgrade_package(
self,
package: str,
installed_version: str,
fetched_version: str,
upgrade_devel=False) -> bool:
upgrade_devel=False,
) -> bool:
"""
Returns True if a package should be upgraded.
"""
if upgrade_devel and is_devel(package):
l.print_debug(
f"Package {package} is devel package. It should be upgraded.")
l.print_debug(f"Package {package} is devel package. It should be upgraded.")
return True
try:
result = int(
subprocess.run(conf.commands.compare_versions(
installed_version, fetched_version),
subprocess.run(
conf.commands.compare_versions(installed_version, fetched_version),
check=True,
stdout=subprocess.PIPE).stdout.decode())
stdout=subprocess.PIPE,
).stdout.decode()
)
should_upgrade = result < 0
l.print_debug(
f"Installed version is: {installed_version}. Available version is {fetched_version}. Should upgrade: {should_upgrade}"
@@ -786,7 +820,8 @@ class ForeignPackageManager:
except (ValueError, subprocess.CalledProcessError) as error:
l.print_error(f"{error}")
raise err.UserFacingError(
"Failed to compare versions using vercmp.") from error
"Failed to compare versions using vercmp."
) from error
class PackageBuilder:
@@ -796,8 +831,12 @@ class PackageBuilder:
always_included_packages = ["base-devel", "git"]
def __init__(self, search: ExtendedPackageSearch, store: l.Store,
resolved_deps: ResolvedDependencies):
def __init__(
self,
search: ExtendedPackageSearch,
store: l.Store,
resolved_deps: ResolvedDependencies,
):
self._search = search
self._store = store
self._resolved_deps = resolved_deps
@@ -850,7 +889,8 @@ class PackageBuilder:
os.chdir(pkgbuild_dir)
git_url_info = self._search.get_package_info(
self._resolved_deps.get_some_pkgname(pkgbase))
self._resolved_deps.get_some_pkgname(pkgbase)
)
# Because all dependencies and packages should be resolved during the creation
# of ResolvedDependencies. git_url should not be None.
@@ -869,16 +909,16 @@ class PackageBuilder:
mkarchroot_env_vars = os.environ.copy()
try:
del mkarchroot_env_vars["GNUPGHOME"]
l.print_debug(
"Removed GNUPGHOME variable from mkarchroot environment.")
l.print_debug("Removed GNUPGHOME variable from mkarchroot environment.")
except KeyError:
pass
subprocess.run(conf.commands.make_chroot(self.chroot_dir,
list(self._pkgs_in_chroot)),
subprocess.run(
conf.commands.make_chroot(self.chroot_dir, list(self._pkgs_in_chroot)),
env=mkarchroot_env_vars,
check=True,
capture_output=conf.suppress_command_output)
capture_output=conf.suppress_command_output,
)
def remove_build_environment(self):
"""
@@ -886,8 +926,9 @@ class PackageBuilder:
"""
shutil.rmtree(conf.build_dir)
def build_packages(self, package_base: str, packages: list[ForeignPackage],
force: bool):
def build_packages(
self, package_base: str, packages: list[ForeignPackage], force: bool
):
"""
Builds package(s) with the same package base.
@@ -906,8 +947,7 @@ class PackageBuilder:
l.print_info(f"Building '{' '.join(package_names)}'.")
chroot_new_pacman_pkgs, chroot_pkg_files = self._get_chroot_packages(
packages)
chroot_new_pacman_pkgs, chroot_pkg_files = self._get_chroot_packages(packages)
pkgbuild_dir = self.pkgbase_dir_map[package_base]
os.chdir(pkgbuild_dir)
@@ -918,19 +958,24 @@ class PackageBuilder:
l.print_info("Installing build dependencies to chroot.")
subprocess.run(conf.commands.install_chroot_packages(
subprocess.run(
conf.commands.install_chroot_packages(
self.chroot_dir,
chroot_new_pacman_pkgs + PackageBuilder.always_included_packages),
chroot_new_pacman_pkgs + PackageBuilder.always_included_packages,
),
check=True,
capture_output=conf.suppress_command_output)
capture_output=conf.suppress_command_output,
)
l.print_info("Making package.")
subprocess.run(conf.commands.make_chroot_pkg(self.chroot_wd_dir,
conf.makepkg_user,
chroot_pkg_files),
subprocess.run(
conf.commands.make_chroot_pkg(
self.chroot_wd_dir, conf.makepkg_user, chroot_pkg_files
),
check=True,
capture_output=conf.quiet_output)
capture_output=conf.quiet_output,
)
for pkgname in package_names:
file = self._find_pkgfile(pkgname, pkgbuild_dir)
@@ -952,16 +997,25 @@ class PackageBuilder:
l.print_info("Removing build dependencies from chroot.")
# FIX: If installed packages are virtual packages, removing them wont succeed.
if len(chroot_new_pacman_pkgs) != 0:
to_remove = []
for p in chroot_new_pacman_pkgs:
if p not in self._pkgs_in_chroot:
to_remove.append(strip_dependency(p))
subprocess.run(conf.commands.remove_chroot_packages(
self.chroot_dir, to_remove),
real_pkgname = (
subprocess.run(
conf.commands.resolve_real_name(self.chroot_dir, p),
check=True,
capture_output=conf.suppress_command_output)
stdout=subprocess.PIPE,
)
.stdout.decode()
.strip()
)
to_remove.append(real_pkgname)
subprocess.run(
conf.commands.remove_chroot_packages(self.chroot_dir, to_remove),
check=True,
capture_output=conf.suppress_command_output,
)
l.print_info(f"Finished building: '{' '.join(package_names)}'.")
@@ -1030,8 +1084,10 @@ class PackageBuilder:
for foreign_pkg in chroot_foreign_pkgs:
entry = self._store.get_package(foreign_pkg)
assert entry is not None, "Build order determines that the dependencies are built \
assert entry is not None, (
"Build order determines that the dependencies are built \
before and thus are found in the cache."
)
_, file = entry
@@ -1070,32 +1126,55 @@ before and thus are found in the cache."
The user is prompted to review the PKGBUILD and confirm if the package should be built.
"""
try:
subprocess.run(conf.commands.git_clone(git_url, "."),
subprocess.run(
conf.commands.git_clone(git_url, "."),
check=True,
capture_output=conf.suppress_command_output)
capture_output=conf.suppress_command_output,
)
if l.prompt_confirm(f"Review PKGBUILD or show diff for {pkgbase}?",
default=True):
latest_reviewed_commit = self._store.pkgbuild_latest_reviewed_commits.get(
pkgbase)
if latest_reviewed_commit is None:
if l.prompt_confirm(
f"Review PKGBUILD or show diff for {pkgbase}?", default=True
):
latest_reviewed_commit = (
self._store.pkgbuild_latest_reviewed_commits.get(pkgbase)
)
git_commit_ids = (
subprocess.run(
conf.commands.git_log_commit_ids(),
check=True,
stdout=subprocess.PIPE,
)
.stdout.decode()
.strip()
.split("\n")
)
if (
latest_reviewed_commit is None
or latest_reviewed_commit not in git_commit_ids
):
for file in os.scandir("."):
if file.is_file() and not file.name.startswith("."):
subprocess.run(conf.commands.review_file(
file.path),
check=True)
subprocess.run(
conf.commands.review_file(file.path), check=True
)
else:
subprocess.run(
conf.commands.git_diff(latest_reviewed_commit),
check=True)
conf.commands.git_diff(latest_reviewed_commit), check=True
)
if l.prompt_confirm("Build this package?", default=True):
commit_id = subprocess.run(
commit_id = (
subprocess.run(
conf.commands.git_get_commit_id(),
check=True,
capture_output=True).stdout.decode().strip()
self._store.pkgbuild_latest_reviewed_commits[
pkgbase] = commit_id
capture_output=True,
)
.stdout.decode()
.strip()
)
self._store.pkgbuild_latest_reviewed_commits[pkgbase] = commit_id
else:
raise err.UserFacingError("Building aborted.")
+46
View File
@@ -264,3 +264,49 @@ class TestSource(unittest.TestCase):
self.source.packages_to_remove(self.currently_installed_packages),
["p4", "A4", "M_A1", "M_A2"],
)
class TestModuleUserServices(unittest.TestCase):
class ModuleWithUserServiceOne(Module):
def __init__(self):
super().__init__("one", True, "0")
def systemd_user_units(self) -> dict[str, list[str]]:
return {
"user": ['foo.service']
}
class ModuleWithUserServiceTwo(Module):
def __init__(self):
super().__init__("two", True, "0")
def systemd_user_units(self) -> dict[str, list[str]]:
return {
"user": ['bar.service']
}
def setUp(self) -> None:
self.source = Source(
pacman_packages=set(),
aur_packages=set(),
user_packages=set(),
ignored_packages=set(),
systemd_units=set(),
systemd_user_units={},
files={},
directories={},
modules={
self.ModuleWithUserServiceOne(),
self.ModuleWithUserServiceTwo()
},
)
self.store = Store()
def test_user_units_to_enable(self):
self.assertDictEqual(
self.source.user_units_to_enable(self.store),
{"user": ["foo.service", "bar.service"]},
)