diff --git a/README.md b/README.md index 4bbd969..18098ac 100644 --- a/README.md +++ b/README.md @@ -3,18 +3,138 @@ A wrapper around `docker compose` for our dockerized Django projects, so the same commands work in every repository. +It reads `docker compose config` to work out which service runs Django and +which one is the database, so nothing here takes a service name or a `-f`. +Everything it does, it does by running `docker`, `git` and the Postgres tools. + ## Installing -You will need Rust installed. Clone repo and run: +You will need Rust installed. Clone the repo and run: + ```bash cargo install --path . ``` -## What it makes of a project +## Django -`ahab status` prints the services it picked, the containers behind them, the -credentials it would use and what the store holds, reporting whatever it cannot -work out rather than stopping at the first thing: +Each of these runs `docker compose run --rm` against the Django service, so it +goes through the image's own entrypoint: + +```bash +ahab django run # anything, in a fresh container +ahab django bash # a shell in a fresh container +ahab django manage # manage.py +ahab django makemigrations +ahab django migrate +ahab django shell +ahab django make-command +``` + +## Postgres + +```bash +ahab postgres dump # pg_dump, custom format +ahab postgres dump -F plain # pg_dump, plain SQL +ahab postgres dump -F tar # pg_dump, tar +ahab postgres dump -F directory # pg_dump, a directory of files +ahab postgres dump -F cluster # pg_dumpall, roles and all databases +ahab postgres dump -F plain -z # any of them, gzipped +ahab postgres import # drop, create, then restore +ahab postgres psql # psql in the database container +``` + +Dumps are written `0600`, under a name of their own until they are complete and +then renamed over the target. + +Arguments reach `psql` untouched, and Docker is only asked for a terminal when +there is one to hand over, so both of these work: + +```bash +ahab postgres psql # interactive +ahab postgres psql -tAc 'select count(*) from auth_user' | wc -l +``` + +### What an import makes of a dump + +The shape is read from the file rather than from its name: + +- a `PGDMP` or `toc.dat` header — an archive, restored with `pg_restore`, as is + a directory produced by `pg_dump -Fd` +- a `pg_dumpall` header — a whole cluster, which creates its own databases and + carries role statements, so the database is dropped but not recreated and the + dump goes to `psql` connected to `postgres` +- gzip magic — decompressed on the way in, whichever of the three is inside +- anything else — SQL, fed to `psql` with `ON_ERROR_STOP` and + `--single-transaction`, so a bad file rolls back instead of half applying + +A cluster restore runs without `ON_ERROR_STOP`, so the expected complaints +about roles that already exist do not stop it. Any other error fails the +import. Importing into a cluster that has never held the database works either +way. + +Containers are stopped before an import starts, so one that fails leaves the +project stopped and says so. + +## Link + +A sandbox that mounts the repository can read anything untracked sitting in it +— a `.env`, a dump, a key. Those paths can be moved out to a store of ahab's +own and symlinked back: the host resolves the link, the sandbox does not. + +```bash +ahab link add .env secrets/ # move out, leave symlinks behind +ahab link restore .env # move it back into the repository +ahab link restore --all # everything this repo has linked +ahab link list # what the store holds for this repository +ahab link check # what a sandbox can still read +ahab link check --porcelain # ` `, for scripts +ahab link check --exit-code # exit 4 when anything is outside the store +``` + +Restoring is the inverse of adding: the file comes back to where it was and the +store keeps nothing. A second checkout linking the same path is left with a +dangling symlink, there being only ever one stored copy. + +Every stored path is in one of three states: + +``` +$ ahab link list +store: /home/you/.local/share/ahab/git.aflabs.org/urnik/afurnik + linked: .env + shadowed: config.local.py + missing: secrets/token +``` + +- `linked` — the symlink is there and points at the store +- `shadowed` — a real file took the path back, so nothing reads the stored copy +- `missing` — the symlink is gone, and nothing reads the stored copy either + +Only a linked path can be moved back, so `restore --all` says how many it left +alone. + +Adding refuses two kinds of path: + +- **a tracked path**, which git would restore anyway. Where one is a symlink + leading out of the repository, `check` reports it under the code `T>` +- **a symlink already leading out of the repository**, since moving it would + move the pointer and leave the contents where they are + +For scripts there is `--porcelain`, and `-z` for filenames that need it: +entries end with a NUL, and the two paths of a symlink leading elsewhere are +separated by one as well, the way `git status -z` reports a rename. + +The store lives under +`${XDG_DATA_HOME:-$HOME/.local/share}/ahab////`, derived +from the Git `origin` remote, or `_local/` when there is no remote to +name it after. Its directories are created `0700`. A remote or checkout name +that needed characters replacing carries a short fingerprint of the original, +so two of them cannot share a directory. + +## What ahab makes of your project + +Running `ahab status` prints the services it picked, the containers behind +them, the credentials it would use and what the store holds. It reports +whatever it cannot work out rather than stopping at the first problem: ``` $ ahab status @@ -32,10 +152,12 @@ store: /home/you/.local/share/ahab/git.aflabs.org/urnik/afurnik ## Seeing what it runs -`-v` prints each Docker command as it runs, `--dry-run` prints the ones it would -run without running them, and `-q` prints only what was asked for, dropping the -progress along the way (Compose's own progress included). All work before or -after the subcommand. +- `-v` — print each Docker command as it runs +- `--dry-run` — print the commands it would run, without running them +- `-q` — print only what was asked for, dropping the progress along the way, + Compose's own included + +All three work before or after the subcommand: ```bash ahab -v django migrate @@ -43,30 +165,50 @@ ahab --dry-run postgres import ./dump ahab -q postgres dump ./dump ``` -Nothing is written under `--dry-run`, filesystem included: `link add` and +Nothing is written under `--dry-run`, the filesystem included: `link add` and `django make-command` say what they would do and leave the tree alone. -## Exit codes +## Service detection -- `0` — It ran and had nothing to report -- `1` — It could not finish -- `2` — The arguments were wrong -- `4` — It ran fine and found something worth reporting, i.e. - `link check --exit-code` with anything outside the store +Services can be named anything; ahab matches on what they are: + +- **postgres** — the service whose image is a Postgres flavour, matching + `postg`, `timescale`, `pgvector` or `citus` +- **django** — the service that both builds an image and sets + `DJANGO_SETTINGS_MODULE`. Where more than one does, a worker beside the web + service say, the one publishing ports wins + +If nothing matches, or two candidates cannot be told apart, ahab says so and +lists the services it looked at rather than guessing. + +The role and database come from `POSTGRES_USER` and `POSTGRES_DB` on the +detected service, falling back to `db` when neither is set. Both have to be +names: a value holding an `=`, a URL or a leading dash is refused, libpq +reading such a name as a whole connection string. + +## The compose file + +Compose finds its own file and ahab does not pass `-f`, so set Docker's own +`COMPOSE_FILE` when the file is not in the working directory, including its +`base.yaml:override.yaml` form. A project's `.env` is a good place for it, +since Docker reads that too: + +``` +COMPOSE_FILE=docker/docker-compose.yaml +``` ## Shell completion -`ahab completions ` writes a completion script to stdout, for Bash, -Elvish, Fish, PowerShell or Zsh: +Completion scripts are written to stdout, for Bash, Elvish, Fish, PowerShell +or Zsh: ```bash ahab completions zsh > ~/.local/share/zsh/completions/_ahab eval "$(ahab completions zsh)" # or one line in .zshrc, never goes stale ``` -Completions are also generated during the build. They land in -`target/*/build/*/out/` by default, and `SHELL_COMPLETIONS_DIR_` -installs a single shell's file straight into place: +They are generated during the build as well, landing in +`target/*/build/*/out/`. To install a single shell's file straight into place: ```bash SHELL_COMPLETIONS_DIR_ZSH=~/.local/share/zsh/completions \ @@ -74,170 +216,20 @@ SHELL_COMPLETIONS_DIR_FISH=~/.config/fish/completions \ cargo install --path . ``` -`SHELL_COMPLETIONS_DIR` writes every shell into one directory instead. +Set `SHELL_COMPLETIONS_DIR` to write every shell into one directory instead. +Either way the path has to be absolute. -## The compose file +## Exit codes -`ahab` does not pass `-f`. Docker Compose finds the file itself, so set -Docker's own `COMPOSE_FILE` when it is not in the working directory, including -its `base.yaml:override.yaml` form. A project's `.env` is a good place for it, -since Docker reads that too: +- `0` — it ran and had nothing to report +- `1` — it could not finish +- `2` — the arguments were wrong +- `4` — it ran fine and found something worth reporting, which today means + `link check --exit-code` with anything outside the store -``` -COMPOSE_FILE=docker/docker-compose.yaml -``` +## Environment variables -## Service detection - -`ahab` finds the services it needs in `docker compose config`, so they can be -named anything: - -- **postgres**: The service whose image is a Postgres flavour, matching - `postg`, `timescale`, `pgvector` or `citus` -- **django**: The service that both builds an image and sets - `DJANGO_SETTINGS_MODULE`. Where more than one does — a worker beside the web - service, say — the one publishing ports wins - -If nothing matches, or two candidates cannot be told apart, `ahab` says so and -lists the services it looked at rather than guessing. Two services that both -build and set the settings module are ambiguous unless exactly one of them -publishes ports. - -`POSTGRES_USER` and `POSTGRES_DB` are read off the detected Postgres service, so -`dropdb`, `createdb`, `pg_restore` and `pg_dump` use the role and database the -project declares, falling back to `db` when it declares neither. Both have to -be names: `ahab` hands them to those tools as a role and a database, and libpq -reads a database name holding an `=` or a URL as a whole connection string, -which would send a dump to whatever server it names. A value that could be read -as something other than a name is refused rather than passed on. - -## Django - -```bash -ahab django run # in a fresh container, through the entrypoint -ahab django bash # shell in a fresh container -ahab django manage # manage.py -ahab django makemigrations -ahab django migrate -ahab django shell -ahab django make-command -``` - -There is no `ahab django test`: which runner a project uses is the project's -choice, and `manage.py test` exits 0 having collected nothing when the tests are -written for pytest, so a wrong guess reads as a pass. Name the runner instead: - -```bash -ahab django run pytest -x tests/ -``` - -## Postgres - -```bash -ahab postgres dump # pg_dump, custom format -ahab postgres dump -F plain # pg_dump, plain SQL -ahab postgres dump -F tar # pg_dump, tar -ahab postgres dump -F directory # pg_dump, a directory of files -ahab postgres dump -F cluster # pg_dumpall, roles and all databases -ahab postgres dump -F plain -z # any of them, gzipped -ahab postgres import # drop, create, then restore -ahab postgres psql # psql in the database container -``` - -A dump is written `0600` and under a name of its own until it is complete, then -renamed over the target: a cluster dump carries every role's password hash, and -the default `0644` would hand it to anyone else with an account on the machine. - -`psql` passes its arguments through and only asks Docker for a terminal when it -has one to hand over, so both of these work: - -```bash -ahab postgres psql # interactive -ahab postgres psql -tAc 'select count(*) from auth_user' | wc -l -``` - -The format of a dump being imported is read from the file rather than its name. -A custom format dump starts with `PGDMP` and a tar one with `toc.dat`, both of -which go to `pg_restore`, as does a directory produced by `pg_dump -Fd`. -Anything else is treated as SQL and fed to `psql` with `ON_ERROR_STOP` and -`--single-transaction`, so a bad file rolls back instead of half applying. -A whole cluster dump from `pg_dumpall` is recognised by its header and handled -differently again: it creates its own databases and carries role statements, so -the database is dropped but not recreated, the dump goes to `psql` connected to -`postgres`, and it runs without `ON_ERROR_STOP` because roles that already exist -report errors that are expected. Any error it does not expect is reported and -fails the import, since `psql` without `ON_ERROR_STOP` exits 0 having applied -only part of the dump. Gzipped dumps are decompressed on the way in, whichever -of the three they hold. A cluster that has never held the database yet is a -valid target either way, so importing into a fresh one works. - -`import` stops the project before it starts, so an import that fails leaves it -stopped and says so: `docker compose up` brings it back. - -## Link - -`ahab link` moves untracked paths out of the repository into an out-of-repo -store and symlinks them back, so a sandbox that mounts the repository sees a -dangling symlink instead of the contents, while the host resolves it as before. - -```bash -ahab link add .env secrets/ # move out, leave symlinks behind -ahab link restore .env # move it back into the repository -ahab link restore --all # everything this repo has linked -ahab link list # what the store holds for this repository -ahab link check # what a sandbox can still read -ahab link check --porcelain # ` `, for scripts -ahab link check --exit-code # exit 4 when anything is outside the store -``` - -`restore` is the inverse of `add`: the file moves out of the store and back to -where it was, and the store keeps nothing. A second checkout linking the same -path is left with a dangling symlink, since there is only ever one stored copy. - -`list` says what state each stored path is in, which `check` cannot see because -it asks Git about the repository rather than reading the store: - -``` -$ ahab link list -store: /home/you/.local/share/ahab/git.aflabs.org/urnik/afurnik - linked: .env - shadowed: config.local.py - missing: secrets/token -``` - -`missing` is a path whose symlink is gone, `shadowed` one that a real file took -back; either way the stored copy is the one nobody reads. - -`restore --all` moves back the linked ones and says how many it left, since -there is nothing to undo for a path whose symlink is gone. - -`-z` is the porcelain format for scripts that must survive any filename: entries -end with a NUL, and the two paths of a symlink leading elsewhere are separated -by one as well, the way `git status -z` reports a rename. - -`check` looks at tracked paths too, since Git tracks symlinks and one can lead -out of the repository without appearing in any untracked listing. `add` cannot -externalize a tracked path, so all `check` can do is report it, under the code -`T>`. - -A symlink already leading out of the repository is not something `add` will -take: moving the link would move the pointer and leave the contents where they -are, so the store would hold a way back out while `check`, seeing a link into -the store, called the repository clean. - -The store lives under -`${XDG_DATA_HOME:-$HOME/.local/share}/ahab////`, derived -from the Git `origin` remote, or `_local/` when there is no remote to -name it after. Its directories are created `0700`: it exists to hold what -should not be readable from the repository, and on a shared machine the default -`0755` would leave that to whoever else has an account. A store path derived -from a remote, or a checkout name, that needed characters replacing carries a -short fingerprint of the original, so two of them cannot land on one directory. - -## Configuration - -Currently `ahab` respects the following environment variables. - -- `AHAB_LINK_ROOT`: Where `ahab link` keeps its store — the same as `--store`, - which takes precedence, and defaults to `${XDG_DATA_HOME:-$HOME/.local/share}/ahab`. - `ahab status` reads it too, so it reports the store the link commands use +- `AHAB_LINK_ROOT` — where `ahab link` keeps its store, the same as `--store`, + which takes precedence. Defaults to + `${XDG_DATA_HOME:-$HOME/.local/share}/ahab`. Read by `ahab status` too, so it + reports the store the link commands actually use