From: Thomas Walker Lynch Date: Tue, 25 Nov 2025 09:29:35 +0000 (+0000) Subject: tool_shared -> shared, bespoke -> {authored, made} X-Git-Url: https://git.reasoningtechnology.com/style/static/git-logo.png?a=commitdiff_plain;h=07703a6cc557681da7d767766af9b07220ec5c6c;p=Harmony.git tool_shared -> shared, bespoke -> {authored, made} --- diff --git a/README.md b/README.md index d340822..db8aabc 100644 --- a/README.md +++ b/README.md @@ -9,15 +9,15 @@ Pick a role, source the env, build your thing, then release. - `env_tester` — test + repro - `env_toolsmith` — shared tools + env wiring -Developers work under `developer/`, testers under `tester/`, toolsmiths wire `tool_shared/` and env scripts. +Developers work under `developer/`, testers under `tester/`, toolsmiths wire `shared/` and env scripts. ## Layout (why it exists) - `document/` — project docs (+ RT conventions in org) - `developer/` — dev code, experiments, dev-specific docs/tools - `tester/` — tests, fixtures, repro steps -- `tool_shared/` — shared tools/env for all roles -- `tool_shared/third_party/` — third-party tools -- `tool_shared/third_party/python/` — your venv lives here (not committed) +- `shared/` — shared tools/env for all roles +- `shared/third_party/` — third-party tools +- `shared/third_party/python/` — your venv lives here (not committed) - `release/` — publishable artifacts - `tmp/` — scratch (gitignored) @@ -28,7 +28,7 @@ Empty directories are tracked with `.githolder` (kept out of release archives). # choose a role (must be sourced) source ./env_developer # or env_tester / env_toolsmith -# create the Python venv under tool_shared/third_party/python/ (literally 'python' instead of 'venv' +# create the Python venv under shared/third_party/python/ (literally 'python' instead of 'venv' ./scripts/python_venv_bootstrap.sh # re-enter later diff --git a/developer/tool/makefile b/developer/tool/makefile index cb41bd4..65408ed 100644 --- a/developer/tool/makefile +++ b/developer/tool/makefile @@ -2,7 +2,7 @@ .SUFFIXES: .EXPORT_ALL_VARIABLES: -RT_INCOMMON := $(REPO_HOME)/tool_shared/third_party/RT-project-share/release +RT_INCOMMON := $(REPO_HOME)/shared/third_party/RT-project-share/release include $(RT_INCOMMON)/make/environment_RT_1.mk .PHONY: usage diff --git a/document/Harmony/00_Project_Structure.org b/document/Harmony/00_Project_Structure.org index 8c08098..dc3acaf 100644 --- a/document/Harmony/00_Project_Structure.org +++ b/document/Harmony/00_Project_Structure.org @@ -64,7 +64,7 @@ The layout below is stable across all Harmony skeleton based projects: | =developer/= | Primary workspace for developers | | =tester/= | Regression and validation workspace for testers | | =tool/= | Project-local tools | -| =tool_shared/= | Shared ecosystem tools | +| =shared/= | Shared ecosystem tools | | =document/= | Documentation (local to project) | | =release/= | Central Working Point for promoted artifacts | | =scratchpad/= | Global scratch (misc experiments) | @@ -116,7 +116,7 @@ This property is set, i.e. this director is entered, by first going to the top l By RT convention, architecture specific build artifacts are not tracked, but rather are built each time the project is cloned. Such build artifacts are placed in =developer/scratchpad/made= and if they are to be shared with the tester, the release script will release them to =release/made_untracted=. ** =tool/= - Developer specific tools. Additional tools will be found under =tool_shared=. If the project is not self contained, then yet additional tools might come from the system environment. + Developer specific tools. Additional tools will be found under =shared=. If the project is not self contained, then yet additional tools might come from the system environment. * Documents @@ -132,7 +132,7 @@ This property is set, i.e. this director is entered, by first going to the top l ** =tester/document/= Documentation for testers. -** =tool_shared/document/= +** =shared/document/= Documentation on installing the shared tools. Note if a tool has a document directory that remains with the tool. This will typically have a list of tools that need to be installed for the project, and notes to help make installs go more smoothly. @@ -142,11 +142,11 @@ This property is set, i.e. this director is entered, by first going to the top l We call the team members who administer the project and install tools the 'toolsmith'. The top level =tool/= directory holds the toolsmith's tools. -** =tool_shared/= +** =shared/= -Shared tools are available to all team members. Those that have been written specifically for the Harmony skeleton or for this project go into the =tool_shared/bespoke= directory. Note the tool =scratchpad=, try =scratchpat help=. +Shared tools are available to all team members. Those that have been written specifically for the Harmony skeleton or for this project go into the =shared/authored= directory. Note the tool =scratchpad=, try =scratchpat help=. -Tools installed from third parties go into the git ignored directory =tool_shared/third_party=. +Tools installed from third parties go into the git ignored directory =shared/third_party=. ** =developer/tool= diff --git a/document/Harmony/01_Workflow_and_Build_Contract.org b/document/Harmony/01_Workflow_and_Build_Contract.org index 6826ca6..e34bcf6 100644 --- a/document/Harmony/01_Workflow_and_Build_Contract.org +++ b/document/Harmony/01_Workflow_and_Build_Contract.org @@ -48,7 +48,7 @@ Responsibilities: 1. Setup the project directory, and keep the project in sync with the Harmony skeleton. 2. Maintain the role environments, apart from the =/tool/env= files which are owned by the respective ==. -3. Install and maintained shared tools, =tool/= and =tool_shared/=, and other tools upon request. +3. Install and maintained shared tools, =tool/= and =shared/=, and other tools upon request. 4. Address issues with project workflow. Propose updates to the Harmony skeleton. diff --git a/env_developer b/env_developer index 7067d75..adb98f6 100644 --- a/env_developer +++ b/env_developer @@ -10,16 +10,16 @@ fi # enter project environment # - source tool_shared/bespoke/env + source shared/authored/env # setup tools # - export PYTHON_HOME="$REPO_HOME/tool_shared/third_party/Python" + export PYTHON_HOME="$REPO_HOME/shared/third_party/Python" if [[ ":$PATH:" != *":$PYTHON_HOME/bin:"* ]]; then export PATH="$PYTHON_HOME/bin:$PATH" fi - RT_gcc="$REPO_HOME/tool_shared/third_party/RT_gcc/release" + RT_gcc="$REPO_HOME/shared/third_party/RT_gcc/release" if [[ ":$PATH:" != *":$RT_gcc:"* ]]; then export PATH="$RT_gcc:$PATH" fi diff --git a/env_tester b/env_tester index 5580c87..c0555c8 100644 --- a/env_tester +++ b/env_tester @@ -10,16 +10,16 @@ fi # enter project environment # - source tool_shared/bespoke/env + source shared/authored/env # setup tools # - export PYTHON_HOME="$REPO_HOME/tool_shared/third_party/Python" + export PYTHON_HOME="$REPO_HOME/shared/third_party/Python" if [[ ":$PATH:" != *":$PYTHON_HOME/bin:"* ]]; then export PATH="$PYTHON_HOME/bin:$PATH" fi - RT_gcc="$REPO_HOME/tool_shared/third_party/RT_gcc/release" + RT_gcc="$REPO_HOME/shared/third_party/RT_gcc/release" if [[ ":$PATH:" != *":$RT_gcc:"* ]]; then export PATH="$RT_gcc:$PATH" fi diff --git a/env_toolsmith b/env_toolsmith index ee09200..1d6b61f 100644 --- a/env_toolsmith +++ b/env_toolsmith @@ -10,17 +10,17 @@ fi # enter project environment # - source tool_shared/bespoke/env + source shared/authored/env # setup tools # initially these will not exist, as the toolsmith installs them # - export PYTHON_HOME="$REPO_HOME/tool_shared/third_party/Python" + export PYTHON_HOME="$REPO_HOME/shared/third_party/Python" if [[ ":$PATH:" != *":$PYTHON_HOME/bin:"* ]]; then export PATH="$PYTHON_HOME/bin:$PATH" fi - RT_gcc="$REPO_HOME/tool_shared/third_party/RT_gcc/release" + RT_gcc="$REPO_HOME/shared/third_party/RT_gcc/release" if [[ ":$PATH:" != *":$RT_gcc:"* ]]; then export PATH="$RT_gcc:$PATH" fi diff --git a/shared/authored/env b/shared/authored/env new file mode 100644 index 0000000..35e91e4 --- /dev/null +++ b/shared/authored/env @@ -0,0 +1,130 @@ +#!/usr/bin/env bash +script_afp=$(realpath "${BASH_SOURCE[0]}") +if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then + echo "$script_afp:: This script must be sourced, not executed." + exit 1 +fi + +# without this bash takes non-matching globs literally +shopt -s nullglob + +# does not presume sharing or world permissions +umask 0077 + +# -------------------------------------------------------------------------------- +# project definition + +# actual absolute director path for this script file + + script_adp(){ + dirname "$script_afp" + } + +# assume this script is located $REPO_HOME/tools_shared/authored and work backwards +# to get $REPO_HOME, etc. + + REPO_HOME=$(dirname "$(dirname "$(script_adp)")") + echo REPO_HOME "$REPO_HOME" + + PROJECT=$(basename "$REPO_HOME") + echo PROJECT "$PROJECT" + + # set the prompt decoration to the name of the project + PROMPT_DECOR=$PROJECT + + export REPO_HOME PROJECT PROMPT_DECOR + +# -------------------------------------------------------------------------------- +# Project wide Tool setup +# + +export VIRTUAL_ENV="$REPO_HOME/shared/third_party/Python" +export PYTHON_HOME="$VIRTUAL_ENV" +unset PYTHONHOME + + +# -------------------------------------------------------------------------------- +# PATH +# precedence: last defined, first discovered + + PATH="$REPO_HOME/shared/third_party/RT-project-share/release/bash:$PATH" + PATH="$REPO_HOME/shared/third_party/RT-project-share/release/amd64:$PATH" + PATH="$REPO_HOME/shared/third_party:$PATH" + PATH="$REPO_HOME/shared/customized:$PATH" + PATH="$REPO_HOME/shared/made:$PATH" + + # Remove duplicates + clean_path() { + PATH=$(echo ":$PATH" | awk -v RS=: -v ORS=: '!seen[$0]++' | sed 's/^://; s/:$//') + } + clean_path + export PATH + +# -------------------------------------------------------------------------------- +# the following functions are provided for other scripts to use. +# at the top of files that make use of these functions put the following line: +# script_afp=$(realpath "${BASH_SOURCE[0]}") +# + + ## script's filename + script_fn(){ + basename "$script_afp" + } + + ## script's dirpath relative to $REPO_HOME + script_fp(){ + realpath --relative-to="${REPO_HOME}" "$script_afp" + } + + ## script's dirpath relative to $REPO_HOME + script_dp(){ + dirname "$(script_fp)" + } + + export -f script_adp script_fn script_dp script_fp + +#-------------------------------------------------------------------------------- +# used by release scripts +# + + install_file() { + if [ "$#" -lt 3 ]; then + echo "env::install_file usage: install_file ... " + return 1 + fi + + perms="${@: -1}" # Last argument is permissions + target_dp="${@: -2:1}" # Second-to-last argument is the target directory + sources=("${@:1:$#-2}") # All other arguments are source files + + if [ ! -d "$target_dp" ]; then + echo "env::install_file no install done: target directory '$target_dp' does not exist." + return 1 + fi + + for source_fp in "${sources[@]}"; do + if [ ! -f "$source_fp" ]; then + echo "env::install_file: source file '$source_fp' does not exist." + return 1 + fi + + target_file="$target_dp/$(basename "$source_fp")" + + if ! install -m "$perms" "$source_fp" "$target_file"; then + echo "env::install_file: Failed to install $(basename "$source_fp") to $target_dp" + return 1 + else + echo "env::install_file: installed $(basename "$source_fp") to $target_dp with permissions $perms" + fi + done + } + + export -f install_file + +# -------------------------------------------------------------------------------- +# closing +# + if [[ -z "$ENV" ]]; then + export ENV=$(script_fp) + fi + diff --git a/shared/authored/git-empty-dir/CLI.py b/shared/authored/git-empty-dir/CLI.py new file mode 100755 index 0000000..2fb22e1 --- /dev/null +++ b/shared/authored/git-empty-dir/CLI.py @@ -0,0 +1,251 @@ +#!/usr/bin/env python3 +# ---------------------------------------------------------------------- +# git-empty-dir :: list/mark/clean empty directories, .gitignore aware +# ---------------------------------------------------------------------- + +import sys +import os +from pathlib import Path + +# The source_sync GitIgnore parser is inside the unpacked tool. +# We assume this directory structure: +# git-empty-dir/ +# CLI.py +# source_sync/ +# GitIgnore.py +# +# That mirrors how your harmony sync tool is structured. + +# Adjust import path so we can load source_sync.* +HERE = Path(__file__).resolve().parent +sys.path.insert(0, str(HERE)) + +from source_sync.GitIgnore import GitIgnore # type: ignore + + +# ---------------------------------------------------------------------- +# helpers +# ---------------------------------------------------------------------- + +def load_gitignore_tree(root: Path): + """ + Build a GitIgnore instance rooted at . + """ + return GitIgnore(str(root)) + +def is_empty_dir(path: Path) -> bool: + """ + A directory is empty if it contains no files or subdirectories. + (Hidden files count; .gitignored children are irrelevant because + behavior here should reflect real filesystem emptiness.) + """ + try: + for _ in path.iterdir(): + return False + return True + except PermissionError: + # treat as non-empty: safer than aborting + return False + + +def has_mark(path: Path, mark_file: str) -> bool: + return (path / mark_file).exists() + + +def sorted_dirs(root: Path): + """ + Produce a list of all directories under root, in parent-before-child order. + Sort rule: + 1. by path length + 2. then lexicographically + """ + all_dirs = [] + for p in root.rglob("*"): + if p.is_dir(): + all_dirs.append(p) + + return sorted( + all_dirs + ,key = lambda p: (len(p.parts), str(p)) + ) + + +# ---------------------------------------------------------------------- +# traversal +# ---------------------------------------------------------------------- + +def visible_dirs(root: Path, ignore_tree, mark_file: str): + """ + Yield all dirs under root, applying: + - skip .git + - apply .gitignore rules (if a dir is ignored, do not descend) + - parent-before-child ordering + """ + for d in sorted_dirs(root): + rel = d.relative_to(root) + + if rel == Path("."): + continue + + # skip .git explicitly + if d.name == ".git": + continue + + # .gitignore filtering + if ignore_tree.check(str(rel)) == "Ignore": + continue + + yield d + + +# ---------------------------------------------------------------------- +# actions +# ---------------------------------------------------------------------- + +def action_list(root, ignore_tree, mark_file, mode): + """ + mode ∈ {"empty","marked","all"} + """ + for d in visible_dirs(root, ignore_tree, mark_file): + if mode == "all": + print(d.relative_to(root)) + continue + + if mode == "marked": + if has_mark(d, mark_file): + print(d.relative_to(root)) + continue + + if mode == "empty": + if is_empty_dir(d): + print(d.relative_to(root)) + continue + + +def action_mark(root, ignore_tree, mark_file, mode): + """ + mode ∈ {"empty","all"} + """ + for d in visible_dirs(root, ignore_tree, mark_file): + if mode == "empty" and not is_empty_dir(d): + continue + try: + (d / mark_file).touch(exist_ok=True) + except Exception: + pass + + +def action_clean(root, ignore_tree, mark_file, mode): + """ + mode ∈ {"nonempty","all"} + """ + for d in visible_dirs(root, ignore_tree, mark_file): + m = d / mark_file + if not m.exists(): + continue + + if mode == "nonempty": + if is_empty_dir(d): + continue + + try: + m.unlink() + except Exception: + pass + + +# ---------------------------------------------------------------------- +# usage +# ---------------------------------------------------------------------- + +USAGE = """ +usage: + git-empty-dir (list|mark|clean) [all|marked|empty] [file-] + git-empty-dir help + git-empty-dir usage + +defaults: + mark-file = .gitkeep + ignores .git + follows .gitignore (no descent into ignored dirs) + +examples: + git-empty-dir list + git-empty-dir list marked file-.githolder + git-empty-dir mark + git-empty-dir clean all +""" + + +# ---------------------------------------------------------------------- +# CLI +# ---------------------------------------------------------------------- + +def CLI(argv): + if len(argv) == 0: + print(USAGE) + return 0 + + cmd = argv[0] + + if cmd in ("help","usage"): + print(USAGE) + return 0 + + # command + if cmd not in ("list","mark","clean"): + print(f"unknown command: {cmd}") + print(USAGE) + return 1 + + # submode + mode = None + mark_file = ".gitkeep" + + for a in argv[1:]: + if a.startswith("file-"): + mark_file = a[5:] + continue + + if a in ("all","empty","marked"): + mode = a + continue + + print(f"unknown argument: {a}") + print(USAGE) + return 1 + + # defaults + if cmd == "list": + if mode is None: + mode = "empty" + elif cmd == "mark": + if mode is None: + mode = "empty" + elif cmd == "clean": + if mode is None: + mode = "nonempty" + + root = Path(".").resolve() + ignore_tree = load_gitignore_tree(root) + + if cmd == "list": + action_list(root, ignore_tree, mark_file, mode) + + elif cmd == "mark": + if mode == "all": + action_mark(root, ignore_tree, mark_file, "all") + else: + action_mark(root, ignore_tree, mark_file, "empty") + + elif cmd == "clean": + if mode == "all": + action_clean(root, ignore_tree, mark_file, "all") + else: + action_clean(root, ignore_tree, mark_file, "nonempty") + + return 0 + + +if __name__ == "__main__": + sys.exit(CLI(sys.argv[1:])) diff --git a/shared/authored/git-empty-dir/Harmony.py b/shared/authored/git-empty-dir/Harmony.py new file mode 120000 index 0000000..112663e --- /dev/null +++ b/shared/authored/git-empty-dir/Harmony.py @@ -0,0 +1 @@ +../source_sync/Harmony.py \ No newline at end of file diff --git a/shared/authored/git-empty-dir/load_command_module.py b/shared/authored/git-empty-dir/load_command_module.py new file mode 120000 index 0000000..87b98be --- /dev/null +++ b/shared/authored/git-empty-dir/load_command_module.py @@ -0,0 +1 @@ +../source_sync/load_command_module.py \ No newline at end of file diff --git a/shared/authored/git-empty-dir/meta.py b/shared/authored/git-empty-dir/meta.py new file mode 100644 index 0000000..dee6439 --- /dev/null +++ b/shared/authored/git-empty-dir/meta.py @@ -0,0 +1,97 @@ +#!/usr/bin/env python3 +# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- + +""" +meta.py - thin wrappers around command modules + +Current responsibilities: + 1. Load the incommon 'printenv' command module (no .py extension) + using load_command_module.load_command_module(). + 2. Expose printenv() here, calling the imported printenv() work + function with default arguments (equivalent to running without + any CLI arguments). + 3. Provide a simple version printer for this meta module. + 4. Provide a small debug tag API (set/clear/has). +""" + +from __future__ import annotations + +import datetime +from load_command_module import load_command_module + + +# Load the incommon printenv module once at import time +_PRINTENV_MODULE = load_command_module("printenv") +_Z_MODULE = load_command_module("Z") + + +# Meta module version +_major = 1 +_minor = 1 +def version_print() -> None: + """ + Print the meta module version as MAJOR.MINOR. + """ + print(f"{_major}.{_minor}") + + +# Debug tag set and helpers +_debug = set([ +]) + + +def debug_set(tag: str) -> None: + """ + Add a debug tag to the meta debug set. + """ + _debug.add(tag) + + +def debug_clear(tag: str) -> None: + """ + Remove a debug tag from the meta debug set, if present. + """ + _debug.discard(tag) + + +def debug_has(tag: str) -> bool: + """ + Return True if the given debug tag is present. + """ + return tag in _debug + + +# Touch the default tag once so static checkers do not complain about +# unused helpers when imported purely for side-effects. +debug_has("Command") + + +def printenv() -> int: + """ + Call the imported printenv() work function with default arguments: + - no null termination + - no newline quoting + - no specific names (print full environment) + - prog name 'printenv' + """ + return _PRINTENV_MODULE.printenv( + False # null_terminate + ,False # quote_newlines + ,[] # names + ,"printenv" + ) + + +def z_format_mtime( + mtime: float +) -> str: + """ + Format a POSIX mtime (seconds since epoch, UTC) using the Z module. + + Uses Z.ISO8601_FORMAT and Z.make_timestamp(dt=...). + """ + dt = datetime.datetime.fromtimestamp(mtime, datetime.timezone.utc) + return _Z_MODULE.make_timestamp( + fmt=_Z_MODULE.ISO8601_FORMAT + ,dt=dt + ) diff --git a/shared/authored/git-empty-dir/source_sync b/shared/authored/git-empty-dir/source_sync new file mode 120000 index 0000000..9fd1d51 --- /dev/null +++ b/shared/authored/git-empty-dir/source_sync @@ -0,0 +1 @@ +../source_sync/ \ No newline at end of file diff --git a/shared/authored/sync/A_minus_B b/shared/authored/sync/A_minus_B new file mode 100755 index 0000000..f6f7bbb --- /dev/null +++ b/shared/authored/sync/A_minus_B @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- + +""" +A_minus_B - CLI test driver for skeleton.tree_dict_A_minus_B(A, B) + +Usage: + A_minus_B +""" + +from __future__ import annotations + +import os +import sys +from typing import Sequence + +import meta +import skeleton + + +def CLI(argv: Sequence[str] | None = None) -> int: + if argv is None: + argv = sys.argv[1:] + + prog = os.path.basename(sys.argv[0]) if sys.argv else "A_minus_B" + + if len(argv) != 2 or argv[0] in ("-h", "--help"): + print(f"Usage: {prog} ") + return 1 + + A_root = argv[0] + B_root = argv[1] + + if not os.path.isdir(A_root): + print(f"{prog}: {A_root}: not a directory") + return 2 + + if not os.path.isdir(B_root): + print(f"{prog}: {B_root}: not a directory") + return 3 + + A = skeleton.tree_dict_make(A_root, None) + B = skeleton.tree_dict_make(B_root, None) + + meta.debug_set("tree_dict_A_minus_B") + + _result = skeleton.tree_dict_A_minus_B(A, B) + + return 0 + + +if __name__ == "__main__": + raise SystemExit(CLI()) diff --git a/shared/authored/sync/CLI.py b/shared/authored/sync/CLI.py new file mode 100755 index 0000000..f7fb0b0 --- /dev/null +++ b/shared/authored/sync/CLI.py @@ -0,0 +1,282 @@ +#!/usr/bin/env python3 +# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- + +""" +CLI.py - Harmony skeleton checker + +Grammar (informal): + + check * [] + + :: | | + + :: version | help | usage + :: environment + :: structure | import | export | suspicious | addendum | all + +Commands are sorted into three sets: + 1. HELP_COMMANDS + 2. NO_OTHER_COMMANDS + 3. HAS_OTHER_COMMANDS + +At runtime, argv commands are classified into four lists: + 1. help_list + 2. no_other_list + 3. has_other_list + 4. unclassified_list + +If the meta debug set contains the tag "print_command_lists", these four lists +are printed. + +If 'environment' appears in no_other_list, the meta.printenv() helper +is invoked to print the environment. + +For commands we compare: + + A = Harmony skeleton tree_dict + B = project tree_dict (path is the last argv token when any + is present before it). +""" + +from __future__ import annotations + +import os +import sys +from typing import Sequence + +import command +import doc +import Harmony +import meta +import skeleton + +# meta.debug_set("print_command_lists") + +# Command tag sets (classification universe) +HELP_COMMANDS: set[str] = set([ + "version" + ,"help" + ,"usage" +]) + +NO_OTHER_COMMANDS: set[str] = set([ + "environment" +]) + +HAS_OTHER_COMMANDS: set[str] = set([ + "structure" + ,"import" + ,"export" + ,"suspicious" + ,"addendum" + ,"all" +]) + + +def command_type(arg: str) -> str: + """ + Classify a single command token. + + Returns: + "Help" if arg is a help command + "NoOther" if arg is a no_other command + "HasOther" if arg is a has_other command + "UnClassified" otherwise + """ + if arg in HELP_COMMANDS: + return "Help" + + if arg in NO_OTHER_COMMANDS: + return "NoOther" + + if arg in HAS_OTHER_COMMANDS: + return "HasOther" + + return "UnClassified" + + +def print_command_lists( + help_list: list[str] + ,no_other_list: list[str] + ,has_other_list: list[str] + ,unclassified_list: list[str] +) -> None: + """ + Print the four classified command lists derived from argv. + """ + print("help_list:", help_list) + print("no_other_list:", no_other_list) + print("has_other_list:", has_other_list) + print("unclassified_list:", unclassified_list) + + +def CLI(argv: Sequence[str] | None = None) -> int: + """ + CLI entrypoint. + + Responsibilities: + 1. Accept argv (or sys.argv[1:] by default). + 2. Classify arguments using command_type(), with the last argument + treated specially to avoid aliasing. + 3. Invoke behaviors implied by the commands. + 4. Return integer status code. + + Argument interpretation: + + Let argv = [a0, a1, ..., aN-1]. + + - If N == 0: + no commands; nothing to do. + + - If N >= 1: + * Classify a0..aN-2. + - If any are UnClassified -> error. + + * If any appear in a0..aN-2: + - aN-1 is treated as path (B_root), not classified. + + * If no appear in a0..aN-2: + - Classify aN-1: + - If UnClassified -> error (unknown command). + - If HasOther -> error (other path not specified). + - Else -> added to Help / NoOther lists. + """ + if argv is None: + argv = sys.argv[1:] + + # No arguments: print usage and exit with status 1. + if len(argv) == 0: + doc.print_usage() + return 1 + + # No arguments: nothing to do (could later decide to print usage). + if len(argv) == 0: + return 0 + + # Split into head (all but last) and last argument + head = argv[:-1] + last = argv[-1] + + help_list: list[str] = [] + no_other_list: list[str] = [] + has_other_list: list[str] = [] + unclassified_list: list[str] = [] + + # 1. Classify head tokens + for arg in head: + ct = command_type(arg) + + if ct == "Help": + help_list.append(arg) + elif ct == "NoOther": + no_other_list.append(arg) + elif ct == "HasOther": + has_other_list.append(arg) + else: + unclassified_list.append(arg) + + # Any unclassified in the head is an error + if len(unclassified_list) > 0: + first_bad = unclassified_list[0] + print(f"Unrecognized command: {first_bad}") + return 5 + + head_has_other = (len(has_other_list) > 0) + + B_root: str | None = None + + if head_has_other: + # 2A. Any in head -> last arg is always path. + B_root = os.path.abspath(last) + else: + # 2B. No in head -> classify last. + ct = command_type(last) + + if ct == "UnClassified": + print(f"Unrecognized command: {last}") + return 5 + + if ct == "HasOther": + print("Other path not specified for has_other command(s).") + return 6 + + if ct == "Help": + help_list.append(last) + elif ct == "NoOther": + no_other_list.append(last) + # ct cannot be HasOther here due to earlier check. + + if meta.debug_has("print_command_lists"): + print_command_lists( + help_list + ,no_other_list + ,has_other_list + ,unclassified_list + ) + + # Help handling + if len(help_list) > 0: + if "version" in help_list: + meta.version_print() + if "usage" in help_list: + doc.print_usage() + if "help" in help_list: + doc.print_help() + return 1 + + ret_val = 0 + + # No-other commands (environment, etc.) + if "environment" in no_other_list: + env_status = meta.printenv() + if env_status != 0: + ret_val = env_status + + # If we still have no has_other commands, we are done. + # (Example: just "environment", or just "help/usage".) + if len(has_other_list) == 0: + return ret_val + + # At this point we know: + # - has_other_list is non-empty + # - B_root must have been set (head_has_other was True) + if B_root is None: + print("Internal error: B_root not set despite has_other commands.") + return 7 + + if not os.path.isdir(B_root): + print(f"Other project path is not a directory: {B_root}") + return 4 + + # Determine Harmony root (A_root) + status, A_root = Harmony.where() + + if status == "not-found": + print("Harmony project not found; normally this command is run from within Harmony.") + return 3 + + if status == "different": + print("Seems we are not running in the Harmony project, will exit.") + return 2 + + # Build tree_dicts for A (Harmony) and B (other project) + A_tree = skeleton.tree_dict_make(A_root, None) + B_tree = skeleton.tree_dict_make(B_root, None) + + # Dispatch the commands + cmd_status = command.dispatch( + has_other_list + ,A_tree + ,B_tree + ,A_root + ,B_root + ) + + if cmd_status != 0: + ret_val = cmd_status + + return ret_val + + +if __name__ == "__main__": + raise SystemExit(CLI()) diff --git a/shared/authored/sync/GitIgnore.py b/shared/authored/sync/GitIgnore.py new file mode 100755 index 0000000..70c6509 --- /dev/null +++ b/shared/authored/sync/GitIgnore.py @@ -0,0 +1,270 @@ +#!/usr/bin/env python3 +# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- + +""" +GitIgnore.py - minimal .gitignore-based helper for Harmony projects + +Behavior: + + 1. During initialization, traverse the project tree rooted at + . + + 2. Whenever a directory contains a '.gitignore' file, record: + - its relative directory path from the project root + - a list of regular expressions compiled from the patterns + in that '.gitignore' file + + These are stored in: + + self.rules: Dict[str, List[Pattern]] + + where the key is the directory RELATIVE to the project root: + "" -> project root (top-level .gitignore) + "src" -> src/.gitignore + "src/module" -> src/module/.gitignore + + 3. check() -> token: + + - is a path relative to the project root. + + - We compute all prefix directories of , including the + root (""), for example: + + path = "a/b/c.py" + prefixes = ["", "a", "a/b"] + + - For each prefix, if there are regexes stored for that directory, + we collect them. + + - We then test ALL collected regexes against the basename of + (the last component only). + + - If ANY regex matches, return 'Ignore'. + Otherwise return 'Accept'. + +Notes: + + * We implement a simplified subset of .gitignore semantics suitable + for your current patterns and add a small base ignore set for + always-ignored names such as '.git'. +""" + +from __future__ import annotations + +import fnmatch +import os +import re +from typing import Dict, List +import Harmony + + +class GitIgnore: + """ + GitIgnore(project_path) + + Attributes: + project_path: + Absolute path to the project root. + + rules: + Mapping from relative directory path -> list of compiled regex + patterns derived from that directory's '.gitignore' file. + + Example: + rules[""] -> patterns from /.gitignore + rules["developer"] -> patterns from developer/.gitignore + + base_patterns: + List of compiled regex patterns applied to the basename of every + checked path, independent of any .gitignore file. Currently used + to always ignore '.git' directories. + """ + + def __init__( + self + ,project_path: str + ) -> None: + """ + Initialize a GitIgnore instance with a path to a project and + scan for '.gitignore' files. + """ + self.project_path: str = os.path.abspath(project_path) + self.rules: Dict[str, List[re.Pattern]] = {} + + # Base patterns: always applied, regardless of .gitignore contents. + # These are matched against basenames only. + self.base_patterns: List[re.Pattern] = [ + re.compile(r"^\.git$") # ignore any basename == ".git" + ] + + self._scan_project() + + def _scan_project(self) -> None: + """ + Traverse the project tree and populate self.rules with entries of + the form: + + -> [Pattern, Pattern, ...] + + where is the directory containing '.gitignore', relative + to the project root ("" for root). + """ + root = self.project_path + + for dirpath, dirnames, filenames in os.walk(root, topdown=True): + if ".gitignore" not in filenames: + continue + + rel_dir = os.path.relpath(dirpath, root) + if rel_dir == ".": + rel_dir = "" + + gitignore_path = os.path.join(dirpath, ".gitignore") + patterns = self._parse_gitignore_file(gitignore_path) + + if patterns: + if rel_dir not in self.rules: + self.rules[rel_dir] = [] + self.rules[rel_dir].extend(patterns) + + def _parse_gitignore_file( + self + ,gitignore_path: str + ) -> List[re.Pattern]: + """ + Parse a single '.gitignore' file into a list of compiled regex patterns. + + Simplified rules: + - Blank lines and lines starting with '#' are ignored. + - Lines containing '/' in the MIDDLE are currently ignored + (future extension). + - Lines ending with '/' are treated as directory name patterns: + '__pycache__/' -> pattern on basename '__pycache__' + - All patterns are treated as name globs and compiled via + fnmatch.translate(), to be matched against basenames only. + """ + patterns: List[re.Pattern] = [] + + try: + with open(gitignore_path, "r", encoding="utf-8") as f: + for raw_line in f: + line = raw_line.strip() + + # Skip comments and blank lines + if not line or line.startswith("#"): + continue + + # Remove trailing '/' for directory patterns (e.g. '__pycache__/') + if line.endswith("/"): + line = line[:-1].strip() + if not line: + continue + + # If there is still a '/' in the line, we do not support this + # pattern in this minimal implementation. + if "/" in line: + continue + + # Compile as a name glob -> regex + regex_text = fnmatch.translate(line) + patterns.append(re.compile(regex_text)) + + except OSError: + # If the .gitignore cannot be read, just skip it. + return patterns + + return patterns + + def check( + self + ,path: str + ) -> str: + """ + Check a path against the collected .gitignore patterns. + + path: + A path relative to the project root. + + Returns: + 'Ignore' if any applicable pattern matches the basename of the path, + otherwise 'Accept'. + """ + # Normalize the incoming path + norm = os.path.normpath(path) + + # If the path is '.' or empty, we accept it + if norm in ("", "."): + return "Accept" + + basename = os.path.basename(norm) + + # First, apply base patterns (always applied). + for pat in self.base_patterns: + if pat.match(basename): + return "Ignore" + + # Build the list of directories that may contribute .gitignore rules. + # + # For path "a/b/c": + # prefixes: ["", "a", "a/b"] + parts = norm.split(os.sep) + + prefixes: List[str] = [""] + prefix = None + for part in parts[:-1]: + if prefix is None: + prefix = part + else: + prefix = os.path.join(prefix, part) + prefixes.append(prefix) + + # Collect all patterns from the applicable .gitignore directories + for rel_dir in prefixes: + dir_patterns = self.rules.get(rel_dir) + if not dir_patterns: + continue + + for pat in dir_patterns: + if pat.match(basename): + return "Ignore" + + return "Accept" + + +def test_GitIgnore() -> int: + """ + 1. Locate the Harmony project root using Harmony.where(). + 2. Create a GitIgnore instance rooted at that path. + 3. Print: + - directories that have .gitignore rules + - directories (relative) that would be ignored by check() + """ + status, Harmony_root = Harmony.where() + + if status == "not-found": + print("Harmony project not found; cannot test GitIgnore.") + return 1 + + if status == "different": + print("Warning: Harmony not found, using nearest .git directory for GitIgnore test.") + + gi = GitIgnore(Harmony_root) + + print(".gitignore rule directories (relative to Harmony root):") + for rel_dir in sorted(gi.rules.keys()): + print(f" {rel_dir if rel_dir else '.'}") + + print("\nDirectories that would be ignored (relative to Harmony root):") + for dirpath, dirnames, filenames in os.walk(Harmony_root, topdown=True): + rel_dir = os.path.relpath(dirpath, Harmony_root) + if rel_dir == ".": + rel_dir = "" + + if gi.check(rel_dir) == "Ignore": + print(f" {rel_dir if rel_dir else '.'}") + + return 0 + + +if __name__ == "__main__": + raise SystemExit(test_GitIgnore()) diff --git a/shared/authored/sync/Harmony.py b/shared/authored/sync/Harmony.py new file mode 100644 index 0000000..9385507 --- /dev/null +++ b/shared/authored/sync/Harmony.py @@ -0,0 +1,133 @@ +#!/usr/bin/env python3 +# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- + +""" +locate the project root +""" + +from __future__ import annotations + +import meta +import os +import sys +from typing import Any, Callable, Dict + +# where +# +# Context / assumptions: +# 1. This module lives somewhere under the Harmony tree, for example: +# /.../Harmony/tool/skeleton/skeleton.py +# 2. CLI.py is run from somewhere inside the same tree (or a clone). +# +# Search behavior: +# 1. Start from the directory containing this file. +# 2. Walk upward towards the filesystem root, with limits: +# a) Do not move up more than 5 levels. +# b) Stop immediately if the current directory contains a +# '.git' subdirectory. +# +# Result classification: +# status is one of: +# 'found' -> we found a directory whose basename is 'Harmony' +# 'different' -> we stopped at a directory that has a '.git' +# subdirectory, but its basename is not 'Harmony' +# 'not-found' -> we hit the 5-level limit or filesystem root +# without finding 'Harmony' or a '.git' directory +# +# Path: +# - In all cases, the returned path is the last directory inspected: +# * the 'Harmony' directory (status 'found'), or +# * the directory with '.git' (status 'different'), or +# * the directory at the 5-level limit / filesystem root +# (status 'not-found'). +# +# Debug printing: +# - If meta.debug_has("print_Harmony_root") is true, print: +# * "The Harmony project root found at: {path}" +# when status == 'found' +# * "Harmony not found, but found: {path}" +# when status == 'different' +# * "Harmony not found." +# when status == 'not-found' +def where() -> tuple[str, str]: + """ + Locate the Harmony root (or best guess). + + Returns: + (status, path) + """ + here = os.path.abspath(__file__) + d = os.path.dirname(here) + + harmony_root = None + status = "not-found" + + max_up = 5 + steps = 0 + + while True: + base = os.path.basename(d) + + # Case 1: exact 'Harmony' directory name + if base == "Harmony": + harmony_root = d + status = "found" + break + + # Case 2: stop at a directory that has a '.git' subdirectory + git_dir = os.path.join(d, ".git") + if os.path.isdir(git_dir): + harmony_root = d + if base == "Harmony": + status = "found" + else: + status = "different" + break + + parent = os.path.dirname(d) + + # Stop if we hit filesystem root + if parent == d: + harmony_root = d + status = "not-found" + break + + steps += 1 + if steps > max_up: + # Reached search depth limit; last inspected directory is d + harmony_root = d + status = "not-found" + break + + d = parent + + if harmony_root is None: + # Extremely defensive; in practice harmony_root will be set above. + harmony_root = d + + root_base = os.path.basename(harmony_root) + + # Warning to stderr if we are not literally in a 'Harmony' directory + if root_base != "Harmony": + sys.stderr.write( + f"WARNING: Harmony root basename is '{root_base}', expected 'Harmony'.\n" + ) + + if meta.debug_has("print_Harmony_root"): + if status == "found": + print(f"The Harmony project root found at: {harmony_root}") + elif status == "different": + print(f"Harmony not found, but found: {harmony_root}") + else: + print("Harmony not found.") + + return status, harmony_root + +def test_where() -> int: + """ + Simple test that prints the Harmony root using the debug flag. + """ + meta.debug_set("print_Harmony_root") + status, _root = where() + return 0 if status != "not-found" else 1 + diff --git a/shared/authored/sync/Harmony_where b/shared/authored/sync/Harmony_where new file mode 100755 index 0000000..9d39f1e --- /dev/null +++ b/shared/authored/sync/Harmony_where @@ -0,0 +1,37 @@ +#!/usr/bin/env python3 +# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- + +""" +Harmony_where - CLI to locate the Harmony project root + +Usage: + Harmony_where + +Prints the status and path returned by Harmony.where(). +""" + +from __future__ import annotations + +import sys + +import Harmony + + +def CLI(argv=None) -> int: + # Ignore argv; no arguments expected + status, Harmony_root = Harmony.where() + + if status == "found": + print(f"Harmony project root found at: {Harmony_root}") + return 0 + + if status == "different": + print(f"Harmony not found, but nearest .git directory is: {Harmony_root}") + return 1 + + print("Harmony project root not found.") + return 2 + + +if __name__ == "__main__": + raise SystemExit(CLI()) diff --git a/shared/authored/sync/README.org b/shared/authored/sync/README.org new file mode 100644 index 0000000..387780d --- /dev/null +++ b/shared/authored/sync/README.org @@ -0,0 +1,278 @@ +#+TITLE: skeleton_compare – Harmony skeleton comparison tool +#+AUTHOR: Reasoning Technology + +* 1. Overview + +1.1 +~skeleton_compare~ compares a Harmony skeleton (=A=) with a derived or legacy project (=B=). + +1.2 +It answers: + +- How has B diverged from A? +- What should be imported back into A? +- What should be exported from A into B? +- Which nodes are misplaced or suspicious? +- Which nodes represent valid project-specific extensions? + +1.3 +The entrypoint in this project is the symlink: + +- =tool/skeleton_compaare= + +which points to: + +- =tool/skeleton_compare_source/CLI.py= + +* 2. Role in the Harmony ecosystem + +2.1 +Harmony defines a skeleton layout (directories, leaves, extension points). + +2.2 +Projects are expected to: + +- start from that skeleton +- add work under approved extension points +- keep core structure aligned over time + +2.3 +Reality diverges: + +- legacy projects that predate Harmony +- projects with ad-hoc edits in skeleton areas +- skeleton evolution over months or years + +2.4 +~skeleton_compare~ provides: + +- a structural comparison +- a semantic comparison (types, topology) +- a chronological comparison (mtimes) +- actionable commands to re-align projects + +* 3. High-level behavior + +3.1 +Tree construction + +1. Build =tree_dict= for A (Harmony skeleton). +2. Build =tree_dict= for B (other project). +3. Attach metadata per relative path: + + - =node_type= :: =directory= | =file= | =other= | =constrained= + - =dir_info= :: =root= | =branch= | =leaf= | =NA= + - =mtime= :: float seconds since epoch + +3.2 +Git ignore + +1. A simplified =.gitignore= model is applied. +2. Some paths (e.g., =.git=) are always ignored. +3. Only paths admitted by this model participate in comparisons. + +3.3 +Topology classification (relative to A) + +1. =in_between= :: under a directory in A, but not under any leaf in A. +2. =below= :: under a leaf directory in A. +3. Neither :: not under any directory known to A (ignored for most commands). + +3.4 +Chronological classification + +1. newer(B,A) :: B node has a newer mtime than A at the same path. +2. older(B,A) :: B node has an older mtime than A at the same path. +3. A-only :: path exists in A but not B. +4. B-only :: path exists in B but not A. + +* 4. Command surface (conceptual) + +4.1 +~structure~ + +1. Compares directory topology. +2. Reports directories that: + + - exist as directories in A + - are missing or non-directories in B + +3. Intended use: + + - detect missing branches in projects + - detect structural drift + +4.2 +~import~ + +1. Direction: B → A. +2. Only considers: + + - nodes in the =in_between= region of B + - that are new or absent in A + +3. Outputs: + + - ~mkdir -p~ commands (when needed) + - ~cp --parents -a~ commands for files + - a comment list for nodes that cannot be handled automatically + (type mismatches, non-file/dir, constrained nodes) + +4. Intended use: + + - mine “good ideas” in B that belong in the skeleton + - keep Harmony evolving based on real projects + +4.3 +~export~ + +1. Direction: A → B. +2. Considers: + + - A-only nodes (present in A, missing in B) + - nodes where A’s file is newer than B’s file + +3. Outputs: + + - ~mkdir -p~ commands for B + - ~cp --parents -a~ commands for files + +4. Intended use: + + - bring B back into alignment with the current Harmony skeleton + - propagate skeleton fixes and improvements into projects + +4.4 +~suspicious~ + +1. Reports nodes in B that are: + + - inside A’s directory structure + - but not under any leaf directory + +2. Intended use: + + - highlight questionable placements + - identify candidates for new skeleton structure + - catch misuse of the skeleton (work living in the “framework” layer) + +4.5 +~addendum~ + +1. Reports nodes in B that are: + + - under leaf directories in A + +2. Intended use: + + - show work added at the intended extension points + - give a quick outline of “project-specific” content layered on Harmony + +4.6 +~all~ + +1. Runs: + + - =structure= + - =import= + - =export= + - =suspicious= + - =addendum= + +2. Intended use: + + - periodic health check of a project against Harmony + - initial analysis when inheriting an old project + +* 5. Safety and behavior guarantees + +5.1 +No direct modification + +1. ~skeleton_compaare~ itself does not modify either tree. +2. It only prints suggested shell commands. +3. A human is expected to review and run those commands (or not). + +5.2 +Constrained and unknown nodes + +1. Some paths are “constrained”: + + - object exists but metadata (e.g., ~mtime~) cannot be safely read + - typical for special files or broken links + +2. These are: + + - classified as =constrained= + - never touched by import/export logic + - surfaced in “not handled automatically” lists + +5.3 +Robust to legacy layouts + +1. A and B are assumed to be non-overlapping roots. +2. B does not have to be a clean Harmony derivative. +3. The tool is designed to: + + - tolerate missing branches + - tolerate ad-hoc additions + - still classify and report differences coherently + +* 6. How to run it + +6.1 +From inside the Harmony repo: + +#+begin_src sh +cd /path/to/Harmony +tool/skeleton_compaare help +tool/skeleton_compaare usage +tool/skeleton_compaare structure ../SomeProject +tool/skeleton_compaare all ../Rabbit +#+end_src + +6.2 +The CLI help (from ~doc.py~) is the canonical reference for: + +1. grammar and argument rules +2. meaning of A and B +3. exact semantics of each command + +This =.org= file is a conceptual overview for Harmony toolsmiths and administrators. + +* 7. Maintenance notes + +7.1 +Core modules + +1. =skeleton_compare_source/skeleton.py= + - tree construction + - topology classification + - “newer/older” logic + - in-between / below partitioning + +2. =skeleton_compare_source/command.py= + - high-level command semantics + - import/export planning and printing + +3. =skeleton_compare_source/CLI.py= + - argument classification + - environment checks + - dispatch to command handlers + +7.2 +Change discipline + +1. CLI behavior and text should be updated in: + + - =doc.py= (help/usage text) + - this =.org= file (conceptual intent) + +2. Any behavioral change that affects: + + - classification rules + - import/export semantics + - constrained handling + + should be reflected here in section 3 or 4. + diff --git a/shared/authored/sync/command.py b/shared/authored/sync/command.py new file mode 100644 index 0000000..0ebb856 --- /dev/null +++ b/shared/authored/sync/command.py @@ -0,0 +1,539 @@ +#!/usr/bin/env python3 +# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- + +""" +command.py - high-level dispatch for Harmony check commands + +Commands (semantics): + + structure: + - Differences in directory structure: directories present in A but + not present as directories in B. + + import: + - Shell copy commands to copy: + * in-between nodes in B that are newer than A (same relative path), or + * in-between nodes in B that do not exist in A at all. + Direction: B -> A + Also emits: + * a mkdir list (directories to create in A) + * an "other" list for type mismatches / non-file/dir nodes. + + export: + - Shell copy commands to copy: + * nodes in A that are newer than B, and + * nodes in A that do not exist in B. + Direction: A -> B + Also emits: + * a mkdir list (directories to create in B) + * an "other" list for type mismatches / non-file/dir nodes. + + suspicious: + - Nodes in B that fall "in between" the Harmony skeleton topology: + under some A directory, but not under any A leaf directory. + (tree_dict_in_between_and_below(A,B).in_between) + + addendum: + - Nodes in B that fall "below" Harmony leaf directories: + added work in appropriate extension locations. + (tree_dict_in_between_and_below(A,B).below) + + all: + - Runs structure, import, export, suspicious, and addendum. +""" + +from __future__ import annotations + +import os +from typing import Any, Dict, List, Tuple + +import skeleton + +TreeDict = Dict[str, Dict[str, Any]] + + +def shell_quote( + s: str +) -> str: + """ + Minimal single-quote shell quoting. + """ + return "'" + s.replace("'", "'\"'\"'") + "'" + + +def _print_header( + title: str +) -> None: + print() + print(f"== {title} ==") + + +# ---------------------------------------------------------------------- +# structure: directories in A that are missing / non-directories in B +# ---------------------------------------------------------------------- +def cmd_structure( + A: TreeDict + ,B: TreeDict +) -> int: + """ + structure: differences in directory structure, directories in A - B. + + We include any path where: + - A[path].node_type == 'directory', and + - either path not in B, or B[path].node_type != 'directory'. + """ + structural: TreeDict = {} + + for path, info_A in A.items(): + if info_A.get("node_type") != "directory": + continue + + info_B = B.get(path) + if info_B is None or info_B.get("node_type") != "directory": + structural[path] = info_A + + if not structural: + _print_header("structure") + print("No structural directory differences (A - B).") + return 0 + + _print_header("structure: directories in A not in B") + skeleton.tree_dict_print(structural) + return 0 + + +# ---------------------------------------------------------------------- +# import: B -> A (mkdir, cp, and "other" list), using in_between_newer +# ---------------------------------------------------------------------- + + +def build_import_commands( + A_tree: TreeDict + ,B_tree: TreeDict + ,A_root: str + ,B_root: str +) -> Tuple[List[str], List[str], List[str]]: + """ + Compute shell commands to update A from B. + + Returns: + (mkdir_cmds, cp_cmds, other_list) + + Semantics: + + mkdir_cmds: + - Directories that are directories in B, but are missing in A. + - We DO NOT auto-resolve type mismatches (e.g. B=directory, + A=file); those go into other_list. + + cp_cmds: + - Files where: + * the path does not exist in A, OR + * the node in A is not a file, OR + * the B copy is newer than A (mtime comparison). + - However, if A has a non-file at that path, we treat it as a + type mismatch and add that path to other_list instead of + emitting a cp command. + + other_list: + - Human-readable notes for: + * type mismatches between A and B, and + * nodes in B that are neither 'file' nor 'directory'. + """ + mkdir_cmds: List[str] = [] + cp_cmds: List[str] = [] + other_list: List[str] = [] + + for rel_path, b_info in B_tree.items(): + b_type = b_info.get("node_type") + rel_display = rel_path if rel_path else "." + + a_info = A_tree.get(rel_path) + a_type = a_info.get("node_type") if a_info is not None else "MISSING" + + # Case 1: B node is neither file nor directory -> other_list + if b_type not in ("file", "directory"): + other_list.append( + f"{rel_display}: A={a_type}, B={b_type}" + ) + continue + + # Case 2: B directory + if b_type == "directory": + if a_info is None: + # Missing in A: copy the directory recursively. + src = os.path.join(B_root, rel_path) if rel_path else B_root + # The destination should be the parent directory in A. + # os.path.join(A_root, rel_path) gives the full path to the new directory in A. + # os.path.dirname of that gives the parent directory. + dst = os.path.dirname(os.path.join(A_root, rel_path)) + mkdir_cmds.append( + f"cp -a {shell_quote(src)} {shell_quote(dst + os.sep)}" + ) + else: + # Exists in A: must also be a directory to be "structurally OK" + if a_type != "directory": + # Type mismatch: do not mkdir, just report + other_list.append( + f"{rel_display}: A={a_type}, B=directory" + ) + continue + + # Case 3: B file + # Decide whether to copy B -> A, or report conflict. + if a_info is None: + # B-only file + src = os.path.join(B_root, rel_path) if rel_path else B_root + # Destination is the parent directory in A, with a trailing slash + dst = os.path.dirname(os.path.join(A_root, rel_path)) + cp_cmds.append( + f"cp -a {shell_quote(src)} {shell_quote(dst + os.sep)}" + ) + continue + + # A has something at this path + if a_type != "file": + # Type mismatch (e.g. A=directory, B=file, or A=other) + other_list.append( + f"{rel_display}: A={a_type}, B=file" + ) + continue + + # Both files: compare mtime + a_mtime = a_info.get("mtime") + b_mtime = b_info.get("mtime") + + if isinstance(a_mtime, (int, float)) and isinstance(b_mtime, (int, float)): + if b_mtime > a_mtime: + src = os.path.join(B_root, rel_path) if rel_path else B_root + # Destination is the parent directory in A, with a trailing slash + dst = os.path.dirname(os.path.join(A_root, rel_path)) + cp_cmds.append( + f"cp -a {shell_quote(src)} {shell_quote(dst + os.sep)}" + ) + + return mkdir_cmds, cp_cmds, other_list + +def cmd_import( + A_tree: TreeDict + ,B_tree: TreeDict + ,A_root: str + ,B_root: str +) -> int: + """ + import: update the skeleton (A) from the project (B), + using only in_between_newer nodes. + """ + inb_newer = skeleton.in_between_newer(A_tree, B_tree) + + mkdir_cmds, cp_cmds, other_list = build_import_commands( + A_tree + ,inb_newer + ,A_root + ,B_root + ) + + print("== import: copy from B -> A (in-between newer only) ==") + print(f"# A root: {A_root}") + print(f"# B root: {B_root}") + print("# Only considering in-between files that are new or absent in A.") + print("#") + + print("# Directories to copy from B -> A (cp -a):") + if mkdir_cmds: + for line in mkdir_cmds: + print(line) + else: + print("# (none)") + print("#") + + print("# Files to copy from B -> A (cp -a):") + if cp_cmds: + for line in cp_cmds: + print(line) + else: + print("# (none)") + print("#") + + print("# Nodes NOT handled automatically (type mismatches / non-file/dir):") + if other_list: + for rel in other_list: + print(f"# {rel}") + else: + print("# (none)") + + return 0 + + +# ---------------------------------------------------------------------- +# export: A -> B (mkdir, cp, and "other" list) +# ---------------------------------------------------------------------- +def build_export_commands( + A_tree: TreeDict + ,B_tree: TreeDict + ,A_root: str + ,B_root: str +) -> Tuple[List[str], List[str], List[str]]: + """ + Compute shell commands to update B from A. + + Returns: + (mkdir_cmds, cp_cmds, other_list) + + Semantics: + + mkdir_cmds: + - Directories that are directories in A, but are missing in B. + - Type mismatches go into other_list. + + cp_cmds: + - Files where: + * the path does not exist in B, OR + * the node in B is not a file, OR + * the A copy is newer than B (mtime comparison). + - If B has a non-file while A has a file, treat as type mismatch. + + other_list: + - Human-readable notes for: + * type mismatches between A and B, and + * nodes in A that are neither 'file' nor 'directory'. + """ + mkdir_cmds: List[str] = [] + cp_cmds: List[str] = [] + other_list: List[str] = [] + + # Sort keys to ensure parent directories are processed before their children. + sorted_paths = sorted(A_tree.keys(), key=len) + included_dirs: Set[str] = set() + + for rel_path in sorted_paths: + a_info = A_tree[rel_path] + a_type = a_info.get("node_type") + rel_display = rel_path if rel_path else "." + + b_info = B_tree.get(rel_path) + b_type = b_info.get("node_type") if b_info is not None else "MISSING" + + # Case 1: A node is neither file nor directory -> other_list + if a_type not in ("file", "directory"): + other_list.append( + f"{rel_display}: A={a_type}, B={b_type}" + ) + continue + + # Check if this path is a child of an already included directory + is_child_of_included_dir = False + for d in included_dirs: + if rel_path.startswith(d + os.sep): + is_child_of_included_dir = True + break + + if is_child_of_included_dir: + continue + + # Case 2: A directory + if a_type == "directory": + if b_info is None: + # Missing in B: copy the directory recursively. + src = os.path.join(A_root, rel_path) if rel_path else A_root + # The destination should be the parent directory in B. + dst = os.path.dirname(os.path.join(B_root, rel_path)) + mkdir_cmds.append( + f"cp -a {shell_quote(src)} {shell_quote(dst + os.sep)}" + ) + included_dirs.add(rel_path) + else: + # Exists in B: must also be directory + if b_type != "directory": + other_list.append( + f"{rel_display}: A=directory, B={b_type}" + ) + continue + + # Case 3: A file + if b_info is None: + # A-only file + src = os.path.join(A_root, rel_path) if rel_path else A_root + # Destination is the parent directory in B, with a trailing slash + dst = os.path.dirname(os.path.join(B_root, rel_path)) + cp_cmds.append( + f"cp -a {shell_quote(src)} {shell_quote(dst + os.sep)}" + ) + continue + + if b_type != "file": + other_list.append( + f"{rel_display}: A=file, B={b_type}" + ) + continue + + # Both files: compare mtime + a_mtime = a_info.get("mtime") + b_mtime = b_info.get("mtime") + + if isinstance(a_mtime, (int, float)) and isinstance(b_mtime, (int, float)): + if a_mtime > b_mtime: + src = os.path.join(A_root, rel_path) if rel_path else A_root + # Destination is the parent directory in B, with a trailing slash + dst = os.path.dirname(os.path.join(B_root, rel_path)) + cp_cmds.append( + f"cp -a {shell_quote(src)} {shell_quote(dst + os.sep)}" + ) + + return mkdir_cmds, cp_cmds, other_list + + +def cmd_export( + A_tree: TreeDict + ,B_tree: TreeDict + ,A_root: str + ,B_root: str +) -> int: + """ + export: show directory creation and copy commands A -> B. + """ + mkdir_cmds, cp_cmds, other_list = build_export_commands( + A_tree + ,B_tree + ,A_root + ,B_root + ) + + print("== export: copy from A -> B ==") + print(f"# A root: {A_root}") + print(f"# B root: {B_root}") + print("#") + + print("# Directories to copy from A -> B (cp -a):") + if mkdir_cmds: + for line in mkdir_cmds: + print(line) + else: + print("# (none)") + print("#") + + print("# Files to copy from A -> B (cp -a):") + if cp_cmds: + for line in cp_cmds: + print(line) + else: + print("# (none)") + print("#") + + print("# Nodes NOT handled automatically (type mismatches / non-file/dir):") + if other_list: + for rel in other_list: + print(f"# {rel}") + else: + print("# (none)") + + return 0 + + +# ---------------------------------------------------------------------- +# suspicious / addendum via in_between_and_below +# ---------------------------------------------------------------------- +def cmd_suspicious( + A: TreeDict + ,B: TreeDict +) -> int: + """ + suspicious: nodes in B that fall 'in between' the Harmony skeleton, + not under leaf directories. + """ + in_between, _below = skeleton.tree_dict_in_between_and_below(A, B) + + _print_header("suspicious: nodes in-between Harmony leaves") + + if not in_between: + print("No suspicious nodes found in B (relative to A).") + return 0 + + skeleton.tree_dict_print(in_between) + return 0 + + +def cmd_addendum( + A: TreeDict + ,B: TreeDict +) -> int: + """ + addendum: nodes in B that fall 'below' Harmony leaf directories. + """ + _in_between, below = skeleton.tree_dict_in_between_and_below(A, B) + + _print_header("addendum: nodes added under Harmony leaves") + + if not below: + print("No addendum nodes found in B (relative to A).") + return 0 + + skeleton.tree_dict_print(below) + return 0 + + +# ---------------------------------------------------------------------- +# Top-level dispatcher +# ---------------------------------------------------------------------- +def dispatch( + has_other_list: List[str] + ,A: TreeDict + ,B: TreeDict + ,A_root: str + ,B_root: str +) -> int: + """ + Dispatch commands. + + has_other_list: + List of command tokens (subset of: + 'structure', 'import', 'export', 'suspicious', 'addendum', 'all'). + + A, B: + tree_dicts for Harmony skeleton (A) and project (B). + + A_root, B_root: + Root paths corresponding to A and B (for copy commands). + """ + cmds = set(has_other_list) + + if "all" in cmds: + cmds.update([ + "structure" + ,"import" + ,"export" + ,"suspicious" + ,"addendum" + ]) + + ordered = [ + "structure" + ,"import" + ,"export" + ,"suspicious" + ,"addendum" + ] + + status = 0 + + for name in ordered: + if name not in cmds: + continue + + if name == "structure": + rc = cmd_structure(A, B) + elif name == "import": + rc = cmd_import(A, B, A_root, B_root) + elif name == "export": + rc = cmd_export(A, B, A_root, B_root) + elif name == "suspicious": + rc = cmd_suspicious(A, B) + elif name == "addendum": + rc = cmd_addendum(A, B) + else: + rc = 0 + + if rc != 0: + status = rc + + return status diff --git a/shared/authored/sync/doc.py b/shared/authored/sync/doc.py new file mode 100644 index 0000000..3198b96 --- /dev/null +++ b/shared/authored/sync/doc.py @@ -0,0 +1,182 @@ +#!/usr/bin/env python3 +# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- + +""" +doc.py - usage and help text for the Harmony 'check' tool + +Grammar (informal): + + * [] + + :: | | + + :: version | help | usage + :: environment + :: structure | import | export | suspicious | addendum | all +""" + +from __future__ import annotations + +import meta +import os +import sys +from typing import TextIO + + +def prog_name() -> str: + """ + Return the program name as invoked by the user. + + Typically: + - basename(sys.argv[0]) when running from the shell. + - Falls back to 'check' if argv[0] is empty. + """ + raw = sys.argv[0] if sys.argv and sys.argv[0] else "check" + base = os.path.basename(raw) or raw + return base + + +def _usage_text(prog: str) -> str: + return f"""\ +Usage: + {prog} * [] + +Where: + :: | | + + :: version | help | usage + :: environment + :: structure | import | export | suspicious | addendum | all +""" + +def _help_text(prog: str) -> str: + return f"""\ +{prog} - Harmony skeleton integrity and metadata checker + +Syntax: + {prog} * [] + +Where: + :: path + :: | | + + :: version | help | usage + :: environment + :: structure | import | export | suspicious | addendum | all + +Argument rules (informal): + 1. commands are processed first, and then the program returns. + Hence if any help commands are present, the remaining commands + are ignored. + + 2. We assume {prog} is run within the Harmony skeleton, or a skeleton + derived directly from it. This is the 'default skeleton', or simply 'A'. + + 3. The path is the directory of a project that is assumed to + be built upon the default skeleton. This second project root is + called 'B'. + + 4. If none of the commands require an path, then + must not be given. If at least one command requires , then + is required. Commands that require a path are called + commands. + + 5. Implementation detail: + All arguments except the final one are interpreted strictly as + command tokens. If any of those are , the final argument + is taken as . If none of the earlier tokens are , + the final argument is also treated as a command token. + +Roots: + A = Skeleton project root (auto-detected). Usually the Harmony skeleton. + B = project root (supplied when required). + +{prog} compares A with B. Differences may come from: + - edits to the skeleton itself, + - edits to skeleton files inside B, + - or new files/directories added to B. +Conceptually, A and B are any two non-overlapping directory trees. + +Command semantics: + structure + - Report directory-structure differences: + directories present in A that are missing in B or not + directories in B. + - Output: table of such directories. + + import + - Update A from B using only "in-between newer" files: + * files in B that lie in the 'in-between' region relative to A, and + * are newer than A or absent from A. + - Also emits: + * directories to create in A, + * files to copy (B -> A), + * nodes that cannot be handled automatically (type mismatches, + constrained nodes, non-file/dir nodes). + - Direction: B -> A + + export + - Update B from A: + * files in A newer than B at the same path, + * files present in A but missing in B. + - Also emits: + * directories to create in B, + * files to copy (A -> B), + * nodes that cannot be handled automatically. + - Direction: A -> B + + suspicious + - Report B nodes that lie "in-between" Harmony leaves: + under a directory from A, but not under any leaf directory of A. + - Indicates questionable placements or missing skeleton structure. + + addendum + - Report B nodes located "below" Harmony leaf directories: + project-specific additions placed in proper extension points. + + all + - Run: structure, import, export, suspicious, addendum (in that order). + +Notes: + - tree_dict traversal respects a simplified .gitignore model plus + always-ignored patterns (e.g. '.git'). + - Timestamps are formatted via the Z helper in UTC (ISO 8601). +""" + +def print_usage( + stream: TextIO | None = None +) -> None: + """ + Print the usage text to the given stream (default: sys.stdout), + using the actual program name as invoked. + """ + if stream is None: + stream = sys.stdout + + text = _usage_text(prog_name()) + stream.write(text) + if not text.endswith("\n"): + stream.write("\n") + + +def print_help( + stream: TextIO | None = None +) -> None: + """ + Print the help text to the given stream (default: sys.stdout), + using the actual program name as invoked. + """ + if stream is None: + stream = sys.stdout + + utext = _usage_text(prog_name()) + htext = _help_text(prog_name()) + + stream.write(utext) + if not utext.endswith("\n"): + stream.write("\n") + + stream.write("\n") + stream.write(htext) + if not htext.endswith("\n"): + stream.write("\n") diff --git a/shared/authored/sync/in_between_and_below b/shared/authored/sync/in_between_and_below new file mode 100755 index 0000000..2993767 --- /dev/null +++ b/shared/authored/sync/in_between_and_below @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- + +""" +in_between_and_below - CLI test driver for skeleton.tree_dict_in_between_and_below(A, B) + +Usage: + in_between_and_below +""" + +from __future__ import annotations + +import os +import sys +from typing import Sequence + +import meta +import skeleton + + +def CLI(argv: Sequence[str] | None = None) -> int: + if argv is None: + argv = sys.argv[1:] + + prog = os.path.basename(sys.argv[0]) if sys.argv else "in_between_and_below" + + if len(argv) != 2 or argv[0] in ("-h", "--help"): + print(f"Usage: {prog} ") + return 1 + + A_root = argv[0] + B_root = argv[1] + + if not os.path.isdir(A_root): + print(f"{prog}: {A_root}: not a directory") + return 2 + + if not os.path.isdir(B_root): + print(f"{prog}: {B_root}: not a directory") + return 3 + + A = skeleton.tree_dict_make(A_root, None) + B = skeleton.tree_dict_make(B_root, None) + + meta.debug_set("tree_dict_in_between_and_below") + + _result = skeleton.tree_dict_in_between_and_below(A, B) + + return 0 + + +if __name__ == "__main__": + raise SystemExit(CLI()) diff --git a/shared/authored/sync/load_command_module.py b/shared/authored/sync/load_command_module.py new file mode 100644 index 0000000..226b6dd --- /dev/null +++ b/shared/authored/sync/load_command_module.py @@ -0,0 +1,103 @@ +#!/usr/bin/env python3 +# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- + +""" +load_command_module.py - locate and import Python command modules from $PATH + +Behavior: + 1. Search $PATH for an executable with the given command name. + 2. Prefer a path containing '/incommon/'. + 3. If only /usr/bin/ is found, raise an error saying we were + looking for the incommon version. + 4. Import the chosen script as a Python module, even if it has no .py + extension, by forcing a SourceFileLoader. +""" + +from __future__ import annotations + +import importlib.util +import os +from importlib.machinery import SourceFileLoader +from types import ModuleType +from typing import List + + +def _find_command_candidates(command_name: str) -> List[str]: + """ + Return a list of absolute paths to executables named `command_name` + found on $PATH. + """ + paths: list[str] = [] + + path_env = os.environ.get("PATH", "") + for dir_path in path_env.split(os.pathsep): + if not dir_path: + continue + candidate = os.path.join(dir_path, command_name) + if os.path.isfile(candidate) and os.access(candidate, os.X_OK): + paths.append(os.path.realpath(candidate)) + + return paths + + +def load_command_module(command_name: str) -> ModuleType: + """ + Locate an executable named `command_name` on $PATH and load it + as a Python module. + + Selection policy: + 1. Prefer any path containing '/incommon/'. + 2. If only /usr/bin/ candidates exist, raise an error + saying we were looking for the incommon version. + 3. If no candidate is found, raise an error. + + Implementation detail: + Because the incommon command may lack a .py suffix, we explicitly + construct a SourceFileLoader rather than relying on the default + extension-based loader resolution. + """ + candidates = _find_command_candidates(command_name) + + incommon_candidates = [ + p + for p in candidates + if "/incommon/" in p + ] + + usrbin_candidates = [ + p + for p in candidates + if p.startswith("/usr/bin/") + ] + + if incommon_candidates: + target = incommon_candidates[0] + elif usrbin_candidates: + raise RuntimeError( + f"Found /usr/bin/{command_name}, but expected the incommon Python " + f"{command_name} module on PATH." + ) + else: + raise RuntimeError( + f"Could not find an incommon '{command_name}' module on PATH." + ) + + module_name = f"rt_incommon_{command_name}" + + loader = SourceFileLoader( + module_name + ,target + ) + spec = importlib.util.spec_from_loader( + module_name + ,loader + ) + if spec is None: + raise RuntimeError(f"Failed to create spec for {command_name} from {target}") + + module = importlib.util.module_from_spec(spec) + # spec.loader is the SourceFileLoader we just created + assert spec.loader is not None + spec.loader.exec_module(module) + + return module diff --git a/shared/authored/sync/make_Harmony_tree_dict b/shared/authored/sync/make_Harmony_tree_dict new file mode 100755 index 0000000..2ed3cea --- /dev/null +++ b/shared/authored/sync/make_Harmony_tree_dict @@ -0,0 +1,48 @@ +#!/usr/bin/env python3 +# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- + +""" +skeleton_test - build and print the Harmony tree_dict + +Usage: + skeleton_test + +Behavior: + 1. Locate the Harmony project root via Harmony.where(). + 2. Enable 'tree_dict_print' debug flag. + 3. Call skeleton.tree_dict_make(Harmony_root, None). + +The skeleton.tree_dict_make() function is expected to call +tree_dict_print() when the 'tree_dict_print' debug flag is set. +""" + +from __future__ import annotations + +import sys + +import Harmony +import meta +import skeleton + + +def CLI(argv=None) -> int: + # No arguments expected + status, Harmony_root = Harmony.where() + + if status == "not-found": + print("Harmony project not found; cannot build tree_dict.") + return 1 + + if status == "different": + print("Warning: Harmony not found, using nearest .git directory for tree_dict.") + + # Enable printing inside tree_dict_make + meta.debug_set("tree_dict_print") + + _tree = skeleton.tree_dict_make(Harmony_root, None) + + return 0 + + +if __name__ == "__main__": + raise SystemExit(CLI()) diff --git a/shared/authored/sync/manus_2025_11_20_215471873762383.txt b/shared/authored/sync/manus_2025_11_20_215471873762383.txt new file mode 100644 index 0000000..70e1f99 --- /dev/null +++ b/shared/authored/sync/manus_2025_11_20_215471873762383.txt @@ -0,0 +1,33 @@ +Conversation with Manus +Started on November 20, 2025 at 08:56 PM Singapore time +08 (GMT+0800) + +--- + +08:56 PM | Thomas Lynch: Hello, I signed up today with 7 days free. However, in the middle of a chat, I was prompted to sign up, which I already did. It then charged me for the month, but we have not been 7 days yet, and it is full charge. Why did it do that? + As a second question is the 'credits'. Did it prompt again because the credits were used? Am I now on the hook for two months of charges in less than a day of use? + If just a few hours use uses up a 'month' of credits, then I really can't afford Manus, and need to back out of this. + +08:56 PM | Manus AI: I understand your concern about the unexpected charge! Let me clarify what likely happened. + About the charge: Our subscriptions operate on a credit-based system (http://help.manus.im/en/articles/11711097-what-are-the-rules-for-credits-consumption-and-how-can-i-obtain-them) where credits are consumed during active task processing. When you run out of credits (even during a trial), you may be prompted to upgrade to continue using the service, which would trigger billing. + You're not charged twice: The billing is monthly from your subscription date (http://help.manus.im/en/articles/11711089-where-can-i-find-the-invoice-after-purchase-can-the-invoice-be-modified), so you'll only have one active subscription. + To check your usage: Visit Settings > Usage to see exactly how your credits were consumed and review transaction details. + Your options: + - Cancel subscription: (http://help.manus.im/en/articles/11711087-how-can-i-cancel-my-subscription) Go to your profile > Account > Manage subscription > Cancel subscription to stop future charges. +- Request refund: You can submit a refund request with your invoice information and reason. We'll review it according to our refund policy. + The good news is you can cancel anytime to prevent future charges, and we do consider refund requests on a case-by-case basis! + +Sources: +What are the rules for credits consumption? And how can I obtain them? (http://help.manus.im/en/articles/11711097-what-are-the-rules-for-credits-consumption-and-how-can-i-obtain-them) +Where can I check my credit consumption/usage? (http://help.manus.im/en/articles/12520235-where-can-i-check-my-credit-consumption-usage) +How can I request a refund for a purchased membership? (http://help.manus.im/en/articles/11711102-how-can-i-request-a-refund-for-a-purchased-membership) +How can I cancel my subscription? (http://help.manus.im/en/articles/11711087-how-can-i-cancel-my-subscription) +Where can I find the invoice after purchase, can the invoice be modified? (http://help.manus.im/en/articles/11711089-where-can-i-find-the-invoice-after-purchase-can-the-invoice-be-modified) + +08:57 PM | Liora from Manus: We will respond to you with the resolution within 48 hours. Please wait for our update! + +09:01 PM | Thomas Lynch: You charged my $46.80, the plan says it costs $33.00 explain. + +09:02 PM | Thomas Lynch: hello? + +--- +Exported from Manus on November 20, 2025 at 09:03 PM Singapore time +08 (GMT+0800) \ No newline at end of file diff --git a/shared/authored/sync/meta.py b/shared/authored/sync/meta.py new file mode 100644 index 0000000..5c8da89 --- /dev/null +++ b/shared/authored/sync/meta.py @@ -0,0 +1,97 @@ +#!/usr/bin/env python3 +# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- + +""" +meta.py - thin wrappers around command modules + +Current responsibilities: + 1. Load the incommon 'printenv' command module (no .py extension) + using load_command_module.load_command_module(). + 2. Expose printenv() here, calling the imported printenv() work + function with default arguments (equivalent to running without + any CLI arguments). + 3. Provide a simple version printer for this meta module. + 4. Provide a small debug tag API (set/clear/has). +""" + +from __future__ import annotations + +import datetime +from load_command_module import load_command_module + + +# Load the incommon printenv module once at import time +_PRINTENV_MODULE = load_command_module("printenv") +_Z_MODULE = load_command_module("Z") + + +# Meta module version +_major = 1 +_minor = 7 +def version_print() -> None: + """ + Print the meta module version as MAJOR.MINOR. + """ + print(f"{_major}.{_minor}") + + +# Debug tag set and helpers +_debug = set([ +]) + + +def debug_set(tag: str) -> None: + """ + Add a debug tag to the meta debug set. + """ + _debug.add(tag) + + +def debug_clear(tag: str) -> None: + """ + Remove a debug tag from the meta debug set, if present. + """ + _debug.discard(tag) + + +def debug_has(tag: str) -> bool: + """ + Return True if the given debug tag is present. + """ + return tag in _debug + + +# Touch the default tag once so static checkers do not complain about +# unused helpers when imported purely for side-effects. +debug_has("Command") + + +def printenv() -> int: + """ + Call the imported printenv() work function with default arguments: + - no null termination + - no newline quoting + - no specific names (print full environment) + - prog name 'printenv' + """ + return _PRINTENV_MODULE.printenv( + False # null_terminate + ,False # quote_newlines + ,[] # names + ,"printenv" + ) + + +def z_format_mtime( + mtime: float +) -> str: + """ + Format a POSIX mtime (seconds since epoch, UTC) using the Z module. + + Uses Z.ISO8601_FORMAT and Z.make_timestamp(dt=...). + """ + dt = datetime.datetime.fromtimestamp(mtime, datetime.timezone.utc) + return _Z_MODULE.make_timestamp( + fmt=_Z_MODULE.ISO8601_FORMAT + ,dt=dt + ) diff --git a/shared/authored/sync/newer b/shared/authored/sync/newer new file mode 100755 index 0000000..30aa373 --- /dev/null +++ b/shared/authored/sync/newer @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- + +""" +newer - CLI test driver for skeleton.tree_dict_newer(A, B) + +Usage: + newer +""" + +from __future__ import annotations + +import os +import sys +from typing import Sequence + +import meta +import skeleton + + +def CLI(argv: Sequence[str] | None = None) -> int: + if argv is None: + argv = sys.argv[1:] + + prog = os.path.basename(sys.argv[0]) if sys.argv else "newer" + + if len(argv) != 2 or argv[0] in ("-h", "--help"): + print(f"Usage: {prog} ") + return 1 + + A_root = argv[0] + B_root = argv[1] + + if not os.path.isdir(A_root): + print(f"{prog}: {A_root}: not a directory") + return 2 + + if not os.path.isdir(B_root): + print(f"{prog}: {B_root}: not a directory") + return 3 + + A = skeleton.tree_dict_make(A_root, None) + B = skeleton.tree_dict_make(B_root, None) + + meta.debug_set("tree_dict_newer") + + _result = skeleton.tree_dict_newer(A, B) + + return 0 + + +if __name__ == "__main__": + raise SystemExit(CLI()) diff --git a/shared/authored/sync/older b/shared/authored/sync/older new file mode 100755 index 0000000..f8ff24d --- /dev/null +++ b/shared/authored/sync/older @@ -0,0 +1,53 @@ +#!/usr/bin/env python3 +# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- + +""" +older - CLI test driver for skeleton.tree_dict_older(A, B) + +Usage: + older +""" + +from __future__ import annotations + +import os +import sys +from typing import Sequence + +import meta +import skeleton + + +def CLI(argv: Sequence[str] | None = None) -> int: + if argv is None: + argv = sys.argv[1:] + + prog = os.path.basename(sys.argv[0]) if sys.argv else "older" + + if len(argv) != 2 or argv[0] in ("-h", "--help"): + print(f"Usage: {prog} ") + return 1 + + A_root = argv[0] + B_root = argv[1] + + if not os.path.isdir(A_root): + print(f"{prog}: {A_root}: not a directory") + return 2 + + if not os.path.isdir(B_root): + print(f"{prog}: {B_root}: not a directory") + return 3 + + A = skeleton.tree_dict_make(A_root, None) + B = skeleton.tree_dict_make(B_root, None) + + meta.debug_set("tree_dict_older") + + _result = skeleton.tree_dict_older(A, B) + + return 0 + + +if __name__ == "__main__": + raise SystemExit(CLI()) diff --git a/shared/authored/sync/skeleton.py b/shared/authored/sync/skeleton.py new file mode 100644 index 0000000..ae272f9 --- /dev/null +++ b/shared/authored/sync/skeleton.py @@ -0,0 +1,576 @@ +#!/usr/bin/env python3 +# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- + +""" +skeleton.py - helpers for working with the Harmony skeleton tree +""" + +from __future__ import annotations + +import os +import sys +from typing import Any, Callable, Dict, List, Set + +import meta +from GitIgnore import GitIgnore +import Harmony + +TreeDict = Dict[str, Dict[str, Any]] + +# tree_dict_make / tree_dict_print +# +# Build a dictionary describing a project tree, respecting GitIgnore. +# +# tree_dict_make(, ) -> tree_dict +# +# () -> bignum | None +# +# Keys of tree_dict: +# - Relative paths from ; the root itself is stored under "". +# +# Values are dicts with: +# 1. 'mtime' : last modification time (float seconds) or None +# 2. 'node_type' : 'file', 'directory', 'other', or 'constrained' +# 3. 'dir_info' : 'NA', 'leaf', 'branch', 'root' +# 4. 'checksum' : present only for file nodes when checksum_fn is +# not None +# +# Traversal: +# - Directories whose relative path GitIgnore.check() marks as +# 'Ignore' are included in tree_dict but not traversed further. +def tree_dict_make( + path: str + ,checksum_fn: Callable[[str], int] | None +) -> Dict[str, Dict[str, Any]]: + """ + Build a tree_dict for the subtree rooted at , respecting GitIgnore. + + Semantics (current): + * Any path (directory or file) for which GitIgnore.check() + returns 'Ignore' is completely omitted from the tree_dict. + * The root directory ('') is always included. + * Directory dir_info: + - 'root' for the root + - 'branch' for directories that have child directories + (after GitIgnore filtering) + - 'leaf' for directories with no child directories + * Non-directory dir_info: + - 'NA' + * Symlinks are classified as file/directory/other based on what + they point to, if accessible. + * If any filesystem access needed for classification/mtime raises, + the node is recorded as node_type='constrained', dir_info='NA', + mtime=None, and we do not attempt checksum. + """ + root = os.path.abspath(path) + gi = GitIgnore(root) + + tree_dict: Dict[str, Dict[str, Any]] = {} + + for dirpath, dirnames, filenames in os.walk(root, topdown=True): + rel_dir = os.path.relpath(dirpath, root) + if rel_dir == ".": + rel_dir = "" + + # Skip ignored directories (except the root). + if rel_dir != "" and gi.check(rel_dir) == "Ignore": + dirnames[:] = [] + continue + + # Filter child directories by GitIgnore so dir_info reflects + # only directories we will actually traverse. + kept_dirnames: List[str] = [] + for dn in list(dirnames): + child_rel = dn if rel_dir == "" else os.path.join(rel_dir, dn) + if gi.check(child_rel) == "Ignore": + dirnames.remove(dn) + else: + kept_dirnames.append(dn) + + # Record the directory node itself + dir_abs = dirpath + try: + dir_mtime = os.path.getmtime(dir_abs) + dir_node_type = "directory" + if rel_dir == "": + dir_info = "root" + elif kept_dirnames: + dir_info = "branch" + else: + dir_info = "leaf" + except OSError: + # Could not stat the directory: treat as constrained. + dir_mtime = None + dir_node_type = "constrained" + dir_info = "NA" + + tree_dict[rel_dir] = { + "mtime": dir_mtime + ,"node_type": dir_node_type + ,"dir_info": dir_info + } + + # For non-ignored directories, record files within + for name in filenames: + abs_path = os.path.join(dirpath, name) + if rel_dir == "": + rel_path = name + else: + rel_path = os.path.join(rel_dir, name) + + if gi.check(rel_path) == "Ignore": + continue + + # Wrap classification + mtime in one try/except so any failure + # marks the node as constrained. + try: + if os.path.islink(abs_path): + # Symlink: classify by target if possible + if os.path.isdir(abs_path): + node_type = "directory" + dir_info_f = "branch" + elif os.path.isfile(abs_path): + node_type = "file" + dir_info_f = "NA" + else: + node_type = "other" + dir_info_f = "NA" + mtime = os.path.getmtime(abs_path) + else: + # Normal node + if os.path.isfile(abs_path): + node_type = "file" + dir_info_f = "NA" + elif os.path.isdir(abs_path): + node_type = "directory" + dir_info_f = "branch" + else: + node_type = "other" + dir_info_f = "NA" + mtime = os.path.getmtime(abs_path) + except OSError: + # Anything that blows up during classification/stat becomes + # constrained; we do not attempt checksum for these. + node_type = "constrained" + dir_info_f = "NA" + mtime = None + + info: Dict[str, Any] = { + "mtime": mtime + ,"node_type": node_type + ,"dir_info": dir_info_f + } + + if node_type == "file" and checksum_fn is not None and isinstance(mtime, (int, float)): + info["checksum"] = checksum_fn(abs_path) + + tree_dict[rel_path] = info + + if meta.debug_has("tree_dict_print"): + tree_dict_print(tree_dict) + + return tree_dict + +def tree_dict_print( + tree_dict: Dict[str, Dict[str, Any]] +) -> None: + """ + Pretty-print a tree_dict produced by tree_dict_make() in fixed-width columns: + + [type] [dir] [mtime] [checksum?] [relative path] + + Only the values are printed in each column (no 'field=' prefixes). + mtime is formatted via the Z module for human readability. + """ + entries: List[tuple[str, str, str, str, str]] = [] + has_checksum = False + + for rel_path in sorted(tree_dict.keys()): + info = tree_dict[rel_path] + display_path = rel_path if rel_path != "" else "." + + type_val = str(info.get("node_type", "")) + dir_val = str(info.get("dir_info", "")) + + raw_mtime = info.get("mtime") + if isinstance(raw_mtime, (int, float)): + mtime_val = meta.z_format_mtime(raw_mtime) + else: + mtime_val = str(raw_mtime) + + if "checksum" in info: + checksum_val = str(info["checksum"]) + has_checksum = True + else: + checksum_val = "" + + entries.append(( + type_val + ,dir_val + ,mtime_val + ,checksum_val + ,display_path + )) + + # Compute column widths + type_w = 0 + dir_w = 0 + mtime_w = 0 + checksum_w = 0 + + for type_val, dir_val, mtime_val, checksum_val, _ in entries: + if len(type_val) > type_w: + type_w = len(type_val) + if len(dir_val) > dir_w: + dir_w = len(dir_val) + if len(mtime_val) > mtime_w: + mtime_w = len(mtime_val) + if has_checksum and len(checksum_val) > checksum_w: + checksum_w = len(checksum_val) + + print("Tree dictionary contents:") + for type_val, dir_val, mtime_val, checksum_val, display_path in entries: + line = " " + line += type_val.ljust(type_w) + line += " " + line += dir_val.ljust(dir_w) + line += " " + line += mtime_val.ljust(mtime_w) + + if has_checksum: + line += " " + line += checksum_val.ljust(checksum_w) + + line += " " + line += display_path + + print(line) + + +def tree_dict_A_minus_B( + A: Dict[str, Dict[str, Any]] + ,B: Dict[str, Dict[str, Any]] +) -> Dict[str, Dict[str, Any]]: + """ + Compute the set difference of two tree_dicts at the key level: + + Result = A \\ B + + That is, return a new tree_dict containing only those entries whose + keys are present in A but NOT present in B. + """ + result: Dict[str, Dict[str, Any]] = {} + + B_keys = set(B.keys()) + + for key, info in A.items(): + if key not in B_keys: + result[key] = info + + if meta.debug_has("tree_dict_A_minus_B"): + tree_dict_print(result) + + return result + + +def tree_dict_in_between_and_below( + A: Dict[str, Dict[str, Any]] + ,B: Dict[str, Dict[str, Any]] +) -> tuple[Dict[str, Dict[str, Any]], Dict[str, Dict[str, Any]]]: + """ + Partition nodes of B into two topology-based sets relative to A: + + 1. in_between: + Nodes in B that lie under at least one directory node in A, + but do NOT lie under any leaf directory of A. + + 2. below: + Nodes in B that lie under at least one leaf directory of A. + + Definitions (relative to A's directory topology): + + - A directory node in A is any key whose info['node_type'] == 'directory'. + + - A leaf directory in A is a directory that has no *other* directory + in A as a proper descendant. The project root ('') is therefore + never a leaf (it always has descendant directories if the tree is + non-trivial). + + - “Lies under”: + * For a path p in B, we look at the chain of directory ancestors + (including the root "") and, if p itself is a directory, p + itself. Any of those that appear as directory keys in A are + considered directory ancestors in A. + + * If any of those ancestors is a leaf in A, p goes to 'below'. + Otherwise, if there is at least one directory ancestor in A, + p goes to 'in_between'. + + - Nodes in B that do not lie under any directory in A are ignored. + + Returns: + (in_between_dict, below_dict), both keyed like B and containing + copies of the info dicts from B. + """ + # 1. Collect all directory keys from A + A_dir_keys: Set[str] = set( + key for key, info in A.items() + if info.get("node_type") == "directory" + ) + + # 2. Compute leaf directories in A + leaf_dirs: Set[str] = set() + + for d in A_dir_keys: + if d == "": + continue + + has_child_dir = False + prefix = d + os.sep + + for other in A_dir_keys: + if other == d: + continue + if other.startswith(prefix): + has_child_dir = True + break + + if not has_child_dir: + leaf_dirs.add(d) + + in_between: Dict[str, Dict[str, Any]] = {} + below: Dict[str, Dict[str, Any]] = {} + + for key, info in B.items(): + # Skip B's root + if key in ("", "."): + continue + + parts = key.split(os.sep) + + # Build directory ancestor chain + node_is_dir = (info.get("node_type") == "directory") + + ancestors: List[str] = [""] + prefix = None + + if node_is_dir: + upto = parts + else: + upto = parts[:-1] + + for part in upto: + if prefix is None: + prefix = part + else: + prefix = os.path.join(prefix, part) + ancestors.append(prefix) + + # Filter ancestors to those that exist as directories in A + ancestors_in_A = [d for d in ancestors if d in A_dir_keys] + + if not ancestors_in_A: + # This B node is not under any directory from A; ignore it. + continue + + # Any leaf ancestor in A? + has_leaf_ancestor = any(d in leaf_dirs for d in ancestors_in_A) + + if has_leaf_ancestor: + below[key] = info + else: + in_between[key] = info + + if meta.debug_has("tree_dict_in_between_and_below"): + merged: Dict[str, Dict[str, Any]] = {} + merged.update(in_between) + merged.update(below) + tree_dict_print(merged) + + return in_between, below + + +def tree_dict_newer( + A: Dict[str, Dict[str, Any]] + ,B: Dict[str, Dict[str, Any]] +) -> Dict[str, Dict[str, Any]]: + """ + Return a dictionary of nodes from B that are newer than their + corresponding nodes in A. + + For each key k: + + - If k exists in both A and B, and + - B[k]['mtime'] > A[k]['mtime'], + + then k is included in the result with value B[k]. + + Keys that are only in B (not in A) are ignored here. + """ + result: Dict[str, Dict[str, Any]] = {} + + for key, info_B in B.items(): + info_A = A.get(key) + if info_A is None: + continue + + mtime_A = info_A.get("mtime") + mtime_B = info_B.get("mtime") + + if mtime_A is None or mtime_B is None: + continue + + if mtime_B > mtime_A: + result[key] = info_B + + if meta.debug_has("tree_dict_newer"): + tree_dict_print(result) + + return result + + +def tree_dict_older( + A: Dict[str, Dict[str, Dict[str, Any]]] + ,B: Dict[str, Dict[str, Dict[str, Any]]] +) -> Dict[str, Dict[str, Any]]: + """ + Return a dictionary of nodes from B that are older than their + corresponding nodes in A. + + For each key k: + + - If k exists in both A and B, and + - B[k]['mtime'] < A[k]['mtime'], + + then k is included in the result with value B[k]. + + Keys that are only in B (not in A) are ignored here. + """ + result: Dict[str, Dict[str, Any]] = {} + + for key, info_B in B.items(): + info_A = A.get(key) + if info_A is None: + continue + + mtime_A = info_A.get("mtime") + mtime_B = info_B.get("mtime") + + if mtime_A is None or mtime_B is None: + continue + + if mtime_B < mtime_A: + result[key] = info_B + + if meta.debug_has("tree_dict_older"): + tree_dict_print(result) + + return result + +def in_between_newer( + A: TreeDict + ,B: TreeDict +) -> TreeDict: + """ + in_between_newer(A, B) -> TreeDict + + Return the subset of B's nodes that: + + 1. Are in the 'in_between' region with respect to A's topology: + - under some directory that exists in A + - NOT under any leaf directory in A + (as defined by tree_dict_in_between_and_below), and + + 2. For file nodes: + - are "newer" than A at the same path, or + - are absent from A. + + More precisely: + - If A has no entry for that path -> include. + - If A has a non-file and B has a file -> include. + - If both are files and B.mtime > A.mtime -> include. + + 3. For constrained nodes: + - are always included, so that higher-level commands (e.g. + 'import') can surface them as "not handled automatically". + + Notes: + - Only file nodes participate in mtime comparisons. + - Nodes with node_type == 'constrained' are passed through without + mtime checks, so that callers can report them separately. + """ + in_between, _below = tree_dict_in_between_and_below(A, B) + + result: TreeDict = {} + + # Keep track of directories already included in the result + included_dirs: Set[str] = set() + + # Sort keys to ensure parent directories are processed before their children. + # This is crucial for the child exclusion logic to work correctly. + sorted_paths = sorted(in_between.keys(), key=len) + + for path in sorted_paths: + b_info = in_between[path] + b_type = b_info.get("node_type") + + # Constrained nodes: always surface so the caller can list them + # under "not handled automatically". + # Check if this path is a child of an already included directory + is_child_of_included_dir = False + for d in included_dirs: + if path.startswith(d + os.sep): + is_child_of_included_dir = True + break + + if is_child_of_included_dir: + continue + + # Constrained nodes: always surface so the caller can list them + # under "not handled automatically". + if b_type == "constrained": + result[path] = b_info + if b_type == "directory": + included_dirs.add(path) + continue + + + + b_mtime = b_info.get("mtime") + a_info = A.get(path) + + # Case 1: path not in A at all -> include (new file/dir in in-between) + if a_info is None: + result[path] = b_info + if b_type == "directory": + included_dirs.add(path) + continue + + # We only do "newer" semantics for regular files. + if b_type != "file": + continue + + a_type = a_info.get("node_type") + + # Case 2: A has non-file, B has file -> include + if a_type != "file": + result[path] = b_info + # Note: b_type must be "file" here due to the check above, so no need + # to check for directory inclusion. + continue + + # Case 3: both are files; compare mtime + a_mtime = a_info.get("mtime") + if ( + isinstance(a_mtime, (int, float)) + and isinstance(b_mtime, (int, float)) + and b_mtime > a_mtime + ): + result[path] = b_info + # Note: b_type must be "file" here, so no need to check for directory inclusion. + + if meta.debug_has("in_between_newer"): + tree_dict_print(result) + + return result + diff --git a/shared/authored/version b/shared/authored/version new file mode 100644 index 0000000..9e30ada --- /dev/null +++ b/shared/authored/version @@ -0,0 +1,2 @@ +Harmony v1.0 2025-11-25 + diff --git a/shared/document/.gitkeep b/shared/document/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/shared/document/install_Python.org b/shared/document/install_Python.org new file mode 100644 index 0000000..14342f9 --- /dev/null +++ b/shared/document/install_Python.org @@ -0,0 +1,75 @@ +#+TITLE: Installing Python in Harmony +#+AUTHOR: Thomas Walker Lynch +#+OPTIONS: toc:2 num:nil + +* Overview + +This document describes how to install a project-local Python environment under: + +#+begin_src bash +shared/third_party/Python +#+end_src + +This environment is shared across the =developer= and =tester= roles and is automatically activated through their respective =env_= scripts. + +* Precondition + +Ensure the following: + +- You are in a POSIX shell with =python3= installed. +- The =python3-venv= package is available (on Debian: =sudo apt install python3-venv=). +- You have sourced the Harmony environment via =env_toolsmith= to initialize =REPO_HOME= and related variables. + +* Step-by-Step Installation + +1. Source the Harmony environment: + #+begin_src bash + source env_toolsmith + #+end_src + +2. Create the virtual environment: + #+begin_src bash + python3 -m venv "$REPO_HOME/shared/third_party/Python" + #+end_src + +3. Activate it temporarily to install required packages: + #+begin_src bash + source "$REPO_HOME/shared/third_party/Python/bin/activate" + pip install --upgrade pip + pip install pytest # Add any shared packages here + deactivate + #+end_src + +4. Rename Python's default activate and deactivate: + Harmony provides its own role-aware environment management. Using Python’s default activation scripts may interfere with prompt logic, PATH order, and role-specific behavior. + + Disable the default scripts by renaming them: + #+begin_src bash + mv "$REPO_HOME/shared/third_party/Python/bin/activate" \ + "$REPO_HOME/shared/third_party/Python/bin/activate_deprecated" + #+end_src + + This ensures that accidental sourcing of Python’s =activate= script won't override Harmony's environment setup. + +5. Verify installation: + #+begin_src bash + ls "$REPO_HOME/shared/third_party/Python/bin/python3" + #+end_src + + The binary should exist and report a working Python interpreter when run. + +* Notes + +- The virtual environment is deliberately named =Python=, not =venv=, to reflect its role as a shared system component. +- Harmony environment scripts define and control =VIRTUAL_ENV=, =PYTHON_HOME=, and =PATH=, making Python activation seamless and uniform. +- There is no need to use Python’s =bin/activate= directly — it is fully replaced by Harmony’s environment logic. + +* Related Files + +- =shared/authored/env= +- =shared/authored/env_source= +- =env_developer=, =env_tester=, =env_toolsmith= + +* Last Verified + +2025-05-19 :: Activate/deactivate renamed post-install. Requires Harmony environment sourcing prior to execution. diff --git a/shared/document/install_generic.org b/shared/document/install_generic.org new file mode 100644 index 0000000..90af13e --- /dev/null +++ b/shared/document/install_generic.org @@ -0,0 +1,81 @@ + +This is the generic install.org doc that comes with the skeleton. + +1. $REPO_HOME/shared/third_party/.gitignore: + + * + !/.gitignore + !/patch + + The only things from the third party directory that will be pushed to the repo origin is the .gitignore file and the patches. + + +2. downloaded tar files etc. go into the directory `upstream` + + $REPO_HOME/shared/upstream + + Typically the contents of upstream are deleted after the install. + +3. for the base install + + cd $REPO_HOME/shared/third_party + do whatever it takes to install tool, as examples: + git clone + tar -xzf ../upstream/tar + ... + + Be sure to add the path to the tool executable(s) in the $REPO_HOME/env_$ROLE files for the $ROLE who uses the tool. + + Assuming you are not also developing the tool, for safety + change each installed git project to a local branch: + + b=__local_$USER + git switch -c "$b" + + +4. Define some variables to simplify our discussion. Lowercase variable names + are not exported from the shell. + + # already set in the environment + # REPO_HOME + # PROJECT + # USER + + # example tool names: 'RT_gcc' 'RT-project share` etc. + tool= + tool_dpath="$REPO_HOME/shared/third_party/$tool" + patch_dpath="$REPO_HOME/shared/patch/" + + +5. create a patch series (from current vendor state → your local edits) + + # this can be repeated and will create an encompassing diff file + + # optionally crate a new branch after cloning the third party tool repo and work from there. You won't make any commits, but in case you plan to ever check the changes in, or have a the bad habit of doing ommits burned into your brain-stem, making a brnch will help. + + # make changes + + cd "$tool_dpath" + + # do your edits + + # Stage edits. Do not commit them!! Be sure you are in the third party + # tool directory when doing `git add -A` and `git diff` commands. + git add -A + + # diff the stage from the current repo to create the patch file + git diff --staged > "$patch_dpath/$tool" + + # the diff file can be added to the project and checked in at the project level. + + +6. how to apply an existing patch + + Get a fresh clone of the tool into $tool_dpath. + + cd "$tool_dpath" + git apply "$patch_dpath/$tool" + + You can see what `git apply` would do by running + + git apply --check /path/to/your/patch_dpath/$tool diff --git a/shared/third_party/.gitignore b/shared/third_party/.gitignore new file mode 100644 index 0000000..0de97f0 --- /dev/null +++ b/shared/third_party/.gitignore @@ -0,0 +1,8 @@ +# Ignore all files +* + +# But don't ignore the .gitignore file itself +!/.gitignore + +# keep the upstream directory +!/upstream diff --git a/shared/third_party/upstream/.gitignore b/shared/third_party/upstream/.gitignore new file mode 100644 index 0000000..aa0e8eb --- /dev/null +++ b/shared/third_party/upstream/.gitignore @@ -0,0 +1,2 @@ +* +!/.gitignore \ No newline at end of file diff --git a/tool/after_pull b/tool/after_pull index 3511c89..946d9ad 100755 --- a/tool/after_pull +++ b/tool/after_pull @@ -21,7 +21,7 @@ notes: import os, sys, stat -# Must match tool_shared/bespoke/env policy: +# Must match shared/authored/env policy: DEFAULT_UMASK = 0o077 # reminder only; effective modes below implement 077 congruence. DIR_MODE_077 = 0o700 @@ -43,7 +43,7 @@ def require_toolsmith_env(): def repo_home(): rh = os.environ.get("REPO_HOME") if not rh: - die("REPO_HOME is not set (did you source tool_shared/bespoke/env?)") + die("REPO_HOME is not set (did you source shared/authored/env?)") return os.path.realpath(rh) def show_path(p, rh): diff --git a/tool/git-tar b/tool/git-tar index 79d5a8c..d6ba1f7 100755 --- a/tool/git-tar +++ b/tool/git-tar @@ -96,9 +96,9 @@ def _import_Z_module(repo_top: pathlib.Path) -> Optional[object]: here = pathlib.Path(__file__).resolve().parent candidates += [here / "Z", here / "Z.py"] candidates += [ - repo_top / "tool_shared" / "third_party" / "RT-project-share" / "release" / "python" / "Z", - repo_top / "tool_shared" / "third_party" / "RT-project-share" / "release" / "python" / "Z.py", - repo_top / "tool_shared" / "third_party" / "RT-project-share" / "release" / "bash" / "Z", + repo_top / "shared" / "third_party" / "RT-project-share" / "release" / "python" / "Z", + repo_top / "shared" / "third_party" / "RT-project-share" / "release" / "python" / "Z.py", + repo_top / "shared" / "third_party" / "RT-project-share" / "release" / "bash" / "Z", ] for d in (pathlib.Path(p) for p in (os.getenv("PATH") or "").split(os.pathsep) if p): p = d / "Z" diff --git a/tool_shared/authored/env b/tool_shared/authored/env deleted file mode 100644 index 0d47fca..0000000 --- a/tool_shared/authored/env +++ /dev/null @@ -1,130 +0,0 @@ -#!/usr/bin/env bash -script_afp=$(realpath "${BASH_SOURCE[0]}") -if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then - echo "$script_afp:: This script must be sourced, not executed." - exit 1 -fi - -# without this bash takes non-matching globs literally -shopt -s nullglob - -# does not presume sharing or world permissions -umask 0077 - -# -------------------------------------------------------------------------------- -# project definition - -# actual absolute director path for this script file - - script_adp(){ - dirname "$script_afp" - } - -# assume this script is located $REPO_HOME/tools_shared/bespoke and work backwards -# to get $REPO_HOME, etc. - - REPO_HOME=$(dirname "$(dirname "$(script_adp)")") - echo REPO_HOME "$REPO_HOME" - - PROJECT=$(basename "$REPO_HOME") - echo PROJECT "$PROJECT" - - # set the prompt decoration to the name of the project - PROMPT_DECOR=$PROJECT - - export REPO_HOME PROJECT PROMPT_DECOR - -# -------------------------------------------------------------------------------- -# Project wide Tool setup -# - -export VIRTUAL_ENV="$REPO_HOME/tool_shared/third_party/Python" -export PYTHON_HOME="$VIRTUAL_ENV" -unset PYTHONHOME - - -# -------------------------------------------------------------------------------- -# PATH -# precedence: last defined, first discovered - - PATH="$REPO_HOME/tool_shared/third_party/RT-project-share/release/bash:$PATH" - PATH="$REPO_HOME/tool_shared/third_party/RT-project-share/release/amd64:$PATH" - PATH="$REPO_HOME/tool_shared/third_party:$PATH" - PATH="$REPO_HOME/tool_shared/customized:$PATH" - PATH="$REPO_HOME"/tool_shared/bespoke:"$PATH" - - # Remove duplicates - clean_path() { - PATH=$(echo ":$PATH" | awk -v RS=: -v ORS=: '!seen[$0]++' | sed 's/^://; s/:$//') - } - clean_path - export PATH - -# -------------------------------------------------------------------------------- -# the following functions are provided for other scripts to use. -# at the top of files that make use of these functions put the following line: -# script_afp=$(realpath "${BASH_SOURCE[0]}") -# - - ## script's filename - script_fn(){ - basename "$script_afp" - } - - ## script's dirpath relative to $REPO_HOME - script_fp(){ - realpath --relative-to="${REPO_HOME}" "$script_afp" - } - - ## script's dirpath relative to $REPO_HOME - script_dp(){ - dirname "$(script_fp)" - } - - export -f script_adp script_fn script_dp script_fp - -#-------------------------------------------------------------------------------- -# used by release scripts -# - - install_file() { - if [ "$#" -lt 3 ]; then - echo "env::install_file usage: install_file ... " - return 1 - fi - - perms="${@: -1}" # Last argument is permissions - target_dp="${@: -2:1}" # Second-to-last argument is the target directory - sources=("${@:1:$#-2}") # All other arguments are source files - - if [ ! -d "$target_dp" ]; then - echo "env::install_file no install done: target directory '$target_dp' does not exist." - return 1 - fi - - for source_fp in "${sources[@]}"; do - if [ ! -f "$source_fp" ]; then - echo "env::install_file: source file '$source_fp' does not exist." - return 1 - fi - - target_file="$target_dp/$(basename "$source_fp")" - - if ! install -m "$perms" "$source_fp" "$target_file"; then - echo "env::install_file: Failed to install $(basename "$source_fp") to $target_dp" - return 1 - else - echo "env::install_file: installed $(basename "$source_fp") to $target_dp with permissions $perms" - fi - done - } - - export -f install_file - -# -------------------------------------------------------------------------------- -# closing -# - if [[ -z "$ENV" ]]; then - export ENV=$(script_fp) - fi - diff --git a/tool_shared/authored/git-empty-dir/CLI.py b/tool_shared/authored/git-empty-dir/CLI.py deleted file mode 100755 index 2fb22e1..0000000 --- a/tool_shared/authored/git-empty-dir/CLI.py +++ /dev/null @@ -1,251 +0,0 @@ -#!/usr/bin/env python3 -# ---------------------------------------------------------------------- -# git-empty-dir :: list/mark/clean empty directories, .gitignore aware -# ---------------------------------------------------------------------- - -import sys -import os -from pathlib import Path - -# The source_sync GitIgnore parser is inside the unpacked tool. -# We assume this directory structure: -# git-empty-dir/ -# CLI.py -# source_sync/ -# GitIgnore.py -# -# That mirrors how your harmony sync tool is structured. - -# Adjust import path so we can load source_sync.* -HERE = Path(__file__).resolve().parent -sys.path.insert(0, str(HERE)) - -from source_sync.GitIgnore import GitIgnore # type: ignore - - -# ---------------------------------------------------------------------- -# helpers -# ---------------------------------------------------------------------- - -def load_gitignore_tree(root: Path): - """ - Build a GitIgnore instance rooted at . - """ - return GitIgnore(str(root)) - -def is_empty_dir(path: Path) -> bool: - """ - A directory is empty if it contains no files or subdirectories. - (Hidden files count; .gitignored children are irrelevant because - behavior here should reflect real filesystem emptiness.) - """ - try: - for _ in path.iterdir(): - return False - return True - except PermissionError: - # treat as non-empty: safer than aborting - return False - - -def has_mark(path: Path, mark_file: str) -> bool: - return (path / mark_file).exists() - - -def sorted_dirs(root: Path): - """ - Produce a list of all directories under root, in parent-before-child order. - Sort rule: - 1. by path length - 2. then lexicographically - """ - all_dirs = [] - for p in root.rglob("*"): - if p.is_dir(): - all_dirs.append(p) - - return sorted( - all_dirs - ,key = lambda p: (len(p.parts), str(p)) - ) - - -# ---------------------------------------------------------------------- -# traversal -# ---------------------------------------------------------------------- - -def visible_dirs(root: Path, ignore_tree, mark_file: str): - """ - Yield all dirs under root, applying: - - skip .git - - apply .gitignore rules (if a dir is ignored, do not descend) - - parent-before-child ordering - """ - for d in sorted_dirs(root): - rel = d.relative_to(root) - - if rel == Path("."): - continue - - # skip .git explicitly - if d.name == ".git": - continue - - # .gitignore filtering - if ignore_tree.check(str(rel)) == "Ignore": - continue - - yield d - - -# ---------------------------------------------------------------------- -# actions -# ---------------------------------------------------------------------- - -def action_list(root, ignore_tree, mark_file, mode): - """ - mode ∈ {"empty","marked","all"} - """ - for d in visible_dirs(root, ignore_tree, mark_file): - if mode == "all": - print(d.relative_to(root)) - continue - - if mode == "marked": - if has_mark(d, mark_file): - print(d.relative_to(root)) - continue - - if mode == "empty": - if is_empty_dir(d): - print(d.relative_to(root)) - continue - - -def action_mark(root, ignore_tree, mark_file, mode): - """ - mode ∈ {"empty","all"} - """ - for d in visible_dirs(root, ignore_tree, mark_file): - if mode == "empty" and not is_empty_dir(d): - continue - try: - (d / mark_file).touch(exist_ok=True) - except Exception: - pass - - -def action_clean(root, ignore_tree, mark_file, mode): - """ - mode ∈ {"nonempty","all"} - """ - for d in visible_dirs(root, ignore_tree, mark_file): - m = d / mark_file - if not m.exists(): - continue - - if mode == "nonempty": - if is_empty_dir(d): - continue - - try: - m.unlink() - except Exception: - pass - - -# ---------------------------------------------------------------------- -# usage -# ---------------------------------------------------------------------- - -USAGE = """ -usage: - git-empty-dir (list|mark|clean) [all|marked|empty] [file-] - git-empty-dir help - git-empty-dir usage - -defaults: - mark-file = .gitkeep - ignores .git - follows .gitignore (no descent into ignored dirs) - -examples: - git-empty-dir list - git-empty-dir list marked file-.githolder - git-empty-dir mark - git-empty-dir clean all -""" - - -# ---------------------------------------------------------------------- -# CLI -# ---------------------------------------------------------------------- - -def CLI(argv): - if len(argv) == 0: - print(USAGE) - return 0 - - cmd = argv[0] - - if cmd in ("help","usage"): - print(USAGE) - return 0 - - # command - if cmd not in ("list","mark","clean"): - print(f"unknown command: {cmd}") - print(USAGE) - return 1 - - # submode - mode = None - mark_file = ".gitkeep" - - for a in argv[1:]: - if a.startswith("file-"): - mark_file = a[5:] - continue - - if a in ("all","empty","marked"): - mode = a - continue - - print(f"unknown argument: {a}") - print(USAGE) - return 1 - - # defaults - if cmd == "list": - if mode is None: - mode = "empty" - elif cmd == "mark": - if mode is None: - mode = "empty" - elif cmd == "clean": - if mode is None: - mode = "nonempty" - - root = Path(".").resolve() - ignore_tree = load_gitignore_tree(root) - - if cmd == "list": - action_list(root, ignore_tree, mark_file, mode) - - elif cmd == "mark": - if mode == "all": - action_mark(root, ignore_tree, mark_file, "all") - else: - action_mark(root, ignore_tree, mark_file, "empty") - - elif cmd == "clean": - if mode == "all": - action_clean(root, ignore_tree, mark_file, "all") - else: - action_clean(root, ignore_tree, mark_file, "nonempty") - - return 0 - - -if __name__ == "__main__": - sys.exit(CLI(sys.argv[1:])) diff --git a/tool_shared/authored/git-empty-dir/Harmony.py b/tool_shared/authored/git-empty-dir/Harmony.py deleted file mode 120000 index 112663e..0000000 --- a/tool_shared/authored/git-empty-dir/Harmony.py +++ /dev/null @@ -1 +0,0 @@ -../source_sync/Harmony.py \ No newline at end of file diff --git a/tool_shared/authored/git-empty-dir/load_command_module.py b/tool_shared/authored/git-empty-dir/load_command_module.py deleted file mode 120000 index 87b98be..0000000 --- a/tool_shared/authored/git-empty-dir/load_command_module.py +++ /dev/null @@ -1 +0,0 @@ -../source_sync/load_command_module.py \ No newline at end of file diff --git a/tool_shared/authored/git-empty-dir/meta.py b/tool_shared/authored/git-empty-dir/meta.py deleted file mode 100644 index dee6439..0000000 --- a/tool_shared/authored/git-empty-dir/meta.py +++ /dev/null @@ -1,97 +0,0 @@ -#!/usr/bin/env python3 -# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- - -""" -meta.py - thin wrappers around command modules - -Current responsibilities: - 1. Load the incommon 'printenv' command module (no .py extension) - using load_command_module.load_command_module(). - 2. Expose printenv() here, calling the imported printenv() work - function with default arguments (equivalent to running without - any CLI arguments). - 3. Provide a simple version printer for this meta module. - 4. Provide a small debug tag API (set/clear/has). -""" - -from __future__ import annotations - -import datetime -from load_command_module import load_command_module - - -# Load the incommon printenv module once at import time -_PRINTENV_MODULE = load_command_module("printenv") -_Z_MODULE = load_command_module("Z") - - -# Meta module version -_major = 1 -_minor = 1 -def version_print() -> None: - """ - Print the meta module version as MAJOR.MINOR. - """ - print(f"{_major}.{_minor}") - - -# Debug tag set and helpers -_debug = set([ -]) - - -def debug_set(tag: str) -> None: - """ - Add a debug tag to the meta debug set. - """ - _debug.add(tag) - - -def debug_clear(tag: str) -> None: - """ - Remove a debug tag from the meta debug set, if present. - """ - _debug.discard(tag) - - -def debug_has(tag: str) -> bool: - """ - Return True if the given debug tag is present. - """ - return tag in _debug - - -# Touch the default tag once so static checkers do not complain about -# unused helpers when imported purely for side-effects. -debug_has("Command") - - -def printenv() -> int: - """ - Call the imported printenv() work function with default arguments: - - no null termination - - no newline quoting - - no specific names (print full environment) - - prog name 'printenv' - """ - return _PRINTENV_MODULE.printenv( - False # null_terminate - ,False # quote_newlines - ,[] # names - ,"printenv" - ) - - -def z_format_mtime( - mtime: float -) -> str: - """ - Format a POSIX mtime (seconds since epoch, UTC) using the Z module. - - Uses Z.ISO8601_FORMAT and Z.make_timestamp(dt=...). - """ - dt = datetime.datetime.fromtimestamp(mtime, datetime.timezone.utc) - return _Z_MODULE.make_timestamp( - fmt=_Z_MODULE.ISO8601_FORMAT - ,dt=dt - ) diff --git a/tool_shared/authored/git-empty-dir/source_sync b/tool_shared/authored/git-empty-dir/source_sync deleted file mode 120000 index 9fd1d51..0000000 --- a/tool_shared/authored/git-empty-dir/source_sync +++ /dev/null @@ -1 +0,0 @@ -../source_sync/ \ No newline at end of file diff --git a/tool_shared/authored/sync/A_minus_B b/tool_shared/authored/sync/A_minus_B deleted file mode 100755 index f6f7bbb..0000000 --- a/tool_shared/authored/sync/A_minus_B +++ /dev/null @@ -1,53 +0,0 @@ -#!/usr/bin/env python3 -# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- - -""" -A_minus_B - CLI test driver for skeleton.tree_dict_A_minus_B(A, B) - -Usage: - A_minus_B -""" - -from __future__ import annotations - -import os -import sys -from typing import Sequence - -import meta -import skeleton - - -def CLI(argv: Sequence[str] | None = None) -> int: - if argv is None: - argv = sys.argv[1:] - - prog = os.path.basename(sys.argv[0]) if sys.argv else "A_minus_B" - - if len(argv) != 2 or argv[0] in ("-h", "--help"): - print(f"Usage: {prog} ") - return 1 - - A_root = argv[0] - B_root = argv[1] - - if not os.path.isdir(A_root): - print(f"{prog}: {A_root}: not a directory") - return 2 - - if not os.path.isdir(B_root): - print(f"{prog}: {B_root}: not a directory") - return 3 - - A = skeleton.tree_dict_make(A_root, None) - B = skeleton.tree_dict_make(B_root, None) - - meta.debug_set("tree_dict_A_minus_B") - - _result = skeleton.tree_dict_A_minus_B(A, B) - - return 0 - - -if __name__ == "__main__": - raise SystemExit(CLI()) diff --git a/tool_shared/authored/sync/CLI.py b/tool_shared/authored/sync/CLI.py deleted file mode 100755 index f7fb0b0..0000000 --- a/tool_shared/authored/sync/CLI.py +++ /dev/null @@ -1,282 +0,0 @@ -#!/usr/bin/env python3 -# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- - -""" -CLI.py - Harmony skeleton checker - -Grammar (informal): - - check * [] - - :: | | - - :: version | help | usage - :: environment - :: structure | import | export | suspicious | addendum | all - -Commands are sorted into three sets: - 1. HELP_COMMANDS - 2. NO_OTHER_COMMANDS - 3. HAS_OTHER_COMMANDS - -At runtime, argv commands are classified into four lists: - 1. help_list - 2. no_other_list - 3. has_other_list - 4. unclassified_list - -If the meta debug set contains the tag "print_command_lists", these four lists -are printed. - -If 'environment' appears in no_other_list, the meta.printenv() helper -is invoked to print the environment. - -For commands we compare: - - A = Harmony skeleton tree_dict - B = project tree_dict (path is the last argv token when any - is present before it). -""" - -from __future__ import annotations - -import os -import sys -from typing import Sequence - -import command -import doc -import Harmony -import meta -import skeleton - -# meta.debug_set("print_command_lists") - -# Command tag sets (classification universe) -HELP_COMMANDS: set[str] = set([ - "version" - ,"help" - ,"usage" -]) - -NO_OTHER_COMMANDS: set[str] = set([ - "environment" -]) - -HAS_OTHER_COMMANDS: set[str] = set([ - "structure" - ,"import" - ,"export" - ,"suspicious" - ,"addendum" - ,"all" -]) - - -def command_type(arg: str) -> str: - """ - Classify a single command token. - - Returns: - "Help" if arg is a help command - "NoOther" if arg is a no_other command - "HasOther" if arg is a has_other command - "UnClassified" otherwise - """ - if arg in HELP_COMMANDS: - return "Help" - - if arg in NO_OTHER_COMMANDS: - return "NoOther" - - if arg in HAS_OTHER_COMMANDS: - return "HasOther" - - return "UnClassified" - - -def print_command_lists( - help_list: list[str] - ,no_other_list: list[str] - ,has_other_list: list[str] - ,unclassified_list: list[str] -) -> None: - """ - Print the four classified command lists derived from argv. - """ - print("help_list:", help_list) - print("no_other_list:", no_other_list) - print("has_other_list:", has_other_list) - print("unclassified_list:", unclassified_list) - - -def CLI(argv: Sequence[str] | None = None) -> int: - """ - CLI entrypoint. - - Responsibilities: - 1. Accept argv (or sys.argv[1:] by default). - 2. Classify arguments using command_type(), with the last argument - treated specially to avoid aliasing. - 3. Invoke behaviors implied by the commands. - 4. Return integer status code. - - Argument interpretation: - - Let argv = [a0, a1, ..., aN-1]. - - - If N == 0: - no commands; nothing to do. - - - If N >= 1: - * Classify a0..aN-2. - - If any are UnClassified -> error. - - * If any appear in a0..aN-2: - - aN-1 is treated as path (B_root), not classified. - - * If no appear in a0..aN-2: - - Classify aN-1: - - If UnClassified -> error (unknown command). - - If HasOther -> error (other path not specified). - - Else -> added to Help / NoOther lists. - """ - if argv is None: - argv = sys.argv[1:] - - # No arguments: print usage and exit with status 1. - if len(argv) == 0: - doc.print_usage() - return 1 - - # No arguments: nothing to do (could later decide to print usage). - if len(argv) == 0: - return 0 - - # Split into head (all but last) and last argument - head = argv[:-1] - last = argv[-1] - - help_list: list[str] = [] - no_other_list: list[str] = [] - has_other_list: list[str] = [] - unclassified_list: list[str] = [] - - # 1. Classify head tokens - for arg in head: - ct = command_type(arg) - - if ct == "Help": - help_list.append(arg) - elif ct == "NoOther": - no_other_list.append(arg) - elif ct == "HasOther": - has_other_list.append(arg) - else: - unclassified_list.append(arg) - - # Any unclassified in the head is an error - if len(unclassified_list) > 0: - first_bad = unclassified_list[0] - print(f"Unrecognized command: {first_bad}") - return 5 - - head_has_other = (len(has_other_list) > 0) - - B_root: str | None = None - - if head_has_other: - # 2A. Any in head -> last arg is always path. - B_root = os.path.abspath(last) - else: - # 2B. No in head -> classify last. - ct = command_type(last) - - if ct == "UnClassified": - print(f"Unrecognized command: {last}") - return 5 - - if ct == "HasOther": - print("Other path not specified for has_other command(s).") - return 6 - - if ct == "Help": - help_list.append(last) - elif ct == "NoOther": - no_other_list.append(last) - # ct cannot be HasOther here due to earlier check. - - if meta.debug_has("print_command_lists"): - print_command_lists( - help_list - ,no_other_list - ,has_other_list - ,unclassified_list - ) - - # Help handling - if len(help_list) > 0: - if "version" in help_list: - meta.version_print() - if "usage" in help_list: - doc.print_usage() - if "help" in help_list: - doc.print_help() - return 1 - - ret_val = 0 - - # No-other commands (environment, etc.) - if "environment" in no_other_list: - env_status = meta.printenv() - if env_status != 0: - ret_val = env_status - - # If we still have no has_other commands, we are done. - # (Example: just "environment", or just "help/usage".) - if len(has_other_list) == 0: - return ret_val - - # At this point we know: - # - has_other_list is non-empty - # - B_root must have been set (head_has_other was True) - if B_root is None: - print("Internal error: B_root not set despite has_other commands.") - return 7 - - if not os.path.isdir(B_root): - print(f"Other project path is not a directory: {B_root}") - return 4 - - # Determine Harmony root (A_root) - status, A_root = Harmony.where() - - if status == "not-found": - print("Harmony project not found; normally this command is run from within Harmony.") - return 3 - - if status == "different": - print("Seems we are not running in the Harmony project, will exit.") - return 2 - - # Build tree_dicts for A (Harmony) and B (other project) - A_tree = skeleton.tree_dict_make(A_root, None) - B_tree = skeleton.tree_dict_make(B_root, None) - - # Dispatch the commands - cmd_status = command.dispatch( - has_other_list - ,A_tree - ,B_tree - ,A_root - ,B_root - ) - - if cmd_status != 0: - ret_val = cmd_status - - return ret_val - - -if __name__ == "__main__": - raise SystemExit(CLI()) diff --git a/tool_shared/authored/sync/GitIgnore.py b/tool_shared/authored/sync/GitIgnore.py deleted file mode 100755 index 70c6509..0000000 --- a/tool_shared/authored/sync/GitIgnore.py +++ /dev/null @@ -1,270 +0,0 @@ -#!/usr/bin/env python3 -# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- - -""" -GitIgnore.py - minimal .gitignore-based helper for Harmony projects - -Behavior: - - 1. During initialization, traverse the project tree rooted at - . - - 2. Whenever a directory contains a '.gitignore' file, record: - - its relative directory path from the project root - - a list of regular expressions compiled from the patterns - in that '.gitignore' file - - These are stored in: - - self.rules: Dict[str, List[Pattern]] - - where the key is the directory RELATIVE to the project root: - "" -> project root (top-level .gitignore) - "src" -> src/.gitignore - "src/module" -> src/module/.gitignore - - 3. check() -> token: - - - is a path relative to the project root. - - - We compute all prefix directories of , including the - root (""), for example: - - path = "a/b/c.py" - prefixes = ["", "a", "a/b"] - - - For each prefix, if there are regexes stored for that directory, - we collect them. - - - We then test ALL collected regexes against the basename of - (the last component only). - - - If ANY regex matches, return 'Ignore'. - Otherwise return 'Accept'. - -Notes: - - * We implement a simplified subset of .gitignore semantics suitable - for your current patterns and add a small base ignore set for - always-ignored names such as '.git'. -""" - -from __future__ import annotations - -import fnmatch -import os -import re -from typing import Dict, List -import Harmony - - -class GitIgnore: - """ - GitIgnore(project_path) - - Attributes: - project_path: - Absolute path to the project root. - - rules: - Mapping from relative directory path -> list of compiled regex - patterns derived from that directory's '.gitignore' file. - - Example: - rules[""] -> patterns from /.gitignore - rules["developer"] -> patterns from developer/.gitignore - - base_patterns: - List of compiled regex patterns applied to the basename of every - checked path, independent of any .gitignore file. Currently used - to always ignore '.git' directories. - """ - - def __init__( - self - ,project_path: str - ) -> None: - """ - Initialize a GitIgnore instance with a path to a project and - scan for '.gitignore' files. - """ - self.project_path: str = os.path.abspath(project_path) - self.rules: Dict[str, List[re.Pattern]] = {} - - # Base patterns: always applied, regardless of .gitignore contents. - # These are matched against basenames only. - self.base_patterns: List[re.Pattern] = [ - re.compile(r"^\.git$") # ignore any basename == ".git" - ] - - self._scan_project() - - def _scan_project(self) -> None: - """ - Traverse the project tree and populate self.rules with entries of - the form: - - -> [Pattern, Pattern, ...] - - where is the directory containing '.gitignore', relative - to the project root ("" for root). - """ - root = self.project_path - - for dirpath, dirnames, filenames in os.walk(root, topdown=True): - if ".gitignore" not in filenames: - continue - - rel_dir = os.path.relpath(dirpath, root) - if rel_dir == ".": - rel_dir = "" - - gitignore_path = os.path.join(dirpath, ".gitignore") - patterns = self._parse_gitignore_file(gitignore_path) - - if patterns: - if rel_dir not in self.rules: - self.rules[rel_dir] = [] - self.rules[rel_dir].extend(patterns) - - def _parse_gitignore_file( - self - ,gitignore_path: str - ) -> List[re.Pattern]: - """ - Parse a single '.gitignore' file into a list of compiled regex patterns. - - Simplified rules: - - Blank lines and lines starting with '#' are ignored. - - Lines containing '/' in the MIDDLE are currently ignored - (future extension). - - Lines ending with '/' are treated as directory name patterns: - '__pycache__/' -> pattern on basename '__pycache__' - - All patterns are treated as name globs and compiled via - fnmatch.translate(), to be matched against basenames only. - """ - patterns: List[re.Pattern] = [] - - try: - with open(gitignore_path, "r", encoding="utf-8") as f: - for raw_line in f: - line = raw_line.strip() - - # Skip comments and blank lines - if not line or line.startswith("#"): - continue - - # Remove trailing '/' for directory patterns (e.g. '__pycache__/') - if line.endswith("/"): - line = line[:-1].strip() - if not line: - continue - - # If there is still a '/' in the line, we do not support this - # pattern in this minimal implementation. - if "/" in line: - continue - - # Compile as a name glob -> regex - regex_text = fnmatch.translate(line) - patterns.append(re.compile(regex_text)) - - except OSError: - # If the .gitignore cannot be read, just skip it. - return patterns - - return patterns - - def check( - self - ,path: str - ) -> str: - """ - Check a path against the collected .gitignore patterns. - - path: - A path relative to the project root. - - Returns: - 'Ignore' if any applicable pattern matches the basename of the path, - otherwise 'Accept'. - """ - # Normalize the incoming path - norm = os.path.normpath(path) - - # If the path is '.' or empty, we accept it - if norm in ("", "."): - return "Accept" - - basename = os.path.basename(norm) - - # First, apply base patterns (always applied). - for pat in self.base_patterns: - if pat.match(basename): - return "Ignore" - - # Build the list of directories that may contribute .gitignore rules. - # - # For path "a/b/c": - # prefixes: ["", "a", "a/b"] - parts = norm.split(os.sep) - - prefixes: List[str] = [""] - prefix = None - for part in parts[:-1]: - if prefix is None: - prefix = part - else: - prefix = os.path.join(prefix, part) - prefixes.append(prefix) - - # Collect all patterns from the applicable .gitignore directories - for rel_dir in prefixes: - dir_patterns = self.rules.get(rel_dir) - if not dir_patterns: - continue - - for pat in dir_patterns: - if pat.match(basename): - return "Ignore" - - return "Accept" - - -def test_GitIgnore() -> int: - """ - 1. Locate the Harmony project root using Harmony.where(). - 2. Create a GitIgnore instance rooted at that path. - 3. Print: - - directories that have .gitignore rules - - directories (relative) that would be ignored by check() - """ - status, Harmony_root = Harmony.where() - - if status == "not-found": - print("Harmony project not found; cannot test GitIgnore.") - return 1 - - if status == "different": - print("Warning: Harmony not found, using nearest .git directory for GitIgnore test.") - - gi = GitIgnore(Harmony_root) - - print(".gitignore rule directories (relative to Harmony root):") - for rel_dir in sorted(gi.rules.keys()): - print(f" {rel_dir if rel_dir else '.'}") - - print("\nDirectories that would be ignored (relative to Harmony root):") - for dirpath, dirnames, filenames in os.walk(Harmony_root, topdown=True): - rel_dir = os.path.relpath(dirpath, Harmony_root) - if rel_dir == ".": - rel_dir = "" - - if gi.check(rel_dir) == "Ignore": - print(f" {rel_dir if rel_dir else '.'}") - - return 0 - - -if __name__ == "__main__": - raise SystemExit(test_GitIgnore()) diff --git a/tool_shared/authored/sync/Harmony.py b/tool_shared/authored/sync/Harmony.py deleted file mode 100644 index 9385507..0000000 --- a/tool_shared/authored/sync/Harmony.py +++ /dev/null @@ -1,133 +0,0 @@ -#!/usr/bin/env python3 -# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- - -""" -locate the project root -""" - -from __future__ import annotations - -import meta -import os -import sys -from typing import Any, Callable, Dict - -# where -# -# Context / assumptions: -# 1. This module lives somewhere under the Harmony tree, for example: -# /.../Harmony/tool/skeleton/skeleton.py -# 2. CLI.py is run from somewhere inside the same tree (or a clone). -# -# Search behavior: -# 1. Start from the directory containing this file. -# 2. Walk upward towards the filesystem root, with limits: -# a) Do not move up more than 5 levels. -# b) Stop immediately if the current directory contains a -# '.git' subdirectory. -# -# Result classification: -# status is one of: -# 'found' -> we found a directory whose basename is 'Harmony' -# 'different' -> we stopped at a directory that has a '.git' -# subdirectory, but its basename is not 'Harmony' -# 'not-found' -> we hit the 5-level limit or filesystem root -# without finding 'Harmony' or a '.git' directory -# -# Path: -# - In all cases, the returned path is the last directory inspected: -# * the 'Harmony' directory (status 'found'), or -# * the directory with '.git' (status 'different'), or -# * the directory at the 5-level limit / filesystem root -# (status 'not-found'). -# -# Debug printing: -# - If meta.debug_has("print_Harmony_root") is true, print: -# * "The Harmony project root found at: {path}" -# when status == 'found' -# * "Harmony not found, but found: {path}" -# when status == 'different' -# * "Harmony not found." -# when status == 'not-found' -def where() -> tuple[str, str]: - """ - Locate the Harmony root (or best guess). - - Returns: - (status, path) - """ - here = os.path.abspath(__file__) - d = os.path.dirname(here) - - harmony_root = None - status = "not-found" - - max_up = 5 - steps = 0 - - while True: - base = os.path.basename(d) - - # Case 1: exact 'Harmony' directory name - if base == "Harmony": - harmony_root = d - status = "found" - break - - # Case 2: stop at a directory that has a '.git' subdirectory - git_dir = os.path.join(d, ".git") - if os.path.isdir(git_dir): - harmony_root = d - if base == "Harmony": - status = "found" - else: - status = "different" - break - - parent = os.path.dirname(d) - - # Stop if we hit filesystem root - if parent == d: - harmony_root = d - status = "not-found" - break - - steps += 1 - if steps > max_up: - # Reached search depth limit; last inspected directory is d - harmony_root = d - status = "not-found" - break - - d = parent - - if harmony_root is None: - # Extremely defensive; in practice harmony_root will be set above. - harmony_root = d - - root_base = os.path.basename(harmony_root) - - # Warning to stderr if we are not literally in a 'Harmony' directory - if root_base != "Harmony": - sys.stderr.write( - f"WARNING: Harmony root basename is '{root_base}', expected 'Harmony'.\n" - ) - - if meta.debug_has("print_Harmony_root"): - if status == "found": - print(f"The Harmony project root found at: {harmony_root}") - elif status == "different": - print(f"Harmony not found, but found: {harmony_root}") - else: - print("Harmony not found.") - - return status, harmony_root - -def test_where() -> int: - """ - Simple test that prints the Harmony root using the debug flag. - """ - meta.debug_set("print_Harmony_root") - status, _root = where() - return 0 if status != "not-found" else 1 - diff --git a/tool_shared/authored/sync/Harmony_where b/tool_shared/authored/sync/Harmony_where deleted file mode 100755 index 9d39f1e..0000000 --- a/tool_shared/authored/sync/Harmony_where +++ /dev/null @@ -1,37 +0,0 @@ -#!/usr/bin/env python3 -# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- - -""" -Harmony_where - CLI to locate the Harmony project root - -Usage: - Harmony_where - -Prints the status and path returned by Harmony.where(). -""" - -from __future__ import annotations - -import sys - -import Harmony - - -def CLI(argv=None) -> int: - # Ignore argv; no arguments expected - status, Harmony_root = Harmony.where() - - if status == "found": - print(f"Harmony project root found at: {Harmony_root}") - return 0 - - if status == "different": - print(f"Harmony not found, but nearest .git directory is: {Harmony_root}") - return 1 - - print("Harmony project root not found.") - return 2 - - -if __name__ == "__main__": - raise SystemExit(CLI()) diff --git a/tool_shared/authored/sync/README.org b/tool_shared/authored/sync/README.org deleted file mode 100644 index 387780d..0000000 --- a/tool_shared/authored/sync/README.org +++ /dev/null @@ -1,278 +0,0 @@ -#+TITLE: skeleton_compare – Harmony skeleton comparison tool -#+AUTHOR: Reasoning Technology - -* 1. Overview - -1.1 -~skeleton_compare~ compares a Harmony skeleton (=A=) with a derived or legacy project (=B=). - -1.2 -It answers: - -- How has B diverged from A? -- What should be imported back into A? -- What should be exported from A into B? -- Which nodes are misplaced or suspicious? -- Which nodes represent valid project-specific extensions? - -1.3 -The entrypoint in this project is the symlink: - -- =tool/skeleton_compaare= - -which points to: - -- =tool/skeleton_compare_source/CLI.py= - -* 2. Role in the Harmony ecosystem - -2.1 -Harmony defines a skeleton layout (directories, leaves, extension points). - -2.2 -Projects are expected to: - -- start from that skeleton -- add work under approved extension points -- keep core structure aligned over time - -2.3 -Reality diverges: - -- legacy projects that predate Harmony -- projects with ad-hoc edits in skeleton areas -- skeleton evolution over months or years - -2.4 -~skeleton_compare~ provides: - -- a structural comparison -- a semantic comparison (types, topology) -- a chronological comparison (mtimes) -- actionable commands to re-align projects - -* 3. High-level behavior - -3.1 -Tree construction - -1. Build =tree_dict= for A (Harmony skeleton). -2. Build =tree_dict= for B (other project). -3. Attach metadata per relative path: - - - =node_type= :: =directory= | =file= | =other= | =constrained= - - =dir_info= :: =root= | =branch= | =leaf= | =NA= - - =mtime= :: float seconds since epoch - -3.2 -Git ignore - -1. A simplified =.gitignore= model is applied. -2. Some paths (e.g., =.git=) are always ignored. -3. Only paths admitted by this model participate in comparisons. - -3.3 -Topology classification (relative to A) - -1. =in_between= :: under a directory in A, but not under any leaf in A. -2. =below= :: under a leaf directory in A. -3. Neither :: not under any directory known to A (ignored for most commands). - -3.4 -Chronological classification - -1. newer(B,A) :: B node has a newer mtime than A at the same path. -2. older(B,A) :: B node has an older mtime than A at the same path. -3. A-only :: path exists in A but not B. -4. B-only :: path exists in B but not A. - -* 4. Command surface (conceptual) - -4.1 -~structure~ - -1. Compares directory topology. -2. Reports directories that: - - - exist as directories in A - - are missing or non-directories in B - -3. Intended use: - - - detect missing branches in projects - - detect structural drift - -4.2 -~import~ - -1. Direction: B → A. -2. Only considers: - - - nodes in the =in_between= region of B - - that are new or absent in A - -3. Outputs: - - - ~mkdir -p~ commands (when needed) - - ~cp --parents -a~ commands for files - - a comment list for nodes that cannot be handled automatically - (type mismatches, non-file/dir, constrained nodes) - -4. Intended use: - - - mine “good ideas” in B that belong in the skeleton - - keep Harmony evolving based on real projects - -4.3 -~export~ - -1. Direction: A → B. -2. Considers: - - - A-only nodes (present in A, missing in B) - - nodes where A’s file is newer than B’s file - -3. Outputs: - - - ~mkdir -p~ commands for B - - ~cp --parents -a~ commands for files - -4. Intended use: - - - bring B back into alignment with the current Harmony skeleton - - propagate skeleton fixes and improvements into projects - -4.4 -~suspicious~ - -1. Reports nodes in B that are: - - - inside A’s directory structure - - but not under any leaf directory - -2. Intended use: - - - highlight questionable placements - - identify candidates for new skeleton structure - - catch misuse of the skeleton (work living in the “framework” layer) - -4.5 -~addendum~ - -1. Reports nodes in B that are: - - - under leaf directories in A - -2. Intended use: - - - show work added at the intended extension points - - give a quick outline of “project-specific” content layered on Harmony - -4.6 -~all~ - -1. Runs: - - - =structure= - - =import= - - =export= - - =suspicious= - - =addendum= - -2. Intended use: - - - periodic health check of a project against Harmony - - initial analysis when inheriting an old project - -* 5. Safety and behavior guarantees - -5.1 -No direct modification - -1. ~skeleton_compaare~ itself does not modify either tree. -2. It only prints suggested shell commands. -3. A human is expected to review and run those commands (or not). - -5.2 -Constrained and unknown nodes - -1. Some paths are “constrained”: - - - object exists but metadata (e.g., ~mtime~) cannot be safely read - - typical for special files or broken links - -2. These are: - - - classified as =constrained= - - never touched by import/export logic - - surfaced in “not handled automatically” lists - -5.3 -Robust to legacy layouts - -1. A and B are assumed to be non-overlapping roots. -2. B does not have to be a clean Harmony derivative. -3. The tool is designed to: - - - tolerate missing branches - - tolerate ad-hoc additions - - still classify and report differences coherently - -* 6. How to run it - -6.1 -From inside the Harmony repo: - -#+begin_src sh -cd /path/to/Harmony -tool/skeleton_compaare help -tool/skeleton_compaare usage -tool/skeleton_compaare structure ../SomeProject -tool/skeleton_compaare all ../Rabbit -#+end_src - -6.2 -The CLI help (from ~doc.py~) is the canonical reference for: - -1. grammar and argument rules -2. meaning of A and B -3. exact semantics of each command - -This =.org= file is a conceptual overview for Harmony toolsmiths and administrators. - -* 7. Maintenance notes - -7.1 -Core modules - -1. =skeleton_compare_source/skeleton.py= - - tree construction - - topology classification - - “newer/older” logic - - in-between / below partitioning - -2. =skeleton_compare_source/command.py= - - high-level command semantics - - import/export planning and printing - -3. =skeleton_compare_source/CLI.py= - - argument classification - - environment checks - - dispatch to command handlers - -7.2 -Change discipline - -1. CLI behavior and text should be updated in: - - - =doc.py= (help/usage text) - - this =.org= file (conceptual intent) - -2. Any behavioral change that affects: - - - classification rules - - import/export semantics - - constrained handling - - should be reflected here in section 3 or 4. - diff --git a/tool_shared/authored/sync/command.py b/tool_shared/authored/sync/command.py deleted file mode 100644 index 0ebb856..0000000 --- a/tool_shared/authored/sync/command.py +++ /dev/null @@ -1,539 +0,0 @@ -#!/usr/bin/env python3 -# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- - -""" -command.py - high-level dispatch for Harmony check commands - -Commands (semantics): - - structure: - - Differences in directory structure: directories present in A but - not present as directories in B. - - import: - - Shell copy commands to copy: - * in-between nodes in B that are newer than A (same relative path), or - * in-between nodes in B that do not exist in A at all. - Direction: B -> A - Also emits: - * a mkdir list (directories to create in A) - * an "other" list for type mismatches / non-file/dir nodes. - - export: - - Shell copy commands to copy: - * nodes in A that are newer than B, and - * nodes in A that do not exist in B. - Direction: A -> B - Also emits: - * a mkdir list (directories to create in B) - * an "other" list for type mismatches / non-file/dir nodes. - - suspicious: - - Nodes in B that fall "in between" the Harmony skeleton topology: - under some A directory, but not under any A leaf directory. - (tree_dict_in_between_and_below(A,B).in_between) - - addendum: - - Nodes in B that fall "below" Harmony leaf directories: - added work in appropriate extension locations. - (tree_dict_in_between_and_below(A,B).below) - - all: - - Runs structure, import, export, suspicious, and addendum. -""" - -from __future__ import annotations - -import os -from typing import Any, Dict, List, Tuple - -import skeleton - -TreeDict = Dict[str, Dict[str, Any]] - - -def shell_quote( - s: str -) -> str: - """ - Minimal single-quote shell quoting. - """ - return "'" + s.replace("'", "'\"'\"'") + "'" - - -def _print_header( - title: str -) -> None: - print() - print(f"== {title} ==") - - -# ---------------------------------------------------------------------- -# structure: directories in A that are missing / non-directories in B -# ---------------------------------------------------------------------- -def cmd_structure( - A: TreeDict - ,B: TreeDict -) -> int: - """ - structure: differences in directory structure, directories in A - B. - - We include any path where: - - A[path].node_type == 'directory', and - - either path not in B, or B[path].node_type != 'directory'. - """ - structural: TreeDict = {} - - for path, info_A in A.items(): - if info_A.get("node_type") != "directory": - continue - - info_B = B.get(path) - if info_B is None or info_B.get("node_type") != "directory": - structural[path] = info_A - - if not structural: - _print_header("structure") - print("No structural directory differences (A - B).") - return 0 - - _print_header("structure: directories in A not in B") - skeleton.tree_dict_print(structural) - return 0 - - -# ---------------------------------------------------------------------- -# import: B -> A (mkdir, cp, and "other" list), using in_between_newer -# ---------------------------------------------------------------------- - - -def build_import_commands( - A_tree: TreeDict - ,B_tree: TreeDict - ,A_root: str - ,B_root: str -) -> Tuple[List[str], List[str], List[str]]: - """ - Compute shell commands to update A from B. - - Returns: - (mkdir_cmds, cp_cmds, other_list) - - Semantics: - - mkdir_cmds: - - Directories that are directories in B, but are missing in A. - - We DO NOT auto-resolve type mismatches (e.g. B=directory, - A=file); those go into other_list. - - cp_cmds: - - Files where: - * the path does not exist in A, OR - * the node in A is not a file, OR - * the B copy is newer than A (mtime comparison). - - However, if A has a non-file at that path, we treat it as a - type mismatch and add that path to other_list instead of - emitting a cp command. - - other_list: - - Human-readable notes for: - * type mismatches between A and B, and - * nodes in B that are neither 'file' nor 'directory'. - """ - mkdir_cmds: List[str] = [] - cp_cmds: List[str] = [] - other_list: List[str] = [] - - for rel_path, b_info in B_tree.items(): - b_type = b_info.get("node_type") - rel_display = rel_path if rel_path else "." - - a_info = A_tree.get(rel_path) - a_type = a_info.get("node_type") if a_info is not None else "MISSING" - - # Case 1: B node is neither file nor directory -> other_list - if b_type not in ("file", "directory"): - other_list.append( - f"{rel_display}: A={a_type}, B={b_type}" - ) - continue - - # Case 2: B directory - if b_type == "directory": - if a_info is None: - # Missing in A: copy the directory recursively. - src = os.path.join(B_root, rel_path) if rel_path else B_root - # The destination should be the parent directory in A. - # os.path.join(A_root, rel_path) gives the full path to the new directory in A. - # os.path.dirname of that gives the parent directory. - dst = os.path.dirname(os.path.join(A_root, rel_path)) - mkdir_cmds.append( - f"cp -a {shell_quote(src)} {shell_quote(dst + os.sep)}" - ) - else: - # Exists in A: must also be a directory to be "structurally OK" - if a_type != "directory": - # Type mismatch: do not mkdir, just report - other_list.append( - f"{rel_display}: A={a_type}, B=directory" - ) - continue - - # Case 3: B file - # Decide whether to copy B -> A, or report conflict. - if a_info is None: - # B-only file - src = os.path.join(B_root, rel_path) if rel_path else B_root - # Destination is the parent directory in A, with a trailing slash - dst = os.path.dirname(os.path.join(A_root, rel_path)) - cp_cmds.append( - f"cp -a {shell_quote(src)} {shell_quote(dst + os.sep)}" - ) - continue - - # A has something at this path - if a_type != "file": - # Type mismatch (e.g. A=directory, B=file, or A=other) - other_list.append( - f"{rel_display}: A={a_type}, B=file" - ) - continue - - # Both files: compare mtime - a_mtime = a_info.get("mtime") - b_mtime = b_info.get("mtime") - - if isinstance(a_mtime, (int, float)) and isinstance(b_mtime, (int, float)): - if b_mtime > a_mtime: - src = os.path.join(B_root, rel_path) if rel_path else B_root - # Destination is the parent directory in A, with a trailing slash - dst = os.path.dirname(os.path.join(A_root, rel_path)) - cp_cmds.append( - f"cp -a {shell_quote(src)} {shell_quote(dst + os.sep)}" - ) - - return mkdir_cmds, cp_cmds, other_list - -def cmd_import( - A_tree: TreeDict - ,B_tree: TreeDict - ,A_root: str - ,B_root: str -) -> int: - """ - import: update the skeleton (A) from the project (B), - using only in_between_newer nodes. - """ - inb_newer = skeleton.in_between_newer(A_tree, B_tree) - - mkdir_cmds, cp_cmds, other_list = build_import_commands( - A_tree - ,inb_newer - ,A_root - ,B_root - ) - - print("== import: copy from B -> A (in-between newer only) ==") - print(f"# A root: {A_root}") - print(f"# B root: {B_root}") - print("# Only considering in-between files that are new or absent in A.") - print("#") - - print("# Directories to copy from B -> A (cp -a):") - if mkdir_cmds: - for line in mkdir_cmds: - print(line) - else: - print("# (none)") - print("#") - - print("# Files to copy from B -> A (cp -a):") - if cp_cmds: - for line in cp_cmds: - print(line) - else: - print("# (none)") - print("#") - - print("# Nodes NOT handled automatically (type mismatches / non-file/dir):") - if other_list: - for rel in other_list: - print(f"# {rel}") - else: - print("# (none)") - - return 0 - - -# ---------------------------------------------------------------------- -# export: A -> B (mkdir, cp, and "other" list) -# ---------------------------------------------------------------------- -def build_export_commands( - A_tree: TreeDict - ,B_tree: TreeDict - ,A_root: str - ,B_root: str -) -> Tuple[List[str], List[str], List[str]]: - """ - Compute shell commands to update B from A. - - Returns: - (mkdir_cmds, cp_cmds, other_list) - - Semantics: - - mkdir_cmds: - - Directories that are directories in A, but are missing in B. - - Type mismatches go into other_list. - - cp_cmds: - - Files where: - * the path does not exist in B, OR - * the node in B is not a file, OR - * the A copy is newer than B (mtime comparison). - - If B has a non-file while A has a file, treat as type mismatch. - - other_list: - - Human-readable notes for: - * type mismatches between A and B, and - * nodes in A that are neither 'file' nor 'directory'. - """ - mkdir_cmds: List[str] = [] - cp_cmds: List[str] = [] - other_list: List[str] = [] - - # Sort keys to ensure parent directories are processed before their children. - sorted_paths = sorted(A_tree.keys(), key=len) - included_dirs: Set[str] = set() - - for rel_path in sorted_paths: - a_info = A_tree[rel_path] - a_type = a_info.get("node_type") - rel_display = rel_path if rel_path else "." - - b_info = B_tree.get(rel_path) - b_type = b_info.get("node_type") if b_info is not None else "MISSING" - - # Case 1: A node is neither file nor directory -> other_list - if a_type not in ("file", "directory"): - other_list.append( - f"{rel_display}: A={a_type}, B={b_type}" - ) - continue - - # Check if this path is a child of an already included directory - is_child_of_included_dir = False - for d in included_dirs: - if rel_path.startswith(d + os.sep): - is_child_of_included_dir = True - break - - if is_child_of_included_dir: - continue - - # Case 2: A directory - if a_type == "directory": - if b_info is None: - # Missing in B: copy the directory recursively. - src = os.path.join(A_root, rel_path) if rel_path else A_root - # The destination should be the parent directory in B. - dst = os.path.dirname(os.path.join(B_root, rel_path)) - mkdir_cmds.append( - f"cp -a {shell_quote(src)} {shell_quote(dst + os.sep)}" - ) - included_dirs.add(rel_path) - else: - # Exists in B: must also be directory - if b_type != "directory": - other_list.append( - f"{rel_display}: A=directory, B={b_type}" - ) - continue - - # Case 3: A file - if b_info is None: - # A-only file - src = os.path.join(A_root, rel_path) if rel_path else A_root - # Destination is the parent directory in B, with a trailing slash - dst = os.path.dirname(os.path.join(B_root, rel_path)) - cp_cmds.append( - f"cp -a {shell_quote(src)} {shell_quote(dst + os.sep)}" - ) - continue - - if b_type != "file": - other_list.append( - f"{rel_display}: A=file, B={b_type}" - ) - continue - - # Both files: compare mtime - a_mtime = a_info.get("mtime") - b_mtime = b_info.get("mtime") - - if isinstance(a_mtime, (int, float)) and isinstance(b_mtime, (int, float)): - if a_mtime > b_mtime: - src = os.path.join(A_root, rel_path) if rel_path else A_root - # Destination is the parent directory in B, with a trailing slash - dst = os.path.dirname(os.path.join(B_root, rel_path)) - cp_cmds.append( - f"cp -a {shell_quote(src)} {shell_quote(dst + os.sep)}" - ) - - return mkdir_cmds, cp_cmds, other_list - - -def cmd_export( - A_tree: TreeDict - ,B_tree: TreeDict - ,A_root: str - ,B_root: str -) -> int: - """ - export: show directory creation and copy commands A -> B. - """ - mkdir_cmds, cp_cmds, other_list = build_export_commands( - A_tree - ,B_tree - ,A_root - ,B_root - ) - - print("== export: copy from A -> B ==") - print(f"# A root: {A_root}") - print(f"# B root: {B_root}") - print("#") - - print("# Directories to copy from A -> B (cp -a):") - if mkdir_cmds: - for line in mkdir_cmds: - print(line) - else: - print("# (none)") - print("#") - - print("# Files to copy from A -> B (cp -a):") - if cp_cmds: - for line in cp_cmds: - print(line) - else: - print("# (none)") - print("#") - - print("# Nodes NOT handled automatically (type mismatches / non-file/dir):") - if other_list: - for rel in other_list: - print(f"# {rel}") - else: - print("# (none)") - - return 0 - - -# ---------------------------------------------------------------------- -# suspicious / addendum via in_between_and_below -# ---------------------------------------------------------------------- -def cmd_suspicious( - A: TreeDict - ,B: TreeDict -) -> int: - """ - suspicious: nodes in B that fall 'in between' the Harmony skeleton, - not under leaf directories. - """ - in_between, _below = skeleton.tree_dict_in_between_and_below(A, B) - - _print_header("suspicious: nodes in-between Harmony leaves") - - if not in_between: - print("No suspicious nodes found in B (relative to A).") - return 0 - - skeleton.tree_dict_print(in_between) - return 0 - - -def cmd_addendum( - A: TreeDict - ,B: TreeDict -) -> int: - """ - addendum: nodes in B that fall 'below' Harmony leaf directories. - """ - _in_between, below = skeleton.tree_dict_in_between_and_below(A, B) - - _print_header("addendum: nodes added under Harmony leaves") - - if not below: - print("No addendum nodes found in B (relative to A).") - return 0 - - skeleton.tree_dict_print(below) - return 0 - - -# ---------------------------------------------------------------------- -# Top-level dispatcher -# ---------------------------------------------------------------------- -def dispatch( - has_other_list: List[str] - ,A: TreeDict - ,B: TreeDict - ,A_root: str - ,B_root: str -) -> int: - """ - Dispatch commands. - - has_other_list: - List of command tokens (subset of: - 'structure', 'import', 'export', 'suspicious', 'addendum', 'all'). - - A, B: - tree_dicts for Harmony skeleton (A) and project (B). - - A_root, B_root: - Root paths corresponding to A and B (for copy commands). - """ - cmds = set(has_other_list) - - if "all" in cmds: - cmds.update([ - "structure" - ,"import" - ,"export" - ,"suspicious" - ,"addendum" - ]) - - ordered = [ - "structure" - ,"import" - ,"export" - ,"suspicious" - ,"addendum" - ] - - status = 0 - - for name in ordered: - if name not in cmds: - continue - - if name == "structure": - rc = cmd_structure(A, B) - elif name == "import": - rc = cmd_import(A, B, A_root, B_root) - elif name == "export": - rc = cmd_export(A, B, A_root, B_root) - elif name == "suspicious": - rc = cmd_suspicious(A, B) - elif name == "addendum": - rc = cmd_addendum(A, B) - else: - rc = 0 - - if rc != 0: - status = rc - - return status diff --git a/tool_shared/authored/sync/doc.py b/tool_shared/authored/sync/doc.py deleted file mode 100644 index 3198b96..0000000 --- a/tool_shared/authored/sync/doc.py +++ /dev/null @@ -1,182 +0,0 @@ -#!/usr/bin/env python3 -# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- - -""" -doc.py - usage and help text for the Harmony 'check' tool - -Grammar (informal): - - * [] - - :: | | - - :: version | help | usage - :: environment - :: structure | import | export | suspicious | addendum | all -""" - -from __future__ import annotations - -import meta -import os -import sys -from typing import TextIO - - -def prog_name() -> str: - """ - Return the program name as invoked by the user. - - Typically: - - basename(sys.argv[0]) when running from the shell. - - Falls back to 'check' if argv[0] is empty. - """ - raw = sys.argv[0] if sys.argv and sys.argv[0] else "check" - base = os.path.basename(raw) or raw - return base - - -def _usage_text(prog: str) -> str: - return f"""\ -Usage: - {prog} * [] - -Where: - :: | | - - :: version | help | usage - :: environment - :: structure | import | export | suspicious | addendum | all -""" - -def _help_text(prog: str) -> str: - return f"""\ -{prog} - Harmony skeleton integrity and metadata checker - -Syntax: - {prog} * [] - -Where: - :: path - :: | | - - :: version | help | usage - :: environment - :: structure | import | export | suspicious | addendum | all - -Argument rules (informal): - 1. commands are processed first, and then the program returns. - Hence if any help commands are present, the remaining commands - are ignored. - - 2. We assume {prog} is run within the Harmony skeleton, or a skeleton - derived directly from it. This is the 'default skeleton', or simply 'A'. - - 3. The path is the directory of a project that is assumed to - be built upon the default skeleton. This second project root is - called 'B'. - - 4. If none of the commands require an path, then - must not be given. If at least one command requires , then - is required. Commands that require a path are called - commands. - - 5. Implementation detail: - All arguments except the final one are interpreted strictly as - command tokens. If any of those are , the final argument - is taken as . If none of the earlier tokens are , - the final argument is also treated as a command token. - -Roots: - A = Skeleton project root (auto-detected). Usually the Harmony skeleton. - B = project root (supplied when required). - -{prog} compares A with B. Differences may come from: - - edits to the skeleton itself, - - edits to skeleton files inside B, - - or new files/directories added to B. -Conceptually, A and B are any two non-overlapping directory trees. - -Command semantics: - structure - - Report directory-structure differences: - directories present in A that are missing in B or not - directories in B. - - Output: table of such directories. - - import - - Update A from B using only "in-between newer" files: - * files in B that lie in the 'in-between' region relative to A, and - * are newer than A or absent from A. - - Also emits: - * directories to create in A, - * files to copy (B -> A), - * nodes that cannot be handled automatically (type mismatches, - constrained nodes, non-file/dir nodes). - - Direction: B -> A - - export - - Update B from A: - * files in A newer than B at the same path, - * files present in A but missing in B. - - Also emits: - * directories to create in B, - * files to copy (A -> B), - * nodes that cannot be handled automatically. - - Direction: A -> B - - suspicious - - Report B nodes that lie "in-between" Harmony leaves: - under a directory from A, but not under any leaf directory of A. - - Indicates questionable placements or missing skeleton structure. - - addendum - - Report B nodes located "below" Harmony leaf directories: - project-specific additions placed in proper extension points. - - all - - Run: structure, import, export, suspicious, addendum (in that order). - -Notes: - - tree_dict traversal respects a simplified .gitignore model plus - always-ignored patterns (e.g. '.git'). - - Timestamps are formatted via the Z helper in UTC (ISO 8601). -""" - -def print_usage( - stream: TextIO | None = None -) -> None: - """ - Print the usage text to the given stream (default: sys.stdout), - using the actual program name as invoked. - """ - if stream is None: - stream = sys.stdout - - text = _usage_text(prog_name()) - stream.write(text) - if not text.endswith("\n"): - stream.write("\n") - - -def print_help( - stream: TextIO | None = None -) -> None: - """ - Print the help text to the given stream (default: sys.stdout), - using the actual program name as invoked. - """ - if stream is None: - stream = sys.stdout - - utext = _usage_text(prog_name()) - htext = _help_text(prog_name()) - - stream.write(utext) - if not utext.endswith("\n"): - stream.write("\n") - - stream.write("\n") - stream.write(htext) - if not htext.endswith("\n"): - stream.write("\n") diff --git a/tool_shared/authored/sync/in_between_and_below b/tool_shared/authored/sync/in_between_and_below deleted file mode 100755 index 2993767..0000000 --- a/tool_shared/authored/sync/in_between_and_below +++ /dev/null @@ -1,53 +0,0 @@ -#!/usr/bin/env python3 -# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- - -""" -in_between_and_below - CLI test driver for skeleton.tree_dict_in_between_and_below(A, B) - -Usage: - in_between_and_below -""" - -from __future__ import annotations - -import os -import sys -from typing import Sequence - -import meta -import skeleton - - -def CLI(argv: Sequence[str] | None = None) -> int: - if argv is None: - argv = sys.argv[1:] - - prog = os.path.basename(sys.argv[0]) if sys.argv else "in_between_and_below" - - if len(argv) != 2 or argv[0] in ("-h", "--help"): - print(f"Usage: {prog} ") - return 1 - - A_root = argv[0] - B_root = argv[1] - - if not os.path.isdir(A_root): - print(f"{prog}: {A_root}: not a directory") - return 2 - - if not os.path.isdir(B_root): - print(f"{prog}: {B_root}: not a directory") - return 3 - - A = skeleton.tree_dict_make(A_root, None) - B = skeleton.tree_dict_make(B_root, None) - - meta.debug_set("tree_dict_in_between_and_below") - - _result = skeleton.tree_dict_in_between_and_below(A, B) - - return 0 - - -if __name__ == "__main__": - raise SystemExit(CLI()) diff --git a/tool_shared/authored/sync/load_command_module.py b/tool_shared/authored/sync/load_command_module.py deleted file mode 100644 index 226b6dd..0000000 --- a/tool_shared/authored/sync/load_command_module.py +++ /dev/null @@ -1,103 +0,0 @@ -#!/usr/bin/env python3 -# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- - -""" -load_command_module.py - locate and import Python command modules from $PATH - -Behavior: - 1. Search $PATH for an executable with the given command name. - 2. Prefer a path containing '/incommon/'. - 3. If only /usr/bin/ is found, raise an error saying we were - looking for the incommon version. - 4. Import the chosen script as a Python module, even if it has no .py - extension, by forcing a SourceFileLoader. -""" - -from __future__ import annotations - -import importlib.util -import os -from importlib.machinery import SourceFileLoader -from types import ModuleType -from typing import List - - -def _find_command_candidates(command_name: str) -> List[str]: - """ - Return a list of absolute paths to executables named `command_name` - found on $PATH. - """ - paths: list[str] = [] - - path_env = os.environ.get("PATH", "") - for dir_path in path_env.split(os.pathsep): - if not dir_path: - continue - candidate = os.path.join(dir_path, command_name) - if os.path.isfile(candidate) and os.access(candidate, os.X_OK): - paths.append(os.path.realpath(candidate)) - - return paths - - -def load_command_module(command_name: str) -> ModuleType: - """ - Locate an executable named `command_name` on $PATH and load it - as a Python module. - - Selection policy: - 1. Prefer any path containing '/incommon/'. - 2. If only /usr/bin/ candidates exist, raise an error - saying we were looking for the incommon version. - 3. If no candidate is found, raise an error. - - Implementation detail: - Because the incommon command may lack a .py suffix, we explicitly - construct a SourceFileLoader rather than relying on the default - extension-based loader resolution. - """ - candidates = _find_command_candidates(command_name) - - incommon_candidates = [ - p - for p in candidates - if "/incommon/" in p - ] - - usrbin_candidates = [ - p - for p in candidates - if p.startswith("/usr/bin/") - ] - - if incommon_candidates: - target = incommon_candidates[0] - elif usrbin_candidates: - raise RuntimeError( - f"Found /usr/bin/{command_name}, but expected the incommon Python " - f"{command_name} module on PATH." - ) - else: - raise RuntimeError( - f"Could not find an incommon '{command_name}' module on PATH." - ) - - module_name = f"rt_incommon_{command_name}" - - loader = SourceFileLoader( - module_name - ,target - ) - spec = importlib.util.spec_from_loader( - module_name - ,loader - ) - if spec is None: - raise RuntimeError(f"Failed to create spec for {command_name} from {target}") - - module = importlib.util.module_from_spec(spec) - # spec.loader is the SourceFileLoader we just created - assert spec.loader is not None - spec.loader.exec_module(module) - - return module diff --git a/tool_shared/authored/sync/make_Harmony_tree_dict b/tool_shared/authored/sync/make_Harmony_tree_dict deleted file mode 100755 index 2ed3cea..0000000 --- a/tool_shared/authored/sync/make_Harmony_tree_dict +++ /dev/null @@ -1,48 +0,0 @@ -#!/usr/bin/env python3 -# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- - -""" -skeleton_test - build and print the Harmony tree_dict - -Usage: - skeleton_test - -Behavior: - 1. Locate the Harmony project root via Harmony.where(). - 2. Enable 'tree_dict_print' debug flag. - 3. Call skeleton.tree_dict_make(Harmony_root, None). - -The skeleton.tree_dict_make() function is expected to call -tree_dict_print() when the 'tree_dict_print' debug flag is set. -""" - -from __future__ import annotations - -import sys - -import Harmony -import meta -import skeleton - - -def CLI(argv=None) -> int: - # No arguments expected - status, Harmony_root = Harmony.where() - - if status == "not-found": - print("Harmony project not found; cannot build tree_dict.") - return 1 - - if status == "different": - print("Warning: Harmony not found, using nearest .git directory for tree_dict.") - - # Enable printing inside tree_dict_make - meta.debug_set("tree_dict_print") - - _tree = skeleton.tree_dict_make(Harmony_root, None) - - return 0 - - -if __name__ == "__main__": - raise SystemExit(CLI()) diff --git a/tool_shared/authored/sync/manus_2025_11_20_215471873762383.txt b/tool_shared/authored/sync/manus_2025_11_20_215471873762383.txt deleted file mode 100644 index 70e1f99..0000000 --- a/tool_shared/authored/sync/manus_2025_11_20_215471873762383.txt +++ /dev/null @@ -1,33 +0,0 @@ -Conversation with Manus -Started on November 20, 2025 at 08:56 PM Singapore time +08 (GMT+0800) - ---- - -08:56 PM | Thomas Lynch: Hello, I signed up today with 7 days free. However, in the middle of a chat, I was prompted to sign up, which I already did. It then charged me for the month, but we have not been 7 days yet, and it is full charge. Why did it do that? - As a second question is the 'credits'. Did it prompt again because the credits were used? Am I now on the hook for two months of charges in less than a day of use? - If just a few hours use uses up a 'month' of credits, then I really can't afford Manus, and need to back out of this. - -08:56 PM | Manus AI: I understand your concern about the unexpected charge! Let me clarify what likely happened. - About the charge: Our subscriptions operate on a credit-based system (http://help.manus.im/en/articles/11711097-what-are-the-rules-for-credits-consumption-and-how-can-i-obtain-them) where credits are consumed during active task processing. When you run out of credits (even during a trial), you may be prompted to upgrade to continue using the service, which would trigger billing. - You're not charged twice: The billing is monthly from your subscription date (http://help.manus.im/en/articles/11711089-where-can-i-find-the-invoice-after-purchase-can-the-invoice-be-modified), so you'll only have one active subscription. - To check your usage: Visit Settings > Usage to see exactly how your credits were consumed and review transaction details. - Your options: - - Cancel subscription: (http://help.manus.im/en/articles/11711087-how-can-i-cancel-my-subscription) Go to your profile > Account > Manage subscription > Cancel subscription to stop future charges. -- Request refund: You can submit a refund request with your invoice information and reason. We'll review it according to our refund policy. - The good news is you can cancel anytime to prevent future charges, and we do consider refund requests on a case-by-case basis! - -Sources: -What are the rules for credits consumption? And how can I obtain them? (http://help.manus.im/en/articles/11711097-what-are-the-rules-for-credits-consumption-and-how-can-i-obtain-them) -Where can I check my credit consumption/usage? (http://help.manus.im/en/articles/12520235-where-can-i-check-my-credit-consumption-usage) -How can I request a refund for a purchased membership? (http://help.manus.im/en/articles/11711102-how-can-i-request-a-refund-for-a-purchased-membership) -How can I cancel my subscription? (http://help.manus.im/en/articles/11711087-how-can-i-cancel-my-subscription) -Where can I find the invoice after purchase, can the invoice be modified? (http://help.manus.im/en/articles/11711089-where-can-i-find-the-invoice-after-purchase-can-the-invoice-be-modified) - -08:57 PM | Liora from Manus: We will respond to you with the resolution within 48 hours. Please wait for our update! - -09:01 PM | Thomas Lynch: You charged my $46.80, the plan says it costs $33.00 explain. - -09:02 PM | Thomas Lynch: hello? - ---- -Exported from Manus on November 20, 2025 at 09:03 PM Singapore time +08 (GMT+0800) \ No newline at end of file diff --git a/tool_shared/authored/sync/meta.py b/tool_shared/authored/sync/meta.py deleted file mode 100644 index 5c8da89..0000000 --- a/tool_shared/authored/sync/meta.py +++ /dev/null @@ -1,97 +0,0 @@ -#!/usr/bin/env python3 -# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- - -""" -meta.py - thin wrappers around command modules - -Current responsibilities: - 1. Load the incommon 'printenv' command module (no .py extension) - using load_command_module.load_command_module(). - 2. Expose printenv() here, calling the imported printenv() work - function with default arguments (equivalent to running without - any CLI arguments). - 3. Provide a simple version printer for this meta module. - 4. Provide a small debug tag API (set/clear/has). -""" - -from __future__ import annotations - -import datetime -from load_command_module import load_command_module - - -# Load the incommon printenv module once at import time -_PRINTENV_MODULE = load_command_module("printenv") -_Z_MODULE = load_command_module("Z") - - -# Meta module version -_major = 1 -_minor = 7 -def version_print() -> None: - """ - Print the meta module version as MAJOR.MINOR. - """ - print(f"{_major}.{_minor}") - - -# Debug tag set and helpers -_debug = set([ -]) - - -def debug_set(tag: str) -> None: - """ - Add a debug tag to the meta debug set. - """ - _debug.add(tag) - - -def debug_clear(tag: str) -> None: - """ - Remove a debug tag from the meta debug set, if present. - """ - _debug.discard(tag) - - -def debug_has(tag: str) -> bool: - """ - Return True if the given debug tag is present. - """ - return tag in _debug - - -# Touch the default tag once so static checkers do not complain about -# unused helpers when imported purely for side-effects. -debug_has("Command") - - -def printenv() -> int: - """ - Call the imported printenv() work function with default arguments: - - no null termination - - no newline quoting - - no specific names (print full environment) - - prog name 'printenv' - """ - return _PRINTENV_MODULE.printenv( - False # null_terminate - ,False # quote_newlines - ,[] # names - ,"printenv" - ) - - -def z_format_mtime( - mtime: float -) -> str: - """ - Format a POSIX mtime (seconds since epoch, UTC) using the Z module. - - Uses Z.ISO8601_FORMAT and Z.make_timestamp(dt=...). - """ - dt = datetime.datetime.fromtimestamp(mtime, datetime.timezone.utc) - return _Z_MODULE.make_timestamp( - fmt=_Z_MODULE.ISO8601_FORMAT - ,dt=dt - ) diff --git a/tool_shared/authored/sync/newer b/tool_shared/authored/sync/newer deleted file mode 100755 index 30aa373..0000000 --- a/tool_shared/authored/sync/newer +++ /dev/null @@ -1,53 +0,0 @@ -#!/usr/bin/env python3 -# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- - -""" -newer - CLI test driver for skeleton.tree_dict_newer(A, B) - -Usage: - newer -""" - -from __future__ import annotations - -import os -import sys -from typing import Sequence - -import meta -import skeleton - - -def CLI(argv: Sequence[str] | None = None) -> int: - if argv is None: - argv = sys.argv[1:] - - prog = os.path.basename(sys.argv[0]) if sys.argv else "newer" - - if len(argv) != 2 or argv[0] in ("-h", "--help"): - print(f"Usage: {prog} ") - return 1 - - A_root = argv[0] - B_root = argv[1] - - if not os.path.isdir(A_root): - print(f"{prog}: {A_root}: not a directory") - return 2 - - if not os.path.isdir(B_root): - print(f"{prog}: {B_root}: not a directory") - return 3 - - A = skeleton.tree_dict_make(A_root, None) - B = skeleton.tree_dict_make(B_root, None) - - meta.debug_set("tree_dict_newer") - - _result = skeleton.tree_dict_newer(A, B) - - return 0 - - -if __name__ == "__main__": - raise SystemExit(CLI()) diff --git a/tool_shared/authored/sync/older b/tool_shared/authored/sync/older deleted file mode 100755 index f8ff24d..0000000 --- a/tool_shared/authored/sync/older +++ /dev/null @@ -1,53 +0,0 @@ -#!/usr/bin/env python3 -# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- - -""" -older - CLI test driver for skeleton.tree_dict_older(A, B) - -Usage: - older -""" - -from __future__ import annotations - -import os -import sys -from typing import Sequence - -import meta -import skeleton - - -def CLI(argv: Sequence[str] | None = None) -> int: - if argv is None: - argv = sys.argv[1:] - - prog = os.path.basename(sys.argv[0]) if sys.argv else "older" - - if len(argv) != 2 or argv[0] in ("-h", "--help"): - print(f"Usage: {prog} ") - return 1 - - A_root = argv[0] - B_root = argv[1] - - if not os.path.isdir(A_root): - print(f"{prog}: {A_root}: not a directory") - return 2 - - if not os.path.isdir(B_root): - print(f"{prog}: {B_root}: not a directory") - return 3 - - A = skeleton.tree_dict_make(A_root, None) - B = skeleton.tree_dict_make(B_root, None) - - meta.debug_set("tree_dict_older") - - _result = skeleton.tree_dict_older(A, B) - - return 0 - - -if __name__ == "__main__": - raise SystemExit(CLI()) diff --git a/tool_shared/authored/sync/skeleton.py b/tool_shared/authored/sync/skeleton.py deleted file mode 100644 index ae272f9..0000000 --- a/tool_shared/authored/sync/skeleton.py +++ /dev/null @@ -1,576 +0,0 @@ -#!/usr/bin/env python3 -# -*- mode: python; coding: utf-8; python-indent-offset: 2; indent-tabs-mode: nil -*- - -""" -skeleton.py - helpers for working with the Harmony skeleton tree -""" - -from __future__ import annotations - -import os -import sys -from typing import Any, Callable, Dict, List, Set - -import meta -from GitIgnore import GitIgnore -import Harmony - -TreeDict = Dict[str, Dict[str, Any]] - -# tree_dict_make / tree_dict_print -# -# Build a dictionary describing a project tree, respecting GitIgnore. -# -# tree_dict_make(, ) -> tree_dict -# -# () -> bignum | None -# -# Keys of tree_dict: -# - Relative paths from ; the root itself is stored under "". -# -# Values are dicts with: -# 1. 'mtime' : last modification time (float seconds) or None -# 2. 'node_type' : 'file', 'directory', 'other', or 'constrained' -# 3. 'dir_info' : 'NA', 'leaf', 'branch', 'root' -# 4. 'checksum' : present only for file nodes when checksum_fn is -# not None -# -# Traversal: -# - Directories whose relative path GitIgnore.check() marks as -# 'Ignore' are included in tree_dict but not traversed further. -def tree_dict_make( - path: str - ,checksum_fn: Callable[[str], int] | None -) -> Dict[str, Dict[str, Any]]: - """ - Build a tree_dict for the subtree rooted at , respecting GitIgnore. - - Semantics (current): - * Any path (directory or file) for which GitIgnore.check() - returns 'Ignore' is completely omitted from the tree_dict. - * The root directory ('') is always included. - * Directory dir_info: - - 'root' for the root - - 'branch' for directories that have child directories - (after GitIgnore filtering) - - 'leaf' for directories with no child directories - * Non-directory dir_info: - - 'NA' - * Symlinks are classified as file/directory/other based on what - they point to, if accessible. - * If any filesystem access needed for classification/mtime raises, - the node is recorded as node_type='constrained', dir_info='NA', - mtime=None, and we do not attempt checksum. - """ - root = os.path.abspath(path) - gi = GitIgnore(root) - - tree_dict: Dict[str, Dict[str, Any]] = {} - - for dirpath, dirnames, filenames in os.walk(root, topdown=True): - rel_dir = os.path.relpath(dirpath, root) - if rel_dir == ".": - rel_dir = "" - - # Skip ignored directories (except the root). - if rel_dir != "" and gi.check(rel_dir) == "Ignore": - dirnames[:] = [] - continue - - # Filter child directories by GitIgnore so dir_info reflects - # only directories we will actually traverse. - kept_dirnames: List[str] = [] - for dn in list(dirnames): - child_rel = dn if rel_dir == "" else os.path.join(rel_dir, dn) - if gi.check(child_rel) == "Ignore": - dirnames.remove(dn) - else: - kept_dirnames.append(dn) - - # Record the directory node itself - dir_abs = dirpath - try: - dir_mtime = os.path.getmtime(dir_abs) - dir_node_type = "directory" - if rel_dir == "": - dir_info = "root" - elif kept_dirnames: - dir_info = "branch" - else: - dir_info = "leaf" - except OSError: - # Could not stat the directory: treat as constrained. - dir_mtime = None - dir_node_type = "constrained" - dir_info = "NA" - - tree_dict[rel_dir] = { - "mtime": dir_mtime - ,"node_type": dir_node_type - ,"dir_info": dir_info - } - - # For non-ignored directories, record files within - for name in filenames: - abs_path = os.path.join(dirpath, name) - if rel_dir == "": - rel_path = name - else: - rel_path = os.path.join(rel_dir, name) - - if gi.check(rel_path) == "Ignore": - continue - - # Wrap classification + mtime in one try/except so any failure - # marks the node as constrained. - try: - if os.path.islink(abs_path): - # Symlink: classify by target if possible - if os.path.isdir(abs_path): - node_type = "directory" - dir_info_f = "branch" - elif os.path.isfile(abs_path): - node_type = "file" - dir_info_f = "NA" - else: - node_type = "other" - dir_info_f = "NA" - mtime = os.path.getmtime(abs_path) - else: - # Normal node - if os.path.isfile(abs_path): - node_type = "file" - dir_info_f = "NA" - elif os.path.isdir(abs_path): - node_type = "directory" - dir_info_f = "branch" - else: - node_type = "other" - dir_info_f = "NA" - mtime = os.path.getmtime(abs_path) - except OSError: - # Anything that blows up during classification/stat becomes - # constrained; we do not attempt checksum for these. - node_type = "constrained" - dir_info_f = "NA" - mtime = None - - info: Dict[str, Any] = { - "mtime": mtime - ,"node_type": node_type - ,"dir_info": dir_info_f - } - - if node_type == "file" and checksum_fn is not None and isinstance(mtime, (int, float)): - info["checksum"] = checksum_fn(abs_path) - - tree_dict[rel_path] = info - - if meta.debug_has("tree_dict_print"): - tree_dict_print(tree_dict) - - return tree_dict - -def tree_dict_print( - tree_dict: Dict[str, Dict[str, Any]] -) -> None: - """ - Pretty-print a tree_dict produced by tree_dict_make() in fixed-width columns: - - [type] [dir] [mtime] [checksum?] [relative path] - - Only the values are printed in each column (no 'field=' prefixes). - mtime is formatted via the Z module for human readability. - """ - entries: List[tuple[str, str, str, str, str]] = [] - has_checksum = False - - for rel_path in sorted(tree_dict.keys()): - info = tree_dict[rel_path] - display_path = rel_path if rel_path != "" else "." - - type_val = str(info.get("node_type", "")) - dir_val = str(info.get("dir_info", "")) - - raw_mtime = info.get("mtime") - if isinstance(raw_mtime, (int, float)): - mtime_val = meta.z_format_mtime(raw_mtime) - else: - mtime_val = str(raw_mtime) - - if "checksum" in info: - checksum_val = str(info["checksum"]) - has_checksum = True - else: - checksum_val = "" - - entries.append(( - type_val - ,dir_val - ,mtime_val - ,checksum_val - ,display_path - )) - - # Compute column widths - type_w = 0 - dir_w = 0 - mtime_w = 0 - checksum_w = 0 - - for type_val, dir_val, mtime_val, checksum_val, _ in entries: - if len(type_val) > type_w: - type_w = len(type_val) - if len(dir_val) > dir_w: - dir_w = len(dir_val) - if len(mtime_val) > mtime_w: - mtime_w = len(mtime_val) - if has_checksum and len(checksum_val) > checksum_w: - checksum_w = len(checksum_val) - - print("Tree dictionary contents:") - for type_val, dir_val, mtime_val, checksum_val, display_path in entries: - line = " " - line += type_val.ljust(type_w) - line += " " - line += dir_val.ljust(dir_w) - line += " " - line += mtime_val.ljust(mtime_w) - - if has_checksum: - line += " " - line += checksum_val.ljust(checksum_w) - - line += " " - line += display_path - - print(line) - - -def tree_dict_A_minus_B( - A: Dict[str, Dict[str, Any]] - ,B: Dict[str, Dict[str, Any]] -) -> Dict[str, Dict[str, Any]]: - """ - Compute the set difference of two tree_dicts at the key level: - - Result = A \\ B - - That is, return a new tree_dict containing only those entries whose - keys are present in A but NOT present in B. - """ - result: Dict[str, Dict[str, Any]] = {} - - B_keys = set(B.keys()) - - for key, info in A.items(): - if key not in B_keys: - result[key] = info - - if meta.debug_has("tree_dict_A_minus_B"): - tree_dict_print(result) - - return result - - -def tree_dict_in_between_and_below( - A: Dict[str, Dict[str, Any]] - ,B: Dict[str, Dict[str, Any]] -) -> tuple[Dict[str, Dict[str, Any]], Dict[str, Dict[str, Any]]]: - """ - Partition nodes of B into two topology-based sets relative to A: - - 1. in_between: - Nodes in B that lie under at least one directory node in A, - but do NOT lie under any leaf directory of A. - - 2. below: - Nodes in B that lie under at least one leaf directory of A. - - Definitions (relative to A's directory topology): - - - A directory node in A is any key whose info['node_type'] == 'directory'. - - - A leaf directory in A is a directory that has no *other* directory - in A as a proper descendant. The project root ('') is therefore - never a leaf (it always has descendant directories if the tree is - non-trivial). - - - “Lies under”: - * For a path p in B, we look at the chain of directory ancestors - (including the root "") and, if p itself is a directory, p - itself. Any of those that appear as directory keys in A are - considered directory ancestors in A. - - * If any of those ancestors is a leaf in A, p goes to 'below'. - Otherwise, if there is at least one directory ancestor in A, - p goes to 'in_between'. - - - Nodes in B that do not lie under any directory in A are ignored. - - Returns: - (in_between_dict, below_dict), both keyed like B and containing - copies of the info dicts from B. - """ - # 1. Collect all directory keys from A - A_dir_keys: Set[str] = set( - key for key, info in A.items() - if info.get("node_type") == "directory" - ) - - # 2. Compute leaf directories in A - leaf_dirs: Set[str] = set() - - for d in A_dir_keys: - if d == "": - continue - - has_child_dir = False - prefix = d + os.sep - - for other in A_dir_keys: - if other == d: - continue - if other.startswith(prefix): - has_child_dir = True - break - - if not has_child_dir: - leaf_dirs.add(d) - - in_between: Dict[str, Dict[str, Any]] = {} - below: Dict[str, Dict[str, Any]] = {} - - for key, info in B.items(): - # Skip B's root - if key in ("", "."): - continue - - parts = key.split(os.sep) - - # Build directory ancestor chain - node_is_dir = (info.get("node_type") == "directory") - - ancestors: List[str] = [""] - prefix = None - - if node_is_dir: - upto = parts - else: - upto = parts[:-1] - - for part in upto: - if prefix is None: - prefix = part - else: - prefix = os.path.join(prefix, part) - ancestors.append(prefix) - - # Filter ancestors to those that exist as directories in A - ancestors_in_A = [d for d in ancestors if d in A_dir_keys] - - if not ancestors_in_A: - # This B node is not under any directory from A; ignore it. - continue - - # Any leaf ancestor in A? - has_leaf_ancestor = any(d in leaf_dirs for d in ancestors_in_A) - - if has_leaf_ancestor: - below[key] = info - else: - in_between[key] = info - - if meta.debug_has("tree_dict_in_between_and_below"): - merged: Dict[str, Dict[str, Any]] = {} - merged.update(in_between) - merged.update(below) - tree_dict_print(merged) - - return in_between, below - - -def tree_dict_newer( - A: Dict[str, Dict[str, Any]] - ,B: Dict[str, Dict[str, Any]] -) -> Dict[str, Dict[str, Any]]: - """ - Return a dictionary of nodes from B that are newer than their - corresponding nodes in A. - - For each key k: - - - If k exists in both A and B, and - - B[k]['mtime'] > A[k]['mtime'], - - then k is included in the result with value B[k]. - - Keys that are only in B (not in A) are ignored here. - """ - result: Dict[str, Dict[str, Any]] = {} - - for key, info_B in B.items(): - info_A = A.get(key) - if info_A is None: - continue - - mtime_A = info_A.get("mtime") - mtime_B = info_B.get("mtime") - - if mtime_A is None or mtime_B is None: - continue - - if mtime_B > mtime_A: - result[key] = info_B - - if meta.debug_has("tree_dict_newer"): - tree_dict_print(result) - - return result - - -def tree_dict_older( - A: Dict[str, Dict[str, Dict[str, Any]]] - ,B: Dict[str, Dict[str, Dict[str, Any]]] -) -> Dict[str, Dict[str, Any]]: - """ - Return a dictionary of nodes from B that are older than their - corresponding nodes in A. - - For each key k: - - - If k exists in both A and B, and - - B[k]['mtime'] < A[k]['mtime'], - - then k is included in the result with value B[k]. - - Keys that are only in B (not in A) are ignored here. - """ - result: Dict[str, Dict[str, Any]] = {} - - for key, info_B in B.items(): - info_A = A.get(key) - if info_A is None: - continue - - mtime_A = info_A.get("mtime") - mtime_B = info_B.get("mtime") - - if mtime_A is None or mtime_B is None: - continue - - if mtime_B < mtime_A: - result[key] = info_B - - if meta.debug_has("tree_dict_older"): - tree_dict_print(result) - - return result - -def in_between_newer( - A: TreeDict - ,B: TreeDict -) -> TreeDict: - """ - in_between_newer(A, B) -> TreeDict - - Return the subset of B's nodes that: - - 1. Are in the 'in_between' region with respect to A's topology: - - under some directory that exists in A - - NOT under any leaf directory in A - (as defined by tree_dict_in_between_and_below), and - - 2. For file nodes: - - are "newer" than A at the same path, or - - are absent from A. - - More precisely: - - If A has no entry for that path -> include. - - If A has a non-file and B has a file -> include. - - If both are files and B.mtime > A.mtime -> include. - - 3. For constrained nodes: - - are always included, so that higher-level commands (e.g. - 'import') can surface them as "not handled automatically". - - Notes: - - Only file nodes participate in mtime comparisons. - - Nodes with node_type == 'constrained' are passed through without - mtime checks, so that callers can report them separately. - """ - in_between, _below = tree_dict_in_between_and_below(A, B) - - result: TreeDict = {} - - # Keep track of directories already included in the result - included_dirs: Set[str] = set() - - # Sort keys to ensure parent directories are processed before their children. - # This is crucial for the child exclusion logic to work correctly. - sorted_paths = sorted(in_between.keys(), key=len) - - for path in sorted_paths: - b_info = in_between[path] - b_type = b_info.get("node_type") - - # Constrained nodes: always surface so the caller can list them - # under "not handled automatically". - # Check if this path is a child of an already included directory - is_child_of_included_dir = False - for d in included_dirs: - if path.startswith(d + os.sep): - is_child_of_included_dir = True - break - - if is_child_of_included_dir: - continue - - # Constrained nodes: always surface so the caller can list them - # under "not handled automatically". - if b_type == "constrained": - result[path] = b_info - if b_type == "directory": - included_dirs.add(path) - continue - - - - b_mtime = b_info.get("mtime") - a_info = A.get(path) - - # Case 1: path not in A at all -> include (new file/dir in in-between) - if a_info is None: - result[path] = b_info - if b_type == "directory": - included_dirs.add(path) - continue - - # We only do "newer" semantics for regular files. - if b_type != "file": - continue - - a_type = a_info.get("node_type") - - # Case 2: A has non-file, B has file -> include - if a_type != "file": - result[path] = b_info - # Note: b_type must be "file" here due to the check above, so no need - # to check for directory inclusion. - continue - - # Case 3: both are files; compare mtime - a_mtime = a_info.get("mtime") - if ( - isinstance(a_mtime, (int, float)) - and isinstance(b_mtime, (int, float)) - and b_mtime > a_mtime - ): - result[path] = b_info - # Note: b_type must be "file" here, so no need to check for directory inclusion. - - if meta.debug_has("in_between_newer"): - tree_dict_print(result) - - return result - diff --git a/tool_shared/authored/version b/tool_shared/authored/version deleted file mode 100644 index 9e30ada..0000000 --- a/tool_shared/authored/version +++ /dev/null @@ -1,2 +0,0 @@ -Harmony v1.0 2025-11-25 - diff --git a/tool_shared/document/.gitkeep b/tool_shared/document/.gitkeep deleted file mode 100644 index e69de29..0000000 diff --git a/tool_shared/document/install_Python.org b/tool_shared/document/install_Python.org deleted file mode 100644 index 4725c04..0000000 --- a/tool_shared/document/install_Python.org +++ /dev/null @@ -1,75 +0,0 @@ -#+TITLE: Installing Python in Harmony -#+AUTHOR: Thomas Walker Lynch -#+OPTIONS: toc:2 num:nil - -* Overview - -This document describes how to install a project-local Python environment under: - -#+begin_src bash -tool_shared/third_party/Python -#+end_src - -This environment is shared across the =developer= and =tester= roles and is automatically activated through their respective =env_= scripts. - -* Precondition - -Ensure the following: - -- You are in a POSIX shell with =python3= installed. -- The =python3-venv= package is available (on Debian: =sudo apt install python3-venv=). -- You have sourced the Harmony environment via =env_toolsmith= to initialize =REPO_HOME= and related variables. - -* Step-by-Step Installation - -1. Source the Harmony environment: - #+begin_src bash - source env_toolsmith - #+end_src - -2. Create the virtual environment: - #+begin_src bash - python3 -m venv "$REPO_HOME/tool_shared/third_party/Python" - #+end_src - -3. Activate it temporarily to install required packages: - #+begin_src bash - source "$REPO_HOME/tool_shared/third_party/Python/bin/activate" - pip install --upgrade pip - pip install pytest # Add any shared packages here - deactivate - #+end_src - -4. Rename Python's default activate and deactivate: - Harmony provides its own role-aware environment management. Using Python’s default activation scripts may interfere with prompt logic, PATH order, and role-specific behavior. - - Disable the default scripts by renaming them: - #+begin_src bash - mv "$REPO_HOME/tool_shared/third_party/Python/bin/activate" \ - "$REPO_HOME/tool_shared/third_party/Python/bin/activate_deprecated" - #+end_src - - This ensures that accidental sourcing of Python’s =activate= script won't override Harmony's environment setup. - -5. Verify installation: - #+begin_src bash - ls "$REPO_HOME/tool_shared/third_party/Python/bin/python3" - #+end_src - - The binary should exist and report a working Python interpreter when run. - -* Notes - -- The virtual environment is deliberately named =Python=, not =venv=, to reflect its role as a shared system component. -- Harmony environment scripts define and control =VIRTUAL_ENV=, =PYTHON_HOME=, and =PATH=, making Python activation seamless and uniform. -- There is no need to use Python’s =bin/activate= directly — it is fully replaced by Harmony’s environment logic. - -* Related Files - -- =tool_shared/bespoke/env= -- =tool_shared/bespoke/env_source= -- =env_developer=, =env_tester=, =env_toolsmith= - -* Last Verified - -2025-05-19 :: Activate/deactivate renamed post-install. Requires Harmony environment sourcing prior to execution. diff --git a/tool_shared/document/install_generic.org b/tool_shared/document/install_generic.org deleted file mode 100644 index add47df..0000000 --- a/tool_shared/document/install_generic.org +++ /dev/null @@ -1,81 +0,0 @@ - -This is the generic install.org doc that comes with the skeleton. - -1. $REPO_HOME/tool_shared/third_party/.gitignore: - - * - !/.gitignore - !/patch - - The only things from the third party directory that will be pushed to the repo origin is the .gitignore file and the patches. - - -2. downloaded tar files etc. go into the directory `upstream` - - $REPO_HOME/tool_shared/upstream - - Typically the contents of upstream are deleted after the install. - -3. for the base install - - cd $REPO_HOME/tool_shared/third_party - do whatever it takes to install tool, as examples: - git clone - tar -xzf ../upstream/tar - ... - - Be sure to add the path to the tool executable(s) in the $REPO_HOME/env_$ROLE files for the $ROLE who uses the tool. - - Assuming you are not also developing the tool, for safety - change each installed git project to a local branch: - - b=__local_$USER - git switch -c "$b" - - -4. Define some variables to simplify our discussion. Lowercase variable names - are not exported from the shell. - - # already set in the environment - # REPO_HOME - # PROJECT - # USER - - # example tool names: 'RT_gcc' 'RT-project share` etc. - tool= - tool_dpath="$REPO_HOME/tool_shared/third_party/$tool" - patch_dpath="$REPO_HOME/tool_shared/patch/" - - -5. create a patch series (from current vendor state → your local edits) - - # this can be repeated and will create an encompassing diff file - - # optionally crate a new branch after cloning the third party tool repo and work from there. You won't make any commits, but in case you plan to ever check the changes in, or have a the bad habit of doing ommits burned into your brain-stem, making a brnch will help. - - # make changes - - cd "$tool_dpath" - - # do your edits - - # Stage edits. Do not commit them!! Be sure you are in the third party - # tool directory when doing `git add -A` and `git diff` commands. - git add -A - - # diff the stage from the current repo to create the patch file - git diff --staged > "$patch_dpath/$tool" - - # the diff file can be added to the project and checked in at the project level. - - -6. how to apply an existing patch - - Get a fresh clone of the tool into $tool_dpath. - - cd "$tool_dpath" - git apply "$patch_dpath/$tool" - - You can see what `git apply` would do by running - - git apply --check /path/to/your/patch_dpath/$tool diff --git a/tool_shared/third_party/.gitignore b/tool_shared/third_party/.gitignore deleted file mode 100644 index 0de97f0..0000000 --- a/tool_shared/third_party/.gitignore +++ /dev/null @@ -1,8 +0,0 @@ -# Ignore all files -* - -# But don't ignore the .gitignore file itself -!/.gitignore - -# keep the upstream directory -!/upstream diff --git a/tool_shared/third_party/upstream/.gitignore b/tool_shared/third_party/upstream/.gitignore deleted file mode 100644 index aa0e8eb..0000000 --- a/tool_shared/third_party/upstream/.gitignore +++ /dev/null @@ -1,2 +0,0 @@ -* -!/.gitignore \ No newline at end of file