-|2022-02-26|` `TWL|brought the README.md into doc/README.odt (still converting), then using online tool to convert the pdf to README.md|\r
-| :-: | :- | :- |\r
-||||\r
-\r
-**The ‘resources’ Project**\r
-1. # **What is it?**\r
-A set of scripts used for the projects found in github.com/thomas-walker-lynch and other places where it has been adopted. These apply to C, C++, Python, Django, Rust, Lisp, etc. This replaces any other script you might be accustom to for entering a ‘virtual environment’. \r
-\r
-|setup|sets up the environment for a project|\r
-| :- | :- |\r
-|pull|fancy shortcut for git pull|\r
-|push|fancy shortcut for git push|\r
-|makefile|a genetic makefile for C/C++ projects|\r
-|makefil-env|default variables for make|\r
-\r
-These scripts have been evolving with each project, so tweaks might be required.\r
-\r
-Assume we have a top level directory called repos, and we have expanded resources\r
-\r
-/home/Thomas/repos/\r
-\r
-` `resources\_repo/\r
-\r
-` `bin/<br>\r
-\r
-` `include/\r
-\r
-` `lib/\r
-\r
-` `media/\r
-\r
-` `tmp/\r
-\r
-` `LICENSE\r
-\r
-` `README.md\r
-\r
-` `projects-init.sh\r
-\r
-1. # **Modifying the resources repo**\r
-If you make some changes and are testing them, then do that on a branch. Merge the branch after it is tested.\r
-\r
-If you do something to the resources repo of general interest, then check it in to master.\r
-\r
-If you need special modifications specific to your needs, i.e. not of general interest even to those who might come in the future and be working in the same language, etc., then make your very own branch.\r
-\r
-` `If you have a fundamentally different way of going about this, some people want it, others don't - then that goes on a branch.\r
-\r
-1. # **Your local directory for holding project, ‘repos’**\r
-\r
-1. make a top level directory for holding all the repos you work on, independent of language, etc.\r
-\r
-` `> mkdir ~/repos\r
-\r
-1. expand the system repo, then inspect and install those script in the system\r
-\r
-\> cd ~/repos\r
-\r
-\> git clone git@github.com:Thomas-Walker-Lynch/system\r
-\r
-Follow the directions from the system repo for installing home, user, and Z. There is not really much to it. ‘Z’ is used for timestamps. home returns the home directory from /etc/passwd.\r
-\r
-1. expand the resources repo\r
-\r
-` `> cd ~/repos\r
-\r
-` `> git clone git@github.com:Thomas-Walker-Lynch/resources\r
-\r
-1. add the following to .bashrc. Note the file lib/bashrc to get started.\r
-\r
-` `> export PATH=~/repos/resources/bin:"$PATH"\r
-\r
-\r
-1. # **The setup script**\r
-\r
-` `This command will open a new shell with the environment in that shell setup for the project.\r
-\r
-` `2022-02-25T13:06:24Z\r
-\r
-` `lander10@beast§~/§\r
-\r
-` `> setup Endian\r
-\r
-` `Hello Emacs\r
-\r
-` `/home/Thomas/repos/bin:/home/Thomas/repos/resources/bin:/usr/local/bin:/usr/bin:/bin\r
-\r
-` `2022-02-25T13:06:28Z [Endian]\r
-\r
-` `lander10@beast§~/repos/Endian§\r
-\r
-` `>\r
-\r
-Currently <tt>setup</tt> echos the path so that they user will understand where the executables\r
-\r
-come from.\r
-\r
-The prompt carries sufficient information for making sense of the transcript when it is read later.\r
-\r
-On the first line, the time shown is UTC in standard iso8601 format. This comes from the ‘/usr/bin/Z’ script.\r
-\r
-Following the time, in square brackets you will see the name of the project. If the square brackets are not there, then no project environment has been setup. If a user attempts to work on a project that has not been setup, he or she is likely to experience many errors.\r
-\r
-On the second line we have the user name, machine name, and current working directory.\r
-\r
-On the third line the familiar `>` appears. Anything you type after the prompt is taken as the command for the shell. It is conventional for script run as root user that this will instead be '#'.\r
-1. # **The generic makefile**\r
-\r
-` `By default a C/C++ project wil have this directory structure:\r
-\r
-` `~/repos/\r
-\r
-` `Endian/\r
-\r
-` `deprecated/\r
-\r
-` `doc/\r
-\r
-` `exec/\r
-\r
-` `lib/\r
-\r
-` `src/\r
-\r
-` `test/\r
-\r
-` `tmp/\r
-\r
-` `try/\r
-\r
-` `LICENSE\r
-\r
-` `makefile\r
-\r
-` `makefile-env\r
-\r
-` `resources/\r
-\r
-` `system/\r
-\r
-` `tmp/\r
-\r
-Note <tt>makefile</tt> and <tt>makefile-env</tt> inside the Endian project are there only to override defaults on the <tt>resources</tt> versions of those files. They are stubs tweak a couple of things and then pass the heavy lifting to <tt>resources/lib/makefile</tt> and <tt>resources/lib/makefile-env</tt>. Leave the <tt>resources/lib/makefile</tt> and <tt>resources/lib/makefile-env</tt> where they are and do not make copies.\r
-\r
-Give source files two suffixes. For example the Endian project source directory appears as:\r
-\r
-` `2022-02-26T10:54:55Z [Endian]\r
-\r
-` `thomas@beast§~/resources§\r
-\r
-` `> cd $PROJECT\_HOME\r
-\r
-` `2022-02-26T10:55:47Z [Endian]\r
-\r
-` `thomas@beast§~/Endian§\r
-\r
-` `> ls src\r
-\r
-` `endian.lib.c endian\_test.cli.c ip4.lib.c ip4\_test.cli.c\r
-\r
-Endian is a C project. For C++ sources the suffix will be <tt>.cc</tt>. I suppose the generic makefile should be improved by adding a variable to configure the string used for suffixes. Right now I think this convention is hard coded.\r
-\r
-The compiled lib files will go into <tt>lib/$PROJECT\_lib.a</tt> while the cli files will be made stand alone and appear in the <tt>exec</tt> directory. Presumably all the <tt>.cli.c</tt> files have a main function defined.\r
-1. # **Push and pull scripts**\r
-These are git command sequence shortcuts. They do the usual things when pushing and pulling from a git repo and also do some checks.\r
-1. # **Other things**\r
-` `Other useful things will be found in the resource tree.\r
-1. # **Projects with submodules**\r
-\r
-` `By convention, projects with submodules have the suffix <tt>\_ensemble</tt>.\r
-\r
-` `1. install the project\r
-\r
-` `Suppose the project with submodules is called <project>\_ensemble\r
-\r
-` `> cd ~/repos\r
-\r
-` `> git clone git@github.com/<user>/<project>\_ensemble.git\r
-\r
-` `> mv <project>\_ensemble <project>\_master\r
-\r
-` `> cd <project>\_master\r
-\r
-` `Note in the line that moves the cloned repo directory to `<project>\_master`, you might\r
-\r
-` `use a different suffix than `master`. Conventionally the suffix is the branch to be\r
-\r
-` `checked out and worked on, but the scripts do not care what it is set to. Inside the\r
-\r
-` `scripts this suffix is called the project ‘version’. If this checkout is for\r
-\r
-` `working on multiple branches, just leave the <tt>\_ensemble</tt>.\r
-\r
-` `2. for submodules that have not yet been added:\r
-\r
-\r
-` `> git submodule add https://github.com/../<resource\_project>\r
-\r
-` `> git submodule add https://github.com/../<resource\_project>\r
-\r
-` `...\r
-\r
-\r
-` `Etc. for the other modules\r
-\r
-` `3. if a submodule is empty, then do the following:\r
-\r
-` `> git submodule init\r
-\r
-` `> git submodule update\r
-\r
-\r
-` `Submodules directories will be empty when the `--recursive` switch is not provided with\r
-\r
-` `the clone. Actually, I prefer not to use `--recursive` and then to follow up with an\r
-\r
-` ``init` and `update` so that it is easier to tell what caused errors.\r
-\r
-1. # **## generally about project security**\r
-\r
-` `0. Contain your development work\r
-\r
-` `Consdider giving the repos directory its own user. If you do add this link so that\r
-\r
-` `the scripts can still have a 'repos' target.\r
-\r
-` `> ln -s . repos\r
-\r
-\r
-` `Alternatively put them in a container or a virtual machine.\r
-\r
-` `1. audit source files and legible scripts\r
-\r
-` `Watch what git pulls. For new things that are source code or scripts, audit those.\r
-\r
-` `Note! ‘.gitignore’ does not apply to pulled content.\r
-\r
-` `2. hidden files\r
-\r
-` `Files that should be audited must not be hidden. Otherwise they are too easy to miss during\r
-\r
-` `the audit.\r
-\r
-` `Hidden files are not seen by file globs, so does not see the <tt>.gitignore</tt> file.\r
-\r
-\r
-` `> ls -ld \*\r
-\r
-` `drwxrwx---. 1 thomas thomas 64 2022-02-26 07:55 bin\r
-\r
-` `drwxrwx---. 1 thomas thomas 142 2022-02-26 07:03 lib\r
-\r
-` `-rw-rw----. 1 thomas thomas 1076 2022-02-25 07:25 LICENSE\r
-\r
-` `drwxrwx---. 1 thomas thomas 20 2022-02-25 07:25 media\r
-\r
-` `-rw-rw----. 1 thomas thomas 141 2022-02-25 07:25 projects-init.sh\r
-\r
-` `-rw-rw----. 1 thomas thomas 18209 2022-02-26 08:02 README.md\r
-\r
-` `drwxrwx---. 1 thomas thomas 84 2022-02-26 05:40 tmp\r
-\r
-\r
-` `Globs not seeing hidden files is generally considered to be a good thing when we do not want usual shell operations\r
-\r
-` `to see it. For example <tt> cp \* </tt> a repo manually we often want a copy of the project, not a copy of the repo\r
-\r
-` `administrative files. There is a separate command for copying a repo, <tt> git clone </tt>.\r
-\r
-` `If we want to 'unhide' dot file from a file glob, then we rename it. That is best. In the case <tt>.gitignore</tt>\r
-\r
-` `<tt>git</tt> is expecting to see the file with that very name. If we make a file that is not hidden and then\r
-\r
-` `link <tt>.gitignore</tt> to it, git will yell at us "too many symbolic links!".\r
-\r
-` `Hence, we can expose a file like <tt>.gitignore</tt> by linking to it (rather than the other way around).\r
-\r
-` `When doing this, typically the link name is the same but without the dot in front of it. Here I expose the\r
-\r
-` `<tt>.gitignore</tt>. \r
-\r
-\r
-` `2022-02-26T13:02:28Z [Endian]\r
-\r
-` `thomas@beast§~/resources§\r
-\r
-` `> ln -s .gitignore gitignore\r
-\r
-` `And now the file glob will pick it up.\r
-\r
-\r
-` `2022-02-26T13:07:13Z [Endian]\r
-\r
-` `thomas@beast§~/resources§\r
-\r
-` `> ls -ld \*\r
-\r
-` `drwxrwx---. 1 thomas thomas 64 2022-02-26 07:55 bin\r
-\r
-` `lrwxrwxrwx. 1 thomas thomas 10 2022-02-26 08:07 gitignore -> .gitignore\r
-\r
-` `drwxrwx---. 1 thomas thomas 142 2022-02-26 07:03 lib\r
-\r
-` `-rw-rw----. 1 thomas thomas 1076 2022-02-25 07:25 LICENSE\r
-\r
-` `drwxrwx---. 1 thomas thomas 20 2022-02-25 07:25 media\r
-\r
-` `-rw-rw----. 1 thomas thomas 141 2022-02-25 07:25 projects-init.sh\r
-\r
-` `-rw-rw----. 1 thomas thomas 20454 2022-02-26 08:14 README.md\r
-\r
-` `drwxrwx---. 1 thomas thomas 84 2022-02-26 05:40 tmp\r
-\r
-\r
-` `2. PATH\r
-\r
-` `Do not put a repo in your execution PATH. When running tests and such call them out locally, <tt>./program</tt>,\r
-\r
-` `or through one of the environment variables initialized by the <tt>setup</tt>.\r
-\r
-` `The <tt>~/repos/resources/bin</tt> directory is an exception to this. We need to maintain the executables used for\r
-\r
-` `maintaing the projects themselves. Be careful to audit any changes to this <tt>resources</tt>.\r
-\r
-` `3. pulled binary executbles\r
-\r
-` `It is best to not pull binary executables then run them, because they can not be audited.\r
-\r
-` `It sometimes happens that a co-developers will compile and create an executable, and then not clean it, and then\r
-\r
-` `accidentally the user poisons the repo with them. Watch your pulls and clones, and remove executables from the\r
-\r
-` `repo, and complain about them. The generic makefile, push, and pull scripts can help with this.\r
-\r
-` `When an unauditable executable arrives in a pull it creates a trust problem. We distrist both the\r
-\r
-` `source and the integrity of the distribution process. This latter problem can be mitigated if\r
-\r
-` `cryptographic signatures are independently available.\r
-\r
-` `Consider running such an executable in a container.\r
-\r
-` `Make sure make your clean targets remove executables before a push.\r
-\r
-` `Be careful with the programs in <tt>resources/bin</tt>.\r
-\r
-` `4. check for the .gitignore and audit it, add to it\r
-\r
-` `If you add the <tt>exec</tt> or <tt>bin</tt> directories to gitignore and then forget to\r
-\r
-` `run make clean, it might prevent accidentally adding executables.\r
-\r
-` `Typical .gitignore files:\r
-\r
-` `env/\r
-\r
-` `tmp/\r
-\r
-` `.\*\r
-\r
-` `!.gitignore\r
-\r
-` `\*~\r
-\r
-` `For a C or C++ project home directory we will also ignore various intermediate files,\r
-\r
-\r
-` `tmp/\r
-\r
-` `bin/\r
-\r
-` `exec/\r
-\r
-` `.\*\r
-\r
-` `!.gitignore\r
-\r
-` `\*~\r
-\r
-` `\*.o\r
-\r
-` `\*.i\r
-\r
-` `\*.s\r
-\r
-` `a.out\r
-\r
-\r
-` `For a python project:\r
-\r
-` `tmp/\r
-\r
-` `.\*\r
-\r
-` `!.gitignore\r
-\r
-` `\_\_pycache\_\_/\r
-\r
-` `\*\*/\*.pyc\r
-\r
-\r
-` `And for a django project:\r
-\r
-\r
-` `tmp/\r
-\r
-` `.\*\r
-\r
-` `!.gitignore\r
-\r
-` `\_\_pycache\_\_/\r
-\r
-` `\*\*/\*.pyc\r
-\r
-` `manage.py\r
-\r
-` `\*\*/migrations\r
-\r
-` `.vscode\r
-\r
-\r
-\## General info and concepts\r
-\r
-` `### What is a ‘project’\r
-\r
-` `The term ‘project’ occurs repeatedly in this document and generally when we talk about code, so it is best to start by\r
-\r
-` `nailing that term down. Generally speaking, a ‘project’ has a well defined final product, along with a list resource\r
-\r
-` `needs, tasks, task dependencies, and a tasks schedule. It is the job of the team members to execute said tasks.\r
-\r
-` `In project management parlance, a group of related projects is called a ‘program’. Well that sure is an unfortunate\r
-\r
-` `choice of terminology for us CS people, so instead we will call a group of related projects a ‘project ensemble’. A\r
-\r
-` `project ensemble itself is also a kind of project, where the tasks are the component projects. Hence we have a\r
-\r
-` `recursive structure. CS people like recursive structures ;-)\r
-\r
-` `Projects are held in git repositories, as project ensembles are also projects, they are also held in git\r
-\r
-` `repositories. When we clone a project ensemble we will find other git repositories have been expanded inside the\r
-\r
-` `directory tree. Those are for the component projects. In git speak we call these component project directory trees\r
-\r
-` `‘submodules’.\r
-\r
-` `When a project ensemble is expanded out, we end up with a directory tree structure where project resources, tools, and\r
-\r
-` `work products are stored.\r
-\r
-\r
-1. # ` `**### Where project code goes**\r
-` `--------\r
-\r
-` `On a typical project we will have three distinct types of code:\r
-\r
-` `1. the application source code\r
-\r
-` `2. the libraries and other resources the application makes use of\r
-\r
-` `3. the tools used for building the source code\r
-\r
-` `Let's give these code types short names:\r
-\r
-` `1. source\r
-\r
-` `2. resources\r
-\r
-` `3. tools\r
-\r
-` `We have various places where we might put code that we need in a program:\r
-\r
-` `1. in the system\r
-\r
-` `2. in the developer's user directory\r
-\r
-` `3. in a project ensemble\r
-\r
-` `4. in a project's home directory.\r
-\r
-\r
-` `We shorten this list of places to:\r
-\r
-` `1. system\r
-\r
-` `2. user\r
-\r
-` `3. ensemble\r
-\r
-` `4. project\r
-\r
-` `Now combining our code and locations into one list:\r
-\r
-` `1. source\r
-\r
-` `1. project\r
-\r
-` `2. resources\r
-\r
-` `1. system\r
-\r
-` `2. user\r
-\r
-` `3. ensemble\r
-\r
-` `3. tools\r
-\r
-` `1. system\r
-\r
-` `2. user\r
-\r
-` `3. ensemble\r
-\r
-` `So there is only one place we will find the application source code that we are\r
-\r
-` `developing, and that is under the project directory. Resources that we may need in\r
-\r
-` `order to compile our code will be found either in the system, the user directory, or in\r
-\r
-` `the ensemble. The same can be said for the tools we will use.\r
-\r
-` `This is what my home directory looks like:\r
-\r
-` `/home/thomas/\r
-\r
-` `bin/\r
-\r
-` `Desktop/\r
-\r
-` `Documents/\r
-\r
-` `Downloads/\r
-\r
-` `repos/\r
-\r
-` `chessgame/\r
-\r
-` `resources/ <--- resources for all projects - expand this repo here\r
-\r
-` `LICENSE\r
-\r
-` `README.md\r
-\r
-` `bin/\r
-\r
-` `makefile-cc\r
-\r
-` `pull\r
-\r
-` `push\r
-\r
-` `rm\_tilda\_files\_tree\r
-\r
-` `setup\r
-\r
-` `subu/\r
-\r
-` `tm/\r
-\r
-` `ws4\_master/ <--- an ensemble directory\r
-\r
-` `LICENSE\r
-\r
-` `README.md\r
-\r
-` `env/ <--- resources specific to the ensemble (wish this was called ‘resources’\r
-\r
-` `bin/\r
-\r
-` `lib/\r
-\r
-` `include/\r
-\r
-` `tmp/\r
-\r
-` `uWebSockets/ <--- resource project directory\r
-\r
-` `ws4/ <--- target project directory\r
-\r
-` `Now looking under my `projects` directory, and expanding out `ws4\_master`:\r
-\r
-\r
-` `ws4\_master/\r
-\r
-` `LICENSE\r
-\r
-` `README.md\r
-\r
-` `env/\r
-\r
-` `bin/\r
-\r
-` `include/\r
-\r
-` `lib/\r
-\r
-` `tmp/\r
-\r
-` `uWebSockets/\r
-\r
-` `ws4/\r
-\r
-\r
-` ``ws4\_master` is a project ensemble. The component projects include `uWebSockets` and\r
-\r
-` ``ws4`. This project ensemble also comes from a git repo.\r
-\r
-` ``ws4\_master/ws4` is the home directory for the project this team is actively working on.\r
-\r
-` `We know it is the active project because it has the same name as the ensemble prefix.\r
-\r
-` `Whereas `ws4\_master/uWebSockets` is someone else's github project. There are other people\r
-\r
-` `working that, but it is not us, and it is being developed in a different\r
-\r
-` `environment. Rather we are just making use of the work product of that project.\r
-\r
-` `Also inside of `ws4\_master` we have a directory called `env`. Frankly, I don't like the\r
-\r
-` `name. I have toyed with calling it `project-share`, but `env` is the name that Python and others\r
-\r
-` `expect. Perhaps make it a symbol link? This directory is used to hold project specific\r
-\r
-` `resources and tools. Note that the contents of `env` are \*not\* pushed to the repo. This\r
-\r
-` `means that custom edits you make to scripts will not be backed up to the repo. I also do\r
-\r
-` `not like `env` because it is not pushed to the repo, but it might be \*pulled\* from it.\r
-\r
-` ``.gitignore` does not affect pulls. This is a security hazard.\r
-\r
-\r
-1. # **## Repo and Directory Naming (again)**\r
-\r
-` `If a repo has submodules in it, I generally give the repo name a suffix of ‘\_ensemble’.\r
-\r
-` `After cloning, if I am only going to work on a given breanch within that clone, I will change\r
-\r
-` `the suffix to the branch name, <project>\_<branch>.\r
-\r
-` `So you ask why do we need more than one directory for the same repo? Well in this case I\r
-\r
-` `am running a web server against the v2.0 branch, and it needs to see the files the v2.0.\r
-\r
-` `The v1.0 directory was the one that was formally being served. There are no servers\r
-\r
-` `pointed at it now, so I should probably delete it. If I ever need v1.0 again I can always\r
-\r
-` `check it out.\r
-\r
-` `> cd ~/repos\r
-\r
-` `> ls\r
-\r
-` `customer\_gateway\_master\r
-\r
-` `customer\_gateway\_v1.0\r
-\r
-` `customer\_gateway\_v2.0\r
-\r
-\r
-` `This is how the `customer\_gateway\_master` project environment directory was made:\r
-\r
-\r
-` `> cd ~/repos\r
-\r
-` `> git clone --recursive --jobs 8 git@github.com:Reasoning-Technology/customer\_gateway\_ensemble.git\r
-\r
-` `> mv customer\_gateway\_ensemble customer\_gateway\_master\r
-\r
-` `When downloading a `<project>\_ensemble` repo, we can expand the submodules at the same time\r
-\r
-` `by including the --recursive switch, as was shown above. If the `--recursive` switch\r
-\r
-` `is not given, the submodules will have to be initialized and updated.\r
-\r
-` `The second directory was then created with the commands:\r
-\r
-\r
-` `> git clone --recursive --jobs 8 git@github.com:Reasoning-Technology/customer\_gateway\_ensemble.git\r
-\r
-` `> mv customer\_gateway\_ensemble customer\_gateway\_v1.0\r
-\r
-` `> cd customer\_gateway\_v1.0\r
-\r
-` `> git checkout v1.0\r
-\r
-\r
-` `Cloning a `git` repository produces a directory tree, which in git speak is\r
-\r
-` `apparently called a `module`.\r
-\r
-` `We may `cd` into a module and clone another module, this will be called a ‘submodule’. A\r
-\r
-` `submodule clone operation requires a special command so that the module will know it is\r
-\r
-` `there:\r
-\r
-\r
-` `git submodule add <repo>\r
-\r
-\r
-` `We might do this because our project depends on other projects, and those other projects\r
-\r
-` `have their own git repositories, or because we are developing more than one project\r
-\r
-` `together with one shared environment.\r
-\r
-` `We then use the submodule just like we would use any other git module. I.e. after we make\r
-\r
-` `changes we must add the changes, then commit them, and then push the submodule.\r
-\r
-` `The parent module only sees the submodule changes when there is a commit in the\r
-\r
-` `submodule. Hence, after there is a commit in a submodule, we must go up to the module, and\r
-\r
-` `then add the submodule, commit, then push the module.\r
-\r
-` `We truly have two layers, and we have to maintain them individually. Luckily we have some\r
-\r
-` `scripts so that we don't to type stuff twice. In the `project-share` project there are two\r
-\r
-` `scripts, one called `push` the other called `pull`. When we run the `push` script it goes\r
-\r
-` `into the project home and does an add, commit, and push. It then goes up to the ensemble\r
-\r
-` `directory and `git add`s the project submodule, commits the change, and then pushes. Finally\r
-\r
-` `it pops directory back to the project. The current `pull` script pulls down all the\r
-\r
-` `submodules and the ensemble.\r
+\r
+This project holds the builds for simple tools and scripts tha are common to\r
+projects, documents that are shared by projects, and useful configuration files.\r
+It will also serves as a project skeleton when it is copmleted.\r
\r
\r
projects / RT-project-share / commitdiff