.. include:: Articles.rst
.. include:: History.rst
.. include:: <isoamsa.txt>

.. include:: Programming-Roles.rst

.. :Blox:`document.css, toc.css, literal.css`


The ``bxpublish`` Program
#########################

.. include:: Navbar-Articles.rst

.. _last: 2022-01-26_1850_

.. footer::

    .. container:: flexfooter

        *Released 30 November 2023.*

        |prev| | |next| | |last|_

.. sectnum::

.. contents:: Contents of this Page


Overview
========

The ``bxpublish`` program generates a copy of a directory structure with
certain files omitted.  It can be used to provide a *publication directory*
as a staging area for uploading a Blox project.  Using ``bxpublish``, it is
possible to prevent staging file which have been recreated without change
in content.  In this way, traffic for upload might be reduced.  It then
suffices to compare the modification dates of the files in the *publication
directory* and those *on the server*.


:file:`publish.txt` Files
=========================

``bxpublish`` reads its specification about which files to include and
which not from configuration files called :file:`publish.txt`.  There are
two kinds of instruction lines in a :file:`publish.txt` file:

1.  *Path Group lines* consist of a *directory pattern* directly followed
    by a colon (without any whitespace in between).  Any whitespace
    preceding the *pattern* and whitespace following the colon will be
    ignored.

2.  *Exclusion/Inclusion lines* (called *definition lines*) are lines
    starting by a ``-`` or a ``+``, followed by a *filename pattern*.
    There can be any amount of whitespace (or none) between the type
    character (``+`` or ``-``) and the filename pattern.  *Inclusion lines*
    use the ``+``, *exclusion lines* the ``-``.  Whitespace at the
    beginning of the line as well as whitespace at its tail is ignored.

The *directory patterns* as well as the *name patterns* are interpreted as
regular expressions.  Space characters *inside* of them will be part of the
pattern.  An example :file:`publish.txt` file is::

    - notneeded\.txt
    + included\.tar\.gz

    dir(ectory)?:
    - .*\.ext

Each *path group line* pertains to the following definition lines, until
another path group line is read.  Because of this behaviour path group
lines partition the definition lines into *groups*.

Comments are started by a hashmark ``#`` and extend till the end of the
line.  Comments are filtered out before any other processing of the line.
Because of the whitespace permitted at the end of *definition lines* and
*path group lines*, whitespace immediately before the hash mark will be
ignored.  Comments might in turn contain any number of further hash marks.
There is no way to escape a hash mark, so it cannot be used as part of a
*path group* or *definition pattern*.


Work Steps
==========

The ``bxpublish`` program will first read all :file:`publish.txt` files, in
order to decide about each and every file in the source directory whether
it needs inclusion or not.  This results in an *inventory* of files which
shall be published.  The ongoing operation of ``bxpublish`` consists of
three steps:

1.  All files in the target directory which are not part of the inventory
    will be pruned.

2.  For each file in the inventory, ``bxpublish`` will examine if it needs
    to be copied, and will copy the file when necessary.

3.  After having copied over files as needed, ``bxpublish`` will check for
    each directory in the destination hierarchy whether it is empty, and
    will prune such empty directories.  The directories are iterated
    *bottom-up*, s.t. directories which do not contain any file on any
    level will turn out empty at the time they are checked, and will be
    pruned.


Copying of Files
================

Files will be copied when the content of the resp. source file differs from
the target file's content.  There are several conditions which can be met
when deciding about a file part of the inventory whether it needs copying
or not:

1.  When the destination exists as a *directory*, the file will be skipped.
    In this case, since the origin exists as a *file*, no files below the
    destination will be part of the inventory, s.t. the destination will be
    eligible for pruning right after the file-pruning step and will be
    deleted in the directory pruning step.  ``bxpublish`` should thus be
    ran again; an appropriate message will be output.  Only in dry run mode
    the destination might survive.

2.  In case the destination does not exist or the size of the destination
    differs from the size of the source, the file will be copied over
    using ``rsync -t``.

3.  When the origin and the destination exist and are of equal size, their
    *content* is compared in order to decide about the copying operation.

    In the beginning of its operation, ``bxpublish`` looks for a *hash
    generator*.  Currently considered are ``sha1sum`` and ``md5sum``.  The
    choice can be overridden manually by the environmental variable
    ``HASHGEN``.  The value ``none`` stands for the absence of any hash
    generator.  The hash generator is expected to produce output starting
    with a file's hash when being called as ``$ <HASHGENERATOR> <FILE>``.

    With a hash generator present, it will be used to compare the contents
    of the origin and the target files, and when the file needs copying,
    ``rsync -t`` will be used on it.

    When no hash generator could be found, the ``-c`` option to ``rsync``
    will be used to hand over the decision to ``rsync``, thus ``rsync
    -tc`` will effectively be employed.

Source files which are content-identical to their destination counterpart
are to be skipped.  In case of the absence of a hash generator, this
condition however cannot be detected, as it is indistinguishable from the
situation of *differing* content with equal size, where copying is
required; so copying cannot be skipped.  Under such circumstances all
source files will be copied over *unconditionally*, possibly with hand-over
of the content question to ``rsync``.  The different reasons for performing
a copy operation are still iterated to provide fine-grained information to
the user at increased verbosity levels.  Notice that ``rsync`` might *skip*
the file *on its own*, when ``rsync -tc`` detects using its built-in hash
algorithm that there are no differences present.  However, only in the
highest verbosity level the user will be able to notice.


Catalogue Generation
====================

To find the :file:`publish.txt` files, ``bxpublish`` looks for directories
beneath the source directory, checking for each directory found whether it
contains such a file.  The list of directory pathes is sorted prior to this
examination.

While reading the :file:`publish.txt` files, a *catalogue* of rules is
generated, one for each *definition line*.  Each such rule has four
components:

1.  Each rule has a *type*, which can either be of *inclusive* (``+``
    lines) or *exclusive* (``-`` lines).
2.  The path component of the examined :file:`publish.txt` file relative to
    the source directory is called the *root* of the rule.
3.  The *path pattern* from the last *path group line* is stored as the
    *path pattern* of the rule.  When no path group line has been read yet
    in the current :file:`publish.txt` file, the *empty string* will be
    used as the path pattern.
4.  Finally, the *filename pattern* of the *definition line* makes up the
    *name pattern* of the rule.

There is a default exclusion rule defined to not publish the
:file:`publish.txt` files.  This rule is rooted at the source directory,
its *path pattern* matches *any path* and its *name pattern* applies
precisely to ``publish.txt``.


Application of the Catalogue to Files
=====================================

For each file within the source directory it is ascertainted for each rule
if the file is *covered* by the given rule:

1.  The path of the file *relative* to the source directory is calculated.
    This will be a path starting with a dot.  When this source-relative
    path *doesn't start* with the rule's *root directory* components, the
    rule does not apply.

2.  Otherwise, the path of the file in question *relative to the rule's
    root path* is computed.  This path is *abbreviated* by stripping
    away the leading ``'./'`` if the path does not consist of the dot '.'
    alone.  When the relative path is precisely the single dot, it is
    abbreviated as the null string.

3.  Now, when the abbreviated path matches the rule's *path pattern*, and
    when the given file's basename also matches the rule's *name pattern*,
    the rule *covers the file*.  If any of the two does not match, the rule
    does *not* cover the file.  In order to be considered matching, the
    path pattern and the name pattern need to match the entire respective
    string.

Definition lines at the top of a :file:`publish.txt` file, before any *path
group* line, will create rules with the empty string as their *path
pattern*.  Such rules thus cover files directly within the *root path* of
the :file:`publish.txt` file.

When multiple rules cover a file, the one defined last is decisive.  When
no rule covers a file in question, the file will be included by default.
Because *only the directory pathes* are sorted during :file:`publish.txt`
file discovery, it is guaranteed that each :file:`publish.txt` file in a
certain directory is read *earlier* than :file:`publish.txt` files in
subfolders of the resp. directory.
