The bxpublish Program

Downloads of the Blox Project | Downloads of the entire Site | Imprint

Contents of this Page

1   Overview

The bxpublish program generates a copy of a directory structure with certain files omitted. It can be used to provide a publication directory as a staging area for uploading a Blox project. Using bxpublish, it is possible to prevent staging file which have been recreated without change in content. In this way, traffic for upload might be reduced. It then suffices to compare the modification dates of the files in the publication directory and those on the server.

2   publish.txt Files

bxpublish reads its specification about which files to include and which not from configuration files called publish.txt. There are two kinds of instruction lines in a publish.txt file:

  1. Path Group lines consist of a directory pattern directly followed by a colon (without any whitespace in between). Any whitespace preceding the pattern and whitespace following the colon will be ignored.
  2. Exclusion/Inclusion lines (called definition lines) are lines starting by a - or a +, followed by a filename pattern. There can be any amount of whitespace (or none) between the type character (+ or -) and the filename pattern. Inclusion lines use the +, exclusion lines the -. Whitespace at the beginning of the line as well as whitespace at its tail is ignored.

The directory patterns as well as the name patterns are interpreted as regular expressions. Space characters inside of them will be part of the pattern. An example publish.txt file is:

- notneeded\.txt
+ included\.tar\.gz

dir(ectory)?:
- .*\.ext

Each path group line pertains to the following definition lines, until another path group line is read. Because of this behaviour path group lines partition the definition lines into groups.

Comments are started by a hashmark # and extend till the end of the line. Comments are filtered out before any other processing of the line. Because of the whitespace permitted at the end of definition lines and path group lines, whitespace immediately before the hash mark will be ignored. Comments might in turn contain any number of further hash marks. There is no way to escape a hash mark, so it cannot be used as part of a path group or definition pattern.

3   Work Steps

The bxpublish program will first read all publish.txt files, in order to decide about each and every file in the source directory whether it needs inclusion or not. This results in an inventory of files which shall be published. The ongoing operation of bxpublish consists of three steps:

  1. All files in the target directory which are not part of the inventory will be pruned.
  2. For each file in the inventory, bxpublish will examine if it needs to be copied, and will copy the file when necessary.
  3. After having copied over files as needed, bxpublish will check for each directory in the destination hierarchy whether it is empty, and will prune such empty directories. The directories are iterated bottom-up, s.t. directories which do not contain any file on any level will turn out empty at the time they are checked, and will be pruned.

4   Copying of Files

Files will be copied when the content of the resp. source file differs from the target file's content. There are several conditions which can be met when deciding about a file part of the inventory whether it needs copying or not:

  1. When the destination exists as a directory, the file will be skipped. In this case, since the origin exists as a file, no files below the destination will be part of the inventory, s.t. the destination will be eligible for pruning right after the file-pruning step and will be deleted in the directory pruning step. bxpublish should thus be ran again; an appropriate message will be output. Only in dry run mode the destination might survive.

  2. In case the destination does not exist or the size of the destination differs from the size of the source, the file will be copied over using rsync -t.

  3. When the origin and the destination exist and are of equal size, their content is compared in order to decide about the copying operation.

    In the beginning of its operation, bxpublish looks for a hash generator. Currently considered are sha1sum and md5sum. The choice can be overridden manually by the environmental variable HASHGEN. The value none stands for the absence of any hash generator. The hash generator is expected to produce output starting with a file's hash when being called as $ <HASHGENERATOR> <FILE>.

    With a hash generator present, it will be used to compare the contents of the origin and the target files, and when the file needs copying, rsync -t will be used on it.

    When no hash generator could be found, the -c option to rsync will be used to hand over the decision to rsync, thus rsync -tc will effectively be employed.

Source files which are content-identical to their destination counterpart are to be skipped. In case of the absence of a hash generator, this condition however cannot be detected, as it is indistinguishable from the situation of differing content with equal size, where copying is required; so copying cannot be skipped. Under such circumstances all source files will be copied over unconditionally, possibly with hand-over of the content question to rsync. The different reasons for performing a copy operation are still iterated to provide fine-grained information to the user at increased verbosity levels. Notice that rsync might skip the file on its own, when rsync -tc detects using its built-in hash algorithm that there are no differences present. However, only in the highest verbosity level the user will be able to notice.

5   Catalogue Generation

To find the publish.txt files, bxpublish looks for directories beneath the source directory, checking for each directory found whether it contains such a file. The list of directory pathes is sorted prior to this examination.

While reading the publish.txt files, a catalogue of rules is generated, one for each definition line. Each such rule has four components:

  1. Each rule has a type, which can either be of inclusive (+ lines) or exclusive (- lines).
  2. The path component of the examined publish.txt file relative to the source directory is called the root of the rule.
  3. The path pattern from the last path group line is stored as the path pattern of the rule. When no path group line has been read yet in the current publish.txt file, the empty string will be used as the path pattern.
  4. Finally, the filename pattern of the definition line makes up the name pattern of the rule.

There is a default exclusion rule defined to not publish the publish.txt files. This rule is rooted at the source directory, its path pattern matches any path and its name pattern applies precisely to publish.txt.

6   Application of the Catalogue to Files

For each file within the source directory it is ascertainted for each rule if the file is covered by the given rule:

  1. The path of the file relative to the source directory is calculated. This will be a path starting with a dot. When this source-relative path doesn't start with the rule's root directory components, the rule does not apply.
  2. Otherwise, the path of the file in question relative to the rule's root path is computed. This path is abbreviated by stripping away the leading './' if the path does not consist of the dot '.' alone. When the relative path is precisely the single dot, it is abbreviated as the null string.
  3. Now, when the abbreviated path matches the rule's path pattern, and when the given file's basename also matches the rule's name pattern, the rule covers the file. If any of the two does not match, the rule does not cover the file. In order to be considered matching, the path pattern and the name pattern need to match the entire respective string.

Definition lines at the top of a publish.txt file, before any path group line, will create rules with the empty string as their path pattern. Such rules thus cover files directly within the root path of the publish.txt file.

When multiple rules cover a file, the one defined last is decisive. When no rule covers a file in question, the file will be included by default. Because only the directory pathes are sorted during publish.txt file discovery, it is guaranteed that each publish.txt file in a certain directory is read earlier than publish.txt files in subfolders of the resp. directory.