Preface
1. Project Status
This software is not yet complete or usable, nor is it supported in any official capacity. If you think this might be useful, let me know (Tom Tyler, ttyler at perforce dot com).
2. Introduction
Directory Structure Image (DSI) is a comprehensive comparison utility, used primarily to detect changes to a directory tree since initial installation. An image can be generated as part of the release process for a software product, and used subsequently to detect changes.
3. Overview
The image identifies elements which can be files, directories, or symlinks. Elements may be specified as absolute paths or paths relative to a variable installation root. In addition to files in an install root, additional elements may be specified that may be scattered sparsely across a filesystem.
Directories can be strictly or loosely controlled. A strictly controlled directory tree is one in which the image comprehends all elements in the tree, and excess or missing files are unexpected. A loosely controlled direcotry is one in which the image specifies specific files, but allows for excess files, directories, and symlinks to exist.
The DSI can be configured to understand a product that may include sparsely deployed files throughout a filesystem.
Detection of changes includes:
-
Newly added files/directories/symlinks.
-
Deleted files/directories/symlinks.
-
Differing last-modified timestamp of existing elements.
-
List files with differing content.
-
Changes in file/directory owner.
-
Changes in file/directory group.
-
Changes in file/directory permissions (e.g. +x bit, etc.)
Command line flags can suppress certain types of diffs that may not be important in certain scenarios, such as group/owner changes, as well as special verifications such as "owner can be anything except root."
This software is powered by the Perforce Helix version management software using Helix native DVCS features to show file and content differences.
4. Command Line Usage
Run the dsi
tool with the -man
flag to display a synopsis of options, plus documentation of commands and options, examples, and other info. Using the -h
flag will display only the synopsis of options.
USAGE for dsi v1.3.2:
Usage varies based on mode, which can be one of: create, diff, info, list, rm, rmall, tags
dsi create [<tag>]
dsi diff [<tag>[@<rev>]]
dsi info
dsi list [<tag>]
dsi rm [<tag>@<rev>]
dsi rmall [<tag>]
dsi tags
Key environment variables:
* DSI_TAG defines a default for <tag>.
* DSI_STORAGE, or specify as '-s <storage_dir>'.
* DSI_IMAGE_ROOT, defaults to $PWD, or specify as '-d <image_root>'.
or
dsi [-h|-man|-V]
DESCRIPTION:
Directory Structure Image (DSI) is a comprehensive comparison utility,
used primarily to detect changes to a directory tree since the initial
installation. A DSI can be generated as part of the release process
for a software product, and then used subsequently to detect changes.
Detection of changes includes:
* Last-modified Timestamp
* Newly added files/directories
* Deleted files/directories
* Changes to contents of files, including showing diffs
* Changes in file/directory owner
* Changes in file/directory group
* Changes in file/directory permissions (e.g. +x bit, etc.)
Command line flags can suppress certain types of diffs that may
not be important in certain scenarios, such as group/owner changes,
as well as special verifications such as 'owner can be anything
except root.'
This software is powered by the Perforce Helix version management
software using Helix native DVCS features to show file and content
differences.
TAGS AND REVISIONS:
A DSI is assigned a tag on creation, which is generally the name of the
product and optionally product version ID, e.g. 'fgs' or 'fgs-1.0'.
DSI tag names can include alphanumeric characters, dashes, underscores,
and dots.
DSI revisions are positive integers associated with a tag and assigned
upon creation.
MODES:
Create, specified with 'create [<tag>]' (or 'c' for short)
Create and store a DSI image with a given tag name.
Diff, specified with 'diff [<tag>]' (or 'd' for short)
Diff a DSI image with the given tag, using the latest version or a specified
revision tag. Optionally, specify the @<rev> to compare against a revision
other than the head revision.
If no flags are specified to limit the output, a comprehensive diff will
report:
* Timestamp differences
* Checksum differences
* Size Differences
The Diff mode supports flags to exlcude certain things from comparison:
Fix, specified with 'fix [<tag>]' (or 'f' for short)
Fix a directory structure as it appears on disk, focing it to match
the image. This will overwrite files in the target structure.
Info, specified with 'info' (or 'i' for short)
Show DSI environment variables, default settings, and version info.
List, specified as 'list [<tag>]' (or 'l' for short)
List stored DSI images for the given tag.
Remove, specified as 'rm [<tag>]@<rev>'
Remove the specified revision for the given tag. For this usage
the revision must be specified.
RemoveAll, specified as 'rmall <tag>'
Remove all images for the given tag.
Tags, specify as 'tags' (or 't' for short)
List available image tags.
<tag>[@<rev>]
Specify the DSI tag and revision. A default tag value can be defined by
setting the DSI_TAG environment variable.
For Diff mode, the following can be specified:
<tag> Explicit value for <tag>, overriding DSI_TAG.
Revision defaults to @LATEST
<tag>@<rev> Explicit definition of tag and revision
@<rev> Uses DSI_TAG for the tag, with explicit rev.
For Create or List modes, the following can be specified:
<tag> Explicit value for <tag>, overriding DSI_TAG.
Revision is not set with Create and List modes.
For Remove mode the tag may be implied (with DSI_TAG) or specified
explicitly. The revision is required.
For RemoveAll mode the tag must be specified explicitly. A revision
is not allowed.
OPTIONS:
-r <image_root>
Specify the root directory in which to create a DSI, or in which to
do a comparsion against stored DSIs. The default is the current
working directory as indicated by $PWD.
-s <storage_dir>
Specify the DSI storage dir. A default can be defined by setting
the DSI_STORAGE environment variable
GENERAL OPTIONS:
-L <log>
Specify the path to a log file to enable logging. Or, specify
'-L on' to enable logging using a default log file name:
/tmp/dsi.v1.3.2.<datestamp>.log
By default, logging is disabled.
NOTE: This script is self-logging if logging is enabled. That is,
output displayed on the screen is simultaneously captured in the log
file. Do not run this script with redirection operators like '> log'
or '2>&1', and do not use 'tee.'
-si Operate silently. All output (stdout and stderr) is redirected to the
log only; no output appears on the terminal. This cannot be used with
'-L off'.
-D Set extreme debugging verbosity.
HELP OPTIONS:
-h Display short help message
-man Display man-style help message
-V Dispay version info for this script and its libraries.
DSI FILE FORMAT:
The directory structure image file format is defined in the
documentation.
FILES:
DSI package definition files are stored in the DSI_STORAGE directory,
and have a .dsi file name suffix.
EXAMPLES:
Example 1: Create a DSI of /p4/common
dsi c -r /p4/common
5. Image File Format and Samples
5.1. Image File Format
Following is the image file format:
# Directory Structure Image (DSI) File Format. # Lines starting with hash ('#') characteres are comments. # Hash characters inline are comments # Support for characters in filenames is limited to ASCII (not extended ASCII) The general format of data lines (i.e. non-commment lines) is: <DirectiveOrElement> <Data> where <DirectiveOrElement> is one of a fixed set of values, and <Data> depends on <DirectiveOrElement>. <DirectiveOrElement> can be one of i, d, D, f, s. The format of <Data> for element and directives types is: i <Tag> <ImageVersion> The 'i' Info entry contains package metadata provided by the user upon DSI creation. There can be only one 'i' entry per DSI file, and it must be the first entry. <Tag> Form: alphanumeric+dash+underbar Sample: FGS Comments: This is the index into the DSI database. <ImageVersion> Form: alphanumeric+dash+underbar Sample: 2022.1.19340 Comments: This is the user-defined version identifier for the image. d <RelPath>|<Owner>:<Group>|<Perms>|<Timestamp> The 'd' is a directory element. <RelPath> is the relative path to the directory from the package root <Owner>:<Group> are the owner/group settings as used by the `chown` command. <Perms> are the file permissions as used by the `chmod` command. <Timestamp> is the file timestamp as reported by the `ls -l` command. D <RelPath>|<Owner>:<Group>|<RecusrivePerms> The 'D' is a shorthand way to specify that all directories under the relative path have the same permissions. (NOT IMPLEMENTED, FOR FUTURE USE) f <RelPath>|<Owner>:<Group>|<Perms>|<Timestamp>|<Checksum> The 'f' is a file element. s <RelPath>|<Owner>:<Group>|<Perms>|<LinkTarget> The 's' is a symbolic link element. <ImageTag> is the tag name for a product, e.g. 'fgs' for the Friendly Greeting System product. The <Owner>:<Group> defines the Operating System owner and group. <Perms> contains permission data, of the form <OwnerPerms>:<GroupPerms>:<OtherPerms>, as reported by `ls`, e.g. rwxr-x---. LIMITATIONS: * DSI works only with regular files, directories, and symlinks. Block special devices and character special devices, sockets, pipes, and other exotic items that can appear in a directory structure are not handled. Handling of such items is undefined. * DSI descriptions only support non-extended ASCII text.
5.2. Sample Image Files
Sample 1: Friendly Greeting System (FGS)
Sample image file:
# Sample Friendly Greeting System file format, with tag 'fgs': i fgs 2020.1 r20.1 26937 Friendly Greeting System d src unset:unset 750 20201123080422 f src/hello.h unset:unset 640 20201123080422 f src/hello.c unset:unset 640 f src/Makefile unset:unset 640 f src/build.sh unset:unset 750
Sample 2: FGS with sparse files.