Configuration and CLI

Every option available on the command-line interface can also be provided using a configuration file.

CLI Options

For any option that acts as a simple on/off switch, you can either just specify the option to imply turning it on, or explicitly pass 0 or false to turn it off, or 1 or true to turn it on. This may be useful if you want to be very clear in a batch file or log about what happens.

For example, the default is for delete to be on (True) and for match_dir to be off (False). So, this:

box **/* -r C:\Temp --delete true --match_dir false

Would be the same as:

box **/* -r C:\Temp

archive_ext

--archive_ext | -ae [<ext>]

If provided, globs will be interpreted as if matching the original file names, and matched files will be matched against any regular expressions provided without the archive extension.

Example:

unbox **/*.csv -ae 7z -re "test.*\.csv"

This would proceed to find files ending in .csv.7z (i.e. previously boxed .csv files) and only archives with names like test123.csv.7z (i.e. starting with test and ending in .csv, ignoring the .7z) would be matched.

create_dirs

--create_dirs | -cd

Switch for unboxing, determining whether to recreate the same directory structure in the target path. When a target location is specified, files will not be boxed or unboxed in-place - if create_dirs is False, all the files in the target location will be created in the same directory. The default is True, which is the useful setting for in-place boxing and unboxing.

It's recommended you review zip_structure and create_dirs before use.

delete

--delete | -d

Remove the files that were boxed immediately after boxing (one file at a time) and remove the archive files that were unboxed immediately after unboxing. The default for this switch is True.

datetime_created

--datetime_created | -dtc <date/time expression> 

Only match files based on their creation date. You have some flexibility in the date/time expression. You can combine multiple expressions with and or or, the point in time or period can be as coarse as a year and as fine as a second, and you can specify that the timestamp for the file has to be either in, before, or after the specified point or period.

Some examples of valid date/time expressions:

< 2023
before 2020-01-01 or on or after 2022-01-01
< 2020-01-01 or >= 2022-01-01
on 2024-03-20 and before 12:00
> 2020-03-14 01:59:20

Note that dates have to be in ISO format (YYYY-MM-DD) and times in zero-padded 24-hour format (HH:MM:SS). So, 2024-03-21 for 21 March 2024, and 01:59 for 1:59 AM or 13:59 for 1:59 PM.

datetime_modified

--datetime_modified | -dtm <date/time expression> 

Only match files based on their most recent modification date. The format for the date/time expression is the same as for datetime_created.

error_log

--error_log | -el <path>

Write an error log file to the name and location specified by <path>. All warnings and error messages will be written (with a timestamp) to the file. The default is to write these messages on screen only.

glob

[--glob | -g] <pattern> [<pattern> [..]]

At least one pattern is required for a box or unbox command. The patterns provided will be processed in order.

group_match

--group_match | -gm

Whether to match files that are grouped with other matched files. See also File Groups. The default is to match grouped files. In short, this means that any files that are normally grouped, like for example files that make up an ArcGIS shape file, will all be matched if at least one of them is matched by a glob pattern.

You can define additional groups using group_add

group_add

--group_add | -ga <path>

In addition to default group definitions used by group_match, you can define your own in a .json format that is structured the same as the groups.json file that accompanies the py7za_box script.

The structure is straightforward:

{
  "group name": [".suffix", ".suffix", ...],
  ...
}

Alternatively, this structure can be part of another .json file, in which case it should be the value of a "groups" key:

{
  ...
  "groups: {
    "group name": [".suffix", ".suffix", ...],
    ...
  }
  ...
}

This allows including it into a configuration file that can be passed to box in addition to (or instead of) command-line options.

Example:

unbox --group_add my_configuration.json

In this case, my_configuration.json would be expected to have a "glob" key, which would contain the glob patterns, since none are given on the command-line.

help

--help | -h

Shows a help text for CLI parameters on the console (including the installed Treewalker version number at the top).

log

--log | -l <path>

Write a log file to the name and location specified by <path>. For each boxing or unboxing operation, source file name, source file size, target file name, and target file size are written in .csv format. The default is not to log to file and to only report progress on the screen.

not_regex

--not_regex | -nre <expr>

The opposite of regex. I.e: any objects with a full, absolute path that match the expression will be skipped. The default is that no regular expression matching is performed (None)

match_dir

--match_dir | -md

Whether glob pattern(s) should match directories. By default only files are matched. If directories are matched, they will be archived as a whole, i.e. the entire directory will be archived as a single file when boxing, and unboxed all at once when unboxing.

match_file

--match_file | -mf

Whether glob pattern(s) should match directories. By default files are matched, and directories are not. In case you want to box a series of directories and need to exclude any files that would otherwise be matched by the glob pattern, you can use this switch to avoid any files from being matched individually.

Example:

box project/* -md -mf False

This command would box each directory in the project directory, but none of the individual files in the same directory.

output

--output | -o  d | default / l | list / q | quiet / s | status / v | verbose

What output to write to the screen. The default option is default which will print a line per archive processed with current status, updating lines as the application goes.

The list option is largely the same, but does not print current status which gets updated like default, this mode is most useful if capturing the output in a file.

Using the verbose option will show each command to 7za.exe as it is issued and enables logging at info level, which means that besides warnings and errors, messages of lesser importance are also logged to standard error.

With status, only live status for the ongoing processes are updated on screen, but there is no scrolling report of every archive as it completes processing. Information about what files have been processed is no registered, unless logged. This mode is most useful if a long runnig operation runs in the background and the user is only interested in overall progress, not what files have been processed.

When quiet, only essential messages like errors are written to the screen.

overwrite

--overwrite | -w  a | all / s | skip / u | rename_new / t | rename_existing 

Controls what should happen if a file is unboxed when the target already exists. The default is to skip unboxing such files (to avoid overwriting changes that were made after boxing). When set to all, all files will be (over)written. When set to rename_new, files with existing targets will be renamed, adding a number to the name, to avoid overwriting the existing file(s). Similarly, the rename_existing option will cause the existing file to be renamed with a new number first, before unboxing the file that takes its place.

parallel

--parallel | -p <n>[x]

How many parallel processes to run at a maximum.

The default is 0, which means all available cores on the system. This can exhaust compute resources on the system to the point of other programs having trouble getting their work done - for example, Remote Desktop sessions to the system may appear to hang or disconnect. If any interactivity with the desktop is required during the run, it is recommended to set --parallel to a lower value. A negative value is interpreted to mean the number of available cores minus that number, so --parallel -2 may be suitable to retain more or less normal access to the system.

You can also specify a multiple of the number of available cores on the system, by appending an x to the value, which can be a floating point value in that case.

Example:

box **/* -p " -6"
box **/* -p 18
box **/* -p .75x

All three of these commands, on a system with 24 cores, would cause the operation to run on 18 cores.

root

--root | -r <path>

All file operations of py7za-box are relative to the given path. The default path is ".", i.e. the current working directory where py7za-box is started. UNC paths are supported.

Example

unbox **/* -r \\server\some\path

regex

--regex | -re <expr>

In case glob patterns don't allow you enough control over what files are processed, you can provide a regular expression to further limit what paths are matched. Note that the expression has to match the full, absolute path of the objects matched. The default is that no regular expression matching is performed (None)

Example:

box **/* -re ".*\d{4}-\d{2} Report.docx?$"

Will match anything that ends in something like "2022-01 Report.docx$" and is an alternative for the glob pattern **/*[0123456789][0123456789][0123456789][0123456789]-[0123456789][0123456789] Report.doc, followed by the same pattern ending in an x.

si

--si | -si 

When file sizes are printed (to screen or log files) in human-readable form, whether to use SI units for file sizes (or the default binary sizes). SI sizes are powers of 1,000 (KB, MB, GB, etc.) while binary sizes are powers of 1,024 (KiB, MiB, GiB, etc.)

target

--target | -t <path>

By default, py7za-box commands work in-place on files in the root location. If you provide a path to --target, outputs will instead be written to that location.

This can be useful if you want to preserve the original and do not want to clutter it by adding archives, or when the root location does not have enough free space to contain the output. Note that options like delete and create_dirs still work as usual.

Example:

unbox **/*.7z --root \\archive\project\directory --target \\file_server\project\directory --delete false

test

--test

Run py7za-box in 'test' mode - this will make the application run as it would otherwise, but it will not actually perform any (un)archiving, file deletion, etc. It will show the 7za.exe commands as they would have been executed.

Example:

box **/* -r D:\test\ --test 

This can output something like:

TEST in ".": 7za.exe a "D:\test\test.mif.7z" "D:\0000\test.mif" -y -sdel
TEST in ".": 7za.exe a "D:\test\test.mid.7z" "D:\0000\test.mid" -y -sdel
TEST in ".": 7za.exe a "D:\test\test.txt.7z" "D:\0000\test.txt" -y -sdel
Matched 3 object(s), start processing in up to 16 parallel processes ...
Completed processing 0 B of files into 0 archives, totaling 0 B.
Took 0:00:00.010001. Skipped 0 files matching glob.
TEST: would have started boxing 3 matches.

test_match

--test_match

Like test, but instead of showing the full 7za.exe commands, it only shows the files that were matched and the order in which the 7za.exe commands for them would be started. Example:

box **/* -r D:\test\ --test_match 

This can output something like:

D:\0000\test.mif
D:\0000\test.mid
D:\0000\test.txt
Matched 3 object(s), start processing in up to 16 parallel processes ...
Completed processing 0 B of files into 0 archives, totaling 0 B.
Took 0:00:00.010001. Skipped 0 files matching glob.
TEST: would have started boxing 3 matches.

Note: if both test and test_match are provided, test_match takes precedence.

unbox

--unbox | unzip | -u

When you run unbox, you effectively run py7za-box with this option set to True, causing it to start unboxing and interpreting the other options accordingly. By default, this option is False, so running py7za-box without this option is nearly identical to running box.

unbox_multi

--unbox_multi | -um

Whether to unzip multi-file archives. The default is False, unboxing will not extract archives that contain more than a single file. When you box a set of files with match_dir you will likely want to unbox them using --unbox_multi.

zip_archives

--zip_archives | -za

Whether to zip matched archives (again). By default, when an existing archive (.7z, .zip, etc.) gets matched by the glob pattern, it is skipped to avoid recompressing the archive. However, if you want this to happen, you can use this switch. A use case for that might be if a file format happens to be recognised as an archive, but is in fact some other more obscure or proprietary file type.

zip_structure

--zip_structure | -zs
 ```

When this option is specified, the subdirectory structure relative to the root directory is included in the archive (there is no effect when unboxing). By default, the directory structure from root is recreated (see [create_dirs](../configuration/#create_dirs)), and there is no need to include the directory structure. But if you set `--create_dirs` to False, you may want to preserve the original location of the files, even though you are creating a flat file structure.  

For example, if you start with two directories in a directory called `project`: `one` and `two`, and both contain a text file, `1.txt` and `2.txt` respectively, then running the following:
```commandline
box **/*.txt -zs -cd False

Will create 2 archives named 1.txt.7z and 2.txt.7z. However, normally these would simply contain 1.txt and 2.txt respectively. With -zs specified, they actually contain project/one/1.txt and project/two/2.txt and thus running:

unbox *.7z

Will recreate the original directory structure and put the files in their correct locations.

It's recommended you review zip_structure and create_dirs before use.

7za

--7za | -7 <arguments>

The passed arguments will be passed as CLI arguments to 7za.exe, after any scripted ones. Since arguments to 7za.exe start with the same character as arguments to py7za-box, you should put double quotes around the arguments, and start with a space.

Example:

box **/* -7 " -mx9 -tzip"

(Note the extra space after the opening quote) This would cause a normal boxing operation, but using .zip as the file format and using maximum compression. Look at the command-line options for 7-zip for a more complete overview: https://sevenzip.osdn.jp/chm/cmdline/switches/index.htm

Glob patterns

There are many online guides on Unix path expansion with glob patterns, or glob expressions, but the key things to know are:

• You can use forward slashes / and backslashes \ interchangeably, but it is recommended to use forward slashes only to avoid unrelated problems with escaping, so prefer model/log/*.txt over model\log\*.txt
• A single asterisk * matches zero or more characters in part of a name, for example abc-*.csv would match every .csv file (or directory with a name that ends in .csv) with a name that starts with abc-.
• A double asterisk ** matches any number of directories in a path, including subdirectories, for example temp/**/model.log would match every file called model.log in every subdirectory of the temp directory.
• If you just need to vary a single character, or a fixed number of characters, you can use the question mark ?, for example logs/2022-01-??.log could be useful to match every log file from January 2022.
• If you need to vary a single character, but only want to match something specific, you can use brackets [], for example output/[AB]*.csv would match any .csv file in the output directory that starts with A or B.

So: */*.txt would match every .txt file in every directory of the root directory (but none of their subdirectories, and also none of the .txt files that sit directly in the root directory), while **/*.txt would match every .txt file in the root directory itself and all of its subdirectories. *.txt would just match all the .txt files in the root directory itself and nothing else.

File Groups

Certain types of files are essentially useless or lose essential context if they are separated from their companion files. One such example are ArcGIS shape files. These consist of a .shp, but also require an accompanying .dbf and several other files, some of which are optional.

py7za-box supports automatically boxing and unboxing these files together, so that you can still have the benefit of good compression on some of them, without having to keep track of what part of the set you compressed or decompressed.

By default, there are very few groups defined in a groups.json that sits in the folder where py7za-box.py is installed (a py7za subfolder of your Python site-packages folder) - the default contents are currently:

{
  "arcgis shape files": [".shp", ".shx", ".dbf", ".sbn", "sbx", ".fbn", ".fbx", ".ain", ".aih", ".atx", "ixs", ".mxs", ".prj", ".xml", ".cpg"],
  "mapinfo tab files": [".tab", ".dat", ".ind", ".map", ".id"],
  "mapinfo mid/mif files": [".mid", ".mif"]
}

You can add your own groups to that file, but that is not recommended as this may cause issues when upgrading or uninstalling the tool. Instead, use the group_add option to point py7za-box to your own .json with a same structure. You can even override the existing groups, if you need, by including the same keys in the dictionary.

Alternatively, you can add a "groups" key to your main .json configuration with settings to drive py7za, which would have to have a dictionary value that's the exacty same as the entire groups.json file.

The behaviour when boxing or unboxing a file that falls into a group is straightforward. For example:

box **/*.shp

This will start boxing all shape files, but include all the group files with the same name (but a different suffix) as well.