Run Directory System

Timestamped, self-contained run directories for reproducible experiments.

Overview

Every HINEC pipeline execution can create an organized, timestamped run directory containing all configuration, intermediate files, outputs, logs, and diagnostics. This ensures that any result can be traced back to the exact parameters and data that produced it.

Directory Structure

text

hinec_runs/
└── run_2025-01-15_14_30_22_hinec_default/
    ├── config/
    │   └── hinec_default.yml          # Copy of configuration used
    ├── output/
    │   ├── sample.mat                 # Processed nim structure
    │   └── tractography_results/
    │       └── tracks_hinec_*.mat     # Tractography output
    ├── logs/
    │   ├── preprocessing.log          # Preprocessing details
    │   ├── tractography.log           # Tracking statistics
    │   └── pipeline.log               # Overall pipeline log
    ├── diagnostics/
    │   ├── fa_histogram.png           # FA distribution
    │   └── quality_report.mat         # Quality metrics
    ├── figures/
    │   └── tractography_whole.png     # Generated visualizations
    ├── run_info.json                  # Run metadata (ID, timestamp, duration)
    └── latest -> .                    # Symlink for convenience

Reproducibility Features

Configuration Copy

The exact YAML file used is copied into the run directory, ensuring parameters are preserved even if the original config changes.

Timestamped Naming

Each run gets a unique timestamp-based directory name, preventing overwrite of previous results.

Processing Logs

Detailed logs record every step, including FSL commands executed, parameters applied, and timing information.

Quality Diagnostics

Automatic quality metrics and FA histograms help verify processing integrity.

Run Metadata

JSON metadata files record the run ID, start/end times, total duration, and system information.

Latest Symlink

A 'latest' symlink always points to the most recent run for quick access.

Creating Run Directories

matlab

% Create a run directory with description
config = load_config_yaml('config/hinec_default.yml');
run_info = create_run_directory('config/hinec_default.yml', ...
    'description', 'First processing run with T1 registration');
 
% Pass run_info to pipeline
main('data/sample', 'output/sample.mat', ...
    'config', config, 'run_info', run_info);

Tip

When using the shell script, run directories are created automatically: ./bin/run_hinec.sh data/sample sample.mat config/hinec_default.yml

Function Reference

Validation