Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

[0.3.0] - 2021-06-08

Added

• API

  • calculate_metrics

    • ppl: Calculate PPL (Perceptual Path Length)

    • ppl_epsilon: Interpolation step size in PPL

    • ppl_reduction: Reduction type to apply to the per-sample output values

    • ppl_sample_similarity: Name of the sample similarity to use in PPL metric computation

    • ppl_sample_similarity_resize: Force samples to this size when computing similarity, unless set to None

    • ppl_sample_similarity_dtype: Check samples are of compatible dtype when computing similarity, unless set to None.

    • ppl_discard_percentile_lower: Removes the lower percentile of samples before reduction

    • ppl_discard_percentile_higher: Removes the higher percentile of samples before reduction

    • ppl_z_interp_mode: Noise interpolation mode in PPL

    • input1_model_z_type: Type of noise accepted by the input1 generator model

    • input1_model_z_size: Dimensionality of noise accepted by the input1 generator model

    • input1_model_num_classes: Number of classes for conditional generation (0 for unconditional) accepted by the input1 generator model

    • input1_model_num_samples: Number of samples to draw from input1 generator model, when it is provided as a path to ONNX model. This option affects the following metrics: ISC, FID, KID

    • input2_model_z_type: Type of noise accepted by the input2 generator model

    • input2_model_z_size: Dimensionality of noise accepted by the input2 generator model

    • input2_model_num_classes: Number of classes for conditional generation (0 for unconditional) accepted by the input2 generator model

    • input2_model_num_samples: Number of samples to draw from input2 generator model, when it is provided as a path to ONNX model. This option affects the following metrics: ISC, FID, KID

• Command line

  • --ppl: Calculate PPL (Perceptual Path Length)

  • --ppl-epsilon: Interpolation step size in PPL

  • --ppl-reduction: Reduction type to apply to the per-sample output values

  • --ppl-sample-similarity: Name of the sample similarity to use in PPL metric computation

  • --ppl-sample-similarity-resize: Force samples to this size when computing similarity, unless set to None

  • --ppl-sample-similarity-dtype: Check samples are of compatible dtype when computing similarity, unless set to None.

  • --ppl-discard-percentile-lower: Removes the lower percentile of samples before reduction

  • --ppl-discard-percentile-higher: Removes the higher percentile of samples before reduction

  • --ppl-z-interp-mode: Noise interpolation mode in PPL

  • --input1-model-z-type: Type of noise accepted by the input1 generator model

  • --input1-model-z-size: Dimensionality of noise accepted by the input1 generator model

  • --input1-model-num-classes: Number of classes for conditional generation (0 for unconditional) accepted by the input1 generator model

  • --input1-model-num-samples: Number of samples to draw from input1 generator model, when it is provided as a path to ONNX model. This option affects the following metrics: ISC, FID, KID

  • --input2-model-z-type: Type of noise accepted by the input2 generator model

  • --input2-model-z-size: Dimensionality of noise accepted by the input2 generator model

  • --input2-model-num-classes: Number of classes for conditional generation (0 for unconditional) accepted by the input2 generator model

  • --input2-model-num-samples: Number of samples to draw from input2 generator model, when it is provided as a path to ONNX model. This option affects the following metrics: ISC, FID, KID

• Support generative model modules as inputs to all metrics

• ONNX and PTH (JIT) model loading via command line functionality to support framework-agnostic metrics calculation

• Noise source types and latent vector interpolation methods can now be registered and dispatched similar to registered inputs

• Registered inputs: stl10-train, stl10-test, stl10-unlabeled

• Registered noise source types: normal, uniform_0_1, unit

• Registered latent vector interpolation methods: lerp, slerp_any, slerp

• Example SNGAN training and evaluation script (examples/sngan_cifar10.py)

• Test for LPIPS fidelity as compared to StyleGAN PyTorch implementation

• Test for feature extraction layer

• Unrecognized command line arguments warning

• Added ReadTheDocs documentation

Changed

• API

  • First input positional argument of calculate_metrics is now expected as a value to kwarg input1

  • Second input (optional) positional argument of calculate_metrics is now expected as a value to kwarg argument input2

  • cache_input1_name renamed to input1_cache_name

  • cache_input2_name renamed to input2_cache_name

  • rng_seed default value from 2020 to 2021

• Command line

  • First input positional argument is now expected as a value to the key --input1

  • Second input (optional) positional argument is now expected as a value to the key --input2

  • --datasets-downloaded renamed to --no-datasets-download

  • --samples-alphanumeric renamed to --no-samples-shuffle

  • --cache-input1-name renamed to --input1-cache-name

  • --cache-input2-name renamed to --input2-cache-name

  • --rng-seed default value from 2020 to 2021

• Change torch.save to an atomic saving operation in all functions of the caching layer, which makes it safe to use torch-fidelity in multiprocessing environment, such as a compute cluster with a shared file system.

Fixed

• #15: Fix ‘64’, ‘192’, and ‘768’ feature layers usage in all metrics

• #8: Fix a missing exception for when KID subset size is larger than the number of samples in one of the inputs

• Fix a missing check that the elements of inputs are actually instances of torch.Tensor

[0.2.0] - 2020-05-05

Added

• Initial release with Inception Score (ISC), Frechet Inception Distance (FID), and Kernel Inception Distance (KID) metrics

• Precision unit tests for all three metrics

• Command line tool and Python API