Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[0.4.0] - Unreleased

Added in 0.4.0

• New metrics: Precision, Recall, F-score

• New registered inputs: cifar100-train, cifar100-val

• New registered feature extractors: clip-vit-b-32, vgg16, dinov2-vit-s-14, dinov2-vit-b-14, dinov2-vit-l-14, dinov2-vit-g-14

• API

  • calculate_metrics

    • prc: Calculate PRC (Precision and Recall)

    • prc_neighborhood: Number of nearest neighbours to consider in PRC

    • prc_batch_size: Batch size in PRC

    • feature_layer_prc: Name of the feature layer to use with PRC metric

    • samples_resize_and_crop: Transform all images found in the directory to a given size and square shape

    • feature_extractor: Accepts a new feature extractors clip-vit-b-32, vgg16, dinov2-vit-s-14, dinov2-vit-b-14, dinov2-vit-l-14, dinov2-vit-g-14

    • feature_extractor_internal_dtype: Allows to change the internal dtype used in the feature extractor’s weights and activations; might be useful to counter numerical issues arising in fp32 implementations, e.g. those seen with the growth of the batch size

    • feature_extractor_compile: Compile feature extractor (experimental: may have negative effect on the metrics numerical precision)

    • kid_kernel: Allows choosing between the default poly (polynomial, default) and rbf (RBF) kernels

    • kid_kernel_rbf_sigma: Specifies RBF kernel sigma in KID

• Command line

  • prc: Calculate PRC (Precision and Recall)

  • prc-neighborhood: Number of nearest neighbours to consider in PRC

  • prc-batch-size: Batch size in PRC

  • feature-layer-prc: Name of the feature layer to use with PRC metric

  • --samples-resize-and-crop: Transform all images found in the directory to a given size and square shape

  • --feature-extractor: Accepts a new feature extractors clip-vit-b-32, vgg16, dinov2-vit-s-14, dinov2-vit-b-14, dinov2-vit-l-14, dinov2-vit-g-14

  • --feature-extractor-internal-dtype: Allows to change the internal dtype used in the feature extractor’s weights and activations; might be useful to counter numerical issues arising in fp32 implementations, e.g. those seen with the growth of the batch size

  • --feature-extractor-compile: Compile feature extractor (experimental: may have negative effect on the metrics numerical precision)

  • --kid-kernel: Allows choosing between the default poly (polynomial, default) and rbf (RBF) kernels

  • --kid-kernel-rbf-sigma: Specifies RBF kernel sigma in KID

Changed in 0.4.0

• Default features for all metrics are now read from the selected feature extractor

• Default feature extractor is now inferred based on the selected metrics

• All tests run in docker now

• API

  • calculate_metrics

    • kid_degree: Deprecated, new name is kid_kernel_poly_degree

    • kid_gamma: Deprecated, new name is kid_kernel_poly_gamma

    • kid_coef0: Deprecated, new name is kid_kernel_poly_coef0

• Command line

  • --kid-degree: Deprecated, new name is --kid-kernel-poly-degree

  • --kid-gamma: Deprecated, new name is --kid-kernel-poly-gamma

  • --kid-coef0: Deprecated, new name is --kid-kernel-poly-coef0

Fixed in 0.4.0

• #19: Adds Precision and Recall metrics

• #42: Fixes missing files in the wheel

• #46: Adds new FID computation code and removed scipy dependency

• #47: Adds new CLIP-based feature extractor

[0.3.0] - 2021-06-08

Added in 0.3.0

• API

  • calculate_metrics

    • ppl: Calculate PPL (Perceptual Path Length)

    • ppl_epsilon: Interpolation step size in PPL

    • ppl_reduction: Reduction type to apply to the per-sample output values

    • ppl_sample_similarity: Name of the sample similarity to use in PPL metric computation

    • ppl_sample_similarity_resize: Force samples to this size when computing similarity, unless set to None

    • ppl_sample_similarity_dtype: Check samples are of compatible dtype when computing similarity, unless set to None.

    • ppl_discard_percentile_lower: Removes the lower percentile of samples before reduction

    • ppl_discard_percentile_higher: Removes the higher percentile of samples before reduction

    • ppl_z_interp_mode: Noise interpolation mode in PPL

    • input1_model_z_type: Type of noise accepted by the input1 generator model

    • input1_model_z_size: Dimensionality of noise accepted by the input1 generator model

    • input1_model_num_classes: Number of classes for conditional generation (0 for unconditional) accepted by the input1 generator model

    • input1_model_num_samples: Number of samples to draw from input1 generator model, when it is provided as a path to ONNX model. This option affects the following metrics: ISC, FID, KID

    • input2_model_z_type: Type of noise accepted by the input2 generator model

    • input2_model_z_size: Dimensionality of noise accepted by the input2 generator model

    • input2_model_num_classes: Number of classes for conditional generation (0 for unconditional) accepted by the input2 generator model

    • input2_model_num_samples: Number of samples to draw from input2 generator model, when it is provided as a path to ONNX model. This option affects the following metrics: ISC, FID, KID

• Command line

  • --ppl: Calculate PPL (Perceptual Path Length)

  • --ppl-epsilon: Interpolation step size in PPL

  • --ppl-reduction: Reduction type to apply to the per-sample output values

  • --ppl-sample-similarity: Name of the sample similarity to use in PPL metric computation

  • --ppl-sample-similarity-resize: Force samples to this size when computing similarity, unless set to None

  • --ppl-sample-similarity-dtype: Check samples are of compatible dtype when computing similarity, unless set to None.

  • --ppl-discard-percentile-lower: Removes the lower percentile of samples before reduction

  • --ppl-discard-percentile-higher: Removes the higher percentile of samples before reduction

  • --ppl-z-interp-mode: Noise interpolation mode in PPL

  • --input1-model-z-type: Type of noise accepted by the input1 generator model

  • --input1-model-z-size: Dimensionality of noise accepted by the input1 generator model

  • --input1-model-num-classes: Number of classes for conditional generation (0 for unconditional) accepted by the input1 generator model

  • --input1-model-num-samples: Number of samples to draw from input1 generator model, when it is provided as a path to ONNX model. This option affects the following metrics: ISC, FID, KID

  • --input2-model-z-type: Type of noise accepted by the input2 generator model

  • --input2-model-z-size: Dimensionality of noise accepted by the input2 generator model

  • --input2-model-num-classes: Number of classes for conditional generation (0 for unconditional) accepted by the input2 generator model

  • --input2-model-num-samples: Number of samples to draw from input2 generator model, when it is provided as a path to ONNX model. This option affects the following metrics: ISC, FID, KID

• Support generative model modules as inputs to all metrics

• ONNX and PTH (JIT) model loading via command line functionality to support framework-agnostic metrics calculation

• Noise source types and latent vector interpolation methods can now be registered and dispatched similar to registered inputs

• Registered inputs: stl10-train, stl10-test, stl10-unlabeled

• Registered noise source types: normal, uniform_0_1, unit

• Registered latent vector interpolation methods: lerp, slerp_any, slerp

• Example SNGAN training and evaluation script (examples/sngan_cifar10.py)

• Test for LPIPS fidelity as compared to StyleGAN PyTorch implementation

• Test for feature extraction layer

• Unrecognized command line arguments warning

• Added ReadTheDocs documentation

Changed in 0.3.0

• API

  • First input positional argument of calculate_metrics is now expected as a value to kwarg input1

  • Second input (optional) positional argument of calculate_metrics is now expected as a value to kwarg argument input2

  • cache_input1_name renamed to input1_cache_name

  • cache_input2_name renamed to input2_cache_name

  • rng_seed default value from 2020 to 2021

• Command line

  • First input positional argument is now expected as a value to the key --input1

  • Second input (optional) positional argument is now expected as a value to the key --input2

  • --datasets-downloaded renamed to --no-datasets-download

  • --samples-alphanumeric renamed to --no-samples-shuffle

  • --cache-input1-name renamed to --input1-cache-name

  • --cache-input2-name renamed to --input2-cache-name

  • --rng-seed default value from 2020 to 2021

• Change torch.save to an atomic saving operation in all functions of the caching layer, which makes it safe to use torch-fidelity in multiprocessing environment, such as a compute cluster with a shared file system.

Fixed in 0.3.0

• #15: Fix ‘64’, ‘192’, and ‘768’ feature layers usage in all metrics

• #8: Fix a missing exception for when KID subset size is larger than the number of samples in one of the inputs

• Fix a missing check that the elements of inputs are actually instances of torch.Tensor

[0.2.0] - 2020-05-05

Added in 0.2.0

• Initial release with Inception Score (ISC), Frechet Inception Distance (FID), and Kernel Inception Distance (KID) metrics

• Numerical precision unit tests for all three metrics

• Command line tool and Python API