API

Here you can find description of functions and their keyword arguments.

torch_fidelity module

torch_fidelity.calculate_metrics(**kwargs)[source]

Calculates metrics for the given inputs. Keyword arguments:

Parameters

  • input1 (str or torch.utils.data.Dataset or GenerativeModelBase) –

    First input, which can be either of the following values:

    Default: None.

  • input2 (str or torch.utils.data.Dataset or GenerativeModelBase) –

    Second input, which can be either of the following values:

    Default: None.

  • cuda (bool) – Sets executor device to GPU. Default: True.

  • batch_size (int) – Batch size used to process images; the larger the more memory is used on the executor device (see cuda). Default: 64.

  • isc (bool) – Calculate ISC (Inception Score). Default: False.

  • fid (bool) – Calculate FID (Frechet Inception Distance). Default: False.

  • kid (bool) – Calculate KID (Kernel Inception Distance). Default: False.

  • ppl (bool) – Calculate PPL (Perceptual Path Length). Default: False.

  • feature_extractor (str) – Name of the feature extractor (see registry). Default: inception-v3-compat.

  • feature_layer_isc (str) – Name of the feature layer to use with ISC metric. Default: logits_unbiased.

  • feature_layer_fid (str) – Name of the feature layer to use with FID metric. Default: “2048”.

  • feature_layer_kid (str) – Name of the feature layer to use with KID metric. Default: “2048”.

  • feature_extractor_weights_path (str) – Path to feature extractor weights (downloaded if None). Default: None.

  • isc_splits (int) – Number of splits in ISC. Default: 10.

  • kid_subsets (int) – Number of subsets in KID. Default: 100.

  • kid_subset_size (int) – Subset size in KID. Default: 1000.

  • kid_degree (int) – Degree of polynomial kernel in KID. Default: 3.

  • kid_gamma (float) – Polynomial kernel gamma in KID (automatic if None). Default: None.

  • kid_coef0 (float) – Polynomial kernel coef0 in KID. Default: 1.0.

  • ppl_epsilon (float) – Interpolation step size in PPL. Default: 1e-4.

  • ppl_reduction (str) – Reduction type to apply to the per-sample output values. Default: mean.

  • ppl_sample_similarity (str) – Name of the sample similarity to use in PPL metric computation (see registry). Default: lpips-vgg16.

  • ppl_sample_similarity_resize (int) – Force samples to this size when computing similarity, unless set to None. Default: 64.

  • ppl_sample_similarity_dtype (str) – Check samples are of compatible dtype when computing similarity, unless set to None. Default: uint8.

  • ppl_discard_percentile_lower (int) – Removes the lower percentile of samples before reduction. Default: 1.

  • ppl_discard_percentile_higher (int) – Removes the higher percentile of samples before reduction. Default: 99.

  • ppl_z_interp_mode (str) – Noise interpolation mode in PPL (see registry). Default: lerp.

  • samples_shuffle (bool) – Perform random samples shuffling before computing splits. Default: True.

  • samples_find_deep (bool) – Find all samples in paths recursively. Default: False.

  • samples_find_ext (str) – List of comma-separated extensions (no blanks) to look for when traversing input path. Default: png,jpg,jpeg.

  • samples_ext_lossy (str) – List of comma-separated extensions (no blanks) to warn about lossy compression. Default: jpg,jpeg.

  • datasets_root (str) – Path to built-in torchvision datasets root. Default: $ENV_TORCH_HOME/fidelity_datasets.

  • datasets_download (bool) – Download torchvision datasets to dataset_root. Default: True.

  • cache_root (str) – Path to file cache for features and statistics. Default: $ENV_TORCH_HOME/fidelity_cache.

  • cache (bool) – Use file cache for features and statistics. Default: True.

  • input1_cache_name (str) – Assigns a cache entry to input1 (when not a registered input) and forces caching of features on it. Default: None.

  • input1_model_z_type (str) – Type of noise, only required when the input is a path to a generator model (see registry). Default: normal.

  • input1_model_z_size (int) – Dimensionality of noise (only required when the input is a path to a generator model). Default: None.

  • input1_model_num_classes (int) – Number of classes for conditional (0 for unconditional) generation (only required when the input is a path to a generator model). Default: 0.

  • input1_model_num_samples (int) – Number of samples to draw (only required when the input is a generator model). This option affects the following metrics: ISC, FID, KID. Default: None.

  • input2_cache_name (str) – Assigns a cache entry to input2 (when not a registered input) and forces caching of features on it. Default: None.

  • input2_model_z_type (str) – Type of noise, only required when the input is a path to a generator model (see registry). Default: normal.

  • input2_model_z_size (int) – Dimensionality of noise (only required when the input is a path to a generator model). Default: None.

  • input2_model_num_classes (int) – Number of classes for conditional (0 for unconditional) generation (only required when the input is a path to a generator model). Default: 0.

  • input2_model_num_samples (int) – Number of samples to draw (only required when the input is a generator model). This option affects the following metrics: FID, KID. Default: None.

  • rng_seed (int) – Random numbers generator seed for all operations involving randomness. Default: 2020.

  • save_cpu_ram (bool) – Use less CPU RAM at the cost of speed. May not lead to improvement with every metric. Default: False.

  • verbose (bool) – Output progress information to STDERR. Default: True.

Returns

Dictionary of metrics with a subset of the following keys:

  • torch_fidelity.KEY_METRIC_ISC_MEAN

  • torch_fidelity.KEY_METRIC_ISC_STD

  • torch_fidelity.KEY_METRIC_FID

  • torch_fidelity.KEY_METRIC_KID_MEAN

  • torch_fidelity.KEY_METRIC_KID_STD

  • torch_fidelity.KEY_METRIC_PPL_MEAN

  • torch_fidelity.KEY_METRIC_PPL_STD

  • torch_fidelity.KEY_METRIC_PPL_RAW

torch_fidelity.register_dataset(name, fn_create)[source]

Registers a new input source.

Parameters

  • name (str) – Unique name of the input source.

  • fn_create (callable) –

    A constructor of a Dataset instance. Callable arguments:

    • root (str): Location where the dataset files may be downloaded.

    • download (bool): Whether to perform downloading or rely on the cached version.

torch_fidelity.register_feature_extractor(name, cls)[source]

Registers a new feature extractor.

Parameters
torch_fidelity.register_sample_similarity(name, cls)[source]

Registers a new sample similarity measure.

Parameters
torch_fidelity.register_noise_source(name, fn_generate)[source]

Registers a new noise source, which can generate samples to be used as inputs to generative models.

Parameters

  • name (str) – Unique name of the noise source.

  • fn_generate (callable) –

    Generator of a random samples of specified type and shape. Callable arguments:

    • rng (numpy.random.RandomState): random number generator state, initialized with seed.

    • shape (torch.Size): shape of the tensor of random samples.

torch_fidelity.register_interpolation(name, fn_interpolate)[source]

Registers a new sample interpolation method.

Parameters

  • name (str) – Unique name of the interpolation method.

  • fn_interpolate (callable) –

    Sample interpolation function. Callable arguments:

    • a (torch.Tensor): batch of the first endpoint samples.

    • b (torch.Tensor): batch of the second endpoint samples.

    • t (float): interpolation coefficient in the range [0,1].

class torch_fidelity.FeatureExtractorBase(name, features_list)[source]

Bases: torch.nn.modules.module.Module

Base class for feature extractors that can be used in calculate_metrics().

Parameters

  • name (str) – Unique name of the subclassed feature extractor, must be the same as used in register_feature_extractor().

  • features_list (list) – List of feature names, provided by the subclassed feature extractor.

forward(input)[source]

Returns a tuple of tensors extracted from the input, in the same order as they are provided by get_provided_features_list().

static get_provided_features_list()[source]

Returns a tuple of feature names, extracted by the subclassed feature extractor.

class torch_fidelity.FeatureExtractorInceptionV3(name, features_list, feature_extractor_weights_path=None, **kwargs)[source]

Bases: torch_fidelity.feature_extractor_base.FeatureExtractorBase

InceptionV3 feature extractor for 2D RGB 24bit images.

Parameters

  • name (str) – Unique name of the feature extractor, must be the same as used in register_feature_extractor().

  • features_list (list) –

    A list of the requested feature names, which will be produced for each input. This feature extractor provides the following features:

    • ’64’

    • ’192’

    • ’768’

    • ’2048’

    • ’logits_unbiased’

    • ’logits’

  • feature_extractor_weights_path (str) – Path to the pretrained InceptionV3 model weights in PyTorch format. Refer to util_convert_inception_weights for making your own. Downloads from internet if None.

class torch_fidelity.GenerativeModelBase[source]

Bases: abc.ABC, torch.nn.modules.module.Module

Base class for generative models that can be used as inputs in calculate_metrics().

Initializes internal Module state, shared by both nn.Module and ScriptModule.

abstract property num_classes

Number of classes used by a conditional generative model. Must return zero for unconditional models.

abstract property z_size

Size of the noise dimension of the generative model (positive integer).

abstract property z_type

Type of the noise used by the generative model (see registry for a list of preregistered noise types, see register_noise_source() for registering a new noise type).

class torch_fidelity.GenerativeModelModuleWrapper(module, z_size, z_type, num_classes, make_copy=False, make_eval=True, cuda=None)[source]

Bases: torch_fidelity.generative_model_base.GenerativeModelBase

Wraps any generative model torch.nn.Module, implements the GenerativeModelBase interface, and provides a few convenience functions.

Parameters

  • module (torch.nn.Module) – A generative model module, taking a batch of noise samples, and producing generative samples.

  • z_size (int) – Size of the noise dimension of the generative model (positive integer).

  • z_type (str) – Type of the noise used by the generative model (see registry for a list of preregistered noise types, see register_noise_source() for registering a new noise type).

  • num_classes (int) – Number of classes used by a conditional generative model. Must return zero for unconditional models.

  • make_copy (bool) – Makes a copy of the model weights if True. Default: False.

  • make_eval (bool) – Switches to torch.nn.Module evaluation mode upon construction if True. Default: True.

  • cuda (bool) – Moves the module on a CUDA device if True, moves to CPU if False, does nothing if None. Default: None.

class torch_fidelity.GenerativeModelONNX(path_onnx, z_size, z_type, num_classes)[source]

Bases: torch_fidelity.generative_model_base.GenerativeModelBase

Wraps ONNX generative model, implements the GenerativeModelBase interface.

Parameters

  • path_onnx (str) – Path to a generative model in ONNX format.

  • z_size (int) – Size of the noise dimension of the generative model (positive integer).

  • z_type (str) – Type of the noise used by the generative model (see registry for a list of preregistered noise types, see register_noise_source() for registering a new noise type).

  • num_classes (int) – Number of classes used by a conditional generative model. Must return zero for unconditional models.

class torch_fidelity.SampleSimilarityBase(name)[source]

Bases: torch.nn.modules.module.Module

Base class for samples similarity measures that can be used in calculate_metrics().

Parameters

name (str) – Unique name of the subclassed sample similarity measure, must be the same as used in register_sample_similarity().

forward(*args)[source]

Returns the value of sample similarity between the inputs.

class torch_fidelity.SampleSimilarityLPIPS(name, sample_similarity_resize=None, sample_similarity_dtype=None, **kwargs)[source]

Bases: torch_fidelity.sample_similarity_base.SampleSimilarityBase

LPIPS sample similarity measure for 2D RGB 24bit images.

Parameters

  • name (str) – Unique name of the sample similarity measure, must be the same as used in register_sample_similarity().

  • sample_similarity_resize (int or None) – Resizes inputs to this size if set, keeps as is if None.

  • sample_similarity_dtype (str) – Coerces tensor dtype to one of the following: ‘uint8’, ‘float32’. This is useful when the inputs are generated by a generative model, to ensure the proper data range and quantization.