# Miscellaneous

## Inputs

ISC and PPL are computed for `input1` only, whereas FID, KID, and MIND are computed between `input1` and `input2`.
PRC is also computed between `input1` and `input2`, where `input1` is treated as the generated (evaluated) set and
`input2` as the real (reference) set: precision measures the fraction of `input1` samples falling within the `input2`
manifold, and recall measures the fraction of `input2` samples falling within the `input1` manifold.

*Note: In v0.4.0-beta, precision and recall values were swapped. Only users who installed this pre-release version directly from GitHub may be affected; no pip-published release contained this bug.*

Each input can be one of the following:
- Registered input, such as a `cifar10-train` string. Registered inputs are commonly used datasets, which can be 
  resolved by name, and are subject to caching;
- Path to a directory with samples;
- Path to a generative model in the ONNX or PTH (JIT) format;
- Instance of `torch.util.data.Dataset`;
- Instance of `torch_fidelity.GenerativeModelBase`.

## Storage, Cache, Datasets

The cached items can occupy quite a lot of space. The default cache root is created under 
`$ENV_TORCH_HOME/fidelity_cache`, which is usually under the `$HOME`. Users with limited home partition should use 
`--cache_root` key to change cache location, or specify `--no-cache` to disable it (not recommended).

Likewise, torchvision datasets may not be suitable for storage under the home directory (default location is
`$ENV_TORCH_HOME/fidelity_datasets`). It can be changed with the key `--datasets-root`. 
If torchvision datasets do not need to be downloaded, it is possible to disable download check using the key 
`--no-datasets-download`.

To save time on recomputations of features and statistics of inputs that do not change often, caching is enabled
on all registered inputs. In addition to that, one can force caching on a path input by assigning a new cache slot name
to such input via `input1_cache_name` or `input2_cache_name` keys, for the first and second positional arguments
respectively.

## Working with Directories of Samples as Inputs

To collect files recursively under the path provided as the input, add `--samples-find-deep` command line key, or set 
the `samples_find_deep` keyword argument to True. 
To change file extensions picked up when traversing the path, specify `--samples-find-ext` command line key or 
the `samples_find_ext` keyword argument.
After the files list is collected, it is sorted alpha-numerically, and then shuffled. If shuffling is not desired, it
can be disabled by setting the `--no-samples-shuffle` key or using the `samples_shuffle` keyword argument.
Since in-the-wild images can be of arbitrary shapes, it is necessary to bring them all to the same canonical size and
square shape, compatible with the evaluation protocol. Specify `--samples_resize_and_crop` to resize all images to match
the given size on the shorter side and then perform center cropping.

## Other Options

Both the `fidelity` command and the API provide several options to tweak the default behavior of the evaluation 
procedure. 
Below is a summary of use cases when changing the defaults is required:

- *Reducing GPU RAM usage*: by default, evaluating `cifar10-train` images with all metrics takes about 2.5 GB of RAM. 
  If it is desired to reduce memory footprint, use `--batch-size` key to reduce batch size. 
  It is not possible to go below a certain threshold required to store the feature extractor model (Inception).

- *Changing device placement*: by default, `fidelity` app decides whether to use GPU judging by the 
  `CUDA_VISIBLE_DEVICES` environment variable of the process. 
  It is possible to override this default behavior by specifying GPU ids with `--gpu` command line key, or forcing 
  computations to CPU with `--cpu` flag.

- *Verifying reproducibility*: For the sake of reproducibility, all pseudo-random numbers generators are pre-seeded 
  with the default value of `--rng_seed` command line key. 
  This affects choosing subsets and splits in metrics and the order of shuffled files from positional arguments.  
  One may want to change this flag to see the effect of seed on the metrics outputs. However, seeds should be kept 
  fixed and reported as a part of the evaluation protocol.  
 
- *Changing verbosity*: verbose mode is enabled by default, so both command line and programmatic interfaces report
  progress to `stderr`. 
  It can be disabled using the `--silent` command line flag, or `verbose` keyword argument. 
  However, it is useful to keep it enabled when extending the code with custom input sources or feature extractors, 
  as it allows to prevent unintended usage of the cache.