seqio.metrics package

MetricValue objects to wrap results being returned by metric funcitons.

class seqio.metrics.Audio(audiodata, sample_rate=44100, max_outputs=3)

An audio example to output to tensorboard.

The format for the audio array should match the format expected for the data parameter described [here](https://www.tensorflow.org/api_docs/python/tf/summary/audio).

class seqio.metrics.CollectingMetric(values)

CollectingMetric interface for seqio evaluation.

actual_compute(task_dataset_as_numpy, task_output_features, target_field_name='targets', cached_targets=None)

Implements the metric computation logics for CollectingMetric.

Parameters:

• task_dataset_as_numpy – Examples in dataset.

• task_output_features – Output features defined in the seqio.Task.

• target_field_name – Field name of the target sequence.

• cached_targets – targets that have been cached by Evaluator and can be supplied here to save time of post-processing targets.

Returns:

A tuple of two items, first item is a dict of metric results, the second

item is targets_and_inferences.

Raises:

NotImplementedError – Must override from_model_output()

classmethod from_model_output(inputs, model_output, features, target_field_name='targets', mask=None, indices_2d=None)

Creates a Metric from model outputs.

class seqio.metrics.Generic(tensor, metadata)

A raw tensor to output to tensorboard.

class seqio.metrics.Histogram(values, bins=None)

A histogram to output to tensorboard.

class seqio.metrics.Image(image, max_outputs=3)

An image to output to tensorboard.

The format for the image array should match the format expected for the data parameter described [here](https://www.tensorflow.org/api_docs/python/tf/summary/image).

class seqio.metrics.LegacyMetric(model_output_type, _metric_fn, _postprocess_fn, metric_fn_kwargs, targets_and_inferences)

Metric class for legacy use-case where metric fn is supplied.

compute()

Computes final metrics from intermediate values.

classmethod empty(metric_fn, postprocess_fn)

Returns an empty instance (i.e. .merge(Metric.empty()) is a no-op).

from_model_output(inputs, model_output, features, target_field_name='targets', mask=None)

Creates a seqio.Metric from model outputs.

Parameters:

• inputs – Examples in dataset.

• model_output – Model output computed by model functions.

• features – Output features defined in seqio.Task.

• target_field_name – Field name of the target sequence.

• mask – A boolean array to indicate which examples in the inputs are included for metric evaluation.

• indices_2d – 2d-indices of examples in the inputs/model_output. First dimension is shard id, the second is the example id within that shard.

Returns:

An instance of Metric.

Raises:

NotImplementedError – Must override from_model_output()

postprocess_fn(targets_or_predictions, **postprocess_kwargs)

Applies the postprocessing to targets or predictions.

replace(**updates)

Returns a new object replacing the specified fields with new values.

class seqio.metrics.Metric(model_output_type)

Base Metric class for seqio evaluation.

classmethod from_model_output(inputs, model_output, features, target_field_name='targets', mask=None, indices_2d=None)

Creates a seqio.Metric from model outputs.

Parameters:

• inputs – Examples in dataset.

• model_output – Model output computed by model functions.

• features – Output features defined in seqio.Task.

• target_field_name – Field name of the target sequence.

• mask – A boolean array to indicate which examples in the inputs are included for metric evaluation.

• indices_2d – 2d-indices of examples in the inputs/model_output. First dimension is shard id, the second is the example id within that shard.

Returns:

An instance of Metric.

Raises:

NotImplementedError – Must override from_model_output()

replace(**updates)

Returns a new object replacing the specified fields with new values.

class seqio.metrics.MetricValue

A base method for the dataclasses that represent tensorboard values.

Task metric_fn`s should output `Mapping[str, MetricValue] which will be written by a Logger.

class seqio.metrics.ModelOutputType(value)

Model output types.

class seqio.metrics.PassthroughLegacyMetric(values)

Makes PassthroughLegacyMetric from metric functions.

classmethod from_metric_fn(metric_fn, postprocess_fn=None)

Creates PassthroughLegacyMetric from metric_fn and postprocess_fn.

Example:

``` squad_cls = PassthroughLegacyMetric.from_metric_fn(

metric_fn=t5_metrics.squd, postprocess_fn=t5_postprocessors.qa)

```

Parameters:

• metric_fn – Function used to compute metric.

• postprocess_fn – Function used to process targets (vocab decoded) and predictions (vocab decoded) before feeding into metric_fn.

Returns:

A Metric that calls metric_fn and postprocess_fn in its .from_model_output().

replace(**updates)

Returns a new object replacing the specified fields with new values.

class seqio.metrics.Scalar(value)

The default tensorflow value, used for creating time series graphs.

class seqio.metrics.Text(textdata)

Text to output to tensorboard, markdown is rendered by tensorboard.

seqio.metrics.globally_sort_model_output(model_output, indices_2d)

Globally sorts model ouputs by the 2d indices of the examples.

The sorting is done first by shard id (first index of the 2d-index) and then by example id (second index of the 2d-index).

Parameters:

• model_output – model outputs of all the examples.

• indices_2d – 2d indices of all the examples.

Returns:

sorted model outputs.

seqio.metrics.remove_padding_examples(model_output, indices_2d, mask)

Removes padding examples indicated by the mask array.

Parameters:

• model_output – model outputs of all the examples (including the padding ones). The padding examples are used to make sure during inference, the inference function receives full batch if the last batch does not enough examples.

• indices_2d – 2d indices of all the examples.

• mask – an array of booleans. 1 indicates valid example, 0 indicates padded example that needs to be removed.

Returns:

2d-indices and model outputs of all the non-padding examples.