API Reference

Core APIs

• seqio.dataset_providers package
  • Interfaces
    • DataSourceInterface
    • DatasetProviderBase
    • DatasetProviderRegistry
  • Data Sources
    • DataSource
    • TfdsDataSource
    • FileDataSource
    • TFExampleDataSource
    • TextLineDataSource
    • ProtoDataSource
    • FunctionDataSource
  • Task
    • Task
    • ShardInfo
    • SourceInfo
    • CacheDatasetPlaceholder
  • Mixture
    • Mixture
    • PyGloveTunableMixture
  • Registry
    • TaskRegistry
    • MixtureRegistry
  • APIs
    • get_dataset()
    • get_mixture_or_task()
    • get_subtasks()
    • maybe_get_mixture_or_task()
• seqio.evaluation package
  • Evaluator
    • Evaluator.eval_tasks
    • Evaluator.cached_model_datasets
    • Evaluator.cached_task_datasets
    • Evaluator.model_feature_shapes
    • Evaluator.loggers
    • Evaluator.close()
    • Evaluator.evaluate()
• seqio.feature_converters package
  • Base Class
    • FeatureConverter
  • Implementations
    • DecoderFeatureConverter
    • EncDecFeatureConverter
    • EncoderFeatureConverter
    • LMFeatureConverter
    • PassThroughFeatureConverter
    • PrePackedEncDecFeatureConverter
    • PrePackedLMFeatureConverter
    • PrePackedPrefixLMFeatureConverter
    • PrefixLMFeatureConverter
    • PrefixSuffixLMFeatureConverter
    • non_padding_position()

Components

• seqio.preprocessors package
  • append_eos()
  • append_eos_after_trim()
  • append_eos_after_trim_impl()
  • apply_feature_converter()
  • preprocess_tensorflow_examples()
  • print_dataset()
  • rekey()
  • tokenize()
  • tokenize_and_append_eos()
  • tokenize_impl()
  • truncate_inputs_left()
• seqio.metrics package
  • Audio
  • CollectingMetric
    • CollectingMetric.actual_compute()
    • CollectingMetric.from_model_output()
  • Generic
  • Histogram
  • Image
  • LegacyMetric
    • LegacyMetric.compute()
    • LegacyMetric.empty()
    • LegacyMetric.from_model_output()
    • LegacyMetric.postprocess_fn()
    • LegacyMetric.replace()
  • Metric
    • Metric.from_model_output()
    • Metric.replace()
  • MetricValue
  • ModelOutputType
  • PassthroughLegacyMetric
    • PassthroughLegacyMetric.from_metric_fn()
    • PassthroughLegacyMetric.replace()
  • Scalar
  • Text
  • globally_sort_model_output()
  • remove_padding_examples()
• seqio.loggers package
  • JSONLogger
  • Logger
    • Logger.output_dir
  • PyLoggingLogger
  • TensorAndNumpyEncoder
    • TensorAndNumpyEncoder.default()
  • TensorBoardLogger
  • TensorBoardLoggerV1
  • skip_none_value_dict_factory()
• seqio.experimental package
  • FewshotDataSource
    • FewshotDataSource.get_dataset()
    • FewshotDataSource.list_shards()
    • FewshotDataSource.supports_arbitrary_sharding
  • add_fully_cached_mixture()
  • add_fully_cached_task()
  • add_task_with_sentinels()
  • disable_registry()
  • fewshot_preprocessor()

Helpers

• seqio.utils package
  • Feature
    • Feature.dtype
  • LazyTfdsLoader
    • LazyTfdsLoader.builder_cls()
    • LazyTfdsLoader.data_dir
    • LazyTfdsLoader.files()
    • LazyTfdsLoader.get_split_params()
    • LazyTfdsLoader.is_memoized()
    • LazyTfdsLoader.load()
    • LazyTfdsLoader.load_shard()
    • LazyTfdsLoader.resolved_tfds_name()
    • LazyTfdsLoader.set_decoders()
    • LazyTfdsLoader.size()
  • TfdsSplit
    • TfdsSplit.dataset
    • TfdsSplit.split
    • TfdsSplit.data_dir
  • add_kwargs_to_transform()
  • dict_to_tfexample()
  • flatten_dict()
  • fully_qualified_class_name()
  • function_name()
  • list_files()
  • make_autoregressive_inputs()
  • map_over_dataset()
  • map_seed_manager()
  • mixing_rate_num_characters()
  • mixing_rate_num_examples()
  • set_preprocessor_seed()
  • stateless_shuffle()
  • tfexample_ragged_length_key()
  • tfexample_sparse_indices_key()
  • tfexample_to_dict()
  • trim_and_pack_dataset()
  • trim_and_pad_dataset()
  • trim_dataset()
  • unflatten_dict()
• seqio.helpers package
  • TruncatedDatasetProvider
    • TruncatedDatasetProvider.caching_permitted
    • TruncatedDatasetProvider.get_dataset()
    • TruncatedDatasetProvider.list_shards()
    • TruncatedDatasetProvider.num_input_examples()
    • TruncatedDatasetProvider.output_features
    • TruncatedDatasetProvider.splits
    • TruncatedDatasetProvider.supports_arbitrary_sharding
  • mixture_or_task_with_new_vocab()
  • mixture_or_task_with_truncated_data()
  • mixture_with_missing_task_splits_removed()
• seqio.beam_utils package
  • GetInfo
  • GetStats
  • PreprocessTask
  • WriteExampleArrayRecord
  • WriteExampleTfRecord
  • WriteJson
  • WriteToArrayRecord
• seqio.test_utils package
  • DataInjector
    • DataInjector.task_name
    • DataInjector.per_split_data
  • FakeLazyTfds
    • FakeLazyTfds.data_dir
    • FakeLazyTfds.files
    • FakeLazyTfds.info
    • FakeLazyTfds.load
    • FakeLazyTfds.load_shard
    • FakeLazyTfds.name
    • FakeLazyTfds.resolved_tfds_name
    • FakeLazyTfds.size
    • FakeLazyTfds.tfds_splits
  • FakeMixtureTest
    • FakeMixtureTest.setUp()
  • FakeTaskTest
    • FakeTaskTest.setUp()
    • FakeTaskTest.tearDown()
    • FakeTaskTest.verify_task_matches_fake_datasets()
  • FakeTfdsInfo
    • FakeTfdsInfo.config_name
    • FakeTfdsInfo.description
    • FakeTfdsInfo.features
    • FakeTfdsInfo.file_format
    • FakeTfdsInfo.homepage
    • FakeTfdsInfo.splits
    • FakeTfdsInfo.version
  • MockVocabulary
    • MockVocabulary.encode()
    • MockVocabulary.encode_tf()
    • MockVocabulary.vocab_size
  • assert_dataset()
  • assert_datasets_eq()
  • assert_datasets_neq()
  • assert_dict_contains()
  • assert_dict_values_equal()
  • create_default_dataset()
  • get_fake_dataset()
  • random_token_preprocessor()
  • split_tsv_preprocessor()
  • test_postprocessing()
  • test_preprocessing()
  • test_preprocessing_single()
  • test_task()
  • test_text_preprocessor()
  • test_token_preprocessor()