Update dependency datasets to v3.5.0

[renovate]

This PR contains the following updates:

Package	Change	Age	Adoption	Passing	Confidence
datasets	3.1.0 -> 3.5.0

---

Release Notes

huggingface/datasets (datasets)

v3.5.0

Compare Source

Datasets Features

• Introduce PDF support (#​7318) by @​yabramuvdi in https://github.com/huggingface/datasets/pull/7325

>>> from datasets import load_dataset, Pdf
>>> repo = "path/to/pdf/folder"  # or username/dataset_name on Hugging Face
>>> dataset = load_dataset(repo, split="train")
>>> dataset[0]["pdf"]
<pdfplumber.pdf.PDF at 0x1075bc320>
>>> dataset[0]["pdf"].pages[0].extract_text()
...

What's Changed

• Fix local pdf loading by @​lhoestq in https://github.com/huggingface/datasets/pull/7466
• Minor fix for metadata files in extension counter by @​lhoestq in https://github.com/huggingface/datasets/pull/7464
• Priotitize json by @​lhoestq in https://github.com/huggingface/datasets/pull/7476

New Contributors

• @​yabramuvdi made their first contribution in https://github.com/huggingface/datasets/pull/7325

Full Changelog: huggingface/datasets@3.4.1...3.5.0

v3.4.1

Compare Source

Bug Fixes

• Fix data_files filtering by @​lhoestq in https://github.com/huggingface/datasets/pull/7459

Full Changelog: huggingface/datasets@3.4.0...3.4.1

v3.4.0

Compare Source

Dataset Features

• Faster folder based builder + parquet support + allow repeated media + use torchvideo by @​lhoestq in https://github.com/huggingface/datasets/pull/7424

  • /!\ Breaking change: we replaced decord with torchvision to read videos, since decord is not maintained anymore and isn't available for recent python versions, see the video dataset loading documentation here for more details. The Video type is still marked as experimental is this version

  from datasets import load_dataset, Video
  
  dataset = load_dataset("path/to/video/folder", split="train")
  dataset[0]["video"]  # <torchvision.io.video_reader.VideoReader at 0x1652284c0>

  • faster streaming for image/audio/video folder from Hugging Face
  • support for metadata.parquet in addition to metadata.csv or metadata.jsonl for the metadata of the image/audio/video files

• Add IterableDataset.decode with multithreading by @​lhoestq in https://github.com/huggingface/datasets/pull/7450

  • even faster streaming for image/audio/video folder from Hugging Face if you enable multithreading to decode image/audio/video data:

  dataset = dataset.decode(num_threads=num_threads)

• Add with_split to DatasetDict.map by @​jp1924 in https://github.com/huggingface/datasets/pull/7368

General improvements and bug fixes

• fix: None default with bool type on load creates typing error by @​stephantul in https://github.com/huggingface/datasets/pull/7426
• Use pyupgrade --py39-plus by @​cyyever in https://github.com/huggingface/datasets/pull/7428
• Refactor string_to_dict to return None if there is no match instead of raising ValueError by @​ringohoffman in https://github.com/huggingface/datasets/pull/7435
• Fix small bugs with async map by @​lhoestq in https://github.com/huggingface/datasets/pull/7445
• Fix resuming after ds.set_epoch(new_epoch) by @​lhoestq in https://github.com/huggingface/datasets/pull/7451
• minor docs changes by @​lhoestq in https://github.com/huggingface/datasets/pull/7452

New Contributors

• @​stephantul made their first contribution in https://github.com/huggingface/datasets/pull/7426
• @​cyyever made their first contribution in https://github.com/huggingface/datasets/pull/7428
• @​jp1924 made their first contribution in https://github.com/huggingface/datasets/pull/7368

Full Changelog: huggingface/datasets@3.3.2...3.4.0

v3.3.2

Compare Source

Bug fixes

• Attempt to fix multiprocessing hang by closing and joining the pool before termination by @​dakinggg in https://github.com/huggingface/datasets/pull/7411
• Gracefully cancel async tasks by @​lhoestq in https://github.com/huggingface/datasets/pull/7414

Other general improvements

• Update use_with_pandas.mdx: to_pandas() correction in last section by @​ibarrien in https://github.com/huggingface/datasets/pull/7407
• Fix a typo in arrow_dataset.py by @​jingedawang in https://github.com/huggingface/datasets/pull/7402

New Contributors

• @​dakinggg made their first contribution in https://github.com/huggingface/datasets/pull/7411
• @​ibarrien made their first contribution in https://github.com/huggingface/datasets/pull/7407
• @​jingedawang made their first contribution in https://github.com/huggingface/datasets/pull/7402

Full Changelog: huggingface/datasets@3.3.1...3.3.2

v3.3.1

Compare Source

Bug fixes

• Fix filter speed regression by @​lhoestq in https://github.com/huggingface/datasets/pull/7408

Full Changelog: huggingface/datasets@3.3.0...3.3.1

v3.3.0

Compare Source

Dataset Features

• Support async functions in map() by @​lhoestq in https://github.com/huggingface/datasets/pull/7384

  • Especially useful to download content like images or call inference APIs

  prompt = "Answer the following question: {question}. You should think step by step."
  async def ask_llm(example):
      return await query_model(prompt.format(question=example["question"]))
  ds = ds.map(ask_llm)

• Add repeat method to datasets by @​alex-hh in https://github.com/huggingface/datasets/pull/7198

  ds = ds.repeat(10)

• Support faster processing using pandas or polars functions in IterableDataset.map() by @​lhoestq in https://github.com/huggingface/datasets/pull/7370

  • Add support for "pandas" and "polars" formats in IterableDatasets
  • This enables optimized data processing using pandas or polars functions with zero-copy, e.g.

  ds = load_dataset("ServiceNow-AI/R1-Distill-SFT", "v0", split="train", streaming=True)
  ds = ds.with_format("polars")
  expr = pl.col("solution").str.extract("boxed\\{(.*)\\}").alias("value_solution")
  ds = ds.map(lambda df: df.with_columns(expr), batched=True)

• Apply formatting after iter_arrow to speed up format -> map, filter for iterable datasets by @​alex-hh in https://github.com/huggingface/datasets/pull/7207

  • IterableDatasets with "numpy" format are now much faster

What's Changed

• don't import soundfile in tests by @​lhoestq in https://github.com/huggingface/datasets/pull/7340
• minor video docs on how to install by @​lhoestq in https://github.com/huggingface/datasets/pull/7341
• Fix typo in arrow_dataset by @​AndreaFrancis in https://github.com/huggingface/datasets/pull/7328
• remove filecheck to enable symlinks by @​fschlatt in https://github.com/huggingface/datasets/pull/7133
• Webdataset special columns in last position by @​lhoestq in https://github.com/huggingface/datasets/pull/7349
• Bump hfh to 0.24 to fix ci by @​lhoestq in https://github.com/huggingface/datasets/pull/7350
• fsspec 2024.12.0 by @​lhoestq in https://github.com/huggingface/datasets/pull/7352
• changes to MappedExamplesIterable to resolve #​7345 by @​vttrifonov in https://github.com/huggingface/datasets/pull/7353
• Catch OSError for arrow by @​lhoestq in https://github.com/huggingface/datasets/pull/7348
• Remove .h5 from imagefolder extensions by @​lhoestq in https://github.com/huggingface/datasets/pull/7374
• Add Pandas, PyArrow and Polars docs by @​lhoestq in https://github.com/huggingface/datasets/pull/7382
• Optimized sequence encoding for scalars by @​lukasgd in https://github.com/huggingface/datasets/pull/7393
• Update docs by @​lhoestq in https://github.com/huggingface/datasets/pull/7395
• Update README.md by @​lhoestq in https://github.com/huggingface/datasets/pull/7396
• Release: 3.3.0 by @​lhoestq in https://github.com/huggingface/datasets/pull/7398

New Contributors

• @​AndreaFrancis made their first contribution in https://github.com/huggingface/datasets/pull/7328
• @​vttrifonov made their first contribution in https://github.com/huggingface/datasets/pull/7353
• @​lukasgd made their first contribution in https://github.com/huggingface/datasets/pull/7393

Full Changelog: huggingface/datasets@3.2.0...3.3.0

v3.2.0

Compare Source

Dataset Features

• Faster parquet streaming + filters with predicate pushdown by @​lhoestq in https://github.com/huggingface/datasets/pull/7309
  • Up to +100% streaming speed
  • Fast filtering via predicate pushdown (skip files/row groups based on predicate instead of downloading the full data), e.g.

    from datasets import load_dataset
    filters = [('date', '>=', '2023')]
    ds = load_dataset("HuggingFaceFW/fineweb-2", "fra_Latn", streaming=True, filters=filters)

Other improvements and bug fixes

• fix conda release worlflow by @​lhoestq in https://github.com/huggingface/datasets/pull/7272
• Add link to video dataset by @​NielsRogge in https://github.com/huggingface/datasets/pull/7277
• Raise error for incorrect JSON serialization by @​varadhbhatnagar in https://github.com/huggingface/datasets/pull/7273
• support for custom feature encoding/decoding by @​alex-hh in https://github.com/huggingface/datasets/pull/7284
• update load_dataset doctring by @​lhoestq in https://github.com/huggingface/datasets/pull/7301
• Let server decide default repo visibility by @​Wauplin in https://github.com/huggingface/datasets/pull/7302
• fix: update elasticsearch version by @​ruidazeng in https://github.com/huggingface/datasets/pull/7300
• Fix typing in iterable_dataset.py by @​lhoestq in https://github.com/huggingface/datasets/pull/7304
• Updated inconsistent output in documentation examples for ClassLabel by @​sergiopaniego in https://github.com/huggingface/datasets/pull/7293
• More docs to from_dict to mention that the result lives in RAM by @​lhoestq in https://github.com/huggingface/datasets/pull/7316
• Release: 3.2.0 by @​lhoestq in https://github.com/huggingface/datasets/pull/7317

New Contributors

• @​ruidazeng made their first contribution in https://github.com/huggingface/datasets/pull/7300
• @​sergiopaniego made their first contribution in https://github.com/huggingface/datasets/pull/7293

Full Changelog: huggingface/datasets@3.1.0...3.2.0

---

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.