python-tokenizers/python-tokenizers.changes

-------------------------------------------------------------------
Wed Mar 19 18:26:11 UTC 2025 - Lucas Mulling <lucas.mulling@suse.com>

- Update to 0.21.1:
  * Update dev version and pyproject.toml
  * Add feature flag hint to README.md
  * Upgrade to PyO3 0.23
  * Fixing the README.md
  * Fix typo in Split docstrings
  * Fix typos
  * Update documentation of Rust feature
  * Fix panic in DecodeStream::step due to incorrect index usage
  * Fixing the stream by removing the read_index altogether
  * Fixing NormalizedString append when normalized is empty
  * Update metadata as Python3.7 and Python3.8 support was dropped
  * Add rustls-tls feature
- Remove define skip_python313 1

-------------------------------------------------------------------
Wed Mar  5 10:32:02 UTC 2025 - Christian Goll <cgoll@suse.com>

- disable python3.13

-------------------------------------------------------------------
Thu Jan  9 15:52:37 UTC 2025 - Andreas Schwab <schwab@suse.de>

- Enable build on riscv64

-------------------------------------------------------------------
Wed Dec 18 14:20:07 UTC 2024 - Soc Virnyl Estela <uncomfyhalomacro@opensuse.org>

- Update to version 0.21.0:
  * More cache options.
  * Disable caching for long strings.
  * Testing ABI3 wheels to reduce number of wheels
  * Adding an API for decode streaming.
  * Decode stream python
  * Fix encode_batch and encode_batch_fast to accept ndarrays again

-------------------------------------------------------------------
Thu Nov  7 11:30:50 UTC 2024 - Soc Virnyl Estela <uncomfyhalomacro@opensuse.org>

- Select only rust tier 1 arches. 
- Update registry.tar.zst dependencies
- Update version to 0.20.3:
  * fix pylist
  * [MINOR:TYP] Fix docstrings
- Updates from 0.20.2:
  * Bump cookie and express in /tokenizers/examples/unstable_wasm/www
  * Fix off-by-one error in tokenizer::normalizer::Range::len
  * Arg name correction: auth_token -> token
  * Unsound call of set_var
  * Add safety comments
  * PyO3 0.22
- Updates from 0.20.1:
  * Update README.md
  * fix benchmark file link
  * [ignore_merges] Fix offsets
  * Bump body-parser and express in /tokenizers/examples/unstable_wasm/www
  * Bump serve-static and express in /tokenizers/examples/unstable_wasm/www
  * Bump send and express in /tokenizers/examples/unstable_wasm/www
  * Bump webpack from 5.76.0 to 5.95.0 in /tokenizers/examples/unstable_wasm/www
  * Fix documentation build
  * style: simplify string formatting for readability

-------------------------------------------------------------------
Sun Nov  3 12:22:13 UTC 2024 - Soc Virnyl Estela <uncomfyhalomacro@opensuse.org>

- Experiment with cargo vendor home registry. See documentation:
  https://github.com/openSUSE-Rust/obs-service-cargo/blob/master/README.md#cargo-vendor-home-registry

-------------------------------------------------------------------
Mon Sep 23 07:19:52 UTC 2024 - Simon Lees <sflees@suse.de>

- Don't use macros for Requires

-------------------------------------------------------------------
Fri Aug 30 05:18:30 UTC 2024 - Simon Lees <sflees@suse.de>

- Update package name back to "huggingface-hub" to match pypi

-------------------------------------------------------------------
Tue Aug 27 05:48:31 UTC 2024 - Guang Yee <gyee@suse.com>

- Update package name "huggingface-hub" to "huggingface_hub" 

-------------------------------------------------------------------
Tue Aug 20 07:27:42 UTC 2024 - Simon Lees <sflees@suse.de>

- Fix testsuite on 15.6

-------------------------------------------------------------------
Sun Aug 18 16:49:56 UTC 2024 - Soc Virnyl Estela <obs@uncomfyhalomacro.pl>

- Replace vendor tarball to zstd compressed vendor tarball
- Force gcc version on leap. Thanks @marv7000 for your zed.spec
- Use `CARGO_*` environmental variables to force generate
  full debuginfo and avoid stripping.
- Enable cargo test in %check.
- Update to version 0.20.0:
  * remove enforcement of non special when adding tokens
  * [BREAKING CHANGE] Ignore added_tokens (both special and not) in the decoder
  * Make USED_PARALLELISM atomic
  * Fixing for clippy 1.78
  * feat(ci): add trufflehog secrets detection
  * Switch from cached_download to hf_hub_download in tests
  * Fix "dictionnary" typo
  * make sure we don't warn on empty tokens
  * Enable dropout = 0.0 as an equivalent to none in BPE
  * Revert "[BREAKING CHANGE] Ignore added_tokens (both special and not) …
  * Add bytelevel normalizer to fix decode when adding tokens to BPE
  * Fix clippy + feature test management.
  * Bump spm_precompiled to 0.1.3
  * Add benchmark vs tiktoken
  * Fixing the benchmark.
  * Tiny improvement
  * Enable fancy regex
  * Fixing release CI strict (taken from safetensors).
  * Adding some serialization testing around the wrapper.
  * Add-legacy-tests
  * Adding a few tests for decoder deserialization.
  * Better serialization error
  * Add test normalizers
  * Improve decoder deserialization
  * Using serde (serde_pyo3) to get str and repr easily.
  * Merges cannot handle tokens containing spaces.
  * Fix doc about split
  * Support None to reset pre_tokenizers and normalizers, and index sequences
  * Fix strip python type
  * Tests + Deserialization improvement for normalizers.
  * add deserialize for pre tokenizers
  * Perf improvement 16% by removing offsets.

-------------------------------------------------------------------
Wed Jul  3 14:55:36 UTC 2024 - Christian Goll <cgoll@suse.com>

- initial commit on rust based python-tokenizers