- Update to 0.21.1:

* Update dev version and pyproject.toml
  * Add feature flag hint to README.md
  * Upgrade to PyO3 0.23
  * Fixing the README.md
  * Fix typo in Split docstrings
  * Fix typos
  * Update documentation of Rust feature
  * Fix panic in DecodeStream::step due to incorrect index usage
  * Fixing the stream by removing the read_index altogether
  * Fixing NormalizedString append when normalized is empty
  * Update metadata as Python3.7 and Python3.8 support was dropped
  * Add rustls-tls feature
- Remove define skip_python313 1

OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=13

python-tokenizers.changes

@@ -1,3 +1,21 @@
+-------------------------------------------------------------------
+Wed Mar 19 18:26:11 UTC 2025 - Lucas Mulling <lucas.mulling@suse.com>
+
+- Update to 0.21.1:
+  * Update dev version and pyproject.toml
+  * Add feature flag hint to README.md
+  * Upgrade to PyO3 0.23
+  * Fixing the README.md
+  * Fix typo in Split docstrings
+  * Fix typos
+  * Update documentation of Rust feature
+  * Fix panic in DecodeStream::step due to incorrect index usage
+  * Fixing the stream by removing the read_index altogether
+  * Fixing NormalizedString append when normalized is empty
+  * Update metadata as Python3.7 and Python3.8 support was dropped
+  * Add rustls-tls feature
+- Remove define skip_python313 1
+
 -------------------------------------------------------------------
 Wed Mar  5 10:32:02 UTC 2025 - Christian Goll <cgoll@suse.com>
 

python-tokenizers.spec

@@ -16,15 +16,13 @@
 #
 
 
-%define skip_python313 1
-
 %if 0%{?suse_version} && 0%{?suse_version} < 1550
 %global force_gcc_version 13
 %endif
 
 %{?sle15_python_module_pythons}
 Name:           python-tokenizers
-Version:        0.21.0
+Version:        0.21.1
 Release:        0
 Summary:        Provides an implementation of today's most used tokenizers
 License:        Apache-2.0

registry.tar.zst

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c6e2a7b1f52a7523f92f36544bd7088ee7a96622f493a7c1a4f353c25078bb74
-size 46535229
+oid sha256:cedae83459a2008d8ab8b27bd2976fadae5df3d4664cddbda0333d510763cd8b
+size 55696377

tokenizers-0.21.0.tar.gz

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:841279ad797d575ed3cf31fc4f30e09e37acbd35028d30c51fc0879ef7ed4094
-size 1544853

tokenizers-0.21.1.tar.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:65dd1f079404161321ece0d1af4f678e5846e3e6056f004600bde776d489bddc
+size 1545849