pool/python-tokenizers
SHA256
17
0
Fork 0
Code Issues Pull Requests Activity
16 Commits 3 Branches 0 Tags
factory
Commit Graph

16 Commits

This Branch
This Branch
All Branches
Author SHA256 Message Date
Ana Guerrero
906241bdac Accepting request 1304759 from science:machinelearning 
OBS-URL: https://build.opensuse.org/request/show/1304759
OBS-URL: https://build.opensuse.org/package/show/openSUSE:Factory/python-tokenizers?expand=0&rev=6
 2025-09-15 17:52:35 +00:00
Guillaume GARDET
ff91514435 - Update to 0.21.4 
* No change, the 0.21.3 release failed, this is just a re-release
- from version 0.21.3
  * Clippy fixes
  * Fixed an introduced backward breaking change in our Rust APIs.
- from version 0.21.2
  * Update the release builds following 0.21.1
  * Replace lazy_static with stabilized std::sync::LazyLock in 1.80
  * Fix no-onig no-wasm builds
  * Fix typos in strings and comments
  * Fix type notation of merges in BPE Python binding
  * Bump http-proxy-middleware from 2.0.6 to 2.0.9 in
    /tokenizers/examples/unstable_wasm/www
  * Fix data path in test_continuing_prefix_trainer_mismatch
  * clippy by @ArthurZucker
  * Update pyo3 and rust-numpy depends for no-gil/free-threading compat
  * Use ApiBuilder::from_env() in from_pretrained function
  * Upgrade onig, to get it compiling with GCC 15
  * Itertools upgrade
  * Bump webpack-dev-server from 4.10.0 to 5.2.1
    in /tokenizers/examples/unstable_wasm/www
  * Bump brace-expansion from 1.1.11 to 1.1.12 in /bindings/node
  * Fix features blending into a paragraph
  * Adding throughput to benches to have a more consistent measure across
  * Upgrading dependencies
  * [docs] Whitespace
  * Hotfixing the stub
  * Bpe clones
  * Fixed Length Pre-Tokenizer
  * Consolidated optimization ahash dary compact str

OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=15
 2025-09-15 09:32:28 +00:00
Ana Guerrero
205d7f2885 Accepting request 1255860 from science:machinelearning 
OBS-URL: https://build.opensuse.org/request/show/1255860
OBS-URL: https://build.opensuse.org/package/show/openSUSE:Factory/python-tokenizers?expand=0&rev=5
 2025-03-25 21:11:43 +00:00
Guillaume GARDET
4e353ff28b - Update to 0.21.1: 
* Update dev version and pyproject.toml
  * Add feature flag hint to README.md
  * Upgrade to PyO3 0.23
  * Fixing the README.md
  * Fix typo in Split docstrings
  * Fix typos
  * Update documentation of Rust feature
  * Fix panic in DecodeStream::step due to incorrect index usage
  * Fixing the stream by removing the read_index altogether
  * Fixing NormalizedString append when normalized is empty
  * Update metadata as Python3.7 and Python3.8 support was dropped
  * Add rustls-tls feature
- Remove define skip_python313 1

OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=13
 2025-03-25 08:40:26 +00:00
Christian Goll
f1cddf0be1 Accepting request 1250365 from home:mslacken:ml 
- disable python3.13

OBS-URL: https://build.opensuse.org/request/show/1250365
OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=12
 2025-03-05 10:32:48 +00:00
Ana Guerrero
e193fc6166 Accepting request 1236795 from science:machinelearning 
OBS-URL: https://build.opensuse.org/request/show/1236795
OBS-URL: https://build.opensuse.org/package/show/openSUSE:Factory/python-tokenizers?expand=0&rev=4
 2025-01-12 10:20:30 +00:00
Guillaume GARDET
d2058b3353 - Enable build on riscv64 
OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=10
 2025-01-10 07:13:13 +00:00
Ana Guerrero
8a92510324 Accepting request 1231946 from science:machinelearning 
OBS-URL: https://build.opensuse.org/request/show/1231946
OBS-URL: https://build.opensuse.org/package/show/openSUSE:Factory/python-tokenizers?expand=0&rev=3
 2024-12-19 20:40:29 +00:00
Guang Yee
995fdb6bb1 - Update to version 0.21.0: 
* More cache options.
  * Disable caching for long strings.
  * Testing ABI3 wheels to reduce number of wheels
  * Adding an API for decode streaming.
  * Decode stream python
  * Fix encode_batch and encode_batch_fast to accept ndarrays again

OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=8
 2024-12-18 17:58:42 +00:00
Ana Guerrero
06028d76d6 Accepting request 1224030 from science:machinelearning 
OBS-URL: https://build.opensuse.org/request/show/1224030
OBS-URL: https://build.opensuse.org/package/show/openSUSE:Factory/python-tokenizers?expand=0&rev=2
 2024-11-14 15:09:19 +00:00
Guang Yee
666a38f444 - Select only rust tier 1 arches. 
- Update registry.tar.zst dependencies
- Update version to 0.20.3:
  * fix pylist
  * [MINOR:TYP] Fix docstrings
- Updates from 0.20.2:
  * Bump cookie and express in /tokenizers/examples/unstable_wasm/www
  * Fix off-by-one error in tokenizer::normalizer::Range::len
  * Arg name correction: auth_token -> token
  * Unsound call of set_var
  * Add safety comments
  * PyO3 0.22
- Updates from 0.20.1:
  * Update README.md
  * fix benchmark file link
  * [ignore_merges] Fix offsets
  * Bump body-parser and express in /tokenizers/examples/unstable_wasm/www
  * Bump serve-static and express in /tokenizers/examples/unstable_wasm/www
  * Bump send and express in /tokenizers/examples/unstable_wasm/www
  * Bump webpack from 5.76.0 to 5.95.0 in /tokenizers/examples/unstable_wasm/www
  * Fix documentation build
  * style: simplify string formatting for readability
- Experiment with cargo vendor home registry. See documentation:
  https://github.com/openSUSE-Rust/obs-service-cargo/blob/master/README.md#cargo-vendor-home-registry

OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=6
 2024-11-13 18:49:50 +00:00
Ana Guerrero
c18aa22fbb Accepting request 1204889 from science:machinelearning 
Fix Requires

OBS-URL: https://build.opensuse.org/request/show/1204889
OBS-URL: https://build.opensuse.org/package/show/openSUSE:Factory/python-tokenizers?expand=0&rev=1
 2024-10-01 15:12:03 +00:00
Jeremy Moffitt
cd6f937b15 - Don't use macros for Requires 
OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=4
 2024-09-24 21:17:04 +00:00
Guillaume GARDET
7e2b0414bb Initial package for openweb-ui ai-maintainers can be set as the maintainer / bugowner. 
OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=3
 2024-09-10 06:26:56 +00:00
Christian Goll
b4c538250d OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=2 2024-08-20 12:45:40 +00:00
Christian Goll
a9df2175f5 initial commit ofrust base python-tokenizers 
OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=1
 2024-07-23 09:21:39 +00:00