pool/python-tokenizers
SHA256
17
0
Fork 0
Code Issues Pull Requests Activity
18 Commits 3 Branches 0 Tags
factory
Commit Graph

18 Commits

This Branch
This Branch
All Branches
Author SHA256 Message Date
Dominique Leuenberger
6c00337f71 Accepting request 1335550 from science:machinelearning 
Automatic submission by obs-autosubmit

OBS-URL: https://build.opensuse.org/request/show/1335550
OBS-URL: https://build.opensuse.org/package/show/openSUSE:Factory/python-tokenizers?expand=0&rev=7
 2026-03-01 21:14:56 +00:00
Adrian Schröter
6dcab67e3e - Update to version 0.22.0: 
* Bump on-headers and compression in /tokenizers/examples/unstable_wasm/www
  * Implement from
  * fix: use AHashMap to fix compile error
  * New stream
  * [docs] Add more decoders
  * Fix missing parenthesis in EncodingVisualizer.calculate_label_colors
  * Update quicktour.mdx re: Issue #1625
  * remove stray comment
  * Fix typo in README
  * RUSTSEC-2024-0436 - replace paste with pastey
  * Tokenizer: Add native async bindings, via py03-async-runtimes.

OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=17
 2026-02-28 10:01:40 +00:00
Ana Guerrero
822772de55 Accepting request 1304759 from science:machinelearning 
OBS-URL: https://build.opensuse.org/request/show/1304759
OBS-URL: https://build.opensuse.org/package/show/openSUSE:Factory/python-tokenizers?expand=0&rev=6
 2025-09-15 17:52:35 +00:00
Guillaume GARDET
2fe9348664 Accepting request 1296354 from home:glaubitz:branches:science:machinelearning 
- Update to 0.21.4
  * No change, the 0.21.3 release failed, this is just a re-release
- from version 0.21.3
  * Clippy fixes
  * Fixed an introduced backward breaking change in our Rust APIs.
- from version 0.21.2
  * Update the release builds following 0.21.1
  * Replace lazy_static with stabilized std::sync::LazyLock in 1.80
  * Fix no-onig no-wasm builds
  * Fix typos in strings and comments
  * Fix type notation of merges in BPE Python binding
  * Bump http-proxy-middleware from 2.0.6 to 2.0.9 in
    /tokenizers/examples/unstable_wasm/www
  * Fix data path in test_continuing_prefix_trainer_mismatch
  * clippy by @ArthurZucker
  * Update pyo3 and rust-numpy depends for no-gil/free-threading compat
  * Use ApiBuilder::from_env() in from_pretrained function
  * Upgrade onig, to get it compiling with GCC 15
  * Itertools upgrade
  * Bump webpack-dev-server from 4.10.0 to 5.2.1
    in /tokenizers/examples/unstable_wasm/www
  * Bump brace-expansion from 1.1.11 to 1.1.12 in /bindings/node
  * Fix features blending into a paragraph
  * Adding throughput to benches to have a more consistent measure across
  * Upgrading dependencies
  * [docs] Whitespace
  * Hotfixing the stub
  * Bpe clones
  * Fixed Length Pre-Tokenizer
  * Consolidated optimization ahash dary compact str

OBS-URL: https://build.opensuse.org/request/show/1296354
OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=15
 2025-09-15 09:32:28 +00:00
Ana Guerrero
921aeeea01 Accepting request 1255860 from science:machinelearning 
OBS-URL: https://build.opensuse.org/request/show/1255860
OBS-URL: https://build.opensuse.org/package/show/openSUSE:Factory/python-tokenizers?expand=0&rev=5
 2025-03-25 21:11:43 +00:00
Guillaume GARDET
a4b73b7ea9 Accepting request 1255038 from home:lmulling:branches:science:machinelearning 
- Update to 0.21.1:
  * Update dev version and pyproject.toml
  * Add feature flag hint to README.md
  * Upgrade to PyO3 0.23
  * Fixing the README.md
  * Fix typo in Split docstrings
  * Fix typos
  * Update documentation of Rust feature
  * Fix panic in DecodeStream::step due to incorrect index usage
  * Fixing the stream by removing the read_index altogether
  * Fixing NormalizedString append when normalized is empty
  * Update metadata as Python3.7 and Python3.8 support was dropped
  * Add rustls-tls feature
- Remove define skip_python313 1

OBS-URL: https://build.opensuse.org/request/show/1255038
OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=13
 2025-03-25 08:40:26 +00:00
Christian Goll
c377c982ee Accepting request 1250365 from home:mslacken:ml 
- disable python3.13

OBS-URL: https://build.opensuse.org/request/show/1250365
OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=12
 2025-03-05 10:32:48 +00:00
Ana Guerrero
13bd454ec3 Accepting request 1236795 from science:machinelearning 
OBS-URL: https://build.opensuse.org/request/show/1236795
OBS-URL: https://build.opensuse.org/package/show/openSUSE:Factory/python-tokenizers?expand=0&rev=4
 2025-01-12 10:20:30 +00:00
Guillaume GARDET
c44150180c Accepting request 1236250 from openSUSE:Factory:RISCV 
- Enable build on riscv64

OBS-URL: https://build.opensuse.org/request/show/1236250
OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=10
 2025-01-10 07:13:13 +00:00
Ana Guerrero
96bbead5a4 Accepting request 1231946 from science:machinelearning 
OBS-URL: https://build.opensuse.org/request/show/1231946
OBS-URL: https://build.opensuse.org/package/show/openSUSE:Factory/python-tokenizers?expand=0&rev=3
 2024-12-19 20:40:29 +00:00
Guang Yee
82d122dfa5 Accepting request 1231910 from home:uncomfyhalomacro:branches:science:machinelearning 
- Update to version 0.21.0:
  * More cache options.
  * Disable caching for long strings.
  * Testing ABI3 wheels to reduce number of wheels
  * Adding an API for decode streaming.
  * Decode stream python
  * Fix encode_batch and encode_batch_fast to accept ndarrays again

OBS-URL: https://build.opensuse.org/request/show/1231910
OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=8
 2024-12-18 17:58:42 +00:00
Ana Guerrero
6bd30b1715 Accepting request 1224030 from science:machinelearning 
OBS-URL: https://build.opensuse.org/request/show/1224030
OBS-URL: https://build.opensuse.org/package/show/openSUSE:Factory/python-tokenizers?expand=0&rev=2
 2024-11-14 15:09:19 +00:00
Guang Yee
5e74175ce9 Accepting request 1222140 from home:uncomfyhalomacro:branches:science:machinelearning 
- Select only rust tier 1 arches. 
- Update registry.tar.zst dependencies
- Update version to 0.20.3:
  * fix pylist
  * [MINOR:TYP] Fix docstrings
- Updates from 0.20.2:
  * Bump cookie and express in /tokenizers/examples/unstable_wasm/www
  * Fix off-by-one error in tokenizer::normalizer::Range::len
  * Arg name correction: auth_token -> token
  * Unsound call of set_var
  * Add safety comments
  * PyO3 0.22
- Updates from 0.20.1:
  * Update README.md
  * fix benchmark file link
  * [ignore_merges] Fix offsets
  * Bump body-parser and express in /tokenizers/examples/unstable_wasm/www
  * Bump serve-static and express in /tokenizers/examples/unstable_wasm/www
  * Bump send and express in /tokenizers/examples/unstable_wasm/www
  * Bump webpack from 5.76.0 to 5.95.0 in /tokenizers/examples/unstable_wasm/www
  * Fix documentation build
  * style: simplify string formatting for readability
- Experiment with cargo vendor home registry. See documentation:
  https://github.com/openSUSE-Rust/obs-service-cargo/blob/master/README.md#cargo-vendor-home-registry

OBS-URL: https://build.opensuse.org/request/show/1222140
OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=6
 2024-11-13 18:49:50 +00:00
Ana Guerrero
d722bec687 Accepting request 1204889 from science:machinelearning 
Fix Requires

OBS-URL: https://build.opensuse.org/request/show/1204889
OBS-URL: https://build.opensuse.org/package/show/openSUSE:Factory/python-tokenizers?expand=0&rev=1
 2024-10-01 15:12:03 +00:00
Jeremy Moffitt
acc2acaead Accepting request 1202913 from home:simotek:branches:science:machinelearning 
- Don't use macros for Requires

OBS-URL: https://build.opensuse.org/request/show/1202913
OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=4
 2024-09-24 21:17:04 +00:00
Guillaume GARDET
89cf12e655 Accepting request 1199768 from home:yeey:OpenWebUI 
Initial package for openweb-ui ai-maintainers can be set as the maintainer / bugowner.

OBS-URL: https://build.opensuse.org/request/show/1199768
OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=3
 2024-09-10 06:26:56 +00:00
Christian Goll
40dd8143cd Accepting request 1194910 from home:uncomfyhalomacro:branches:science:machinelearning 
OBS-URL: https://build.opensuse.org/request/show/1194910
OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=2
 2024-08-20 12:45:40 +00:00
Christian Goll
d7b204d0fe Accepting request 1185138 from home:mslacken:ml 
initial commit ofrust base python-tokenizers

OBS-URL: https://build.opensuse.org/request/show/1185138
OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=1
 2024-07-23 09:21:39 +00:00