pool/python-tokenizers
SHA256
17
0
Fork 0
Code Issues Pull Requests Activity

Commit Graph

  • 6c00337f71 Accepting request 1335550 from science:machinelearning factory Dominique Leuenberger 2026-03-01 21:14:56 +00:00
  • 6dcab67e3e - Update to version 0.22.0: * Bump on-headers and compression in /tokenizers/examples/unstable_wasm/www * Implement from * fix: use AHashMap to fix compile error * New stream * [docs] Add more decoders * Fix missing parenthesis in EncodingVisualizer.calculate_label_colors * Update quicktour.mdx re: Issue #1625 * remove stray comment * Fix typo in README * RUSTSEC-2024-0436 - replace paste with pastey * Tokenizer: Add native async bindings, via py03-async-runtimes. Adrian Schröter 2026-02-28 10:01:40 +00:00
  • 822772de55 Accepting request 1304759 from science:machinelearning Ana Guerrero 2025-09-15 17:52:35 +00:00
  • 2fe9348664 Accepting request 1296354 from home:glaubitz:branches:science:machinelearning Guillaume GARDET 2025-09-15 09:32:28 +00:00
  • 205d7f2885 Accepting request 1255860 from science:machinelearning leap-16.1 leap-16.0 Ana Guerrero 2025-03-25 21:11:43 +00:00
  • 921aeeea01 Accepting request 1255860 from science:machinelearning Ana Guerrero 2025-03-25 21:11:43 +00:00
  • 4e353ff28b - Update to 0.21.1: * Update dev version and pyproject.toml * Add feature flag hint to README.md * Upgrade to PyO3 0.23 * Fixing the README.md * Fix typo in Split docstrings * Fix typos * Update documentation of Rust feature * Fix panic in DecodeStream::step due to incorrect index usage * Fixing the stream by removing the read_index altogether * Fixing NormalizedString append when normalized is empty * Update metadata as Python3.7 and Python3.8 support was dropped * Add rustls-tls feature - Remove define skip_python313 1 Guillaume GARDET 2025-03-25 08:40:26 +00:00
  • a4b73b7ea9 Accepting request 1255038 from home:lmulling:branches:science:machinelearning Guillaume GARDET 2025-03-25 08:40:26 +00:00
  • f1cddf0be1 Accepting request 1250365 from home:mslacken:ml Christian Goll 2025-03-05 10:32:48 +00:00
  • c377c982ee Accepting request 1250365 from home:mslacken:ml Christian Goll 2025-03-05 10:32:48 +00:00
  • e193fc6166 Accepting request 1236795 from science:machinelearning Ana Guerrero 2025-01-12 10:20:30 +00:00
  • 13bd454ec3 Accepting request 1236795 from science:machinelearning Ana Guerrero 2025-01-12 10:20:30 +00:00
  • d2058b3353 - Enable build on riscv64 Guillaume GARDET 2025-01-10 07:13:13 +00:00
  • c44150180c Accepting request 1236250 from openSUSE:Factory:RISCV Guillaume GARDET 2025-01-10 07:13:13 +00:00
  • 8a92510324 Accepting request 1231946 from science:machinelearning Ana Guerrero 2024-12-19 20:40:29 +00:00
  • 96bbead5a4 Accepting request 1231946 from science:machinelearning Ana Guerrero 2024-12-19 20:40:29 +00:00
  • 995fdb6bb1 - Update to version 0.21.0: * More cache options. * Disable caching for long strings. * Testing ABI3 wheels to reduce number of wheels * Adding an API for decode streaming. * Decode stream python * Fix encode_batch and encode_batch_fast to accept ndarrays again Guang Yee 2024-12-18 17:58:42 +00:00
  • 82d122dfa5 Accepting request 1231910 from home:uncomfyhalomacro:branches:science:machinelearning Guang Yee 2024-12-18 17:58:42 +00:00
  • 06028d76d6 Accepting request 1224030 from science:machinelearning Ana Guerrero 2024-11-14 15:09:19 +00:00
  • 6bd30b1715 Accepting request 1224030 from science:machinelearning Ana Guerrero 2024-11-14 15:09:19 +00:00
  • 666a38f444 - Select only rust tier 1 arches. - Update registry.tar.zst dependencies - Update version to 0.20.3: * fix pylist * [MINOR:TYP] Fix docstrings - Updates from 0.20.2: * Bump cookie and express in /tokenizers/examples/unstable_wasm/www * Fix off-by-one error in tokenizer::normalizer::Range::len * Arg name correction: auth_token -> token * Unsound call of set_var * Add safety comments * PyO3 0.22 - Updates from 0.20.1: * Update README.md * fix benchmark file link * [ignore_merges] Fix offsets * Bump body-parser and express in /tokenizers/examples/unstable_wasm/www * Bump serve-static and express in /tokenizers/examples/unstable_wasm/www * Bump send and express in /tokenizers/examples/unstable_wasm/www * Bump webpack from 5.76.0 to 5.95.0 in /tokenizers/examples/unstable_wasm/www * Fix documentation build * style: simplify string formatting for readability - Experiment with cargo vendor home registry. See documentation: https://github.com/openSUSE-Rust/obs-service-cargo/blob/master/README.md#cargo-vendor-home-registry Guang Yee 2024-11-13 18:49:50 +00:00
  • 5e74175ce9 Accepting request 1222140 from home:uncomfyhalomacro:branches:science:machinelearning Guang Yee 2024-11-13 18:49:50 +00:00
  • c18aa22fbb Accepting request 1204889 from science:machinelearning Ana Guerrero 2024-10-01 15:12:03 +00:00
  • d722bec687 Accepting request 1204889 from science:machinelearning Ana Guerrero 2024-10-01 15:12:03 +00:00
  • cd6f937b15 - Don't use macros for Requires Jeremy Moffitt 2024-09-24 21:17:04 +00:00
  • acc2acaead Accepting request 1202913 from home:simotek:branches:science:machinelearning Jeremy Moffitt 2024-09-24 21:17:04 +00:00
  • 7e2b0414bb Initial package for openweb-ui ai-maintainers can be set as the maintainer / bugowner. Guillaume GARDET 2024-09-10 06:26:56 +00:00
  • 89cf12e655 Accepting request 1199768 from home:yeey:OpenWebUI Guillaume GARDET 2024-09-10 06:26:56 +00:00
  • b4c538250d OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=2 Christian Goll 2024-08-20 12:45:40 +00:00
  • 40dd8143cd Accepting request 1194910 from home:uncomfyhalomacro:branches:science:machinelearning Christian Goll 2024-08-20 12:45:40 +00:00
  • a9df2175f5 initial commit ofrust base python-tokenizers Christian Goll 2024-07-23 09:21:39 +00:00
  • d7b204d0fe Accepting request 1185138 from home:mslacken:ml Christian Goll 2024-07-23 09:21:39 +00:00