Christian Goll
a9df2175f5
OBS-URL: https://build.opensuse.org/package/show/science:machinelearning/python-tokenizers?expand=0&rev=1
76 lines
2.4 KiB
RPMSpec
76 lines
2.4 KiB
RPMSpec
#
|
|
# spec file for package python-tokenizers
|
|
#
|
|
# Copyright (c) 2024 SUSE LLC
|
|
#
|
|
# All modifications and additions to the file contributed by third parties
|
|
# remain the property of their copyright owners, unless otherwise agreed
|
|
# upon. The license for this file, and modifications and additions to the
|
|
# file, is the same license as for the pristine package itself (unless the
|
|
# license for the pristine package is not an Open Source License, in which
|
|
# case the license is the MIT License). An "Open Source License" is a
|
|
# license that conforms to the Open Source Definition (Version 1.9)
|
|
# published by the Open Source Initiative.
|
|
|
|
# Please submit bugfixes or comments via https://bugs.opensuse.org/
|
|
#
|
|
|
|
|
|
%{?!python_module:%define python_module() python-%{**} python3-%{**}}
|
|
|
|
Name: python-tokenizers
|
|
Version: 0.19.1
|
|
Release: 0
|
|
Summary: Provides an implementation of today's most used tokenizers
|
|
License: Apache-2.0
|
|
URL: https://github.com/huggingface/tokenizers
|
|
Source0: https://github.com/huggingface/tokenizers/archive/refs/tags/v%{version}.tar.gz#/tokenizers-%{version}.tar.gz
|
|
Source1: vendor.tar.gz
|
|
BuildRequires: %{python_module devel}
|
|
BuildRequires: %{python_module maturin}
|
|
BuildRequires: %{python_module pip}
|
|
BuildRequires: %{python_module setuptools}
|
|
BuildRequires: cargo-packaging
|
|
BuildRequires: gcc-c++
|
|
BuildRequires: fdupes
|
|
BuildRequires: python-rpm-macros
|
|
BuildRequires: python-rpm-macros
|
|
%python_subpackages
|
|
|
|
%description
|
|
Provides an implementation of today's most used tokenizers, with a focus on
|
|
performance and versatility.
|
|
* Train new vocabularies and tokenize, using today's most used tokenizers.
|
|
* Extremely fast (both training and tokenization), thanks to the Rust
|
|
implementation. Takes less than 20 seconds to tokenize a GB of text on a
|
|
server's CPU.
|
|
* Easy to use, but also extremely versatile.
|
|
* Designed for research and production.
|
|
* Normalization comes with alignments tracking. It's always possible to get the
|
|
part of the original sentence that corresponds to a given token.
|
|
* Does all the pre-processing: Truncate, Pad, add the special tokens your model
|
|
needs.
|
|
|
|
%prep
|
|
%autosetup -p1 -n tokenizers-%{version}
|
|
cd bindings/python
|
|
tar xzf %{S:1}
|
|
|
|
%build
|
|
cd bindings/python
|
|
%pyproject_wheel
|
|
|
|
%install
|
|
cd bindings/python
|
|
%pyproject_install
|
|
%python_expand %fdupes %{buildroot}/%{$python_sitearch}/*
|
|
|
|
%check
|
|
|
|
%files %{python_files}
|
|
%license LICENSE
|
|
%doc README.md
|
|
%{python_sitearch}/tokenizers*
|
|
|
|
%changelog
|