SLFO-pool/python-fastparquet
SHA256
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
main
python-fastparquet/python-fastparquet.changes

515 lines
20 KiB
Plaintext
Raw Permalink Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

-------------------------------------------------------------------
Mon Jan 22 12:41:30 UTC 2024 - Daniel Garcia <daniel.garcia@suse.com>
- Do not run tests in s390x, bsc#1218603
-------------------------------------------------------------------
Tue Dec 5 12:23:32 UTC 2023 - Dirk Müller <dmueller@suse.com>
- update to 2023.10.0:
* Datetime units in empty() with tz (#893)
* Fewer inplace decompressions for V2 pages (#890
* Allow writing categorical column with no categories (#888)
* Fixes for new numpy (#886)
* RLE bools and DELTA for v1 pages (#885, 883)
-------------------------------------------------------------------
Mon Sep 11 21:29:16 UTC 2023 - Dirk Müller <dmueller@suse.com>
- update to 2023.8.0:
* More general timestamp units (#874)
* ReadTheDocs V2 (#871)
* Better roundtrip dtypes (#861, 859)
* No convert when computing bytes-per-item for str (#858)
-------------------------------------------------------------------
Sat Jul 1 20:05:36 UTC 2023 - Arun Persaud <arun@gmx.de>
- update to version 2023.7.0:
* Add test case for reading non-pandas parquet file (#870)
* Extra field when cloning ParquetFile (#866)
-------------------------------------------------------------------
Fri Apr 28 08:10:46 UTC 2023 - Dirk Müller <dmueller@suse.com>
- update to 2023.4.0:
* allow loading categoricals even if not so in the pandas metadata,
when a column is dict-encodedand we only have one row-group (#863)
 * apply dtype to the columns names series, even when selecting no
columns (#861, 859)
 * don't make strings while estimating bye column size (#858)
 * handle upstream depr (#857, 856)
-------------------------------------------------------------------
Thu Feb 9 15:55:08 UTC 2023 - Arun Persaud <arun@gmx.de>
- update to version 2023.2.0:
* revert one-level set of filters (#852)
* full size dict for decoding V2 pages (#850)
* infer_object_encoding fix (#847)
* row filtering with V2 pages (#845)
-------------------------------------------------------------------
Wed Feb 8 18:25:03 UTC 2023 - Arun Persaud <arun@gmx.de>
- specfile:
* remove fastparquet-pr835.patch, implemented upstream
- update to version 2023.1.0:
* big improvement to write speed
* paging support for bigger row-groups
* pandas 2.0 support
* delta for big-endian architecture
-------------------------------------------------------------------
Mon Jan 2 20:38:49 UTC 2023 - Ben Greiner <code@bnavigator.de>
- Update to 2022.12.0
* check all int32 values before passing to thrift writer
* fix type of num_rows to i64 for big single file
- Release 2022.11.0
* Switch to calver
* Speed up loading of nullable types
* Allow schema evolution by addition of columns
* Allow specifying dtypes of output
* update to scm versioning
* fixes to row filter, statistics and tests
* support pathlib.Paths
* JSON encoder options
- Drop fastparquet-pr813-updatefixes.patch
-------------------------------------------------------------------
Fri Dec 23 09:18:39 UTC 2022 - Guillaume GARDET <guillaume.gardet@opensuse.org>
- Add patch to fox the test test_delta_from_def_2 on
aarch64, armv7 and ppc64le:
* fastparquet-pr835.patch
-------------------------------------------------------------------
Fri Oct 28 15:47:41 UTC 2022 - Ben Greiner <code@bnavigator.de>
- Update to 0.8.3
* improved key/value handling and rejection of bad types
* fix regression in consolidate_cats (caught in dask tests)
- Release 0.8.2
* datetime indexes initialised to 0 to prevent overflow from
randommemory
* case from csv_to_parquet where stats exists but has not nulls
entry
* define len and bool for ParquetFile
* maintain int types of optional data tha came from pandas
* fix for delta encoding
- Add fastparquet-pr813-updatefixes.patch gh#dask/fastparquet#813
-------------------------------------------------------------------
Tue Apr 26 11:02:27 UTC 2022 - Ben Greiner <code@bnavigator.de>
- Update to 0.8.1
* fix critical buffer overflow crash for large number of columns
and long column names
* metadata handling
* thrift int32 for list
* avoid error storing NaNs in column stats
-------------------------------------------------------------------
Sat Jan 29 21:36:38 UTC 2022 - Ben Greiner <code@bnavigator.de>
- Update to 0.8.0
* our own cythonic thrift implementation (drop thrift dependency)
* more in-place dataset editing ad reordering
* python 3.10 support
* fixes for multi-index and pandas types
- Clean test skips
-------------------------------------------------------------------
Sun Jan 16 13:34:53 UTC 2022 - Ben Greiner <code@bnavigator.de>
- Clean specfile from unused python36 conditionals
- Require thrift 0.15.0 (+patch) for Python 3.10 compatibility
* gh#dask/fastparquet#514
-------------------------------------------------------------------
Sat Nov 27 20:34:53 UTC 2021 - Arun Persaud <arun@gmx.de>
- update to version 0.7.2:
* Ability to remove row-groups in-place for multifile datasets
* Accept pandas nullable Float type
* allow empty strings and fix min/max when there is no data
* make writing statistics optional
* row selection in to_pandas()
-------------------------------------------------------------------
Sun Aug 8 15:13:55 UTC 2021 - Ben Greiner <code@bnavigator.de>
- Update to version 0.7.1
* Back compile for older versions of numpy
* Make pandas nullable types opt-out. The old behaviour (casting
to float) is still available with ParquetFile(...,
pandas_nulls=False).
* Fix time field regression: IsAdjustedToUTC will be False when
there is no timezone
* Micro improvements to the speed of ParquetFile creation by
using simple simple string ops instead of regex and
regularising filenames once at the start. Effects datasets with
many files.
- Release 0.7.0
* This version institutes major, breaking changes, listed here,
and incremental fixes and additions.
* Reading a directory without a _metadata summary file now works
by providing only the directory, instead of a list of
constituent files. This change also makes direct of use of
fsspec filesystems, if given, to be able to load the footer
metadata areas of the files concurrently, if the storage
backend supports it, and not directly instantiating
intermediate ParquetFile instances
* row-level filtering of the data. Whereas previously, only full
row-groups could be excluded on the basis of their parquet
metadata statistics (if present), filtering can now be done
within row-groups too. The syntax is the same as before,
allowing for multiple column expressions to be combined with
AND|OR, depending on the list structure. This mechanism
requires two passes: one to load the columns needed to create
the boolean mask, and another to load the columns actually
needed in the output. This will not be faster, and may be
slower, but in some cases can save significant memory
footprint, if a small fraction of rows are considered good and
the columns for the filter expression are not in the output.
Not currently supported for reading with DataPageV2.
* DELTA integer encoding (read-only): experimentally working,
but we only have one test file to verify against, since it is
not trivial to persuade Spark to produce files encoded this
way. DELTA can be extremely compact a representation for
slowly varying and/or monotonically increasing integers.
* nanosecond resolution times: the new extended "logical" types
system supports nanoseconds alongside the previous millis and
micros. We now emit these for the default pandas time type,
and produce full parquet schema including both "converted" and
"logical" type information. Note that all output has
isAdjustedToUTC=True, i.e., these are timestamps rather than
local time. The time-zone is stored in the metadata, as
before, and will be successfully recreated only in fastparquet
and (py)arrow. Otherwise, the times will appear to be UTC. For
compatibility with Spark, you may still want to use
times="int96" when writing.
* DataPageV2 writing: now we support both reading and writing.
For writing, can be enabled with the environment variable
FASTPARQUET_DATAPAGE_V2, or module global fastparquet.writer.
DATAPAGE_VERSION and is off by default. It will become on by
default in the future. In many cases, V2 will result in better
read performance, because the data and page headers are
encoded separately, so data can be directly read into the
output without addition allocation/copies. This feature is
considered experimental, but we believe it working well for
most use cases (i.e., our test suite) and should be readable
by all modern parquet frameworks including arrow and spark.
* pandas nullable types: pandas supports "masked" extension
arrays for types that previously could not support NULL at
all: ints and bools. Fastparquet used to cast such columns to
float, so that we could represent NULLs as NaN; now we use the
new(er) masked types by default. This means faster reading of
such columns, as there is no conversion. If the metadata
guarantees that there are no nulls, we still use the
non-nullable variant unless the data was written with
fastparquet/pyarrow, and the metadata indicates that the
original datatype was nullable. We already handled writing of
nullable columns.
-------------------------------------------------------------------
Tue May 18 14:41:46 UTC 2021 - Ben Greiner <code@bnavigator.de>
- Update to version 0.6.3
* no release notes
* new requirement: cramjam instead of separate compression libs
and their bindings
* switch from numba to Cython
-------------------------------------------------------------------
Fri Feb 12 14:50:18 UTC 2021 - Dirk Müller <dmueller@suse.com>
- skip python 36 build
-------------------------------------------------------------------
Thu Feb 4 17:50:32 UTC 2021 - Jan Engelhardt <jengelh@inai.de>
- Use of "+=" in %check warrants bash as buildshell.
-------------------------------------------------------------------
Wed Feb 3 21:43:10 UTC 2021 - Ben Greiner <code@bnavigator.de>
- Skip the import without warning test gh#dask/fastparquet#558
- Apply the Cepl-Strangelove-Parameter to pytest
(--import-mode append)
-------------------------------------------------------------------
Sat Jan 2 21:04:30 UTC 2021 - Benjamin Greiner <code@bnavigator.de>
- update to version 0.5
* no changelog
- update test suite setup -- install the .test module
-------------------------------------------------------------------
Sat Jul 18 18:13:53 UTC 2020 - Arun Persaud <arun@gmx.de>
- specfile:
* update requirements: version numbers and added packaging
- update to version 0.4.1:
* nulls, fixes #504
* deps: Add missing dependency on packaging. (#502)
-------------------------------------------------------------------
Thu Jul 9 14:04:10 UTC 2020 - Marketa Calabkova <mcalabkova@suse.com>
- Update to 0.4.0
* Changed RangeIndex private methods to public ones
* Use the python executable used to run the code
* Add support for Python 3.8
* support for numba > 0.48
- drop upstreamed patch use-python-exec.patch
-------------------------------------------------------------------
Mon Apr 6 06:54:36 UTC 2020 - Tomáš Chvátal <tchvatal@suse.com>
- Add patch to use sys.executable and not call py2 binary directly:
* use-python-exec.patch
-------------------------------------------------------------------
Mon Apr 6 06:50:26 UTC 2020 - Tomáš Chvátal <tchvatal@suse.com>
- Update to 0.3.3:
* no upstream changelog
-------------------------------------------------------------------
Fri Oct 25 17:50:50 UTC 2019 - Todd R <toddrme2178@gmail.com>
- Drop broken python 2 support.
- Testing fixes
-------------------------------------------------------------------
Sat Aug 3 15:10:41 UTC 2019 - Arun Persaud <arun@gmx.de>
- update to version 0.3.2:
* Only calculate dataset stats once (#453)
* Fixes #436 (#452)
* Fix a crash if trying to read a file whose created_by value is not
set
* COMPAT: Fix for pandas DeprecationWarning (#446)
* Apply timezone to index (#439)
* Handle NaN partition values (#438)
* Pandas meta (#431)
* Only strip _metadata from end of file path (#430)
* Simple nesting fix (#428)
* Disallow bad tz on save, warn on load (#427)
-------------------------------------------------------------------
Tue Jul 30 14:23:21 UTC 2019 - Todd R <toddrme2178@gmail.com>
- Fix spurious test failure
-------------------------------------------------------------------
Mon May 20 15:12:11 CEST 2019 - Matej Cepl <mcepl@suse.com>
- Clean up SPEC file.
-------------------------------------------------------------------
Tue Apr 30 14:28:46 UTC 2019 - Todd R <toddrme2178@gmail.com>
- update to 0.3.1
* Add schema == (__eq__) and != (__ne__) methods and tests.
* Fix item iteration for decimals
* List missing columns in error message
* Fix tz being None case
- Update to 0.3.0
* Squash some warnings and import failures
* Improvements to in and not in operators
* Fixes because pandas released
-------------------------------------------------------------------
Sat Jan 26 17:05:09 UTC 2019 - Arun Persaud <arun@gmx.de>
- specfile:
* update copyright year
- update to version 0.2.1:
* Compat for pandas 0.24.0 refactor (#390)
* Change OverflowError message when failing on large pages (#387)
* Allow for changes in dictionary while reading a row-group column
(#367)
* Correct pypi project names for compression libraries (#385)
-------------------------------------------------------------------
Thu Nov 22 22:47:24 UTC 2018 - Arun Persaud <arun@gmx.de>
- update to version 0.2.0:
* Don't mutate column list input (#383) (#384)
* Add optional requirements to extras_require (#380)
* Fix "broken link to parquet-format page" (#377)
* Add .c file to repo
* Handle rows split across 2 pages in the case of a map (#369)
* Fixes 370 (#371)
* Handle multi-page maps (#368)
* Handle zero-column files. Closes #361. (#363)
-------------------------------------------------------------------
Sun Sep 30 16:22:56 UTC 2018 - Arun Persaud <arun@gmx.de>
- specfile:
* update url
* make %files section more specific
- update to version 0.1.6:
* Restrict what categories get passed through (#358)
* Deep digging for multi-indexes (#356)
* allow_empty is the default in >=zstandard-0.9 (#355)
* Remove setup_requires from setup.py (#345)
* Fixed error if a certain partition is empty, when writing a
partioned (#347)
* Allow UTF8 column names to be read (#342)
* readd test file
* Allow for NULL converted type (#340)
* Robust partition names (#336)
* Fix accidental multiindex
* Read multi indexes (#331)
* Allow reading from any file-like (#330)
* change `parquet-format` link to apache repo (#328)
* Remove extra space from api.py (#325)
* numba bool fun (#324)
- changes from version 0.1.5:
* Fix _dtypes to be more efficient, to work with files with lots of
columns (#318)
* Buildfix (#313)
* Use LZ4 block compression for compatibility with parquet-cpp
(#314) (#315)
* Fix typo in ParquetFile docstring (#312)
* Remove annoying print() when reading file with CategoricalDtype
index (#311)
* Allow lists of multi-file data-sets (#309)
* Acceleate dataframe.empty for small/medium sizes (#307)
* Include dictionary page in column size (#306)
* Fix for selecting columns which were used for partitioning (#304)
* Remove occurances of np.fromstring (#303)
* Add support for zstandard compression (#296)
* Int96time order (#298)
- changes from version 0.1.4:
* Add handling of keyword arguments for compressor (#294)
* Fix setup.py duplication (#295)
* Integrate pytest with setup.py (#293)
* Get setup.py pytest to work. (#287)
* Add LZ4 support (#292)
* Update for forthcoming thrift release (#281)
* If timezones are in pandas metadata, assign columns as required
(#285)
* Pandas import (#284)
* Copy FMDs instead of mutate (#279)
* small fixes (#278)
* fixes to get benchmark to work (#276)
* backwards compat with Dask
* Fix test_time_millis on Windows (#275)
* join paths os-independently (#271)
* Adds int32 support for object encoding (#268)
* Fix a couple small typos in documentation (#267)
* Partition order should be sorted (#265)
* COMPAT: Update thrift (#264)
* Speedups result (#253)
* Remove thrift_copy
* Define `__copy__` on thrift structures
* Update rtd deps
- changes from version 0.1.3:
* More care over append when partitioning multiple columns
* Sep for windows cats filtering
* Move pytest imports to tests/ remove requirememnt
* Special-case only zeros
* Cope with partition values like "07"
* fix for s3
* Fix for list of paths rooted in the current directory
* add test
* Explicit file opens
* update docstring
* Refactor partition interpretation
* py2 fix
* Error in test changed
* Better error messages when failed to cnovert on write
- changes from version 0.1.2:
* Revert accidental removal of s3 import
* Move thrift things together, and make thrift serializer for pickle
* COMPAT: for new pandas CategoricalDtype
* Fixup for backwards seeking.
* Fix some test failures
* Protptype version using thrift instead of thriftpy
* Not all mergers have cats
* Revert accidental deletion
* remove warnings
* Sort keys in json for metadata
* Check column chunks for categories sizes
* Account for partition dir names with numbers
* Fix map/list doc
* Catch more stats errors
* Prevent pandas auto-names being given to index
- changes from version 0.1.1:
* Add workaround for single-value-partition
* update test
* Simplify and fix for py2
* Use thrift encoding on statistics strings
* remove redundant SNAPPY from supported compressions list
* Fix statistics
* lists again
* Always convert int96 to times
* Update docs
* attribute typo
* Fix definition level
* Add test, clean columns
* Allow optional->optional lists and maps
* Flatten schema to enable loading of non-repeated columns
* Remove extra file
* Fix py2
* Fix "in" filter to cope with strings that could be numbers
* Allow pip install without NumPy or Cython
- changes from version 0.1.0:
* Add ParquetFile attribute documentation
* Fix tests
* Enable append to an empty dataset
* More warning words and check on partition_on
* Do not fail stats if there are no row-groups
* Fix "numpy_dtype"->"numpy_type
* "in" was checking range not exact membership of set
* If metadata gives index, put in columns
* Fix pytest warning
* Fail on ordering dict statistics
* Fix stats filter
* clean test
* Fix ImportWarning on Python 3.6+
* TEST: added updated test file for special strings used in filters
* fix links
* [README]: indicate dependency on LLVM 4.0.x.
* Filter stats had unfortunate converted_type check
* Ignore exceptions in val_to_num
* Also for TODAY
* Very special case for partition: NOW should be kept as string
* Allow partition_on; fix category nuls
* Remove old category key/values on writing
* Implement writing pandas metadata and auto-setting cats/index
* Pandas compatability
* Test and fix for filter on single file
* Do not attempt to recurse into schema elements with zero childrean
-------------------------------------------------------------------
Thu Jun 7 20:41:31 UTC 2018 - jengelh@inai.de
- Fixup grammar./Replace future aims with what it does now.
-------------------------------------------------------------------
Thu May 3 14:07:08 UTC 2018 - toddrme2178@gmail.com
- Use %license tag
-------------------------------------------------------------------
Thu May 25 12:19:26 UTC 2017 - toddrme2178@gmail.com
- Initial version
Reference in New Issue View Git Blame Copy Permalink