rpm/duperemove
SHA256
1
0
Fork 0
forked from pool/duperemove
Code Pull Requests Activity
71 Commits 1 Branch 0 Tags
factory
Commit Graph

39 Commits

Author SHA256 Message Date
Dirk Mueller
2c4d3a704c Accepting request 1128968 from home:jubalh:branches:filesystems 
- Update to 0.14.1:
  * file_scan: satisfy clang CFI
  * file_scan: walk_dir: improve rootfs support
  * dbfile: get_duplicate_files: exclude inlined files before
    looking for duplicates
  * filerec: fiemap_scan_extent: fix typo: we must check the
    return from get_extent

OBS-URL: https://build.opensuse.org/request/show/1128968
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=68
 2023-11-27 06:36:16 +00:00
Dirk Mueller
526ce195e2 Accepting request 1127541 from home:jubalh:branches:filesystems 
- Update to 0.14:
  * Batching has been reimplemented on top of the dedupe_seq.
  * The "scan" phase has been reimplemented (see 8264336 for details).
  * Filesystem locking has been implemented. See f3947e9 for details.

OBS-URL: https://build.opensuse.org/request/show/1127541
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=66
 2023-11-19 18:07:17 +00:00
David Sterba
550f46da46 Accepting request 1114307 from home:jubalh:branches:filesystems 
- Update to 0.13:
  * Add a new dedupe option: [no]rescan_files. It will increase
    performance in some use cases.
  * New behaviors from v0.12 has been consolidated. Extent-based
    lookup is always enabled, as is fiemap. The v2
    hashfile is no longer supported.
  * Hashfile are now updated after deduplication, to reflect the
    new physical offsets. This avoid (re)deduplicating extents
    in some cases.
  * Partial mode has been enhanced to support batching. The
    overall performance of this mode (which was previously known
    as "block-based mode") has been improved.
  * All files are now open in readonly mode.
  * Hashfile version has been increased to reflect the new
    database behaviors. Previous hashfiles are not compatible.
  * Always compute a hash for the entire file. This let us
    deduplicate same files easily, regardless of their
    extents mappings.
  * Deduplicating only parts of a file can be disabled using
    the [no]only_whole_files dedupe option.
  * Hashfiles with unsupported features or hash algorithm are
    now recreated transparently. Migration of the old content
    is not implemented.
  * Relative exclude patterns are no longer silently ingested.
    Such patterns are now rebuilt on top of the current
    working directory.
  * Batching is now set to 1024 by default.

OBS-URL: https://build.opensuse.org/request/show/1114307
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=64
 2023-09-29 12:59:48 +00:00
David Sterba
d91e3b0cd3 Accepting request 1111318 from home:pgajdos 
- build for 15

OBS-URL: https://build.opensuse.org/request/show/1111318
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=62
 2023-09-18 15:20:47 +00:00
David Sterba
da9b80fbb7 Accepting request 1099030 from home:jubalh:branches:filesystems 
- Update to 0.12:
  * Duplication lookup is now based on extents. This leads to a
    massive increase of the performances. Block-based lookup is
    still possible via --dedupe-options=partial.
  * Following that change, a new hashfile format has been introduced.
    Previous hashfile format is still supported when extents lookup
    are disabled, this is not recommended.
  * Batching has been implemented. When enabled with the -B <batchsize>
    option, duperemove will run the deduplication phase every <batchsize>
    scanned files. This is meant to help running duperemove on large
    dataset, with small blocksize, or on memory-constrained systems.
  * All hash algorithm has been removed and replaced by xxh128. This
    variant is as robust as murmur3 while being faster. Choosing a
    hash function via the --hash option has been removed. Hashfiles
    built with other algorithm must be removed.

OBS-URL: https://build.opensuse.org/request/show/1099030
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=60
 2023-07-17 16:36:57 +00:00
Dirk Mueller
bd5f8f9365 Accepting request 951355 from home:jubalh:branches:filesystems 
- Update to 0.11.3:
  * Increase open file limit. (#269)
  * Create hash database file with 600 permission for improved
    security. (#262)
  * Read more data per pread, for v2 hashfile format this reduces
    the overall number of syscalls made which in turns results
    in better performance.
  * Fix truncated file handling, eliminating a an infinite
    loop case. (#255)

OBS-URL: https://build.opensuse.org/request/show/951355
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=59
 2022-02-03 21:01:17 +00:00
David Sterba
592fd6403a Accepting request 846138 from home:nikbor:branches:filesystems 
- Update to v0.11.2
  - Improved detection of hyperthreading
  - Multiple minor fixes and cleanups
  - Updated xxhash algorithm to latest release
  - Fixed extent collision handling
  - Fix overflow scanning in extents

OBS-URL: https://build.opensuse.org/request/show/846138
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=57
 2020-11-05 11:08:59 +00:00
David Sterba
da158864ea Accepting request 814537 from home:pluskalm:branches:filesystems 
- Fix building with gcc10

OBS-URL: https://build.opensuse.org/request/show/814537
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=55
 2020-07-07 21:00:36 +00:00
Mark Fasheh
9e08d2ffb2 Accepting request 654107 from home:mfasheh:branches:filesystems 
- Use source tarball from github

OBS-URL: https://build.opensuse.org/request/show/654107
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=53
 2018-12-04 22:04:12 +00:00
Mark Fasheh
35bcabda67 Accepting request 653159 from home:mfasheh:branches:filesystems 
- Update to v0.11.1
  - Add 'quiet' mode - duperemove will only print errors and a
    short summary of any dedupe.
  - Revert block dedupe default, it is causing excessive
    fragmentation on users systems.

OBS-URL: https://build.opensuse.org/request/show/653159
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=52
 2018-12-03 21:07:58 +00:00
Dirk Mueller
12617fe472 Accepting request 607705 from home:mfasheh:branches:filesystems 
- Update to v0.11
  - Fix a minor fd leak in hyperthreading detection code
  - Clean up some typos in documentation
  - Some build fixes
  - Removed patch: v0.11.beta4_to_git_f95e429.patch

OBS-URL: https://build.opensuse.org/request/show/607705
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=51
 2018-05-16 06:53:41 +00:00
Mark Fasheh
da623b2bae Accepting request 432087 from home:mfasheh:branches:filesystems 
- Update to v0.11.beta4 git commit f95e429
  * Use proper len of tail blocks during block dedupe
  * run_dedupe: avoid size_list corruption in push_blocks()
  - Added patch: v0.11.beta4_to_git_f95e429.patch

OBS-URL: https://build.opensuse.org/request/show/432087
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=49
 2016-10-01 04:52:10 +00:00
Dirk Mueller
027db8aa5b Accepting request 430642 from home:mfasheh:branches:filesystems 
- Update to duperemove v0.11.beta4
  - Many man page and FAQ updates
  - Much lower memory usage (changed to use block-dedupe by default)
  - removed unused sha256 module

OBS-URL: https://build.opensuse.org/request/show/430642
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=47
 2016-09-28 16:31:10 +00:00
Mark Fasheh
fb71cfcf60 Accepting request 428313 from home:mfasheh:branches:filesystems 
- Update to duperemove v0.11.beta3
  - Bugfixes
  - Large (2.5x) performance improvement to find dupes stage
  - Added '--cpu-threads' for user control of cpu bound tasks

OBS-URL: https://build.opensuse.org/request/show/428313
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=45
 2016-09-16 19:58:20 +00:00
Mark Fasheh
75aeed8ea2 Accepting request 426198 from home:mfasheh:branches:filesystems 
- Update to duperemove v0.11.beta2
  - Many bugfixes
  - Improved dedupe coverage with better extent selection algorithm
  - Improved (easier to read) output for csum stage
  - Progress counter added for dedupe stage
  - Removed patch v0.11.beta1_to_git_d4b9818.patch
  - Removed patch v0.11-d4b9818_to_82234f5.patch

OBS-URL: https://build.opensuse.org/request/show/426198
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=43
 2016-09-09 23:23:59 +00:00
Mark Fasheh
148f1a8c2f Accepting request 423277 from home:mfasheh:branches:filesystems 
- Update to v0.11.beta1 git commit 82234f5
  - Set file->dedupe_seq if we drop out in add_file_db()
  - add_file_db(): collapse all cases of ino/subvol/filename being
    different
  - Simplify the logic in add_file_db()
  - add_file_db(): check for same filename in filerec table
  - Store filerecs in a tree by filename

OBS-URL: https://build.opensuse.org/request/show/423277
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=42
 2016-08-26 22:19:57 +00:00
Mark Fasheh
c5fa21bf39 Accepting request 421850 from home:mfasheh:branches:filesystems 
- Update to git commit d4b9818:
  - Set scan flag on renamed file
  - Remove db record if a file was renamed
  - More vprints in add_file_db()
  - Fix confused error handling in dbfile_load_one_filerec()
  - Return 0 if user specified --help
  - Don't list '-?' as a valid option
  - add_file_db() Properly account for __add_file() skipping a file
  - Add hashstats to the man page

OBS-URL: https://build.opensuse.org/request/show/421850
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=41
 2016-08-24 19:46:51 +00:00
Mark Fasheh
4262370af0 Accepting request 419113 from home:mfasheh:branches:filesystems 
- Update to duperemove v0.11.beta1
  - Greatly increased performance during extent search
  - Hashfiles are reusable now, Duperemove will only scan files which have
    changed since the last time it was run.
  - Better selection of target extent during dedupe
    - reduces fragmentation of deduped space
  - Support for dedupe on xfs (experimental)

OBS-URL: https://build.opensuse.org/request/show/419113
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=39
 2016-08-12 23:32:36 +00:00
Dirk Mueller
0b7d08b2bb Accepting request 383214 from home:pluskalm:branches:filesystems 
- Use pretty name for tarball
- Cleanup spec file with spec-cleaner
- Use pkgconfig style dependnencies

OBS-URL: https://build.opensuse.org/request/show/383214
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=37
 2016-04-05 08:54:46 +00:00
Mark Fasheh
289d32af0b Accepting request 338308 from home:mfasheh:branches:filesystems 
- Update to duperemove v0.10
  - Documentation updates for latest stable version
    - document --fdupes mode in manpage
    - update man page headers
    - update FAQ

OBS-URL: https://build.opensuse.org/request/show/338308
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=35
 2015-10-12 21:04:40 +00:00
Mark Fasheh
076400119a Accepting request 317205 from home:mfasheh:branches:filesystems 
- Update to duperemove v0.10.beta4
  - Better memory usage with hashstats utility

OBS-URL: https://build.opensuse.org/request/show/317205
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=33
 2015-07-16 22:05:12 +00:00
Mark Fasheh
b5779bf30d Accepting request 315978 from home:mfasheh:branches:filesystems 
- Update to duperemove v0.10.beta3
  - fixes many crashes in the dedupe stage
  - fixed an issue with our alignment function which would throw dedupe into
    a loop
  - improvements to output formatting
  - better logic when throwing out already-deduped extents won't give as
    many false positives

OBS-URL: https://build.opensuse.org/request/show/315978
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=31
 2015-07-10 20:05:07 +00:00
Mark Fasheh
fa628957a3 Accepting request 314546 from home:mfasheh:branches:filesystems 
- Update to duperemove v0.10.beta2
  - multi-threaded dedupe stage (increased performance)
  - hashfile mode for drastically improved memory consumption
  - now uses murmur3 hash internally for improved cpu usage
  - can now take input from 'fdupes'

OBS-URL: https://build.opensuse.org/request/show/314546
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=30
 2015-06-30 17:05:51 +00:00
Mark Fasheh
3bc9c75563 Accepting request 294669 from home:mfasheh:branches:filesystems 
- Update with correct tarball (last one didn't have version number bump)

OBS-URL: https://build.opensuse.org/request/show/294669
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=28
 2015-04-06 19:47:44 +00:00
Mark Fasheh
83c2ffbc3f Accepting request 294667 from home:mfasheh:branches:filesystems 
- Update to duperemove v0.09.3
  - Fix hang during file scan phase (bsc#926098)
- References: bsc#926098

OBS-URL: https://build.opensuse.org/request/show/294667
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=27
 2015-04-06 19:34:28 +00:00
Mark Fasheh
8642a0f974 Accepting request 293347 from home:mfasheh:branches:filesystems 
- Update to duperemove v0.09.2
  - Now compiles on SLE11 targets
  - Fallback to stat on filesystems that don't report file type from
    readdir
  - Don't assert on failed open during dedupe
- References: bsc#923277 bsc#924577

OBS-URL: https://build.opensuse.org/request/show/293347
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=25
 2015-03-26 23:13:54 +00:00
Mark Fasheh
49a9ba41ce Accepting request 291619 from home:mfasheh:branches:filesystems 
- Update to duperemove v0.09.1
  - Fallback to stat on filesystems that don't report file type from
    readdir

OBS-URL: https://build.opensuse.org/request/show/291619
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=23
 2015-03-19 17:35:20 +00:00
Mark Fasheh
9f1ab85115 Accepting request 283378 from home:mfasheh:branches:filesystems 
- Update to duperemove v0.09
  - start with first node in free_compare_tree
  - fix small typos in duperemove man page
  - add test_progs variable in Makefile
- References: bsc#915354 FATE#318171
- Removed patch: do-not-install-test-binary.patch

OBS-URL: https://build.opensuse.org/request/show/283378
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=21
 2015-01-29 22:00:59 +00:00
Tomáš Chvátal
647ca5d71b - Cleanup a bit with spec-cleaner and fix few tiny issues 
- Use install phase from upstream Makefile
- Move csum-test to test progs to prevent its install:
  * do-not-install-test-binary.patch

OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=19
 2015-01-20 10:03:23 +00:00
Mark Fasheh
c68886a53f Accepting request 264495 from home:mfasheh:branches:filesystems 
- Update to duperemove v0.09.beta5
  - Documentation updates
    - FAQ and README are more relevant now
    - added man pages for show-shared-extents and hashstats programs
    - updated duperemove man page, and duperemove usage() function
  - Have show-shared-extents take a file list as arguments.
  - Change default of --lookup-extents option back to 'no'
  - Write hash type into hashfile header, check against what hash we were
    compiled with.

OBS-URL: https://build.opensuse.org/request/show/264495
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=17
 2014-12-09 04:21:54 +00:00
David Sterba
a129e793a3 Accepting request 262093 from home:mfasheh:branches:filesystems 
- Update to duperemove v0.09.beta3
  - Fix leak of directory fd during file scan
  - Fix EMFILES (too many file descriptors) error during dedupe
  - Fix corner case with dedupe leaving a file open and not-queued
  - Support '-x' (one file system) option
  - Add option to turn off extent lookup during csum phase
    - Useful if running against snapshotted volumes
  - show-shared-extents program to help users examine file state before or
    after dedupe.

OBS-URL: https://build.opensuse.org/request/show/262093
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=15
 2014-11-18 09:01:35 +00:00
David Sterba
770f64b84d Accepting request 260710 from home:Andreas_Schwab:Factory 
- Build with %optflags

OBS-URL: https://build.opensuse.org/request/show/260710
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=14
 2014-11-14 15:58:37 +00:00
David Sterba
a978b77f4f Accepting request 259065 from home:mfasheh:branches:filesystems 
- Update to duperemove v0.09.beta2
  - fix memory leak
  - fix hardlink detection on btrfs
  - print file number status during csum phase
  - print a status bar during extent seearch
  - several bugfixes and performance improvements to extent search
- Removed patch: 001-fix-build.patch

OBS-URL: https://build.opensuse.org/request/show/259065
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=12
 2014-10-31 12:59:50 +00:00
David Sterba
564c99233b add changelog 
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=10
 2014-09-29 08:35:57 +00:00
Mark Fasheh
98d7e67413 Accepting request 252570 from home:mfasheh:branches:filesystems 
- update to duperemove v0.09.beta1
- Added patch: 001-fix-build.patch

OBS-URL: https://build.opensuse.org/request/show/252570
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=9
 2014-09-27 00:34:50 +00:00
Mark Fasheh
4738202378 Accepting request 230974 from home:mfasheh:branches:filesystems 
- update to duperemove v0.08
  - fixes many bugs
  - a few more usability improvements
  - adds fiemap support to better estimate space savings

OBS-URL: https://build.opensuse.org/request/show/230974
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=7
 2014-04-22 22:11:22 +00:00
Mark Fasheh
a22294d4bb Accepting request 229768 from home:mfasheh:branches:filesystems 
- update to duperemove v0.07
- fix usability of command line arguments
- large documentation update

OBS-URL: https://build.opensuse.org/request/show/229768
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=5
 2014-04-11 17:47:32 +00:00
Mark Fasheh
51e8417acb Accepting request 229675 from home:mfasheh:branches:filesystems 
- update to duperemove v0.06. This adds several fixes and features:
        - fixes bnc#871804 (duperemove not looping on entire range)
- also includes important usability fixes
- updates hashing library to libgcrypt to reflect upstream

OBS-URL: https://build.opensuse.org/request/show/229675
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=3
 2014-04-11 03:21:03 +00:00
Mark Fasheh
34dba07346 Accepting request 225754 from home:mfasheh 
I want to maintain duperemove and btrfs-extent-same in Factory and would like to use filesystems as the devel project.

OBS-URL: https://build.opensuse.org/request/show/225754
OBS-URL: https://build.opensuse.org/package/show/filesystems/duperemove?expand=0&rev=1
 2014-03-12 19:45:27 +00:00