adamm/git-importer
1
0
Fork 1
forked from importers/git-importer
Code Pull Requests Activity
109 Commits 10 Branches 0 Tags 1.8 MiB
9114c2fff8
Commit Graph

109 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Stephan Kulow
9114c2fff8 Change debug output for downloading files 2022-11-06 10:16:42 +01:00
Stephan Kulow
834cf61634 Use proper user info in commits 2022-11-06 09:53:52 +01:00
Stephan Kulow
a294c0f670 Readd the skipping of _staging_workflow file 
A repository with 150k commits is just very hard to work with -
especially if 99% of them are worthless
 2022-11-06 08:29:17 +01:00
Stephan Kulow
7bc4d6c8b1 Make downloading a little more careful for races 
As we're downloading packages in parallel, it could happen that we
copy a file that isn't fully copied yet
 2022-11-06 08:24:11 +01:00
Stephan Kulow
bd5bd5a444 Don't reset the .gitattributes file 
Just change it if it existed before
 2022-11-04 21:02:18 +01:00
Stephan Kulow
4e1d5b42ca Only validate the MD5 if we downloaded - trust the file system 2022-11-04 21:02:18 +01:00
Stephan Kulow
0bcc0183c9 Load the proxy data for is_text as well 
Otherwise the text state changes over time
 2022-11-04 21:02:18 +01:00
Stephan Kulow
7f88e0cc5c Run in the same process if there is only package 
Debugging is much easier without Process Pool
 2022-11-04 21:02:18 +01:00
coolo
4cc0a23d4e Merge pull request 'Run many packages in parallel to avoid overhead and make use of CPUS' (#16) from parallize_packages into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/16
 2022-11-04 10:04:15 +01:00
Stephan Kulow
a457a16a50 Limit the workers to 8 
This is hard coding the limit, we may want to make this configurable
but for now the machines supposed to run this code are very similiar
 2022-11-04 10:00:28 +01:00
Stephan Kulow
60ced896a3 Fix condition for export 2022-11-04 09:58:36 +01:00
Stephan Kulow
33a5733cb9 Create the git repos in multiple processes 
Threads appear to be too dangerous for this
 2022-11-04 07:48:17 +01:00
Stephan Kulow
d21ce571f5 Refresh the packages in multiple threads 2022-11-03 22:04:45 +01:00
Stephan Kulow
ab38332642 Allow to import multiple packages in one go 
This way we avoid duplicating all startup and SQL queries
 2022-11-03 20:14:56 +01:00
coolo
651bd94771 Merge pull request 'Fix merge points creating a cross' (#15) from debug_firewalld into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/15
 2022-11-03 15:35:52 +01:00
Stephan Kulow
dd5e26b779 Clarify which of the candidates is the right one - removing assert 2022-11-03 15:29:58 +01:00
Stephan Kulow
f2019db8ff Ignore merge point candidates that create crosses 
In OBS you can create submit requests for revisions that are behind
the last merge point, in git you can't - so we ignore them.

Fixes #14
 2022-11-03 15:19:51 +01:00
Stephan Kulow
ef7755c771 Add firewalld test showing a broken tree 2022-11-03 15:19:21 +01:00
coolo
6b26f8ff96 Merge pull request 'Fix the import of breeze and other packages' (#13) from add_export into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/13
 2022-11-03 15:19:06 +01:00
Stephan Kulow
ed4b7367eb Reset branch if the devel branch is based on Factory 
This happens in packages that change their devel project over time. Then
the commit in the devel project no longer has the parent in the devel branch
but is based on factory
 2022-11-03 15:12:07 +01:00
Stephan Kulow
f5b3e42165 Add a test case that switches devel project in its life time 2022-11-03 15:06:12 +01:00
nkrapp
6dd3cf3eba Merge pull request 'implement file caching' (#11) from file-cache into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/11
 2022-11-03 14:24:22 +01:00
nkrapp
8aed76e52a
change cached file naming pattern 2022-11-03 14:22:19 +01:00
nkrapp
639096b548
optimize cached file locations and add option for cache directory 2022-11-03 14:12:32 +01:00
nkrapp
7678967ae0
implement file caching 
to prevent having to download files multiple times
 2022-11-03 14:05:11 +01:00
coolo
74f5cd901e Merge pull request 'Keep a reference to the database in DBRevision' (#12) from add_export into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/12
 2022-11-03 08:16:30 +01:00
Stephan Kulow
1c54a74ecd Download the full revision 2022-11-02 20:55:09 +01:00
Stephan Kulow
c2294d6200 Add a default LFS .gitattributes for now 
Otherwise some packages will break to import
 2022-11-02 18:27:17 +01:00
Stephan Kulow
ba7436f10c Keep a reference to the database in DBRevision 
To avoid passing the db to all actions
 2022-11-02 18:27:09 +01:00
coolo
75f9f56a57 Merge pull request 'Fix up some code after aplanas' continued review' (#10) from add_export into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/10
 2022-11-02 18:05:04 +01:00
Stephan Kulow
172242891d Fix up some code after aplanas' continued review 2022-11-02 15:22:24 +01:00
coolo
6ac0f90ec7 Merge pull request 'Export the built tree as git repo' (#9) from add_export into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/9
 2022-11-02 14:17:24 +01:00
Stephan Kulow
05cf792b26 Add the file_md5 to the download function so it can cache and verify 2022-11-02 13:35:45 +01:00
Stephan Kulow
a1ead29734 Extend documention and use some more pythonier loops 2022-11-02 13:29:18 +01:00
Stephan Kulow
05a5e6aea7 Don't refresh packages we're already looking at 2022-11-02 10:52:53 +01:00
Stephan Kulow
68ded48be1 Don't crash on user with missing realname either 2022-11-02 10:52:37 +01:00
Stephan Kulow
fce8aac001 Import creator users as well 2022-11-02 08:59:25 +01:00
Stephan Kulow
bbf1bc2fda Fetch source projects of requests 
We do not care for current devel project, but for the projects we saw
requests from
 2022-11-02 08:50:54 +01:00
Stephan Kulow
8a863498db Move default repos into a subdirectory that is git ignored 2022-11-02 08:01:22 +01:00
Stephan Kulow
c4654dd896 Split GitExporter out of Importer class 2022-11-02 07:59:25 +01:00
Stephan Kulow
9de0d6e6c5 Rename Exporter to TestExporter to make it more obvious 2022-11-02 07:39:04 +01:00
Stephan Kulow
a8dfd80fdd Reduce zsh fixture data 2 years to ease test runtime 
The interesting parts are before 2012 anyway
 2022-11-02 07:24:24 +01:00
Stephan Kulow
4ff9b9771a Split out Flat generator to be able to test it 2022-11-02 07:20:53 +01:00
Stephan Kulow
c94d13d74e Don't crash on packages without merges ever (very few packages) 2022-11-01 19:30:41 +01:00
Stephan Kulow
ab8120ca53 Don't crash on last_node 2022-11-01 19:02:29 +01:00
Stephan Kulow
b2cadb8c01 Don't crash on packages that didn't get updates in devel 2022-11-01 18:44:59 +01:00
Stephan Kulow
578fb2a30a Change tree pruning algorithm 
The first merge we see in Factory determines if we keep the devel
commits in the factory chain or cut that branch.
 2022-11-01 13:52:15 +01:00
Stephan Kulow
9aaba03bcc Add llvm13 as test case as I'm missing rev 3 in the export 2022-11-01 11:45:20 +01:00
Stephan Kulow
e6a401d8ac Remove old history handling 2022-11-01 11:37:30 +01:00
Stephan Kulow
9ed8abad2b Make database usage the default 
Some cleanup of no longer used functions
 2022-11-01 11:23:40 +01:00