importers/git-importer
3
0
Fork 2
Code Issues Pull Requests Packages Projects Releases Wiki Activity
135 Commits 4 Branches 0 Tags 1.8 MiB
56cbe0a125
Commit Graph

135 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
nkrapp
7678967ae0
implement file caching 
to prevent having to download files multiple times
 2022-11-03 14:05:11 +01:00
coolo
74f5cd901e Merge pull request 'Keep a reference to the database in DBRevision' (#12) from add_export into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/12
 2022-11-03 08:16:30 +01:00
Stephan Kulow
1c54a74ecd Download the full revision 2022-11-02 20:55:09 +01:00
Stephan Kulow
c2294d6200 Add a default LFS .gitattributes for now 
Otherwise some packages will break to import
 2022-11-02 18:27:17 +01:00
Stephan Kulow
ba7436f10c Keep a reference to the database in DBRevision 
To avoid passing the db to all actions
 2022-11-02 18:27:09 +01:00
coolo
75f9f56a57 Merge pull request 'Fix up some code after aplanas' continued review' (#10) from add_export into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/10
 2022-11-02 18:05:04 +01:00
Stephan Kulow
172242891d Fix up some code after aplanas' continued review 2022-11-02 15:22:24 +01:00
coolo
6ac0f90ec7 Merge pull request 'Export the built tree as git repo' (#9) from add_export into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/9
 2022-11-02 14:17:24 +01:00
Stephan Kulow
05cf792b26 Add the file_md5 to the download function so it can cache and verify 2022-11-02 13:35:45 +01:00
Stephan Kulow
a1ead29734 Extend documention and use some more pythonier loops 2022-11-02 13:29:18 +01:00
Stephan Kulow
05a5e6aea7 Don't refresh packages we're already looking at 2022-11-02 10:52:53 +01:00
Stephan Kulow
68ded48be1 Don't crash on user with missing realname either 2022-11-02 10:52:37 +01:00
Stephan Kulow
fce8aac001 Import creator users as well 2022-11-02 08:59:25 +01:00
Stephan Kulow
bbf1bc2fda Fetch source projects of requests 
We do not care for current devel project, but for the projects we saw
requests from
 2022-11-02 08:50:54 +01:00
Stephan Kulow
8a863498db Move default repos into a subdirectory that is git ignored 2022-11-02 08:01:22 +01:00
Stephan Kulow
c4654dd896 Split GitExporter out of Importer class 2022-11-02 07:59:25 +01:00
Stephan Kulow
9de0d6e6c5 Rename Exporter to TestExporter to make it more obvious 2022-11-02 07:39:04 +01:00
Stephan Kulow
a8dfd80fdd Reduce zsh fixture data 2 years to ease test runtime 
The interesting parts are before 2012 anyway
 2022-11-02 07:24:24 +01:00
Stephan Kulow
4ff9b9771a Split out Flat generator to be able to test it 2022-11-02 07:20:53 +01:00
Stephan Kulow
c94d13d74e Don't crash on packages without merges ever (very few packages) 2022-11-01 19:30:41 +01:00
Stephan Kulow
ab8120ca53 Don't crash on last_node 2022-11-01 19:02:29 +01:00
Stephan Kulow
b2cadb8c01 Don't crash on packages that didn't get updates in devel 2022-11-01 18:44:59 +01:00
Stephan Kulow
578fb2a30a Change tree pruning algorithm 
The first merge we see in Factory determines if we keep the devel
commits in the factory chain or cut that branch.
 2022-11-01 13:52:15 +01:00
Stephan Kulow
9aaba03bcc Add llvm13 as test case as I'm missing rev 3 in the export 2022-11-01 11:45:20 +01:00
Stephan Kulow
e6a401d8ac Remove old history handling 2022-11-01 11:37:30 +01:00
Stephan Kulow
9ed8abad2b Make database usage the default 
Some cleanup of no longer used functions
 2022-11-01 11:23:40 +01:00
Stephan Kulow
9554fea7e1 Reuse the repository directory by storing a state yaml 
Not using the database for that so that removing the repository directory will
automatically recreate it
 2022-11-01 11:22:58 +01:00
Stephan Kulow
2168c898a2 Add users not known to the FAKE_ACCOUNTS 
This is technically incorrect but we need to handle them all the same anyway
 2022-11-01 09:12:42 +01:00
Stephan Kulow
cfab0a522b Download .spec and .changes file only for now 2022-10-31 16:31:10 +01:00
Stephan Kulow
bfdade8ecf Create a flat list of commits to do from the tree 
Walk the node tree and record the parents, then reverse the tree so we
can have the exact order in which to create git commits
 2022-10-31 12:28:12 +01:00
Stephan Kulow
8a3db6c183 Collect the requests for revisions in one DB call 
For smaller packages this doesn't matter as much, but for large ones
the N+1 queries can sum up badly
 2022-10-31 08:03:01 +01:00
Stephan Kulow
822329279d Switch to walker for tree print 2022-10-31 07:03:19 +01:00
Alberto Planas
6f83425538 Merge pull request 'Add a test case for a simpler and newer package' (#8) from add_second_test into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/8
 2022-10-27 15:19:47 +02:00
Stephan Kulow
83544e45d7 Add a test case for a simpler and newer package 2022-10-27 15:15:41 +02:00
coolo
62511d15bc Merge pull request 'Add actual tree and a test case for zsh' (#7) from add_tree into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/7
 2022-10-27 14:04:52 +02:00
Stephan Kulow
e636f3ed9d Print the calculated tree for now 2022-10-27 13:34:04 +02:00
Stephan Kulow
7a9e9ce334 Add actual tree and a test case for zsh 2022-10-27 13:16:50 +02:00
Stephan Kulow
65000cd586 Fix even more issues with the linked_revs considered cache 2022-10-27 10:16:20 +02:00
coolo
996b90dd7a Merge pull request 'Fix calculation of the tree data' (#5) from fix_fake into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/5
 2022-10-27 10:01:33 +02:00
Stephan Kulow
38e4996280 Fix the fake revisions after introduction of cache 
The way I stored it in the linked_revs was the wrong way
 2022-10-27 07:35:08 +02:00
Stephan Kulow
9bd6643e8a Fix files_hash calculcation and fake revisions 2022-10-26 22:18:16 +02:00
Stephan Kulow
1c846e963c Import files of revisions more reliable and faster 2022-10-26 21:47:39 +02:00
coolo
22b70d8bb7 Merge pull request 'Build a Tree of commits (at least prep it)' (#4) from tree_builder into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/4
 2022-10-26 19:33:13 +02:00
Stephan Kulow
4dc7e972a2 Make use of db.cursor() in with blocks 2022-10-26 15:49:14 +02:00
Stephan Kulow
217bbcadb5 Run the formatter from make all 2022-10-26 14:57:08 +02:00
Stephan Kulow
1346223822 Reshuffle schema versions after rebase 2022-10-26 14:56:38 +02:00
Stephan Kulow
9dbf3b9094 Import all projects before checking files - otherwise we end with 0s 2022-10-26 14:56:38 +02:00
Stephan Kulow
d961c13614 Ignore commits after the merge 2022-10-26 14:56:38 +02:00
Stephan Kulow
2784171f75 Create fake revisions for every commit in the base of a linked package 
This is a rather complex operation, but whenever a package changes in
Factory, the inherited package gets a shadow revision consisting of the
3-way merge. If this happens due to a request being accepted, this is
actually in most cases also commited by 'buildservice-autocommit',
so we're making sure this is always happening (and are actually
duplicating revisions in cases that we filter out later as empty
commits).

To differenciate the fake revisions from the real revisions, I add
a fraction part
 2022-10-26 14:56:38 +02:00
Stephan Kulow
d17e60a608 Add another table to store linked_revs 
We need to create fake revisions when packages were touched that are
linked themselves
 2022-10-26 14:20:28 +02:00