Commit Graph

45 Commits

Author SHA1 Message Date
Stephan Kulow
4692d47120 Make the refresh a debug output, not info 2022-11-16 09:05:36 +01:00
Stephan Kulow
be8fb2ab94 Fix fake revision creation 2022-11-06 12:27:36 +01:00
Stephan Kulow
5ae02a413d Store API URL in the revision table
Will be important once we get into SLE
2022-11-06 10:57:32 +01:00
Stephan Kulow
a457a16a50 Limit the workers to 8
This is hard coding the limit, we may want to make this configurable
but for now the machines supposed to run this code are very similiar
2022-11-04 10:00:28 +01:00
Stephan Kulow
d21ce571f5 Refresh the packages in multiple threads 2022-11-03 22:04:45 +01:00
Stephan Kulow
ab38332642 Allow to import multiple packages in one go
This way we avoid duplicating all startup and SQL queries
2022-11-03 20:14:56 +01:00
Stephan Kulow
ba7436f10c Keep a reference to the database in DBRevision
To avoid passing the db to all actions
2022-11-02 18:27:09 +01:00
Stephan Kulow
05a5e6aea7 Don't refresh packages we're already looking at 2022-11-02 10:52:53 +01:00
Stephan Kulow
bbf1bc2fda Fetch source projects of requests
We do not care for current devel project, but for the projects we saw
requests from
2022-11-02 08:50:54 +01:00
Stephan Kulow
c4654dd896 Split GitExporter out of Importer class 2022-11-02 07:59:25 +01:00
Stephan Kulow
4ff9b9771a Split out Flat generator to be able to test it 2022-11-02 07:20:53 +01:00
Stephan Kulow
578fb2a30a Change tree pruning algorithm
The first merge we see in Factory determines if we keep the devel
commits in the factory chain or cut that branch.
2022-11-01 13:52:15 +01:00
Stephan Kulow
e6a401d8ac Remove old history handling 2022-11-01 11:37:30 +01:00
Stephan Kulow
9ed8abad2b Make database usage the default
Some cleanup of no longer used functions
2022-11-01 11:23:40 +01:00
Stephan Kulow
9554fea7e1 Reuse the repository directory by storing a state yaml
Not using the database for that so that removing the repository directory will
automatically recreate it
2022-11-01 11:22:58 +01:00
Stephan Kulow
cfab0a522b Download .spec and .changes file only for now 2022-10-31 16:31:10 +01:00
Stephan Kulow
bfdade8ecf Create a flat list of commits to do from the tree
Walk the node tree and record the parents, then reverse the tree so we
can have the exact order in which to create git commits
2022-10-31 12:28:12 +01:00
Stephan Kulow
8a3db6c183 Collect the requests for revisions in one DB call
For smaller packages this doesn't matter as much, but for large ones
the N+1 queries can sum up badly
2022-10-31 08:03:01 +01:00
Stephan Kulow
822329279d Switch to walker for tree print 2022-10-31 07:03:19 +01:00
Stephan Kulow
e636f3ed9d Print the calculated tree for now 2022-10-27 13:34:04 +02:00
Stephan Kulow
7a9e9ce334 Add actual tree and a test case for zsh 2022-10-27 13:16:50 +02:00
Stephan Kulow
65000cd586 Fix even more issues with the linked_revs considered cache 2022-10-27 10:16:20 +02:00
Stephan Kulow
38e4996280 Fix the fake revisions after introduction of cache
The way I stored it in the linked_revs was the wrong way
2022-10-27 07:35:08 +02:00
Stephan Kulow
9bd6643e8a Fix files_hash calculcation and fake revisions 2022-10-26 22:18:16 +02:00
Stephan Kulow
1c846e963c Import files of revisions more reliable and faster 2022-10-26 21:47:39 +02:00
Stephan Kulow
4dc7e972a2 Make use of db.cursor() in with blocks 2022-10-26 15:49:14 +02:00
Stephan Kulow
217bbcadb5 Run the formatter from make all 2022-10-26 14:57:08 +02:00
Stephan Kulow
9dbf3b9094 Import all projects before checking files - otherwise we end with 0s 2022-10-26 14:56:38 +02:00
Stephan Kulow
2784171f75 Create fake revisions for every commit in the base of a linked package
This is a rather complex operation, but whenever a package changes in
Factory, the inherited package gets a shadow revision consisting of the
3-way merge. If this happens due to a request being accepted, this is
actually in most cases also commited by 'buildservice-autocommit',
so we're making sure this is always happening (and are actually
duplicating revisions in cases that we filter out later as empty
commits).

To differenciate the fake revisions from the real revisions, I add
a fraction part
2022-10-26 14:56:38 +02:00
Stephan Kulow
d17e60a608 Add another table to store linked_revs
We need to create fake revisions when packages were touched that are
linked themselves
2022-10-26 14:20:28 +02:00
Stephan Kulow
089b215f4c Build a revision tree 2022-10-26 14:16:22 +02:00
dbeb534108
cleanup context managers 2022-10-26 13:46:02 +02:00
1656d73340
fixed formatting and other suggestions 2022-10-26 11:58:01 +02:00
f14c609db2
Add batch import 2022-10-26 09:30:02 +02:00
10cd7e4e38
execute user import 2022-10-24 18:12:44 +02:00
Stephan Kulow
fa9ec8618e Load requests into the DB - not sure yet what for :) 2022-10-19 16:35:33 +02:00
Stephan Kulow
771595ba85 Catch invalid XML in links 2022-10-18 21:36:58 +02:00
Stephan Kulow
8dcae5df50 Remove debug output 2022-10-18 19:31:41 +02:00
Stephan Kulow
89c0335812 Import file list of revisions 2022-10-18 19:29:25 +02:00
Stephan Kulow
d747a3a761 Calculate the linked revision 2022-10-18 15:40:11 +02:00
Stephan Kulow
fb130bbf6f Import packages links point to 2022-10-18 14:57:23 +02:00
Stephan Kulow
95412bc834 Prepare link collection 2022-10-18 13:13:52 +02:00
Stephan Kulow
25b45c5073 Split Revision into OBS and DB 2022-10-18 12:17:43 +02:00
Stephan Kulow
8563334b07 Add experimental import into DB 2022-10-17 20:39:03 +02:00
Stephan Kulow
a82562b794 Move modules into lib subdir 2022-10-17 19:56:41 +02:00