importers/git-importer
3
0
Fork 2
Code Issues Pull Requests Packages Projects Releases Wiki Activity
146 Commits 4 Branches 0 Tags 1.8 MiB
5a353c98d3
Commit Graph

146 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Dirk Müller
5a353c98d3
Add tasks 2024-05-17 11:46:18 +02:00
Dirk Müller
1fc466d15b
Add monitor for commits 2024-05-17 11:40:19 +02:00
Dirk Müller
39fde7744a
Code cleanup 2024-05-16 15:47:45 +02:00
Dirk Müller
f5ffc83a69
Remove double quoting of url parameters 
makeurl quotes by itself, so this was messing it up
 2024-05-16 11:49:14 +02:00
Dirk Müller
d0ccf83684
Revert "Try to fetch the element as deleted if initial access failed" 
The OBS api has been fixed to provide an automatic fallback via
https://github.com/openSUSE/open-build-service/pull/15655

This reverts commit c9e07e536f.
 2024-05-16 11:49:14 +02:00
Dirk Müller
b0ffb01c59
cleanups 2024-05-16 11:49:14 +02:00
Dirk Müller
28d5c6e606
Switch to psycopg rather than psycopg2 
It's a bit more modern and uses dedicated c bindings
 2024-05-16 11:49:14 +02:00
Dirk Mueller
1e22c2895a Merge pull request 'Switch to sha-256 git repo and use git tools again' (#23) from adamm/git-importer:main into main 
Reviewed-on: #23
 2024-05-16 11:48:36 +02:00
Adam Majer
5da7861c2a Switch to sha-256 git repo and use git tools again 2024-04-09 11:40:26 +02:00
Dirk Müller
c9e07e536f
Try to fetch the element as deleted if initial access failed 
The reference to the object might be already deleted by when the
request is failing. plus setting deleted=0 is rejected by the API.
So try with deleted=1 if and only if the previous access failed.
 2023-12-07 18:30:36 +01:00
Dirk Müller
dc0f33354e
Failing to LFS register should abort the import 2023-12-07 18:29:56 +01:00
Dirk Müller
56cbe0a125
Avoid multi-threading races on import 
There seems to be races when using db cursors from multiple threads. as
found by import issues after switching to a newer computer that has
performance and energy efficient cores.

As this is not particularly performance critical, convert to single
threaded use which makes it work again
 2023-11-28 23:36:44 +01:00
Dirk Müller
4353f015c8
Switch to localhost:9999 which is provided via a ssh tunnel 
The port is no longer directly exposed, so we need to ssh tunnel it
 2023-11-22 14:39:55 +01:00
Dirk Müller
9cbe0899bc
Remove unused import 2023-06-19 13:19:52 +02:00
Dirk Müller
9e80a64fe0
Change hostname references from gitea.opensuse.org to src.opensuse.org 2023-06-19 10:59:56 +02:00
Dirk Müller
12001b1640
Commit local changes 2023-04-18 22:31:38 +02:00
Stephan Kulow
3797ea178a Merge pull request 'Add a list of packages no longer existing' (#22) from add_gone into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/22
 2023-02-09 10:23:35 +01:00
Stephan Kulow
999dcabcfa Add a list of packages no longer existing 
I made this a file and not a DB that is automatically maintained as I think
for now adding an entry in there should be done manually - OBS being OBS
packages might look have gone for a brief moment and reappar the day after.
 2022-12-02 11:00:31 +01:00
Alberto Planas
9962673eff Merge pull request 'Add force push for the devel branch' (#21) from add_force into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/21
 2022-12-02 09:35:40 +01:00
Stephan Kulow
7b20c03256 Add force push for the devel branch 
As devel branches can change in case of factory reverts we need to force
push. Factory branch shouldn't be affected, so not force pushing there
 2022-12-02 09:12:11 +01:00
Stephan Kulow
4692d47120 Make the refresh a debug output, not info 2022-11-16 09:05:36 +01:00
coolo
d311d54f26 Merge pull request 'Also treat some more mimetypes as text' (#20) from add_further_mimetypes into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/20
 2022-11-15 07:28:05 +01:00
Stephan Kulow
dddc54ab1c Remove ProcessPool from exporting 
It's ignoring exceptions and makes debugging way too hard to justify
what's happening
 2022-11-11 16:33:44 +01:00
Stephan Kulow
4d1ca8d882 Also treat some more mimetypes as text 2022-11-11 16:22:18 +01:00
Stephan Kulow
7861a7e9b0 Fix LFS register (it needs json not data) 
Refactored the LFS Oid handling in its class of its own and
add a way to recheck all LFS handles (or re-register)
 2022-11-09 08:32:18 +01:00
coolo
f5b29886ae Merge pull request 'No longer rely on external service for LFS tracking' (#18) from add_lfs into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/18
 2022-11-08 11:00:34 +01:00
coolo
d1a8a3288d Merge pull request 'Push to the remote when the repo changed' (#19) from push_it_baby into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/19
 2022-11-08 09:48:56 +01:00
Stephan Kulow
9f6c8f62e7 Push to the remote when the repo changed 2022-11-08 09:32:03 +01:00
Stephan Kulow
3e1fbaa1c3 Migrate the ProxySHA256 data into postgresql DB 
The calculation of the sha256 and the mimetype is local due to that
 2022-11-07 21:50:31 +01:00
Stephan Kulow
e1b32999f0 Fix confusion about User constructor 2022-11-07 16:04:44 +01:00
coolo
86490b51dd Merge pull request 'Fix the maintenance of .gitattributes file' (#17) from fix_lfs_attributes into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/17
 2022-11-07 13:26:36 +01:00
Stephan Kulow
da5de04171 Add packages to consider 2022-11-07 07:29:20 +01:00
Stephan Kulow
be8fb2ab94 Fix fake revision creation 2022-11-06 12:27:36 +01:00
Stephan Kulow
9e895e34b6 Adding a gitea remote when creating the git repo 2022-11-06 12:18:16 +01:00
Stephan Kulow
5e495dbd95 Fancy up the git commit message 2022-11-06 11:46:04 +01:00
Stephan Kulow
5ae02a413d Store API URL in the revision table 
Will be important once we get into SLE
 2022-11-06 10:57:32 +01:00
Stephan Kulow
f1457e8f8e Move git commit message creation into class 2022-11-06 10:16:42 +01:00
Stephan Kulow
9114c2fff8 Change debug output for downloading files 2022-11-06 10:16:42 +01:00
Stephan Kulow
834cf61634 Use proper user info in commits 2022-11-06 09:53:52 +01:00
Stephan Kulow
a294c0f670 Readd the skipping of _staging_workflow file 
A repository with 150k commits is just very hard to work with -
especially if 99% of them are worthless
 2022-11-06 08:29:17 +01:00
Stephan Kulow
7bc4d6c8b1 Make downloading a little more careful for races 
As we're downloading packages in parallel, it could happen that we
copy a file that isn't fully copied yet
 2022-11-06 08:24:11 +01:00
Stephan Kulow
bd5bd5a444 Don't reset the .gitattributes file 
Just change it if it existed before
 2022-11-04 21:02:18 +01:00
Stephan Kulow
4e1d5b42ca Only validate the MD5 if we downloaded - trust the file system 2022-11-04 21:02:18 +01:00
Stephan Kulow
0bcc0183c9 Load the proxy data for is_text as well 
Otherwise the text state changes over time
 2022-11-04 21:02:18 +01:00
Stephan Kulow
7f88e0cc5c Run in the same process if there is only package 
Debugging is much easier without Process Pool
 2022-11-04 21:02:18 +01:00
coolo
4cc0a23d4e Merge pull request 'Run many packages in parallel to avoid overhead and make use of CPUS' (#16) from parallize_packages into main 
Reviewed-on: https://gitea.opensuse.org/importers/git-importer/pulls/16
 2022-11-04 10:04:15 +01:00
Stephan Kulow
a457a16a50 Limit the workers to 8 
This is hard coding the limit, we may want to make this configurable
but for now the machines supposed to run this code are very similiar
 2022-11-04 10:00:28 +01:00
Stephan Kulow
60ced896a3 Fix condition for export 2022-11-04 09:58:36 +01:00
Stephan Kulow
33a5733cb9 Create the git repos in multiple processes 
Threads appear to be too dangerous for this
 2022-11-04 07:48:17 +01:00
Stephan Kulow
d21ce571f5 Refresh the packages in multiple threads 2022-11-03 22:04:45 +01:00