Run many packages in parallel to avoid overhead and make use of CPUS #16

Merged
Ghost merged 5 commits from parallize_packages into main 2022-11-04 10:04:15 +01:00

5 Commits

Author SHA1 Message Date
Stephan Kulow
a457a16a50 Limit the workers to 8
This is hard coding the limit, we may want to make this configurable
but for now the machines supposed to run this code are very similiar
2022-11-04 10:00:28 +01:00
Stephan Kulow
60ced896a3 Fix condition for export 2022-11-04 09:58:36 +01:00
Stephan Kulow
33a5733cb9 Create the git repos in multiple processes
Threads appear to be too dangerous for this
2022-11-04 07:48:17 +01:00
Stephan Kulow
d21ce571f5 Refresh the packages in multiple threads 2022-11-03 22:04:45 +01:00
Stephan Kulow
ab38332642 Allow to import multiple packages in one go
This way we avoid duplicating all startup and SQL queries
2022-11-03 20:14:56 +01:00