importers/git-importer
5
1
Fork 2
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity

Run many packages in parallel to avoid overhead and make use of CPUS #16

Merged
Ghost merged 5 commits from parallize_packages into main 2022-11-04 10:04:15 +01:00
Conversation 3 Commits 5 Files Changed 7 +150 -150
7 changed files with 150 additions and 150 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit 33a5733cb9 - Show all commits

4
Makefile
View File

@@ -1,7 +1,7 @@
all:
isort *.py lib/*py tests/*py
autoflake -r --in-place --remove-unused-variables .
black .
autoflake --in-place --remove-unused-variables *.py lib/*py tests/*py
black *.py lib/*py tests/*py
test:
python3 -m unittest -v tests/*.py

21
git-importer.py
View File

@@ -1,6 +1,7 @@
#!/usr/bin/python3
import argparse
import concurrent.futures
import logging
import pathlib
import sys
@@ -42,6 +43,12 @@ PROJECTS = [
]
def export_package(package, repodir, cachedir, gc):
exporter = GitExporter(URL_OBS, "openSUSE:Factory", package, repodir, cachedir)
exporter.set_gc_interval(gc)
exporter.export_as_git()
def main():
parser = argparse.ArgumentParser(description="OBS history importer into git")
parser.add_argument("packages", help="OBS package names", nargs="*")
@@ -106,12 +113,14 @@ def main():
importer = Importer(URL_OBS, "openSUSE:Factory", args.packages)
importer.import_into_db()
for package in args.packages:
exporter = GitExporter(
URL_OBS, "openSUSE:Factory", package, args.repodir, args.cachedir
)
exporter.set_gc_interval(args.gc)
exporter.export_as_git()
with concurrent.futures.ProcessPoolExecutor() as executor:
aplanas marked this conversation as resolved Outdated
aplanas commented 2022-11-04 08:02:06 +01:00
Outdated
Review
Copy Link

Making max_workers to None will create one process per CPU. That is OK in my desktop, but laptop report like 32 of them, and I am sure this will kill OBS. Maybe create a parameter with some low default?

Making `max_workers` to `None` will create one process per CPU. That is OK in my desktop, but laptop report like 32 of them, and I am sure this will kill OBS. Maybe create a parameter with some low default?
Ghost commented 2022-11-04 09:54:10 +01:00
Outdated
Review
Copy Link

I guess I can just hard code to 8. Would that work for you?

I guess I can just hard code to 8. Would that work for you?
aplanas commented 2022-11-04 09:54:57 +01:00
Outdated
Review
Copy Link

seems safer

seems safer
fs = [
executor.submit(
export_package, package, args.repodir, args.cachedir, args.gc
)
for package in args.packages
]
concurrent.futures.wait(fs)
if __name__ == "__main__":

5
lib/proxy_sha256.py
View File

@@ -41,7 +41,9 @@ class ProxySHA256:
self.hashes = dict()
return
logging.debug("Retrieve all previously defined SHA256")
response = requests.get(f"http://source.dyn.cloud.suse.de/package/{package}")
response = requests.get(
f"http://source.dyn.cloud.suse.de/package/{package}", timeout=5
)
if response.status_code == 200:
json = response.json()
self.hashes = json["shas"]
@@ -67,6 +69,7 @@ class ProxySHA256:
"url": url,
"package": package,
},
timeout=10,
)
if response.status_code != 200:
raise Exception(f"Redirector error on {self.url} for {url}")