------------------------------------------------------------------- Mon May 20 07:41:15 UTC 2019 - coolo@suse.com - Update to version 4.6.1558084113.18c6f052: * Revert "Improve code for settings expansion" * Update documentation about job scheduling * Add job template settings when scheduling new jobs * Show job template parameter via regular REST API * Allow YAML import/export of job template settings * Support scaling MAX_JOB_TIME * Add job template settings table * Add test for saving needles via Git * Validate properties of OpenQA::Git before using them * Fix comparision of cluster ID for dependency graph (#2068) * Set user correctly in OpenQA::Git instantiation ------------------------------------------------------------------- Mon May 13 12:07:50 UTC 2019 - coolo@suse.com - Update to version 4.6.1557739992.f9fbeb06: * Avoid leaking GITHUB_TOKEN * Skip UEFI_PFLASH_VARS asset when UEFI is disabled * Add an own result jobs which exceeded the timeout * Split off git support into its own class * Adapt 13-admin.t to the new sorting-enabled tables * Fix remaining jshint warnings in admintable.js * Adjust sorting of admin tables so empty strings come last * Improve styling of help icon beside search box * Improve regex-enabled search of admin tables * Tweak sorting and filtering in admin data tables * Remove admin table code for column type settings * Make admin tables use DataTables correctly * Improve code for settings expansion * Move job settings expansion to its own module * Use CodeMirror with YAML editor * Inherit specified settings when create job with 'Jobs post' * Show an error message in the editor if the group changed * Clean up 08-jobtemplates.t * Do not add new testsuites when importing job templates * Add YAML-based template editor UI * Add non-trivial test suites to exercise YAML validation * Import of job template specification from YAML * Support click_point in needle editor controller code * Fix issues reported by jshint in shapes.js * Test adding/removing a click point * Allow only one click point per needle * Allow adding click coordinates in the needle editor * Improve JSON parse errors in openqa-clone-custom-git-refspec * Retry after rsync exits with status 24 * Add non-fatal messages during scheduling to results * Allow authenticated calls to the github api * Remove deprecated qemu-kvm and use qemu-arm, qemu-ppc and qemu-x86 instead * apparmor: rk permission on hdd contents ------------------------------------------------------------------- Wed Apr 17 14:34:07 UTC 2019 - coolo@suse.com - Update to version 4.6.1555478309.f6dfdd9b: * Clone only the latest revision of the SLE needles (#2048) * Revert "openQA.spec: multiline macros for services and dependencies" (#2051) * Update architecture diagram in accordance with PR 2043 * Fix the remaining issues of deleting workers * Get rid of a few warnings that make test results on Travis hard to read * Be a little more strict about WebSocket server response codes * There should be no need for AssetPack in the websocket server * Trust connections from localhost to the websocket server to avoid complex config file requirements * The websocket server is no longer a dbus service * Clean up tests and remove a few connection warnings * The websocket server is no longer a dbus service * Move ws_send from dbus to HTTP * Move ws_send_job from dbus to HTTP * Remove ws_send_all dbus function, which was actually unused * Remove first dbus function from the websocket server * Add an HTTP API to the websocket server * Don't return status 402 on worker deletion failure * Fix indentation in admin_worker.js * Make OpenQA::Events::emit_event a non-static member function * Update AppArmor profile for usrMerge (boo#1132350) * Add cacheservice as part of openqa-worker.target * openQA.spec: multiline macros for services and dependencies ------------------------------------------------------------------- Sun Apr 14 17:57:10 UTC 2019 - coolo@suse.com - Update to version 4.6.1554809752.3a77164a: * Fix mock session logins to use only public Mojolicious APIs and create users with a proper resultset * Prevent multiple limit_* tasks from running in parallel * Allow deleting offline workers via API and web UI * Don't duplicate settings in iso_create/cancel events * Adapt product log test * Add test for re-scheduling product * Migrate 'iso_create' events to scheduled products * Show scheduled products from new table in UI * Test async flag for iso post route * Move constants for job dependencies to their own file * Add API route to query scheduled product * Add async flag to ISO post route and table to keep track * Cleanup scheduling functions ------------------------------------------------------------------- Sun Mar 31 17:49:35 UTC 2019 - coolo@suse.com - Update to version 4.6.1553599469.57d6b3ca: * Properly escape bug titles * YAML output for job templates and schema validation (#1999) * Make the websocket server testable as a normal Mojolicious application * Add a model class to store worker state * Move more functions into helpers * Move dbus functions into helpers * Add controllers to the websocket server application * Install test distri deps on bootstrap * Turn websocket server into a real Mojolicious app * Use Mojo::Promise in enqueue_and_keep_track * docker: Add IO::Scalar dependency * doc: Reflect renaming virtio_screen.pm to serial_screen.pm * worker: Upload serial_terminal.txt for svirt backend * Unify schema access in OpenQA::WebAPI and fix the issue where the db helper was called on the application * Use result_p to track Minion job instead of timers * Improve logging non-scalar error info of GRU/Minion tasks * Extract common code for saving and deleting needles * Fix test for dependency graph * Don't let needle-related tasks fail if the cause is uncritical * Make needle deletion a Minion job * Setting to not cancel parallel parents with still-pending children * Add hints to fix dependencies when using devel ------------------------------------------------------------------- Fri Mar 15 05:22:32 UTC 2019 - coolo@suse.com - Update to version 4.6.1552567428.80082fe5: * Test create and delete events for job templates * Unify schema access with a singleton method * Use route placeholder types whenever possible for cleaner routes * Add link to relevant documentation in the job group settings * Cache result of OpenQA::..::JobGroup::load_properties * Allow disabling bug carry over on job group level * Remove dead validation code and replace it with route placeholder types * Pass affected_rows correctly in openqa_jobtemplate_create ------------------------------------------------------------------- Tue Mar 12 21:22:25 UTC 2019 - coolo@suse.com - Update to version 4.6.1552406543.b3e49dcb: * Remove check for $OpenQA::Utils::app * Move get_bug from Result::Bugs to ResultSet::Bugs * Add bug to db already on comment create or update * Add support for links to jira tickets * Rename tmp_schema to search_path_for_tests, which is a little more descriptive * Allow monitoring of Gru/Minion with a new public /admin/influxdb/minion API endpoint * Log job group events via audit log * Make openqa-webui.service depend on openqa-gru.service ------------------------------------------------------------------- Sun Mar 10 13:22:04 UTC 2019 - coolo@suse.com - Update to version 4.6.1552078716.6b292c9a: * Test deleting job from worker's job history * Move another class into its own .pm file and do not use private functions/methods across modules/classes anymore * Remove all uses of the enterprise operator and use a more consistent coding style in the affected classes * Add loading indication when saving needle ------------------------------------------------------------------- Fri Mar 08 05:22:22 UTC 2019 - coolo@suse.com - Update to version 4.6.1551963862.0a6b30be: * Prevent deletion of worker when deleting its current job * Prevent broken rendering of unknown result types * Add unxz/xz to apparmor profile to fix poo#48743 * Don't hardcode remote and branch names for saving needles ------------------------------------------------------------------- Tue Mar 05 21:22:30 UTC 2019 - coolo@suse.com - Update to version 4.6.1551797932.a5aadf24: * Remove safe_call from OpenQA::Utils since it is no longer needed * Cover more Gru special cases with tests * Remove resource allocator from architecture diagram * Mention the possible race condition in a comment * Bypass Gru special cases for non-Gru tasks * Use a completely new task for testing so we can replicate more edge cases in the future * Be a little more defensive with removing Gru tasks * Allow pure Minion jobs (without a Gru id) to bypass special case handling * Do not delete Gru tasks if they are not finished in Minion yet ------------------------------------------------------------------- Sat Mar 02 17:37:11 UTC 2019 - coolo@suse.com - Update to version 4.6.1551454083.e27d66d8: * Extend apparmor profile for all Git commands we use * Correct limit_assets task arguments * Allow test vars for openqa-clone-custom-git-refs * Fix updating needle Git repository * Fix uniform box widths of dependency graph in current Firefox * Move JSON::Validator and YAML::XS to t_requires * Drop the `cpanfile` changes * Add YAML::XS and JSON::Validator dependencies * worker: Skip announcing stop if isotovideo already exited * worker: Fix "final status update" * worker: Refactor _stop_job_kill_and_upload * worker: Remove extra handling for dead children * worker: Make API calls when stopping job non-critical * worker: Treat upload failures as API failures * Raise an exception if deploying the SQLite database fails and handle an existing database more gracefully * Do not reset the job queue every time the cache service is restarted * Test cache service migrations * Use Mojo::SQLite migrations to handle schema upgrades in the cache service gracefully * Remove unused imports and IPC objects * Remove the resource allocator * Move barrier handling out of the resource allocator * Move mutex handling out of the resource allocator * Remove first dbus method and use OpenQA::Schema directly * Declare the DBIx::Class dependency properly * Move OpenQA::Resource::Jobs handling from the resource allocator to the webapi ------------------------------------------------------------------- Tue Feb 26 21:22:42 UTC 2019 - coolo@suse.com - Update to version 4.6.1551101052.7f891940: * Use well-defined worker config for fullstack test(s) * Adapt tests to default WORKER_CLASS * Assign default WORKER_CLASS when posting job * Assign default WORKER_CLASS when scheduling ISO ------------------------------------------------------------------- Sat Feb 23 17:39:53 UTC 2019 - coolo@suse.com - Update to version 4.6.1550673294.3fb7464e: * worker: Log error returned by JSON API * Render error in all cases in create_artefact ------------------------------------------------------------------- Tue Feb 19 14:11:16 UTC 2019 - coolo@suse.com - Update to version 4.6.1550584043.5098c32d: * Log isotovideo startup more detailed ------------------------------------------------------------------- Tue Feb 12 21:22:24 UTC 2019 - coolo@suse.com - Update to version 4.6.1549966436.c3938033: * Revert support for ASSET_\d+_URL * openqa-clone-custom-git-refspec: make use of GROUP var * Improve error handling when changing priority ------------------------------------------------------------------- Sun Feb 10 13:22:09 UTC 2019 - coolo@suse.com - Update to version 4.6.1549799415.315b994a: * Fix modules filter wrong results while using comma * Show bug reporting links also on text results * Prevent download tasks for the same asset running in parallel * Improve Minion error handling when saving needle * Allow to run Minion jobs in UI tests * Migrate intermediate database changes * Don't use might_have relationship for job scenario * Show externally skipped module count in resultstats * Use same style for needle editor warnings as for regular flash messages * Save needles via Minion * Format Git errors more nicely when committing needle * Rebase needle directory before commiting changes * Use commit_git_return_error in Step.pm to propagate error * Skip result skipped on calculate overall result ------------------------------------------------------------------- Tue Feb 05 15:29:17 UTC 2019 - coolo@suse.com - Update to version 4.6.1549379724.2b90641c: * Split configured hosts by whitespace * stopping job: Proceed on errors during announcement * Let devel mode fullstack test also pass on state cancelled * stopping job: Change order of POST requests * Prevent error about cmd srv connection when test ends * Make code for engine startup more clear * Add architecture diagram ------------------------------------------------------------------- Sun Feb 03 07:29:14 UTC 2019 - coolo@suse.com - Update to version 4.6.1549026855.08485c7c: * Remove SQL files for 2017 * Update serial terminal function (#1982) * Remove image migration jobs that were only used for very old migrations * Lift the parallel job limit for the gru worker and add more fine grained limits to individual tasks ------------------------------------------------------------------- Thu Jan 31 12:36:58 UTC 2019 - coolo@suse.com - Update to version 4.6.1548936964.47070d70: * spec: Fix missing mention of 'openqa-clone-custom-git-refspec' in script/ * Add test check for scenario within error reporting * docs: Add documentation for triggering jobs based on any git refspec * Add script 'openqa-clone-custom-git-refspec' * Consider parent group comments for important builds * Cache result of function call * Add scenario description to infopanel of test * Fix typo in reference to DeveloperSessions * Fix another regression from c9735699 when no sync is needed * apparmor: Fix access to git templates * Switch to four spaces as requested, modify error ignoring * Fix flaky test regarding needle update * Improve error handling in worker cache /status route * openqa-bootstrap-container: do more error checking, fix a hang ------------------------------------------------------------------- Sat Jan 26 02:08:43 UTC 2019 - coolo@suse.com - Update to version 4.6.1548420629.e8e80a4b: * Fix regression of test caching failing introduced by c9735699 * worker: Fix two perl warnings ------------------------------------------------------------------- Wed Jan 23 17:22:28 UTC 2019 - coolo@suse.com - Update to version 4.6.1548078204.c9f60161: * Use nicer syntax for default vars * Add (carried over) bugref to job.done AMQP event ------------------------------------------------------------------- Fri Jan 18 17:28:35 UTC 2019 - coolo@suse.com - Update to version 4.6.1547830304.62f9aaaa: * Allow overwriting container bootstrap repo * Remove Failed modules filter from UI * Add UI tests for filter by module * Add search by module in all jobs of job group * Prevent github to think we have perl6 code * Better check for existing DBs in bootstrap * bootstrap: make errors fatal and deal with re-runs * apparmor: Further fixes for custom git clones * Fix external reporting if scenario description not present * Improve code for scenario description and add test * Add test suite description for bug reporting * apparmor: Fix custom git repo clone "env" denied * clone_job: Add convenience option '--within-instance' * Also set assigned jobs to parallel_failed (#1951) * Fix a few issues reported by jshint * Prevent the live log from becoming too long * Handle undefined `$text` in `find_bugref` and `find_bugrefs` * Wrap text in live log so long lines are readable * Allow custom CASEDIR/PRODUCTDIR/ASSETDIR ------------------------------------------------------------------- Thu Jan 10 10:08:18 UTC 2019 - coolo@suse.com - Update to version 4.6.1547114886.81dabdea: * 21-needles.t: Use `discard_changes` instead of re-calling `find` * Add bootstrap hint for Leap 15.0 * Fix "Can't locate object method new via package JSON" introduced by 95cb6601 * Check for correct perltidy version on execution * Fix if condition and leap repo prios for bootstrap * Improve a few details in worker code * worker: Improve handling API failures * Handle unexpected errors on status update more nicely * worker: Fix handling errors caused by connection issues with web UI * Show currently executed function in developer mode * Allow to pause on the next command ------------------------------------------------------------------- Sat Dec 22 13:41:18 UTC 2018 - coolo@suse.com - Update to version 4.6.1545406149.53968c1e: * Activate strict and warnings for all tests and do not use indirect object syntax * Add tool to bootstrap openQA with only one command * Fix comments and reference the usage messages of the minion job/worker commands in usage messages * Test setting last needle update * Add examples for run and list commands * Add Minion tests and make the run command a subclass of the worker command * Use better variable names in t/21-needles.t * Prevent wrong 'New needle ...' notifications in needle editor * Rename Docker-related targets and scripts * Improve documentation about running tests without Docker * Remove obsolete docker/testing * Improve documentation regarding Docker * Use --oneshot to make it more obvious what is happening * Improve job restart * Allow submitting margin and match level via return key * Hide tooltips when hiding developer mode elements * Remove 'Cancel job' button from developer mode * Reuse many more parts of Minion for Gru * Add a new OpenQA::WebAPI::Command namespace so there is no need for hidden classes in plugins ------------------------------------------------------------------- Thu Dec 20 05:41:42 UTC 2018 - coolo@suse.com - Update to version 4.6.1545230885.56b79eeb: * Use OPENQA_TEST_IPC mock in 37-limit_assets.t (#1935) * Reformat with new perltidy version * Adjust comment for pauseOnScreenMismatch * Allow test in devel mode fullstack test to pass * Remove 'Error from livehandler daemon' prefix * devel mode: Test version check * devel mode: Check whether os-autoinst is compatible * devel mode: Allow to pause on check_screen timeout * Improve and fix asset caching doc * Fix typos * Makefile: Prevent docker "build" and "run" to run in parallel * Add basic documentation for OpenQA::Events ------------------------------------------------------------------- Mon Dec 17 13:44:07 UTC 2018 - coolo@suse.com - Update to version 4.6.1545054233.21210ce3: * Use a fixed version of perltidy * Prevent 'Summary of HASH(...)' if multiple distris/versions specified * .travis.yml: Delete not necessary stage overrides * docs: Make commands to call more prominent with backticks * Makefile: Delete unused target docker-test-travis * Do not set the MOJO_LOG_LEVEL environment variable by default for more granular logging * Reformat everything for the latest version of perltidy * Support filtering by test in group overview again * worker: Validate job sent by web UI "job_grab" * Use localized signal handling instead of global sigtrap for modules * Set transaction isolation level for asset cleanup accordingly * asset cleanup: Check whether $res_max_job is set before comparing * Fix copyright year * Add license to new module * Move memory limit handling into a plugin * Move all global events to OpenQA::Events * Add OpenQA::Events module to replace random events being added to Mojo classes * Improve documentation for schedule control * Extend and improve asset documentation * Don't treat UEFI_PFLASH_VARS as asset if path is absolute ------------------------------------------------------------------- Thu Dec 13 05:40:52 UTC 2018 - coolo@suse.com - Update to version 4.6.1544609268.19a4cf68: * Fix test for popovers in 15-admin-workers.t for real * Improve "Refusing 'grab_job'" debug message * Test worker broken state * Add 'broken' state to the worker * Prevent worker to accept jobs if caching not available * Update specfile line for restarting apparmor ------------------------------------------------------------------- Tue Dec 11 07:06:36 UTC 2018 - coolo@suse.com - Update to version 4.6.1544510066.913d8f26: * Update license of test spec-file to current SPDX format (#1915) * Use the boring standard return value for all Perl modules * Merge hashes properly * Test handling of assets with invalid name * Prevent registering empty assets * Skip cleanup of assets with empty name ------------------------------------------------------------------- Fri Dec 07 09:40:53 UTC 2018 - coolo@suse.com - Update to version 4.6.1544166689.47296c23: * Disable INSTALL_FROM_CPAN by default (#1863) * Don't leave a limit_assets task hanging after 14-grutasks.t (#1882) * Do not require perlcritic on sle-12 (#1910) * When detecting asset keys, don't use the _URL form for ASSET (#1907) * Add Perl::Critic::Freenode to the test container (#1906) * Fix devel fullstack mode test * Make devel fullstack test more verbose * openQA.spec: Add missing runtime requirement for client (#1905) * doc: Mention select_serial_terminal() in serial terminal doc * Test rendering external table * Don't show categories in table for external results * Add filter for external results table * Use circle to display results in external table * Add separate table for external results from parser * Move logic for test detail steps from view into controller * Fedmsg: use Python 3 logger command * Extend documentation on the developer mode panel * Add WORKER_HOSTNAME to workers.ini template * Document re-needling workflow ------------------------------------------------------------------- Mon Dec 03 10:10:38 UTC 2018 - coolo@suse.com - Update to version 4.6.1543830229.359b5dd9: * Add Perl::Critic::Freenode to openQA.spec * Use Perl::Critic::Freenode to determine good coding style * Make all modules pass Perl::Critic::Freenode with severity level 4 (stern) * Activate strict and warnings consistently for all modules * Fix network dependency for cache service * Remove DEBUG and LOGDIR environment variables and change cache services to use the test mode * Test skipping timeout in developer fullstack test * devel fullstack test: Don't fail if renamed needles already there * devel mode: Allow to skip assert/check_screen timeout * Remove srand workaround that does nothing anymore since Mojolicious 5.76 (2015) ------------------------------------------------------------------- Wed Nov 28 11:53:44 UTC 2018 - coolo@suse.com - Update to version 4.6.1543400978.50e006bc: * Fix worker-cacheservice failing on startup due to no network * docs: Extend development tricks with dynamic test override * Fix typo in installation documentation * Replace Mojolicious::Commands with Mojo::Server::Daemon in test modules, make sure all test modules use strict, and fix a few issues caused by not using strict before * Restart openqa-worker-cacheservice on failure * Allow caching of externally downloaded assets ------------------------------------------------------------------- Mon Nov 26 21:40:33 UTC 2018 - coolo@suse.com - Update to version 4.6.1543249675.02bb8eff: * Exit worker with non-zero return code in error cases * Do not import from_json, there is already an inherited method with the same name * Add comments to point at Mojo::JSON being used to set up booleans * Debug 25-cache-service.t * Make perltidy happy * Convert a few cases back to Cpanel::JSON::XS that require special configurations (mostly for tests, but there might be underlying protocol requirements too) * Document steps to debug developer mode setup * Convert most uses of Cpanel::JSON::XS to Mojo::JSON, which will take care of using an optimized encoder and avoiding boolean warnings * Use fixed header for test result overview table * Check for JavaScript errors while waiting for result panel * Prevent selenium test to accidentally click on the VNC notice * Move note about VNC into developer panel * Update existing unique flash messages * Expand devel controls by default if paused and owning the session (#1880) * Add rsync BuildRequires and Recommends to RPM spec (#1881) * Add a new is_admin_js helper to generate JavaScript booleans for use in templates * Add a test for browsing the admin pages in readonly mode * Use OPENQA_TEST_IPC in developer mode unit test (#1868) * Add test to check link for entire build * Add link to test results for multiple job grops * Call _reschedule after schedule concluded * Remove unused variable $force in _reschedule() * Fix comment in scheduler code * Fix various uses of the is_admin helper in templates to generate JavaScript that resulted in empty lists, because the helper returns a Perl boolean value instead of a JavaScript boolean * Revert "Fix deprecation warnings regarding Mojo::Transaction::error" * Fix line break in failed module box * Fix formatting * Fix deprecation warnings regarding Mojo::Transaction::error * Combine multiple distri/version/flavor/build filter with logical or * Fix filter for architecture * Upload video timestamp file when it exists (#1873) * Fix formatting * Use Mojo::File::slurp in check_tabmode * Improve error handling of Step controller * Handle `app` and `tx` attributes of Controller being weakened * Do not check group name by dragging group * Minor fixes for WritingTests doc (#1865) * Add 2 new dependencies to the travis container: (#1864) * Update all uses of Mojo::Transaction::success to their Mojolicious 8.06 equivalent * Fix a few new uses of tempdir * Fix various incorrect uses of Mojo::File::tempdir * docs: Extend definition of *softfailed* to "known issues" (#1837) * Refuse updating job group name with empty or blank (#1844) * Apparmor: Add /dev/hugepages (#1857) * Fix broken in section of inter machine dependencies (#1859) * Wrap up and polish * Do not use wantarray in GRU's enqueue() * Update POD OpenQA::Worker::Cache::Client's SYNOPSIS * Avoid to race to get available minion workers * Drop PID from log * Move Minion workers under a single service * Cleanup * Fix typo * Adapt gru tests * Generate same lock for asset/host in the request object * Fix rsync output in minion task * Refactor and update Documentation * Use the cache service to perform rsync tests caching * Create Minion task for rsync execution * Use the host folder to cache tests/needles * Do not check if child is running if it's not there * Reduce the lock expiration time * Trim spaces from left and right in assets name * Hold the db handle during transactions * Add test for Minion task registration and execution * specfile: DBD-SQLite needs to be > 1.51, required by Minion::Backend::SQLite * Add tests and cover also subprocess execution * Add function to generate session token * Read and set limit from config file, defaults to 50GB * Makefile: install Worker Cache Service systemd unit * Add POD for Cache Service and extend docs/Installing.asciidoc * Add service file and adapt openQA.spec * Avoid useless quotes on hash key * Debug - adapt compile-check-all * Address to PR comments - also add test for checking output in cache-service.t * Drop unused function * Write cache output into os-autoinst log * Make possible to retrieve cache output * Add debug messages while enqueuing download * Asset path needs to be a string for vars * Make dequeue and status post * Adapt fullstack to cache path change * Extract domain parsing to _base_url and use it also in the client to resolve asset path * Adapt tests * Get the cache path right * Incomplete job if cache service is down * tests: Wait also for worker minion in fullstack * Use defaults in isotovideo engine * Update defaults and remove protoyped function from Cache::Client * tests: Add test for available_workers() * Add available_workers() to Cache::Client * Fullstack: move sqlite cache check up * Cache: make retry() return always the response object * Fullstack: Wait cache to be ready * Adapt tests and utils * Drop unneeded default * Use OpenQA::Worker::Client inside the cache task * Start to adapt fullstack test * Adapt to PBP * Add dep on Minion::Backend::SQLite * Move common testing functions to Test::Utils as they will be reused also in fullstack test * Add Mojo::SQLite dependency * Use the Cache service within the worker * Add accessors to retrieve asset path and check if exists * Remove unneded log line and cleanup check_limits() * Finishing porting the Cache.pm code to Mojo::SQLite * Rework cleanup strategy, avoid endless loops - update etag fails now * Use Minion Admin plugin with the cache service * Drop downloading, Cache.pm now records just etag and size * Use Devel::Cover in cache-service tests to enhance coverage * Adapt cache tests - still fails on size calculation * Move Cache.pm code to Mojo::SQLite * Add dequeue() and tests for it * Fix and test enqueued tasks * Do not use list_jobs from Minion to gather statuses, causes races * Add default usage tests and use minion lock to detect if it's in use * Fix test copyright header * Add standard client usage * Add Cache service client * Do not lock anymore on assets * Add test for cache service * Add script to start cache service * Create separate class for Minion Server and worker * Do not log an error when trying to unlink an nonexisting file (#1854) * Prevent filter on test overview to reveal old jobs * Allow to create dependency for jobs on different machines * Test that filtering overview does not reveal old jobs * Add dependency for test in os-autoinst (#1851) * Handle skipped test result in IPA parser * Show bug title on mouseover * Fix job templates JavaScript * Fix typo in networking documentation * Regenerate docs * devel mode: Suppress connection errors if no module running * Make 13-admin.t more verbose in case of failure * Use $LEAP_VERSION in documentation consistently * Use 'not yet' instead of 'never started' on /admin/workers/X (#1841) * Extend unit tests for developer mode UI * Don't replace preview container when updating results * Load new test details when paused in the middle of module * Add START_AFTER_TEST/PARALLEL_WITH to dep graph tooltip (#1840) * Allow to scroll up and down using arrow keys on test details (#1842) * Fix testhelper wait_for_session_info * Lazy-initialize devel mode when switching to live view tab * Make 'can't remove ...' warnings debug messages (#1836) * Create new db schema version * Add tests for test module flags * Add special icon for the always rollback flag * Add always_rollback flag description * Add regex filter for admin tables (#1831) * Simplify Schema::Result::Jobs::running_modinfo * Test progress bars on /tests * Fix progress bars on /tests * Fix prio field in /tests and add a test for it (#1834) * Hide cloned jobs from job dependency graph (#1833) * Add route to poll stats of running and scheduled jobs (#1829) * Allow softfail and softfailed as step result * Prevent rendering undef/refs as resborder * Move errors regarding developer mode into the devel panel (#1830) * Test error handling of status/image/progress upload * Recompute asset status after cleanup for /admin/assets * Fix misleading error message on non-UEFI jobs (#1825) * Save asset status cache under /var/lib/openqa/webui/cache * Don't serve assets table while cleanup is ongoing * Update apparmor profile for current Tumbleweed for x86 workers (#1827) * worker: Stop job if considered dead by the web UI (2) * worker: Stop job if considered dead by the web UI (#1824) * Prevent multiple limit_assets tasks to run in parallel * Spawn limit_assets task in a consistent way * Polish a few details on the /admin/assets table * Allow to trigger asset cleanup task on /admin/assets * Prevent warning when rendering assigned jobs on 'All tests' page (#1823) * Fix issues noted by jshint * Make displaying assets by group faster * Set asset table width to 100% * Show last update of assets table * Improve speed of /admin/assets * Fix dependencies of openQA-client package (#1819) * Use less-ugly gravatar icons already used in OBS and progress (#1822) * Fix padding of tabs in testresults * Polish code for dependency graph * Clean CSS * Ensure filter icon and last tab button don't overlap * Add help text to dependency graph * Rename part_of_cluster to has_dependencies * Remove old dependency table * Test dependency graph * Improve appearance of dependency graph * Compute cluster for dependency graph * Fix clipping and alignment of dependency graph * Make dependency graph a bit more fancy * Display dependency graph using dagre-d3 * Handle logging errors (#1815) * Use italic font to display default prio (#1818) * Fix openQA-client package for openSUSE (#1816) * Stop worker when an unhandled exception occurs (#1809) * Limit new needles sent to os-autoinst on resume to 100 * Use Mojo::File in t/ui/07-file.t * Add 'perl(File::Touch)' to Dockerfile * Test needle download * Allow command server to download new needles on resume ------------------------------------------------------------------- Wed Oct 03 06:37:58 UTC 2018 - coolo@suse.com - Update to version 4.6.1538413557.f9e2a131: * Adapt tests to logging behavior of Mojolicious 8 (#1814) * Use transaction when computing the asset status * Make only cleanup fail on inconsistent asset status * Print max job when detecting ongoing scheduling during cleanup * Fix removal of scheduling assets * Document asset cleanup ------------------------------------------------------------------- Sat Sep 29 18:38:25 UTC 2018 - coolo@suse.com - Update to version 4.6.1538057874.2d8d0618: * Update AppArmor profile for Leap 15.0 (#1810) * Allow overwriting job priority when posting isos (#1807) * Dump the JSON templates as ascii * Fix jobs fixtures * Increase debugging information in asset removal * Round the remaining days to make tests less blinky * Fix wrongly assigned states in existing data * Prevent setting the state to SKIPPED * Test whether t_finished is assigned when setting a final job state ------------------------------------------------------------------- Wed Sep 26 06:38:31 UTC 2018 - coolo@suse.com - Update to version 4.6.1537939546.7aa7ed58: * all tests page: Fix parameter passing for timestamp rendering (#1806) * Make asset cleanup more verbose (#1803) * Fix 'last_match' param of AJAX route for needle table (#1802) * Use AJAX for all tables on /tests * Don't pre-compute IDs for finished jobs when loading HTML * Optimize all tests page * doc: remove mention of openQA-devel package (#1800) * Make sure multimachine jobs get a worker eventually * Some code restructing in the scheduler function * Delete empty settings in clone job * Display progress bar when downloading asset * Cache infos about scheduled jobs * Limit logs and results for groupless jobs (#1798) * Release job networks more often (#1795) * Fix removal of untracked assets (#1796) * Enable the scheduler on updates (and the webui is enabled) (#1785) * docs: Write Leap installation repo in more timeless way (#1791) * Run inserting screenshots in a transaction (#1792) * Update AppArmor profile to allow worker to use cgroups * Set specific slice for worker service * Better detect of systemd slice * Inform worker about devel session to reduce posting status info (#1789) * Correct typo (#1790) * Require RabbitMQ::Client 0.2 * Make AMQP sending less fragile * Make web socket and livehandler daemons listen on IPv4 and IPv6 (#1787) * Prevent worker to endlessly retry non-critical API calls (#1788) * Fix typo in t/api/02-iso.t * Make asset download feature usable by both "jobs post" and "isos post" * docs: Cover different asset types and asset options ------------------------------------------------------------------- Thu Sep 13 15:13:01 UTC 2018 - coolo@suse.com - Update to version 4.6.1536756702.f75303c1: * Improve error log of AMQP plugin (#1784) * Treat failure when posting upload progress as non-critical * Allow regular HTTP and WS being passed though reverse proxy * Fix random failures in developer mode unit tests * Add unit test for posting progress upload * Add HMAC time stamp also on livehandler daemon * Store upload progress in database * Prevent use of uninitialized value when posting upload progress * Check for upload progress in developer fullstack test * Use API authentification for post upload progress route * Test handling of upload progress in UI * Show needle editor button not unless upload complete * Pass upload progress from worker via liveviewhandler to browser * Open needle editor in developer fullstack test * Fix one more corner case for blocked_by (#1781) * Simplify reschedule code * Make the overview page responsive again (#1780) * Sort scheduled jobs via database (#1779) * Fedmsg plugin: emit 'standardized' fedmsg (#1758) * Fix blocked jobs not appearing in scheduled table * Increase max execution time per job in fullstack test * fullstack.t: Distinguish between not found and wrong permissions * Add more checks in 15-comments.t to debug random failure * OpenQA::Parser::Format::JUnit include error and skipped details (#1778) * Update serial failures detection documentation * Prevent asset-releated tests being disturbed by unrelated files * Test computing asset status * Optimize querying asset status for admin table * Allow rendering asset table as JSON ------------------------------------------------------------------- Sun Sep 09 11:03:30 UTC 2018 - coolo@suse.com - Update to version 4.6.1536432963.ac9f540e: * Fix the vlan assignment and release * Remove debug that prints to STDERR * Clean up 05-scheduler-dependencies * Take the full cluster into account, not just the first level * Avoid undefined warnings from rescheduling * Do not reschedule cluster jobs multiple times * Do not require websocket in scheduler (#1777) * Extend the test coverage for blocked_by * Fixed blocked_by calculation not to create loops * Schedule the jobs manually in fullstack test * git clone only the HEAD of os-autoinst * Do only copy back the cache on travis * Make 2 debug messages in WS server info * Wait for the scheduler to reshedule job 3 * Die early in full-stack test * Don't output 'Scheduling new jobs' if there are 0 free workers * Add some means to debug full-stack test * Only talk about cpanm if it's actually true * Run chromium with --no-sandbox not to require admin caps * Revert PR 1747 * Allow to enable monitoring using mojo-status (#1763) * Catch misconfigured test suites creating cycles (#1775) * Make tidy fast by only looking at changed files (#1773) * Ignore file docker.env created by make target docker.env * Update docker-related documentation * fullstack/Docker test: Skip os-autoinst build only conditionally * fullstck test: Improve ignoring JS errors * fullstack test: Prevent auto page reload when waiting for result panel (#1770) * Improve help popover for job group/template prio * Prevent Firefox to misalign arrows of prio input * Disable job template prio editor for non-admins * Test using default prio from job group when scheduling ISO * Add tests for setting prio via job templates API * Improve error msg and response for update of job template API * Allow job template to inherit prio from group * Allow amending job template priority * Use default prio when adding new job template to existing medium * Avoid using Mojo::ByteStream directly when rendering JSON * Adapt t/16-utils-runcmd.t for e2771ee * Fix sorting worker jobs by finished * Make 33-developer-mode.t more stable * Prevent use of uninitialized $current_running in Worker/Jobs.pm * Test opening needle editor for running tests * Improve 404 error pages of needle editor for running jobs * Test pausing on assert screen timeout * Consider only new console messages in fullstack test * Adjust upload_status for pausing on assert_screen timeout * Fix typo occured -> occurred * Test UI changes for pausing on assert_screen timeout * devel mode: Hide executed modules when test status updated * devel mode: Show reason for pause * needle editor: Show 'back to live view' if in devel mode * devel mode: Allow to open needle editor when paused * devel mode: Add UI elements for pausing at timeout * Use git -C for commits * Don't check for .git directory when making git commits ------------------------------------------------------------------- Tue Aug 21 18:55:46 UTC 2018 - coolo@suse.com - Update to version 4.6.1534873283.be929573: * Remove entrypoint from specfile sources * Makefile: Build env file for docker with needed environmental variables * Fix Mojolicious version requirement * Emit parent group ID in AMQP comment event * Use subtests in AMQP test * Makefile: Avoid also to parse env with spaces to travis * tests: Fix builds on master * Adapt fullstack tests to changes in os-autoinst * Make javascript_console method less verbose when there are no errors * test: disable broken test * Do not call unblock introduced in 184d0c5d5 * Revert "Calculate blocked by jobs on creation in job resultset" * Do not display blocked count * Do not calculate blocked jobs * Drop unused import * Revert "Merge pull request #1729 from mudler/blocked_by" * Revert "Fix usage of $blocked in test results template (#1740)" * Revert "Merge pull request #1741 from coolo/fix_blocked_by" * Makefile: cleanup * tests: INSTALL_FROM_CPAN by default * docker: Pass environment to docker * docker: try to build our image with package upgrades and custom entrypoint while running tests * docker: Upgrade container packages before running tests * Fix suggested command for adding devel repo ------------------------------------------------------------------- Fri Aug 10 15:29:12 UTC 2018 - coolo@suse.com - Update to version 4.6.1533914931.14b73490: * Use the transaction to check if it's empty (and avoid warnings) * Upload: Keep the transaction around * Calculate blocked_by status of restarted jobs * Remove systemd dependencies on scheduler * Fix usage of $blocked in test results template (#1740) * Make limit for keeping untracked assets configurable * Test inheritance and overriding of job/parent group properties * Read default job/parent group properties for cleanup from settings * Update openQA documentation regarding serial failures detection * Make UEFI_PFLASH_VARS non critical on local workers * Print autoinst-log.txt after Travis testrun * Fix full-stack.t * Adapt test of group overview JSON route for parent comments * Test tagging on parent group level * Test commenting on parent group level * Allow pinned comments on parent group overview * Support commenting on parent-group level * Test overrides for job settings passed to clone job script * Reduce indentation in clone_job_apply_settings * Require at least IO Socket SSL 2.009 * Pass worker setting WORKER_HOSTNAME to web UI * Add tests for comments carryover if module contains bugref in details * developer mode: Don't show same error twice * Unify disable_bootstrap_animations and disable_bootstrap_fade_animation * Carry comment over if test suite failed with same serial failure * developer mode: Show flash messages on errors * Test processing state changes from os-autoinst * Consider fullpath as a valid asset * Minor refactor for clone_job.pl and add unit test * Add global settings support for clone job * Treat UEFI_PFLASH_VARS as non critical asset * Change striped name from IPA results * Remove debug leftover * Highlight blocked jobs with orange colour * Do not hide scheduled jobs if blocked * Calculate blocked by jobs on creation in job resultset * Release vlan networks only if other jobs in the same cluster aren't running anymore * Disable worker shuffling in 05-scheduler-* tests * Add scheduler function to disable worker shuffling during computation * Adapt job_grab tests * Adapt scheduler fullstack test to latest changes * Follow the rename of get_websocket_api_version * blocked_by calculation: Don't block on self * Fix timer reset on quit * Calculate blocked_by state later in the process * Reset the scheduler tick if not summoned * Simply scheduler logic - but a lot * Fix blocked_by for chained parallel jobs * Disable pp in IPC - a little too noisy on allocating * Filtering for scheduler * Prepare a hash of jobs * Select all scheduled jobs and filter them for free workers * Remove BUSY_BACKOFF and CONGESTION_CONTROL * Remove CAPTURE_LOOP_AVOIDANCE option * Remove OPENQA_SCHEDULER_FIND_JOB_ATTEMPTS option * Add UEFI_PFLASH_VARS to detect_asset_keys * Mark all restarted (cluster) jobs in UI * Add IPA format to OpenQA Parser * Improve logging in LiveViewHandler * Assign os-autoinst tx not before actually connected * Ignore 'Data frame received after close' for now * Extend unit tests for live view handler * Refactor LiveViewHandler * devel mode: Prevent endless, instant reconnects on error * Mention how to actually enable openqa-ssl.conf under openSUSE * Use wss protocol for websocket connection when page loaded via https * Fix javascript errors on running.js (#1719) * Update README.md ------------------------------------------------------------------- Tue Jul 10 13:39:38 UTC 2018 - coolo@suse.com - Update to version 4.6.1531161673.f7fc21dd: * Remove docker-tests during install * Don't display blocked jobs as scheduled * Make registering assets transactional (#1711) * docs: Add note on sending new lines to a serial terminal (#1714) * clone_job: Make _GROUP and _GROUP_ID override each other (#1716) * Add blocked_by to jobs - calculated on job changes * WIP: Move commands to run docker tests to separate script * Make parser test less verbose * Make error messages in setup_share_dir more specific * Allow to use a custom os-autoinst checkout * Remove print() leftover from debugging * ws proxy: Add tx after last client-specific error handled * Don't fail full-stack.t when already assigned * Prevent use of global variables in LiveViewHandler.pm * Make developer mode accessible to operators (#1707) * needle editor: Assign new background image instantly * developer mode: Show only modules which are still ahead * Adjust overflow behavior of dashboard progress bars * Use _GROUP_ID in clone_job.pl * Refactor test code for jobs post * Test specifying group by ID on jobs post * live view: Connect to ws server when testStatus.running is set * Fix bugs in form controls for devel mode * Show developer mode form only when connected via web sockets * Fix help popover in developer panel * devel mode: Handle case when module list still unknown * Never assume it is the own devel session when not logged in * Don't update developer mode controls until confirmed * Rephrase to avoid 'developer session' * Make full-stack test more stable (#1701) * Replace 'opened by' with 'owned by' * Prevent resetting the module to pause at on page load * Show link to ws console only in development mode * Remove useless debug printing in live view handler * Don't use Test::MockModule::redefine() * Improve error handling of livehandler daemon * Test UI controls also in developer fullstack test * Cancel job when quitting the developer session * Don't quit developer session when disconnecting from os-autoinst * Add tests for developer UI controls * Fix importing determine_web_ui_web_socket_url * Start developer session via submit button * Set status code when finishing ws connections * Avoid using Mojo::IOLoop->one_tick() in developer unit tests * Adapt to removal of waiting state * Fix style issues mentioned by @foursixnine in review * Extend unit tests for status-only route * Fix broken HTML on test details page * Check that VNC instructions are shown when test is paused * Document variables for developer mode * Hide entire developer panel when test not running * Adapt existing developer test to UI changes * Update module select for devel mode like modules under details * Handle reconnects in the client * Notify clients about removed sessions as well * Keep track of status-only and devel ws tx separately * Show VNC instruction only for running jobs * Pick the right route (either developer session or status only) * Add UI controls for developer session * Move methods to build proxy url to OpenQA::Utils * Remove deprecated ui test * Allow developer-instructions to be modified by status updates * Add status route to websocket proxy * Fix typos * Connect via livehandler proxy to isotovideo to get updates * Update service file for livehandler service * Remove unnecesary options for livehandler service * Update instructions * Expose VNC port to developer * Add UI controls for developer session * Fix formatting of Contributing.asciidoc * Document running UI tests with Docker in non-headless mode ------------------------------------------------------------------- Tue Jun 26 12:54:52 UTC 2018 - coolo@suse.com - Update to version 4.6.1530017682.0f384112: * Allow 'assigned' state in fullstack.t (#1699) * Move constants from Schema::Result::Jobs to a separate package (#1696) * Remove leftovers of interacitve mode (#1697) * Adapt fullstack test for the logging changes * Adapt tests for the display of pid in the log in debug mode * Print pids of processes in logs if log level is debug * Avoid db locks on asset_lookup * Treat the case when we could not update asset in critical section * More protection for critical section in the cache * Add other cache tests and more refactoring * Try to bring coverage back * Add locking test, and remove useless query in fullstack.t * Reduce the critical section * Avoid Perl 5.18 complain * No need to import getcwd * Do not run the cache test in parallel * Reduce parallel processes for concurrency test to avoid travis to crash * Wrap also toggle_asset_lock in lock_section * Flip download = 0 from the same function used to toggle on/off * Make the cache test more stable - add also option for stress test * Add concurrent test for cache * No need to init explictly anymore * No need to export get_asset anymore * Adapt cache and cache test to more OOP style * Use transactions and wrap rollbacks in eval calls for further debug. Also enable exclusive transaction in toggle_asset_lock * Fix wrapping of parser results (#1695) ------------------------------------------------------------------- Tue Jun 19 18:37:47 UTC 2018 - coolo@suse.com - Update to version 4.6.1529433451.a70c6e82: * Remove running_or_waiting method * Remove waiting state that was leftover from 5e65496f * Ensure proxy is connected to os-autoinst in devel test * Allow status as a ws_proxy command * Make waiting for paused more robust against disconnects * Remove unused parameter in wait_for_developer_console_contains_log_message * Let the web socket console automatically reconnect * Ensure ws connection won't timeout in fullstack tests * Improve check for developer console in fullstack test * Remove link to developer console * Use docker image from OBS (#1685) * Let tmpfiles.d create the mount point "share" (#1689) * Add basic barrier documentation * Improve documentation about mutexes * Remove $isodir from Utils (only used in tests) * Remove /tests/X/iso route * Remove hdddir and otherdir from Utils * Fix waiting for development session * Don't fake D-Bus when running the devel fullstack test * Propagate os-autoinst URL from worker to web UI * Fix CSS for developer console * Fix path for liveviewhandler in Apache config * Adapt execution of developer test to latest changes * Fix perl critic error * Open the 2nd window/tab in devel mode test correctly * Handle errors happening before dev session correctly * Add unit test for developer mode * Remove unneccassary D-bus env for livehandler * Refactor live view handler to use regular methods * developer console: Handle case when port is NaN * Refactor full-stack.t and 33-developer_mode.t * Add test for developer mode to Travis test matrix * Add test for developer mode * Workaround broken syntax highlighting of full-stack.t in Kate * Allow to start Selenium driver directly via start_driver * Improve passing parameter to OpenQA::Test::Database::create() * Fix ID of developer session info * Increase inactivity timeout for openqa-livehandler * ws client: Broadcast messages to all JavaScript client * ws proxy: Try/catch only JSON decoding * Allow login as a 2nd administrator via fake auth * Make logout accessible via regular GET * Install service file for openqa-livehandler * Use transaction to delete development session * Move ws connection for developer session to its own service * Keep track of developer session, limit to one session per job * Validate commands before passing to os-autoinst * Make the websocket proxy route basically work * Allow to use websocket console for proxy route as well * Add basic structure for developer mode web socket route * Make fullstack test more verbose * Prevent warning in fullstack test * Test pausing and resuming jobs * Add web socket console for isotovideo commands * Display incomplete jobs in next&previous tab (#1682) * Update documentation with NAT configuration using firewalld (#1683) * Categorize UEFI PFLASH drives as HDD assets (#1684) * Try caching * Easier way to pass exit status from docker-tests * Change message level when cgroups cannot be created (#1681) * Fix typo in the travis.yml and fix the grep regular expression for the status checking (#1679) * Adjust popover margin on audit log (#1678) * Change from su to sudo in the docker image * Docker shell scripts cleanup * Allow where parameter for mutex unlocking (#1677) * Documentation for running tests in docker * Enable codecoverage * Add support for os-autoinst * Add DBUS support. * Force user permissions 1000 to be compatible with opensuse * Moved the DB from the host to the container * First release of the dockerfile to create container to run unit and integration tests * Remove job_set_running - as jobs are no longer waiting * Remove job_set_waiting - jobs aren't WAITING anymore * Remove current interactive mode ------------------------------------------------------------------- Sat Jun 16 05:41:15 UTC 2018 - coolo@suse.com - initial package