in 21.08 tree.
* CVE-2021-43337:
For sites using the new AccountingStoreFlags=job_script and/or job_env
options, an issue was reported with the access control rules in SlurmDBD
that will permit users to request job scripts and environment files that
they should not have access to. (Scripts/environments are meant to only be
accessible by user accounts with administrator privileges, by account
coordinators for jobs submitted under their account, and by the user
themselves.)
- changes from 21.08.3:
* This includes a number of fixes since the last release a month ago,
including one critical fix to prevent a communication issue between
slurmctld and slurmdbd for sites that have started using the new
AccountingStoreFlags=job_script functionality.
OBS-URL: https://build.opensuse.org/package/show/network:cluster/slurm?expand=0&rev=193
- update to 21.08.2
- major change:
* removed of support of the TaskAffinity=yes option in cgroup.conf. Please
consider using "TaskPlugins=cgroup,affinity" in slurm.conf as an option.
- minor changes and bugfixes:
* slurmctld - fix how the max number of cores on a node in a partition are
calculated when the partition contains multi*socket nodes. This in turn
corrects certain jobs node count estimations displayed client*side.
* job_submit/cray_aries - fix "craynetwork" GRES specification after changes
introduced in 21.08.0rc1 that made TRES always have a type prefix.
* Ignore nonsensical check in the slurmd for [Pro|Epi]logSlurmctld.
* Fix writing to stderr/syslog when systemd runs slurmctld in the foreground.
* Fix issue with updating job started with node range.
* Fix issue with nodes not clearing state in the database when the slurmctld
is started with clean*start.
* Fix hetjob components > 1 timing out due to InactiveLimit.
* Fix sprio printing -nan for normalized association priority if
PriorityWeightAssoc was not defined.
* Disallow FirstJobId=0.
* Preserve job start info in the database for a requeued job that hadn't
registered the first time in the database yet.
* Only send one message on prolog failure from the slurmd.
* Remove support for TaskAffinity=yes in cgroup.conf.
* accounting_storage/mysql - fix issue where querying jobs via sacct
*-whole-hetjob=yes or slurmrestd (which automatically includes this flag)
could in some cases return more records than expected.
* Fix issue for preemption of job array task that makes afterok dependency
fail. Additionally, send emails when requeueing happens due to preemption.
* Fix sending requeue mail type.
* Properly resize a job's GRES bitmaps and counts when resizing the job.
OBS-URL: https://build.opensuse.org/request/show/923425
OBS-URL: https://build.opensuse.org/package/show/network:cluster/slurm?expand=0&rev=190
- updated to 21.08.1 with following bug fixes:
* Fix potential memory leak if a problem happens while allocating GRES for
a job.
* If an overallocation of GRES happens terminate the creation of a job.
* AutoDetect=nvml: Fatal if no devices found in MIG mode.
* Print federation and cluster sacctmgr error messages to stderr.
* Fix off by one error in --gpu-bind=mask_gpu.
* Add --gpu-bind=none to disable gpu binding when using --gpus-per-task.
* Handle the burst buffer state "alloc-revoke" which previously would not
display in the job correctly.
* Fix issue in the slurmstepd SPANK prolog/epilog handler where configuration
values were used before being initialized.
* Restore a step's ability to utilize all of an allocations memory if --mem=0.
* Fix --cpu-bind=verbose garbage taskid.
* Fix cgroup task affinity issues from garbage taskid info.
* Make gres_job_state_validate() client logging behavior as before 44466a4641.
* Fix steps with --hint overriding an allocation with --threads-per-core.
* Require requesting a GPU if --mem-per-gpu is requested.
* Return error early if a job is requesting --ntasks-per-gpu and no gpus or
task count.
* Properly clear out pending step if unavailable to run with available
resources.
* Kill all processes spawned by burst_buffer.lua including decendents.
* openapi/v0.0.{35,36,37} - Avoid setting default values of min_cpus,
job name, cwd, mail_type, and contiguous on job update.
* openapi/v0.0.{35,36,37} - Clear user hold on job update if hold=false.
* Prevent CRON_JOB flag from being cleared when loading job state.
* sacctmgr - Fix deleting WCKeys when not specifying a cluster.
* Fix getting memory for a step when the first node in the step isn't the
first node in the allocation.
OBS-URL: https://build.opensuse.org/request/show/919668
OBS-URL: https://build.opensuse.org/package/show/network:cluster/slurm?expand=0&rev=186
- updated to 21.08.1, major changes:
* A new "AccountingStoreFlags=job_script" option to store the job scripts
directly in SlurmDBD.
* Added "sacct -o SubmitLine" format option to get the submit line
of a job/step.
* Changes to the node state management so that nodes are marked as PLANNED
instead of IDLE if the scheduler is still accumulating resources while
waiting to launch a job on them.
* RS256 token support in auth/jwt.
* Overhaul of the cgroup subsystems to simplify operation, mitigate a number
of inherent race conditions, and prepare for future cgroup v2 support.
* Further improvements to cloud node power state management.
* A new child process of the Slurm controller called "slurmscriptd"
responsible for executing PrologSlurmctld and EpilogSlurmctld scripts,
which significantly reduces performance issues associated with enabling
those options.
* A new burst_buffer/lua plugin allowing for site-specific asynchronous job
data management.
* Fixes to the job_container/tmpfs plugin to allow the slurmd process to be
restarted while the job is running without issue.
* Added json/yaml output to sacct, squeue, and sinfo commands.
* Added a new node_features/helpers plugin to provide a generic way to change
settings on a compute node across a reboot.
* Added support for automatically detecting and broadcasting shared libraries
for an executable launched with "srun --bcast".
* Added initial OCI container execution support with a new --container option
to sbatch and srun.
* Improved "configless" support by allowing multiple control servers to be
specified through the slurmd --conf-server option, and send additional
configuration files at startup including cli_filter.lua.
OBS-URL: https://build.opensuse.org/request/show/915777
OBS-URL: https://build.opensuse.org/package/show/network:cluster/slurm?expand=0&rev=184
- Updated to 20.11.8:
* slurmctld - fix erroneous "StepId=CORRUPT" messages in error logs.
* Correct the error given when auth plugin fails to pack a credential.
* Fix unused-variable compiler warning on FreeBSD in fd_resolve_path().
* acct_gather_filesystem/lustre - only emit collection error once per step.
* Add GRES environment variables (e.g., CUDA_VISIBLE_DEVICES) into the
interactive step, the same as is done for the batch step.
* Fix various potential deadlocks when altering objects in the database
dealing with every cluster in the database.
* slurmrestd:
- handle slurmdbd connection failures without segfaulting.
- fix segfault for searches in slurmdb/v0.0.36/jobs.
- remove (non-functioning) users query parameter for
slurmdb/v0.0.36/jobs from openapi.json
- fix segfault in slurmrestd db/jobs with numeric queries
- add argv handling for job/submit endpoint.
- add description for slurmdb/job endpoint.
* slurmrestd/dbv0.0.36:
- Fix values dumped in job state/current and
job step state.
- Correct description for previous state property.
* srun:
- fix broken node step allocation in a heterogeneous allocation.
- leave SLURM_DIST_UNKNOWN as default for --interactive.
* Fail step creation if -n is not multiple of --ntasks-per-gpu.
* job_container/tmpfs - Fix slowdown on teardown.
* Fix problem with SlurmctldProlog where requeued jobs would never launch.
* job_container/tmpfs - Fix issue when restarting slurmd where the namespace
mount points could disappear.
* sacct:
OBS-URL: https://build.opensuse.org/request/show/903744
OBS-URL: https://build.opensuse.org/package/show/network:cluster/slurm?expand=0&rev=183
- Updated to 20.11.7 which fixes CVE-2021-31215 (bsc#1186024)
- New featuresi from 20.11.7:
* slurmd - handle configless failures gracefully instead of hanging
indefinitely.
* select/cons_tres - fix Dragonfly topology not selecting nodes in the same
leaf switch when it should as well as requests with *-switches option.
* Fix issue where certain step requests wouldn't run if the first node in the
job allocation was full and there were idle resources on other nodes in
the job allocation.
* Fix deadlock issue with <Prolog|Epilog>Slurmctld.
* torque/qstat - fix printf error message in output.
* When adding associations or wckeys avoid checking multiple times a user or
cluster name.
* Fix wrong jobacctgather information on a step on multiple nodes
due to timeouts sending its the information gathered on its node.
* Fix missing xstrdup which could result in slurmctld segfault on array jobs.
* Fix security issue in PrologSlurmctld and EpilogSlurmctld by always
prepending SPANK_ to all user-set environment variables. CVE-2021-31215.
- New features from 20.11.6:
* Fix sacct assert with the --qos option.
* Use pkg-config --atleast-version instead of --modversion for systemd.
* common/fd - fix getsockopt() call in fd_get_socket_error().
* Properly handle the return from fd_get_socket_error() in _conn_readable().
* cons_res - Fix issue where running jobs were not taken into consideration
when creating a reservation.
* Avoid a deadlock between job_list for_each and assoc QOS_LOCK.
* Fix TRESRunMins usage for partition qos on restart/reconfig.
* Fix printing of number of tasks on a completed job that didn't request
tasks.
* Fix updating GrpTRESRunMins when decrementing job time is bigger than it.
OBS-URL: https://build.opensuse.org/request/show/893087
OBS-URL: https://build.opensuse.org/package/show/openSUSE:Factory/slurm?expand=0&rev=59
- New featuresi from 20.11.7:
* slurmd - handle configless failures gracefully instead of hanging
indefinitely.
* select/cons_tres - fix Dragonfly topology not selecting nodes in the same
leaf switch when it should as well as requests with *-switches option.
* Fix issue where certain step requests wouldn't run if the first node in the
job allocation was full and there were idle resources on other nodes in
the job allocation.
* Fix deadlock issue with <Prolog|Epilog>Slurmctld.
* torque/qstat - fix printf error message in output.
* When adding associations or wckeys avoid checking multiple times a user or
cluster name.
* Fix wrong jobacctgather information on a step on multiple nodes
due to timeouts sending its the information gathered on its node.
* Fix missing xstrdup which could result in slurmctld segfault on array jobs.
* Fix security issue in PrologSlurmctld and EpilogSlurmctld by always
prepending SPANK_ to all user-set environment variables. CVE-2021-31215.
- New features from 20.11.6:
* Fix sacct assert with the --qos option.
* Use pkg-config --atleast-version instead of --modversion for systemd.
* common/fd - fix getsockopt() call in fd_get_socket_error().
* Properly handle the return from fd_get_socket_error() in _conn_readable().
* cons_res - Fix issue where running jobs were not taken into consideration
when creating a reservation.
* Avoid a deadlock between job_list for_each and assoc QOS_LOCK.
* Fix TRESRunMins usage for partition qos on restart/reconfig.
* Fix printing of number of tasks on a completed job that didn't request
tasks.
* Fix updating GrpTRESRunMins when decrementing job time is bigger than it.
OBS-URL: https://build.opensuse.org/package/show/network:cluster/slurm?expand=0&rev=179
- Udpate to 20.11.04
* Fix node selection for advanced reservations with features.
* mpi/pmix: Handle pipe failure better when using ucx.
* mpi/pmix: include PMIX_NODEID for each process entry.
* Fix job getting rejected after being requeued on same node that died.
* job_submit/lua - add "network" field.
* Fix situations when a reoccuring reservation could erroneously skip a
period.
* Ensure that a reservations [pro|epi]log are ran on reoccuring reservations.
* Fix threads-per-core memory allocation issue when using CR_CPU_MEMORY.
* Fix scheduling issue with --gpus.
* Fix gpu allocations that request --cpus-per-task.
* mpi/pmix: fixed print messages for all PMIXP_* macros
* Add mapping for XCPU to --signal option.
* Fix regression in 20.11 that prevented a full pass of the main scheduler
from ever executing.
* Work around a glibc bug in which "0" is incorrectly printed as "nan"
which will result in corrupted association state on restart.
* Fix regression in 20.11 which made slurmd incorrectly attempt to find the
parent slurmd address when not applicable and send incorrect reverse*tree
info to the slurmstepd.
* Fix cgroup ns detection when using containers (e.g. LXC or Docker).
* scrontab - change temporary file handling to work with emacs.
- Removed check-for-lipmix.so.MAJOR.patch
- Added: load-pmix-major-version.patch
OBS-URL: https://build.opensuse.org/request/show/874647
OBS-URL: https://build.opensuse.org/package/show/network:cluster/slurm?expand=0&rev=173
- Update to 20.11.03
- This release includes a major functional change to how job step launch is
handled compared to the previous 20.11 releases. This affects srun as
well as MPI stacks - such as Open MPI - which may use srun internally as
part of the process launch.
One of the changes made in the Slurm 20.11 release was to the semantics
for job steps launched through the 'srun' command. This also
inadvertently impacts many MPI releases that use srun underneath their
own mpiexec/mpirun command.
For 20.11.{0,1,2} releases, the default behavior for srun was changed
such that each step was allocated exactly what was requested by the
options given to srun, and did not have access to all resources assigned
to the job on the node by default. This change was equivalent to Slurm
setting the --exclusive option by default on all job steps. Job steps
desiring all resources on the node needed to explicitly request them
through the new '--whole' option.
In the 20.11.3 release, we have reverted to the 20.02 and older behavior
of assigning all resources on a node to the job step by default.
This reversion is a major behavioral change which we would not generally
do on a maintenance release, but is being done in the interest of
restoring compatibility with the large number of existing Open MPI (and
other MPI flavors) and job scripts that exist in production, and to
remove what has proven to be a significant hurdle in moving to the new
release.
Please note that one change to step launch remains - by default, in
20.11 steps are no longer permitted to overlap on the resources they
have been assigned. If that behavior is desired, all steps must
explicitly opt-in through the newly added '--overlap' option.
Further details and a full explanation of the issue can be found at:
https://bugs.schedmd.com/show_bug.cgi?id=10383#c63
OBS-URL: https://build.opensuse.org/request/show/864993
OBS-URL: https://build.opensuse.org/package/show/network:cluster/slurm?expand=0&rev=171
- Add support for configuration files from external plugins.
While built-in plugins have their configuration added in slurm.conf,
external SPANK plugins add their configuration to plugstack.conf
To allow packaging easily spank plugins, their configuration files
should be added independently at /etc/spack/plugstack.conf.d and
plugstack.conf should be left with an oneliner including all the
files under /etc/spack/plugstack.conf.d
OBS-URL: https://build.opensuse.org/request/show/860690
OBS-URL: https://build.opensuse.org/package/show/network:cluster/slurm?expand=0&rev=164
- Update to 20.11.02
* Fix older versions of sacct not working with 20.11.
* Fix slurmctld crash when using a pre-20.11 srun in a job allocation.
* Correct logic problem in _validate_user_access.
* Fix libpmi to initialize Slurm configuration correctly.
- Update to 20.11.01
* Fix spelling of "overcomited" to "overcomitted" in sreport's cluster
utilization report.
* Silence debug message about shutting down backup controllers if none are
configured.
* Don't create interactive srun until PrologSlurmctld is done.
* Fix fd symlink path resolution.
* Fix slurmctld segfault on subnode reservation restore after node
configuration change.
* Fix resource allocation response message environment allocation size.
* Ensure that details->env_sup is NULL terminated.
* select/cray_aries - Correctly remove jobs/steps from blades using NPC.
* cons_tres - Avoid max_node_gres when entire node is allocated with
--ntasks-per-gpu.
* Allow NULL arg to data_get_type().
* In sreport have usage for a reservation contain all jobs that ran in the
reservation instead of just the ones that ran in the time specified. This
matches the report for the reservation is not truncated for a time period.
* Fix issue with sending wrong batch step id to a < 20.11 slurmd.
* Add a job's alloc_node to lua for job modification and completion.
* Fix regression getting a slurmdbd connection through the perl API.
* Stop the extern step terminate monitor right after proctrack_g_wait().
* Fix removing the normalized priority of assocs.
* slurmrestd/v0.0.36 - Use correct name for partition field:
"min nodes per job" -"min_nodes_per_job".
OBS-URL: https://build.opensuse.org/request/show/859114
OBS-URL: https://build.opensuse.org/package/show/network:cluster/slurm?expand=0&rev=162
- Update to version 20.11.0
Slurm 20.11 includes a number of new features including:
* Overhaul of the job step management and launch code, alongside improved
GPU task placement support.
* A new "Interactive Step" mode of operation for salloc.
* A new "scrontab" command that can be used to submit and manage
periodically repeating jobs.
* IPv6 support.
* Changes to the reservation logic, with new options allowing users
to delete reservations, allowing admins to skip the next occurance of a
repeated reservation, and allowing for a job to be submitted and eligible
to run within multiple reservations.
* Dynamic Future Nodes - automatically associate a dynamically
provisioned (or "cloud") node against a NodeName definition with matching
hardware.
* An experimental new RPC queuing mode for slurmctld to reduce thread
contention on heavily loaded clusters.
* SlurmDBD integration with the Slurm REST API.
Also check
https://github.com/SchedMD/slurm/blob/slurm-20-11-0-1/RELEASE_NOTES (forwarded request 852039 from eeich)
OBS-URL: https://build.opensuse.org/request/show/853268
OBS-URL: https://build.opensuse.org/package/show/openSUSE:Factory/slurm?expand=0&rev=50
- Update to version 20.11.0
Slurm 20.11 includes a number of new features including:
* Overhaul of the job step management and launch code, alongside improved
GPU task placement support.
* A new "Interactive Step" mode of operation for salloc.
* A new "scrontab" command that can be used to submit and manage
periodically repeating jobs.
* IPv6 support.
* Changes to the reservation logic, with new options allowing users
to delete reservations, allowing admins to skip the next occurance of a
repeated reservation, and allowing for a job to be submitted and eligible
to run within multiple reservations.
* Dynamic Future Nodes - automatically associate a dynamically
provisioned (or "cloud") node against a NodeName definition with matching
hardware.
* An experimental new RPC queuing mode for slurmctld to reduce thread
contention on heavily loaded clusters.
* SlurmDBD integration with the Slurm REST API.
Also check
https://github.com/SchedMD/slurm/blob/slurm-20-11-0-1/RELEASE_NOTES
OBS-URL: https://build.opensuse.org/request/show/852039
OBS-URL: https://build.opensuse.org/package/show/network:cluster/slurm?expand=0&rev=160
- Updated to 20.02.5, changes:
* Fix leak of TRESRunMins when job time is changed with --time-min
* pam_slurm - explicitly initialize slurm config to support configless mode.
* scontrol - Fix exit code when creating/updating reservations with wrong
Flags.
* When a GRES has a no_consume flag, report 0 for allocated.
* Fix cgroup cleanup by jobacct_gather/cgroup.
* When creating reservations/jobs don't allow counts on a feature unless
using an XOR.
* Improve number of boards discovery
* Fix updating a reservation NodeCnt on a zero-count reservation.
* slurmrestd - provide an explicit error messages when PSK auth fails.
* cons_tres - fix job requesting single gres per-node getting two or more
nodes with less CPUs than requested per-task.
* cons_tres - fix calculation of cores when using gres and cpus-per-task.
* cons_tres - fix job not getting access to socket without GPU or with less
than --gpus-per-socket when not enough cpus available on required socket
and not using --gres-flags=enforce binding.
* Fix HDF5 type version build error.
* Fix creation of CoreCnt only reservations when the first node isn't
available.
* Fix wrong DBD Agent queue size in sdiag when using accounting_storage/none.
* Improve job constraints XOR option logic.
* Fix preemption of hetjobs when needed nodes not in leader component.
* Fix wrong bit_or() messing potential preemptor jobs node bitmap, causing
bad node deallocations and even allocation of nodes from other partitions.
* Fix double-deallocation of preempted non-leader hetjob components.
* slurmdbd - prevent truncation of the step nodelists over 4095.
* Fix nodes remaining in drain state state after rebooting with ASAP option.
- changes from 20.02.4:
OBS-URL: https://build.opensuse.org/request/show/845108
OBS-URL: https://build.opensuse.org/package/show/network:cluster/slurm?expand=0&rev=156
