Moab Cluster Suite Integration Guide
Overview
Moab Cluster Suite configuration is quite complicated and is beyond the scope of any documents we could supply with SLURM. The best resource for Moab configuration information is the online documents at Cluster Resources Inc.: http://www.clusterresources.com/products/mwm/docs/slurmintegration.shtml.
Configuration
First, download the Moab scheduler kit from their web site
http://www.clusterresources.com/pages/products/moab-cluster-suite.php.
Note: Use Moab version 5.0.0 or higher and SLURM version 1.1.28
or higher.
SLURM configuration
slurm.conf
Set the slurm.conf scheduler parameters as follows:
SchedulerType=sched/wiki2 SchedulerPort=7321
Running multiple jobs per mode can be accomplished in two different ways. The SelectType=select/cons_res parameter can be used to let SLURM allocate the individual processors, memory, and other consumable resources (in SLURM version 1.2.1 or higher). Alternately, SelectType=select/linear or SelectType=select/bluegene can be used with the Shared=yes or Shared=force parameter in partition configuration specifications.
The default value of SchedulerPort is 7321.
SLURM commands
Note that the srun --immediate option is not compatible with Moab. All jobs must wait for Moab to schedule them rather than being scheduled immediately by SLURM.
wiki.conf
SLURM's wiki configuration is stored in a file specific to the wiki-plugin named wiki.conf. This file should be protected from reading by users. It only needs to be readable by SlurmUser (as configured in slurm.conf) and only needs to exist on computers where the slurmctld daemon executes. More information about wiki.conf is available in a man page distributed with SLURM.
The currently supported wiki.conf keywords include:
AuthKey is a DES based encryption key used to sign communications between SLURM and Maui or Moab. This use of this key is essential to insure that a user not build his own program to cancel other user's jobs in SLURM. This should be no more than 32-bit unsigned integer and match the the encryption key in Maui (--with-key on the configure line) or Moab (KEY parameter in the moab-private.cfg file). Note that SLURM's wiki plugin does not include a mechanism to submit new jobs, so even without this key nobody could run jobs as another user.
EPort is an event notification port in Moab. When a job is submitted to or terminates in SLURM, Moab is sent a message on this port to begin an attempt to schedule the computer. This numeric value should match EPORT configured in the moab.cnf file.
EHost is the event notification host for Moab. This identifies the computer on which the Moab daemons executes which should be notified of events. By default EHost will be identical in value to the ControlAddr configured in slurm.conf.
EHostBackup is the event notification backup host for Moab. Names the computer on which the backup Moab server executes. It is used in establishing a communications path for event notification. By default EHostBackup will be identical in value to the BackupAddr configured in slurm.conf.
ExcludePartitions is used to identify partitions whose jobs are to be scheduled directly by SLURM rather than Moab. This only effects jobs which are submitted using Slurm commands (i.e. srun, salloc or sbatch, NOT msub from Moab). These jobs will be scheduled on a First-Come-First-Served basis. This may provide faster response times than Moab scheduling. Moab will account for and report the jobs, but their initiation will be outside of Moab's control. Note that Moab controls for resource reservation, fair share scheduling, etc. will not apply to the initiation of these jobs. If more than one partition is to be scheduled directly by Slurm, use a comma separator between their names.
HostFormat controls the format of job task lists built by Slurm and reported to Moab. The default value is "0", for which each host name is listed individually, once per processor (e.g. "tux0:tux0:tux1:tux1:..."). A value of "1" uses Slurm hostlist expressions with processor counts (e.g. "tux[0-16]*2"). This is currently experimental.
JobAggregationTime is used to avoid notifying Moab of large numbers of events occurring about the same time. If an event occurs within this number of seconds since Moab was last notified of an event, another notification is not sent. This should be an integer number of seconds. The default value is 10 seconds. The value should match JOBAGGREGATIONTIME configured in the moab.cnf file.
JobPriority controls the scheduling of newly arriving jobs in SLURM. SLURM can either place all newly arriving jobs in a HELD state (priority = 0) and let Moab decide when and where to run the jobs or SLURM can control when and where to run jobs. In the later case, Moab can modify the priorities of pending jobs to re-order the job queue or just monitor system state. Possible values are "hold" and "run" with "hold" being the default.
Here is a sample wiki.conf file
# wiki.conf # SLURM's wiki plugin configuration file # # Matches KEY in moab-private.cfg AuthKey=123456789 # # Slurm directly schedules jobs in the debug partitions ExcludePartitions=debug # # Have Moab control job scheduling JobPriority=hold # # Moab event notification port, matches EPORT in moab.cfg EPort=15017 # Moab event notification host, where the Moab daemon runs #EHost=tux0 # # Moab event notification throttle, matches JOBAGGREGATIONTIME # in moab.cfg (integer value in seconds) JobAggregationTime=15
Moab Configuration
Moab has support for SLURM's WIKI interface by default. Specify this interface in the moab.cfg file as follows:
SCHEDCFG[base] MODE=NORMAL RMCFG[slurm] TYPE=WIKI:SLURM AUTHTYPE=CHECKSUM
In moab-private.cfg specify the private key as follows:
CLIENTCFG[RM:slurm] KEY=123456789
Insure that this file is protected from viewing by users.
Last modified 17 August 2007