NAME

       oarsub - OAR job submission command.

SYNOPSIS

       oarsub [OPTIONS] <command>

       oarsub [OPTIONS] -I

       oarsub [OPTIONS] -r "<START DATE|now>[, <END DATE>]" [<command>]

       oarsub [OPTIONS] -C [<JOB ID>]

DESCRIPTION

       One uses oarsub to submit a job to the OAR resources and jobs management system.  A job is defined by a
       command to run and optionally the description of a set of wanted resources to execute a task.  The
       command can be any executable program or script file, or a script command line.

       The OAR scheduler is in charge of providing a set of resources which matches the oarsub resource request
       (see the -l option). Once scheduled and launched, the job executes the command on the first node
       (machine) of the assigned resources, with some environment variables set (e.g. the assigned resources
       list, the job duration a.k.a walltime, ...). If many nodes (machines) are assigned to the job, it is let
       to the user's command to dispatch an execution on all nodes.

       OAR defines two kinds of jobs: batch jobs and advance reservation jobs.

   BATCH JOBS
       A batch job lets the OAR scheduler decide when to execute the job, depending on when the requested
       resources are available. If not using the -I option, oarsub returns immediately: the job is queued (use
       the oarstat command to list the queued jobs). When using the -I option, oarsub waits until the job can
       start then opens a interactive shell on the first of the assigned nodes.

   ADVANCE RESERVATION JOBS
       Calling oarsub with the -r option creates an advance reservation job instead of a batch job. An advance
       reservation job lets the user decide when the job should start (and optionally end). oarsub waits for a
       validation of the resources availability. Whether a set of resources matching the resource request of the
       job is available, an OK or KO message is returned. OK means that the job will run at the requested date.
       KO means that the job is refused.

OPTIONS

       -I, --interactive
           Request an interactive job.

       -C, --connect [<JOB ID>]
           Connect to a running job. Job id may be omitted if only one job is running.

       -l, --resource <DESCRIPTION>[,<WALLTIME>]
           Set  the  description  of  the  requested  resources  for  the  job and a walltime which specifies an
           estimation of the maximum duration of the job (the job process can terminate earlier).  The  walltime
           format is hours:minutes:seconds (seconds and minutes can be omitted).

             oarsub -l /host=4/cpu=1,walltime=2:00:00

           Multiple  -l  options can be given to define a moldable job: a job accepting different combination of
           resource definitions. Example:

             oarsub -l cpu=2,walltime=20:00:00 -l cpu=4,walltime=10:00:00 -l cpu=8,walltime=5:00:00 ./script.sh

           OAR will allocate resources to the job accordingly to one of the 3 given definitions depending of the
           load of the cluster, and preferring earliest possible start.

           One can also request different groups of resources, for example:

             oarsub -l "{mem > 64}/host=1+{mem < 48}/host=3",walltime=1:00:00 -I

           The job requests to run on 1 host with the property "mem" > 64 and 3 hosts with property "mem" <  48.
           The syntax between braces, {...}, is the same as the one used for -p option.

       --array <NUMBER>
           Submit  an  array  job  containing "NUMBER" subjobs. All the subjobs share the same array_id but each
           subjob is independent and has its own job_id. All the subjobs have the same characteristics  (script,
           requirements) and can be identified by an environment variable $OAR_ARRAY_INDEX.

           Array jobs can neither be Interactive (-I) nor a reservation (-r).

       --array-param-file <FILE>
           Submit a parametric array job. Each non-empty line of "FILE" defines the parameters for a subjob. All
           the  subjobs  have  the  same  characteristics  (script,  requirements)  and  can be identified by an
           environment variable $OAR_ARRAY_INDEX.   '#' is the comment sign.

           Parameters are given as arguments to the command.

           Parametric array jobs can neither be Interactive (-I) nor a reservation (-r).

       -S, --scanscript
           Request oarsub to scan the script file given as the job command, for extra options to  apply  to  the
           job (lines starting with #OAR)

       -q, --queue <QUEUE>
           Set the queue to submit the job to.

       -p, --property "<SQL WHERE CLAUSE>"
           Add  constraints to the wanted resource for the job, using the resources properties.  Constraints use
           the SQL WHERE CLAUSE syntax.

       -r, --reservation "<START DATE|now>[, <END DATE>]"
           Request that the job starts (and optionally ends) at a specified time. Using this option, the job  is
           called an advance reservation.

           Start  and end dates use the YYYY-MM-DD hh:mm:ss format. If YYYY-MM-DD is omitted, it defaults to the
           current day. In hh:mm:ss, ss and mm can be omitted. Quotes are only required when spaces are present.
           When an end date is provided the job walltime is  inferred,  unless  provided  in  the  -l  resources
           request  which  prevails. The special keyword now can be used as the start date to request an advance
           reservation job that starts just now.

       --checkpoint <DELAY>
           Enable the checkpointing mechanism for the job.  A signal will  be  sent  DELAY  seconds  before  the
           walltime to the first processus of the job (on the first node of the job resources).

       --signal <#SIG>
           Specify the signal to used to trigger the checkpointing. Use signal numbers (see kill -l), default is
           12 (SIGUSR2).

       -t, --type <TYPE>
           Specify  a  specific  type  (besteffort,  timesharing, idempotent, cosystem, deploy, noop, container,
           inner, token:xxx=yy,... )

           Notes:
             - a job with the besteffort type will be scheduled with the lowest priority and will be killed if a
           "normal" job needs its resources.
             - a job with the idempotent type will be automatically resubmitted if its exit code is 99  and  its
           duration > 60s.
             - a job with the idempotent and besteffort types but with no checkpoint delay will automatically be
           resubmitted whenever killed by OAR before its normal termination to execute a non-besteffort jobs.
             -  a  job  with  the  idempotent and besteffort types with a checkpoint delay will be automatically
           resubmitted if its exit code is 99 and its duration >  60s  (that  is,  as  if  only  idempotent  was
           present).
             -  a  job with the noop type does nothing except reserving the resources. It is ended at the end of
           it walltime or when using the oardel command.

           The -t option can be used several times to provide different job types to the job.

       -d, --directory <DIR>
           Specify the directory where to launch the command (default is current directory)

       --project <TXT>
           Specify a name of a project the job belongs to.

       -n, --name <TXT>
           Specify an arbitrary name for the job.

       -a, --anterior <OAR JOB ID>
           Previously submitted job that the new job execution must depend on. The new job will only start  upon
           the end of the previous one.

       --notify <TXT>
           Specify a notification method (email or command to execute).  Ex:
            --notify "mail:name@domain.com"
            --notify "exec:/path/to/script args"

           Arguments are job_id, job_name, TAG, comment

           TAG can be:
             - RUNNING: when the job is launched
             - END: when the job is finished normally
             - ERROR: when the job is finished abnormally
             - INFO: used when oardel is called on the job
             - SUSPENDED: when the job is suspended
             - RESUMING: when the job is resumed

           By default all TAGs are triggered. It is possible to specify which TAGs must be triggered.  Ex:
            --notify "[END,ERROR]mail:name@domain.com"
            --notify "[RUNNING]mail:name@domain.com"
            --notify "[RUNNING,END,ERROR]exec:/path/to/script args"

       --resubmit <OAR JOB ID>
           Resubmit the given job as a new one.

       -k, --use-job-key
           Activate  the  job-key  mechanism:  a job-key will be generated for the job, which can be used my the
           oarsh command in place of the OAR_JOB_ID. That job-key also allows one to connect to  a  job  from  a
           machine  which  is  outside the OAR cluster the job belong to (e.g. in a grid of OAR clusters), given
           the job-key is available on that machine.

           The job-key mechanism may be activated by default in the configuration of your OAR cluster.  In  that
           case, the -k option is useless.

           Please  note that this option is mostly useful when used together with the -e option or the -i option
           (see below).

       -e, --export-job-key-to-file
           Export the job-key to a file (the  %jobid%  pattern  is  automatically  replaced  in  the  filename).
           Warning: the file will be overwritten, whenever it already exists.

       -i, --import-job-key-from-file
           Import the job-key to use (you may reuse an exported job-key from a previous job for instance) from a
           file,  instead of generating a new one. One may also use the OAR_JOB_KEY_FILE environment variable to
           set the job-key file.

       --import-job-key-inline
           Import the job-key to use inline (as text in the command line), instead of generating a new one.

       -O <FILE>
           Specify the file that will store the standard output stream of the job.  The  %jobid%  and  %jobname%
           patterns are automatically replaced.

       -E <FILE>
           Specify  the  file  that  will  store the standard error stream of the job. The %jobid% and %jobname%
           patterns are automatically replaced.

       --hold
           Set the job state into Hold instead of Waiting, so that it is not scheduled as long  as  not  resumed
           (the oarresume command allows one to turn it back into the Waiting state).

       -D, --dumper
           Print the command results in Perl's Data::Dumper format.

       -X, --xml
           Print the command results in the XML format.

       -Y, --yaml
           Print the command results in the YAML format.

       -J, --json
           Print the command results in the JSON format.

       -h, --help
           Print this help message.

       -V, --version
           Print the version of OAR.

       -v  Verbose  mode,  oarsub  and admissions rules (if compatible) will print more information on terminal.
           Multiple -v can be used to increase the level of verbosity (max: 2).

ENVIRONMENT

       OAR_FILE_NODES or OAR_NODE_FILE or OAR_NODEFILE
           Pathname to the file containing the list of the nodes that are assigned to the job.

       OAR_JOB_NAME
           Name of the job as given using the -n option.

       OAR_JOB_ID or OAR_JOBID
           Id of the job. Each job get a  unique  job  identifier.  This  identifier  can  be  use  to  retrieve
           information  about the job using oarstat, or to connect to a running job using oarsub -C or oarsh for
           instance.

       OAR_ARRAY_ID or OAR_ARRAYID
           Array Id of the job. Every array job gets an unique array  identifier  that  is  shared  by  all  the
           subjobs of the array job. This identifier can be used to identify the different subjobs pertaining to
           a same array job. Array Id can also be used to deal with all the subjobs of a given array at once (by
           means of the --array  option in the case of oarstat, oarhold, oarresume and oardel). NB: regular jobs
           are array jobs with only one subjob.

       OAR_JOB_INDEX or OAR_JOBINDEX
           Array  Index  of  the  job: within an array job, each subjob gets a unique (for a given array) index,
           starting from 0, which can be used to identify the subjob.

       OAR_JOB_WALLTIME and OAR_JOB_WALLTIME_SECONDS
           Walltime of the job in the hh:mm:ss format and in seconds.

       OAR_RESOURCE_PROPERTIES_FILE
           Pathname to the file containing the list of all resources assigned for the job, and their value.  See
           oarprint also.

       OAR_PROJECT_NAME
           Name of the project the job is part of, as given using the --project option.

       OAR_STDOUT and OAR_STDERR
           Pathname  to  the  files storing the standard output and standard error of the job executable, if not
           running in interactive mode.

       OAR_WORKING_DIRECTORY aka OAR_WORKDIR aka OAR_O_WORKDIR
           Working directory for the job. The job executable will be executed in that directory,  on  the  first
           node which is allocated to the job.

       OAR_JOB_KEY_FILE
           Key  file  to  use  for the submission (or oarsh) if using the job-key mechanism (-k or --use-job-key
           option). One may provide the job-key to import using the -i or --import-job-key-from-file  option  as
           well.

SCRIPT

       When  submitting  a  job  using  a script file, that script file can contain some extra OAR options, with
       lines starting with #OAR and using the same option syntax as described above.

EXAMPLES

   Job submission with arguments
        oarsub -l /nodes=4 -I

        oarsub -q default -l /nodes=10/cpu=3,walltime=50:30:00 -p "switch = 'sw1'" /home/username/path/to/my/prog

        oarsub -r "2009-04-27 11:00:00" -l /nodes=12/cpu=2

        oarsub -C 154

   Array job submission with 10 identical subjobs
        oarsub -l /nodes=4 /home/usename/path/to/my/prog --array 10

   Parametric array job submission
        oarsub /home/users/toto/prog --array-param-file /home/username/path/to/params.txt

       With /home/username/path/to/param.txt containing for instance:

        # my param file
        # a subjob with a single parameter
        p100
        # a subjob without parameter
        ""
        # a subjob with two strings as parameters
        "arg1a arg1b arg1c" "arg2a arg2b"

   Job submission with a job script
        oarsub -S /home/username/path/to/my/script.sh

       With /home/username/path/to/my/script.sh containing for instance:

        #!/bin/bash
        #OAR -l /nodes=4/cpu=1,walltime=3:15:00
        #OAR -p switch = 'sw3' or switch = 'sw5'
        #OAR -t besteffort
        #OAR -t type2
        #OAR -k
        #OAR -e /path/to/job/key
        #OAR --stdout stdoutfile.log
        /home/username/path/to/my/prog

SEE ALSO

       oarprint(1), oarsh(1), oardel(1), oarstat(1), oarnodes(1), oarhold(1), oarresume(1)

COPYRIGHTS

        Copyright 2003-2021 Laboratoire d'Informatique de Grenoble (http://www.liglab.fr). This software is licensed under the GNU General Public License Version 2 or above. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

oarsub                                             2025-03-24                                          oarsub(1)