4.1   sacct Slurm Calls¶

Simply put, the “sacct” slurm call allows us to look at the resources used by a code after it was run with a slurm script of some kind. In my case, I used sacct in order to get the average memory usage of a job and reported that as the memory usage. An example of such a call would be as follows:

sacct -u thrush --format=JobID,JobName,AveRSS,Elapsed

Here, sacct looks for slurm jobs associated with my username, and then gives me the job IDs, the job name that I assigned it, the average memory used for the number of cores used, and the time it took for the job to run. Thus, since I’m using one core, I can say that the average memory reported is the memory used.

4.2   /usr/bin/time Linux Calls¶

The /usr/bin/time calls work slightly differently as these are generic Linux commands. These calls (shown below) simply look at the real-time memory usage of the command-line code call by looking into the /usr/bin/time files and reporting back the time the job to be completed, the user name, the system used, and the memory utilized.

/usr/bin/time -f "\n%E elapsed, \n%U user, \n%S system, \n%M memory\n" processCcd.py /datasets/hsc/repo --rerun private/thrush/RF --id visit=11382 ccd=0..8^10..103

Once the job is completed, the data from /usr/bin/time is printed on the screen, where it can be recorded. This is done so as to record the memory usage of a job that is not using slurm to run, but instead is run on the command line.

5.1   processCcd.py Trial¶

Prior to running the tests below, the baseline memory usage was established by running a similar trial as those below by using processCcd.py. The processCcd.py code accomplishes the same task as the codes shown in the singleFrameDriver section, however processCcd.py is much more transparent with its memory usage and seems to have reasonable memory usage. The code used in this case is that shown below:

/usr/bin/time -f "\n%E elapsed, \n%U user, \n%S system, \n%M memory\n" processCcd.py /datasets/hsc/repo --rerun private/thrush/RF --id visit=11382 ccd=0..8^10..103

After running the code snippet above 5 times and averaging those results together, the average memory usage reported by /usr/bin/time was found to be 2,288,000 K.

5.2   Different Number of Visits and CCDs¶

Before testing the methods discussed below, it is important to see first how memory depends on the number of visits and ccds that will be used. In order to ascertain this scaling, three different trials were run for the batch-type method, the /usr/bin/time method and the homemade slurm script method: one visit and one ccd, one visit with two ccds, and two visits with one ccd.

singleFrameDriver.py /datasets/hsc/repo --rerun $DIR --batch-type slurm --mpiexec='-bind-to socket' --job Memtest --id visit=11382 ccd=0 --cores 1 --time 400

singleFrameDriver.py /datasets/hsc/repo --rerun $DIR --batch-type slurm --mpiexec='-bind-to socket' --job Memtest --id visit=11382 ccd=0^1 --cores 1 --time 400

singleFrameDriver.py /datasets/hsc/repo --rerun $DIR --batch-type slurm --mpiexec='-bind-to socket' --job Memtest --id visit=11382^11690 ccd=0 --cores 1 --time 400

/usr/bin/time -f "\n%E elapsed, \n%U user, \n%S system, \n%M memory\n" singleFrameDriver.py /datasets/hsc/repo --rerun $DIR --id visit=11382 ccd=0 --cores 1

/usr/bin/time -f "\n%E elapsed, \n%U user, \n%S system, \n%M memory\n" singleFrameDriver.py /datasets/hsc/repo --rerun $DIR --id visit=11382 ccd=0^1 --cores 1

/usr/bin/time -f "\n%E elapsed, \n%U user, \n%S system, \n%M memory\n" singleFrameDriver.py /datasets/hsc/repo --rerun $DIR --id visit=11382^11690 ccd=0 --cores 1

#!/bin/bash -l

#SBATCH -p debug
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -t 03:00:00
#SBATCH -J test

srun singleFrameDriver.py /datasets/hsc/repo --rerun private/thrush/RD --id ccd=0 visit=11382 --cores 1

6.1   Results¶

The average memory for the –slurm method was found to be 306,500 K. As you can see, this is much lower than the baseline memory shown in section 5.1 above.

The /usr/bin/time method, which is most like the baseline trial from section 5.1, has an average memory usage of 3,155,000 K (averaged over 5 trials).

The homemade slurm script had memory usage at 398,000 K (averaged over 5 trials) as reported by sacct for AveRSS after the trials were run. This is slightly higher than expected from a code that is so similar to the Batch trial described above.

Finally, the salloc method gave an average memory usage of 310,500 K, which was found by looking into the top.txt file created by the salloc call above. This is not so shocking as salloc simply acts as an interactive slurm session, so although this call looks quite different from the batch-type results, they are essentially the same.

7.1   Results¶

After running the /usr/bin/time trial, the memory usage was found to be approximately 912400 K. However, the –slurm trial only reported a memory usage of 2592K, while the hand-made slurm script reported a memory usage of 371100 K. All of these computations took approximately 20 minutes to complete, on average.

These results mirror those of the singleFrameDriver trials above in that the largest memory usage belongs to the /usr/bin/time run, while the smallest memory usage belongs to the –slurm job. As stated in the singleFrameDriver section above, I believe that /usr/bin/time is more accurate in reporting its memory usage simply because it accounts for SWAP memory usage as well as normal memory usage, thus giving a more holistic view of the situation.

8.1   Results¶

The –slurm method used 2600 K of memory in order to work. However, the handmade slurm script used 365,512 K of memory in order to work. Both of these seem strangely low, but they did finish successfully. On the other hand, the /usr/bin/time trial used 1,727,120K of memory. These results echo those given above for the singleFrameDriver and coaddDriver codes.