Using VASP on Stampede/Stampede2 (XSEDE)

Update: At the time that I created this blog, guides for using XSEDE and TACC were pretty sparse. TACC now has its own guide. Even so, I'll leave this post up, as I think it does have a few more details that are still pretty useful.

Disclaimer: Despite some good notes online and a few kind people who responded to my desperate email pings, I really struggled to learn this largely on my own, so do keep that in mind if you find anything silly - comments to improve are appreciated!!!

Quick Description: NSF GRFP Recipients and Honorable Mention Awardees are provided with access to XSEDE, but the details for utilizing this cyberinfrastructure (for VASP and in general) are sparse and not accommodating towards the typical graduate student. Here, I go through the specifics of using VASP on Stampede. The software is preinstalled on this cluster, and so you simply need to get your license and user registration checked to us it.

The Point: Using XSEDE (Stampede) for VASP.

Prerequisites: An XSEDE allocation with access to Stampede (now Stampede2).

Notes:

1. You will then need to contact Hang Liu, administrator for usage of VASP on Stampede. Email him your VASP license number and he will confirm that you are registered under that account number before granting you access.
2. You will first need to download WinSCP and Putty (or whatever SSH client you would prefer to use).
3. SSH into Stampede
• On Windows, I like using WinSCP and Putty: 
• On MAC (just got a MacBook, and am still adjusting): Shell 
• Shell > New Remote Connection 
• Under Server, click + and add “stampede2.tacc.utexas.edu” 
• Then add username to user 
• 
  
  
  
4. You will be prompted for your password and then your TACC authentication (text message code, which I forward to my google voice for ease).   
• Your default directory will be your folder in your group's folder in the home folder
• For example, "pwd" gives me: /home1/[5 digit number]/username
  
• I like to work in the work folder instead - it has more space.
•  cd /work/[5 digit number]/[username]
• You will run into weird submission/run errors when you run out of space in your folder. Ideally, you should be moving your completed work into archive (you should apply for space on Ranch in your XSEDE proposal), but honestly, I only ever did that once a year.
5. For job submissions, Stampede uses SLURM. I often refer to this chart to remember what to do: https://slurm.schedmd.com/rosetta.pdf
• Here’s an example of a job submission script (adapted from what the kind Hang Liu sent to me): 
•  
  
  
  
• You can find more information on how to format this submission script here, do note, on Stampede you only had to request a certain number of nodes. On Stampede2, you request a number of nodes (-N) and how many mpi tasks (-n) you want to run on each note.
• A general rule of thumb to pick -n choose a number that is a power of 2 closest to the number of atoms in your system (roughly, calculating each atom is like a task). Then, I will need to distribute the tasks amongst nodes. Each KNL node on Stampede2 has 68 cores, but ~4 of those must be reserved for processes like talking to other nodes, so treat it like you actually only have access to 64 tasks per node. So, say I have 72 atoms, I will set "-n 64". Since this is how many I can access with one node, then I set "-N 1".
• 
• There are actually two types of submission scripts (which can be set with -p): normal and development. As you might, guess, typically normal is used. But to test out code, a development submission (which has access to a shorter queue) is available to you. It will only run for 2 hours (and won't submit properly if you try to go over that number).  
• So I would submit this (test.job) with "sbatch test.job". I then check on my jobs by searching everything under my user account with "squeue -u [username]". If I made a mistake, I use "scancel <jobID#>". And that's pretty much it!
• It's generally a good idea to run rough VASP jobs before attaining higher accuracy optimization and finally getting the specific files you need (e.g. CHGCAR, DOSCAR). Here's a link to a folder of INCARs I've used previously, but take with a grain of salt how useful they will be to you.
6. I personally like editing (everything) in Notepad++ and using the WinSCP drag and drop feature to move things around. On the fly, I usually just use the built-in VI editor. (e.g. "vi test.job")
• here's a link to vi commands (pretty much just "i", Esc, ":q", and/or ":wq")
• VI is also useful, because I like to check on jobs from my phone (plenty of SSH apps out there, but I use JuiceSSH)
7.  ........there's so much to know so this might be updated later (then again, this is my first update in years)