A bit of bragging

The point: Not a real post, just some happy news.

A few months ago, I took down this blog, as I'd become a little embarrassed about it. I started this blog on VASP and DFT when I first started graduate school with no experience in DFT calculations and minimal programming abilities. It's scary to know that anyone on the internet can see how much of a beginner I was (and still am). Even now, after coming a long way and learning a lot more, I'm primarily an experimentalist that only dabbles in DFT...

Anway, I decided to bring the blgo back because it does seem like it's still helpful to people. I was very surprised to find that my blog had been cited in a publication! Wow!

 

 

Yes... it's from last year, but I only saw it last week....

I usually try to keep these notes short and straight to the point, but I wanted to take some time to say that I'm pretty happy if these notes are useful to others. I apologize that I don't update frequently and that sometimes my notes aren't the best. I hope to someday clean things up so that they're something that I'm more proud of. Even so, thanks for reading.

I'll leave off with some interesting stats snapshots:

Cheers!

VASP Code Performance Scaling: Testing with Au/graphene

Quick Description: When applying for a supercomputer allocation, you will need to justify the amount of resources that you're applying for. In the case of XSEDE, you can apply for a startup allocation, and the point of this is largely to do some baseline calculations from which you can project how much to ask for in your next application.

The Point: Here's how to estimate the supercomputer resources you will require.

Prerequisites: Ability to gain an XSEDE allocation with access to Stampede (now Stampede2).

Notes:  

1. For any sort of experiment, you need a control experiment to compare with. In DFT, this calculation can usually serve as a lower bound estimate for the resource consumption of your subsequent (likely more complicated) calculations. (NOTE: Here, I'm assuming that you are working with similar systems of interest and tweaking the POSCAR slightly between calculations. That is to say that most of your atoms,  their elemental composition, and they geometric coordinates should stay largely the same. At the very least, your cell size should not be changing.)
   
• In my case, I'm studying metal clusters on graphene. Without going into too much detail, this system is used as a test case for exploring other metals and their interaction with graphene surfaces, where we assume the time necessary to complete calculations on similar systems will roughly match that of this control. 
• [removed]
  
2. Unfortunately, I decided to remove a picture of our scaling performance plot, because my PI was uncomfortable with me sharing it online. Essentially, the plot shows how the performance time scales with the number of cores used for the calculation. Up until a certain point, increasing the number of cores will decrease the calculation time. Eventually, this speed-up factor will level off, at which point you have determined the number of cores you will likely use on your subsequent calculations. 
3. To get a rough idea of how many cores you need, VASP utilizes MPI parallelization, assigning one MPI rank to one core, and Larsson generally suggests that one core be assigned per atom in a VASP job. Since our baseline calculation consisted of 72 atoms, we guessed that 64 cores (using powers of 2 is recommended) would work best, which is what our performance scaling calculations showed. In our case, all simulations were performed on the Stampede2 computing cluster with Knights Landing (KNL) nodes, which each have 68 cores. This means that (if you are increasingly doubling your number of cores, as you should), a single node should run 64 cores max. Per TACC’s Best Known Practices for KNL Nodes 2, 4 cores should be left unutilized anyway. 
4. Again, in our case, we tested up to 128 cores, consisting of 2 nodes with 64 cores each. This resulted in a worse run time than 1 node with 64 cores. Some of this poor performance is due to communication inefficiencies between the nodes. Obviously, using one node is also easier to keep track of. :)
5. For our work, we intended to model variations of our control calculations with at most 76 atoms total, making 1 node and 64 cores an appropriate configuration to ensure fast run times and efficient SU estimation for Stampede2. For a  completely optimized job, total run times was around 12 hours. We then counted the number of variations we hoped to try and padded on a few extras hours for good measure. This calculator from XSEDE may also prove useful to you.
   

Using VASP on Stampede/Stampede2 (XSEDE)

Update: At the time that I created this blog, guides for using XSEDE and TACC were pretty sparse. TACC now has its own guide. Even so, I'll leave this post up, as I think it does have a few more details that are still pretty useful.

Disclaimer: Despite some good notes online and a few kind people who responded to my desperate email pings, I really struggled to learn this largely on my own, so do keep that in mind if you find anything silly - comments to improve are appreciated!!!

Quick Description: NSF GRFP Recipients and Honorable Mention Awardees are provided with access to XSEDE, but the details for utilizing this cyberinfrastructure (for VASP and in general) are sparse and not accommodating towards the typical graduate student. Here, I go through the specifics of using VASP on Stampede. The software is preinstalled on this cluster, and so you simply need to get your license and user registration checked to us it.

The Point: Using XSEDE (Stampede) for VASP.

Prerequisites: An XSEDE allocation with access to Stampede (now Stampede2).

Notes:

1. You will then need to contact Hang Liu, administrator for usage of VASP on Stampede. Email him your VASP license number and he will confirm that you are registered under that account number before granting you access.
2. You will first need to download WinSCP and Putty (or whatever SSH client you would prefer to use).
3. SSH into Stampede
• On Windows, I like using WinSCP and Putty: 
• On MAC (just got a MacBook, and am still adjusting): Shell 
• Shell > New Remote Connection 
• Under Server, click + and add “stampede2.tacc.utexas.edu” 
• Then add username to user 
• 
  
  
  
4. You will be prompted for your password and then your TACC authentication (text message code, which I forward to my google voice for ease).   
• Your default directory will be your folder in your group's folder in the home folder
• For example, "pwd" gives me: /home1/[5 digit number]/username
  
• I like to work in the work folder instead - it has more space.
•  cd /work/[5 digit number]/[username]
• You will run into weird submission/run errors when you run out of space in your folder. Ideally, you should be moving your completed work into archive (you should apply for space on Ranch in your XSEDE proposal), but honestly, I only ever did that once a year.
5. For job submissions, Stampede uses SLURM. I often refer to this chart to remember what to do: https://slurm.schedmd.com/rosetta.pdf
• Here’s an example of a job submission script (adapted from what the kind Hang Liu sent to me): 
•  
  
  
  
• You can find more information on how to format this submission script here, do note, on Stampede you only had to request a certain number of nodes. On Stampede2, you request a number of nodes (-N) and how many mpi tasks (-n) you want to run on each note.
• A general rule of thumb to pick -n choose a number that is a power of 2 closest to the number of atoms in your system (roughly, calculating each atom is like a task). Then, I will need to distribute the tasks amongst nodes. Each KNL node on Stampede2 has 68 cores, but ~4 of those must be reserved for processes like talking to other nodes, so treat it like you actually only have access to 64 tasks per node. So, say I have 72 atoms, I will set "-n 64". Since this is how many I can access with one node, then I set "-N 1".
• 
• There are actually two types of submission scripts (which can be set with -p): normal and development. As you might, guess, typically normal is used. But to test out code, a development submission (which has access to a shorter queue) is available to you. It will only run for 2 hours (and won't submit properly if you try to go over that number).  
• So I would submit this (test.job) with "sbatch test.job". I then check on my jobs by searching everything under my user account with "squeue -u [username]". If I made a mistake, I use "scancel <jobID#>". And that's pretty much it!
• It's generally a good idea to run rough VASP jobs before attaining higher accuracy optimization and finally getting the specific files you need (e.g. CHGCAR, DOSCAR). Here's a link to a folder of INCARs I've used previously, but take with a grain of salt how useful they will be to you.
6. I personally like editing (everything) in Notepad++ and using the WinSCP drag and drop feature to move things around. On the fly, I usually just use the built-in VI editor. (e.g. "vi test.job")
• here's a link to vi commands (pretty much just "i", Esc, ":q", and/or ":wq")
• VI is also useful, because I like to check on jobs from my phone (plenty of SSH apps out there, but I use JuiceSSH)
7.  ........there's so much to know so this might be updated later (then again, this is my first update in years)

       

Pymatgen (for generating DOS) - Windows 8 Instructions

Another way to do band diagrams using pymatgen. You'll need a few other things like cygwin and platmatlib to actually get pymatgen going. For full instr;uctions on actually generating the band diagram and DOS plots, use this graphene example here: https://www.nsc.liu.se/~pla/vasptools/

Rough Notes...

First, install Cygwin

In install include: 

(in the link it will tell you that you need gcc4, but this is the default gcc now)
 

http://pymatgen.org/installation.html#windows-tested-on-win7-64-bit

https://github.com/materialsproject/pymatgen

 

Also, add all python packages and lapack and blas and any c libraries…

Install matplotlib

 

Download matplotlib from sourceforge and unzip in Cygwin directory (wherever), to install it…

 

If you get a freetype error….

http://berlinbrowndev.blogspot.com/2012/01/python-matplotlib-plotting-setup-for.html

 

 

Apt-get in cygwin is apt-cyg

import matplotlib

matplotlib.use('agg')

http://matplotlib.1069221.n5.nabble.com/Error-when-running-code-on-remote-machine-how-can-I-overpass-this-td19784.html

since running remotely you want to turn on ‘agg’ so that it doesn’t try to launch in the shell