Applying for an XSEDE Allocation

Quick Description: NSF GRFP Recipients and Honorable Mention Awardees are provided with access to XSEDE. (For many other reasons, one should consider applying for NSF GRFP, but many awardees are unaware of their supercomputer privileges. If you have  collaborator who has recipient or honorable mention status, it's quite possible that they are not using their supercomputer time, and they probably wouldn't mind sharing with you...) Depending on what sort of work you would like to use supercomputer time for, you should apply with a certain computational goal in mind, for specific clusters that host the software you want, and for a reasonable amount of space and time. In line with the rest of this blog, here I show you how to apply for an allocation Stampede to gain access to VASP. However, the general process could be used as a guideline for another allocation application. I spent a lot of time reading up how to to apply and what qualified for a successful application. This is not necessary and please don't waste your time on such effort. Instead, see below.

The Point: If you have NSF or are honorable mention, apply for your XSEDE allocation in ~30 min.

Prerequisites: An XSEDE allocation.

Notes:

1. Log into your XSEDE account (you may have to activate this using your NSF GRFP fastlane login and other information).
2. Navigating to the Allocations page and Submit/Review Request tab, click "Start a New Submission" and choose "Guided Submission".
• 
  
  
  
  
  
• You should see the next round of available applications. Obviously, you may have to wait for a new round to open up, but they typically have several a year so there should be no issue. When I applied, it took me one week to hear back that I had been approved.
3. From here, it should be fairly obvious how to fill out each section, but there are many examples you can find on the XSEDE site.
   
4. If you are a graduate student, make sure to identify your NSF status. "On the first page (PI information) you will see a box to check if you are an NSF fellow. Check that box, and on the "Documents" section, please upload award letter." - Stephen Deems
• This is important because a) normally graduate students are not allowed to be PIs on a supercomputer allocation; b) you will be pretty much automatically approved for your allocation since you are already NSF approved
5. You should soon recieve an email like the one below.
• 
  
  
  
  

Electronic Convergence

Quick Description: We can treat electronic and ionic convergence as separate steps so as to mathematically simplify solving Schrodinger's Equation. More elaborately put: "The forces on both electrons and nuclei due to their electric charge are of the same order of magnitude, and so the changes which occur in their momenta as a result of these forces must also be the same. One might, therefore, assume that the actual momenta of the electrons and nuclei were of similar magnitude. In this case, since the nuclei are so much more massive than the electrons, they must accordingly have much smaller velocities. Thus it is plausible that on the typical time-scale of the nuclear motion, the electrons will very rapidly relax to the instantaneous ground-state configuration, so that in solving the time-independent Schrödinger equation... we can assume that the nuclei are stationary and solve for the electronic ground-state first, and then calculate the energy of the system in that configuration and solve for the nuclear motion." - Peter Haynes

The Point: Sometimes you will have problems reaching electronic convergence in a single ionic step. Here are a few things to try.

Notes: These notes are collected from quite a few different forum posts (here, here, and here) with some help from my labmate, Jessica McKinley. The vasp manual is a really great resource for this problem too.

• Increase NELM, which sets the max number of electronic self consistency steps. However, "Normally, there is no need to change the default value: if the self-consistency loop does not converge within 40 steps, it will probably not converge at all." - VASP guide
• Set "ALGO = Normal" in your INCAR.
• Add a delay by setting NELMDL. This will increase the number of non-self-consistent to improve the pre-convergence of the wavefunction. You can either set this at the beginning of the job (by using a negative number) or by adding this delay at every ionic step (by using a positive number). This delay is especially important when the SC-convergence is bad, as is the case surfaces, molecules/clusters, chains, or charge density difference plots.  This delay can speed up convergence and can be essential to achieving convergence at all.
• Play with mixing parameters AMIX and BMIX, and/or AMIX_MAX and BMIX_MAX (for the spin-polarized runs). Reduce these parameters (perhaps a few steps with linear mixing first to generate an initial WAVECAR), then rerun from there with the default mixing parameters. Make the preconvergence easier and then attempt to reach true convergence. It may help to use L(S)DA+U. If you have f-elements or d-elements, set LMAXMIX to 6 or 4, respectively. 

Corrugation Comparison

Warning, these notes are still in development...

Quick Description: If we have two structures that we wish to compare, one method to use is to take a look at their corrugation. We can look both at the average height of each structure and how much that height varies (corrugation) for each structure. This gives us an idea of what they should appear as individually and relatively when we are using STM. Moreover, we can use the height information to set the tip-sample distances to be the same for between structures, comparing their STS measurements at various tip-sample distances. Keep in mind that an STS measurement at one height for one structure and at another height for the other structure is like "comparing apples and oranges" (as Prof. Bartels would say), so it's important to analyze in this manner that puts each structure in comparable states.

The Point: Comparing two structures using corrugation.

Prerequisites: Optimized jobs of each structures with WAVECAR files and completed STS calculations  generated for both.

Notes: Although I discuss several ways of obtaining a simulated STM image, in this section, I describe everything in terms of using Jon's code because it outputs the necessary values that I want.

1. We want to use the same scale for each job. Using Jon's ExampleSTM code, we can generate a simulated STM image for one structure. For example, here I have an image of a 2H MoS2 structure on Cu(111). At a certain tip-sample distance, this will output three relevant values for you: scale, lowest point, and highest point. You can think of the difference between the latter two values as your image corrugation.
   
2. Repeat the same process for your second structure, in my case 1T MoS2 on Cu(111). Using the same scale value you just acquired, use this value instead when running "DispSTM" on your second structure. (Both simulated STM images should be generated with the same scale value.) This will move the tip to a location where it experiences a similar current/charge value from your system.
   
3. If you take the average of the corrugation values from each system, you can obtain the average height of each system using the following equation.
   
4. So now you can compare the heights and corrugations of each system at different tip height settings. Additionally, you can compare the difference that this makes in STS measurements. With Jon's ExampleIV Code, we can set the z values ourselves. Disregarding the offset we use in the above equation, we set the z value to be the height of the tip in the STM simulation. This requires the following change in the code.
   
5. Now you can form a table for comparing tip height vs. structure vs. STS (bandgap).

Optional: You may consider using a "more realistic" STM tip by applying median and Gaussian filters to the isosurface data. This involves delving into the molXML code and editing idxTable. Also, you may find that your corrugation and height values become weird at a certain distance away from the surface. I'm still trying to work out what this means, but according to Profesor Bartels, "maybe at the point of inflection (where the first deriv changes sign) we run out of the accuracy of our method".

STM Imaging Techniques

Quick Description: There are tons of places on the internet that you can go to in order to understand the principles under which STMs operate. However, achieving a good image can be a bit trickier and less ideal than these explanations. Over the past three years, I've worked with 5 different STMs in 2 different labs and have googled a lot in fits of desperation. I've never found really good documentation on this sort of stuff, so I thought I'd include it here, even if it is a bit tangential to other posts in the blog. 

The Point: Getting good STM images.

Prerequisites: A working STM and patience. (If I ever have time, I'd also like to write a post on debugging STM problems.)

Notes: Some techniques are system specific, but just in case it is useful to you, I'll include all the tips and tricks I've picked up over time here. Remember that each STM and sample are unique and require different parameters; at the end of the day, you probably will need to play with all the parameters and see what works for you.

• Low voltage scanning:  A lower voltage brings the tip closer the sample, allowing the probe to more closely follow small features. This provides a higher resolution sample. BUT, you must be careful to have a clean and flat surface. In this mode, the tip is grazing the surface of the sample and can easily be damaged or pick up dirt.

• Shaking the voltage/tip: Quickly pulsing the tip from high to low voltage and back literally shakes the tip and also changes its state dramatically. This can allow you to shake off dirt and/or unwanted instabilities at the tip's apex. You can manually change the voltage very fast, but some STMs have a software feature that will do this for you.

• High voltage scanning (for moving molecules or for cleaning the tip): You may want to scan at higher voltages in two situations. One, if you want to clean your tip, you can sort of blast the tip (~10V) and sample with a high voltage over an area a few times before running away from this likely dirty location and reducing back down to a normal scanning voltage. Two, if you have a lot of molecules free flowing on your surface, your tunneling current and feedback loop can be disturbed but this "sea of molcules". Back your tip away with a high voltage and try using a large gain to distinguish the features of your surface.

• Crashing the tip: This is a bit of a sketchy technique, but if used very carefully and in moderation, you can use this technique to brush off your tip, thereby sharpening and/or cleaning it. To do so, lower the tip until the tunneling current disappears and immediately retract away from the sample.

• Scanning at the edges of your piezos: One trick that we find useful in our lab is to scan towards the limit of the x and y raster piezos. This often reduces noise (perhaps by stiffening the piezos a bit more).

• Gain values: "Optimize the I-Gain and P-Gain values. These values control the feedback loop that adjusts the tip to maintain a constant tunneling current. Increasing these values will give a sharper image up to a point; increasing them too much leads to instability and a noisy image." - Cornell CNS's Intro to STM PDF

• Scan speed: "When you are in a scan range and z range close to where you should see atoms, a good thing to do is speed up the time/line value to avoid these little temperature changes.  A suggested value is 0.06 s/line.  If you can't get this value, make sure you experiment by using one lower than the time/line value you previously were using for the larger scan range values." - Cornell CNS's Intro to STM PDF

• Check for Noise: This may seem obvious, but still... check that you don't have any dangling wires (or any wires with tension on them), things in the surrounding area causing vibration, or the like. If your system uses any sort of spring or air flotation noise suppression, check that those are in place properly. And if other people, experiments, whatever are causing vibrations, try to figure out a way around having that interfere with your experiments. Try looking at the FFT of your current to see if there is a certain frequency contributing to your noisy images. This may clue you in to the source of the noise. Of course this can be filtered out later, but it's better not to have at all in the first place. 
• 60 Hz is likely electrical noise, perhaps coming from a grounding issue
• If you have a inner/outer cryostat cooling system, then it's possible that the inner cryostat (essentially a long thin tube) is misaligned and therefor crashing into the side of the outer cryo. This can be fixed by either rebalancing the table or adjusting the bellow top from which the inner cryo hangs.

Have any more tips and tricks? Please drop a comment! :)

Maintaining the Clusters

Quick Description: Often, we will need to update or fix our clusters, but to do this, we will need need to turn off the computer properly before modifying it and restarting it.

The Point: Take a cluster down. Put it back online.

Prerequisites: SSH Access, such as through using PuTTY.

Notes:

1. Stopping a Job
• To properly stop a job, copy over the STOPCAR file:
• To stop after an ionic step, send the command "cp ~/STOPCAR ."
• To stop after an ionic step, send the command "cp ~/STOPCAR.abort ." but this is pretty much the same as the "qdel" command (as in, this method isn't as nice)
2. Taking down a cluster

• Become a super user by using the command "su" (password required). If later you want to exit being a super user just use "exit"
• SSH into the computer that you want using "ssh <name of cluster>" (make sure you go into the cluster you want because you don't want to accidentally take down vasp1, since it controls everything.
• Enter "shutdown now" - this will close the cluster and then kick you out back into vasp1
• If you want to check you can use "pbsnodes -a" which will show you everything and "pbsnodes -l" will show you all the nodes that are down. Alternatively you can SSH into the cluster you just took down and if it refuses you, then it's down. Some computers will allow you back on because they restart automatically, you will need to physically turn them off.
3. Physically turning off a cluster
• Cluster is located in the Pierce Annex bottom floor lab. The computers are labeled and organized in power strips in the back. You can use the power button or just the power cord. (Note: vasp4, vasp5, and vasp6 are slow.)
• Sweet. Now you can change ram or put in a new hard drive or whatever you want.
• Turn on with the power switch again. (Except for vasp3 which doesn't have one... so just use the power cord.)
4. Restarting a cluster
   
   

Other notes....kind of disorganized still...

Nersc:

useful command for changing from dos CRLF to linux LF (ASCII):
dos2unix [filename] <--(Use only when you use a Windows machine to create INCAR, POSCAR, …)

checking text format: (to check and make sure dos2unix worked).
file [filename]

using vi to see special characters:
vi [filename]
:set list


Torque

Commands for the External Hard drive (which is currently dead):
mount --bind olddir newdir
mount -o uid=username,gid=groupname /dev/sdc /path/to/mount

uid=bartels,gid=users

mount -o remount /home/bartels/sharedData/share2

Related to any NSF mount (what your would type in options on directories to export):
ro,root_squash,sync,no_subtree_check


Check OpenSUSE installation instructions
How to know how much swap space there is:
>ssh vasp(# you wish to view)
>top (to view the swap)
>q (to exit top)

How to give a node more swap space than it currently has (if possible):
umount -a
fuser -m  /home   (then get rid of those processes using kill)
try: umount /home
      umount -a
-in yast, go to harddrives
add partition

-outside yast
mkswap /dev/sda3  (or /dev/sda4 or whatever it is) ... might not be necessary if partition was added via yast
swapon /dev/sda3


df [display hd statistics]
df -h [human readable]
(The commands above, particularly the df –h?, will show you the info that appears at the bottom of the Servlet test page. Should show you what extra drives are mounted and the local harddrives)

To check the cpu info on any server:
cat /proc/cpuinfo [determine freq etc.]

If you restart vasp1 and things won’t work do this:
start NFS server on vasp1 through YAST [if not already started -- yast2]

pbs_mom [on each compute node]
pbs_server
***** not any more: pbs_sched ... -> now called maui
maui
disown –a

If something is wrong with the scheduling system:
(probably have to be su to do this)
momctl -s [shutdown mom]
qterm [shutdown pbs]
schedctl -k [shutdown maui]

To take nodes on and offline (must be su):
pbsnodes -o nodename (takes a node offline)
pbsnodes -c nodename (put the node back online)

Allows you to run 2 8 machines per job (doesn’t always work):
qsub -l nodes=2:ppn=8

If you wanted your job to run on one specific machine:
qsub -l nodes=vasp3:ppn=24 ~/g24vasp

Runs all 2 node machines on one job:
qsub -l nodes=3:ppn=2:proc2 ~/g2vasp

Probably kills everything:
momctl -c 'all'

Shows everything that has happened to a job (note: job must be currently running and must be su)):
tracejob <jobid>

Code to suspend and resume a job (not sure if it works):
qsig -s suspend <jobid> [as su]
qsig -s resume <jobid>

See install nodes manual:
dynamically update nodes info:
  qmgr
  qmgr -c "set node vasp4 properties += dual"

How to kill a job:
qdel <jobid>

MAUI commands: (google them)
maui
mjobctl -s <jobid>
schedctl -k [shutdown maui]
runjob <jobid>
runjob -s <jobid> [this seems to be the best way to suspend]
setspri 10 <jobid>
sethold -b <jobid>   [can do more than 1 job]
releasehold -b <jobid>

Most likely the computer normally mounts to media/New Volume but we wanted to mount it to /home/… so everyone could see it:
mount --bind /media/New\ Volume /home/bartels/sharedData/share2

Commensurate Pieces (Rotate+Cut)

Quick Description: For a project we are working on, we are looking at a film "adsorbed" onto the substrate at varying degrees of rotation. So we take our film in whatever unit cell size we see fit, and we use it as a sort of cookie cutter, cutting out the unit cell of the substrate beneath.

The Point: Using one unit cell to cut out another in an infinitely (or simply many) tiled system.

Prerequisites: You will need to do the math (geometry, algebra, and some crystallography skills) and plan out the model using Corel or something to check that you have two commensurable pieces to join together. There may be a bit of compression or expansion require by the film. Keep track of this scaling value.

Notes:

1. First tile film and substrate out in a large space to ensure that when we rotate the film, there is enough overlap between the two pieces.
• Matlab: "plot_super(<film+substrate>,<unitcell>,[<xrep> <yrep> <zrep>])"
2. Rotate the film.
• Matlab: "rot(<filmname>,<rotation in degrees>)"
  
3. Choose your shape.
• Matlab: The "scatter3" plotting function can be useful in seeing where two groups (say the top layer of your substrate and the bottom layer of your film) overlap in this new system. (Hint: color them differently and use as few layers as possible to prevent confusion.)

• Matlab: Use the tagging option in the plot to pick out the atoms that define your unit cell - remember, you are moving in v1 and v2 directions to the first repeat of your originating atom. You can then export the tags to your workspace to pick up their exact positions and use them to define your shape (make sure to work in an order from corner to corner that draw out the perimeter of your unit cell, in total you will need 4 points).
4. Subtract the appropriate coordinates from one another to define your unit vectors v1 and v2. Your unit cell should be those two vectors with a third vector added that is simply: "0 0 <vacuum>" 
5. Compress with your compression factor (multiply). This applies to your rotated film and the unit vectors that you just defined.
6. Cut your shape. You will do this both for the film and the substrate - double check each separately to make sure you don't pick up extra atoms on the edges.
• Matlab: "<reduced>=reduce(<original>,<shape>)"
  
• Again, you can use the "scatter3" plot function to check coordinates and delete them if necessary from your variable (use the spreadsheet view).
7. Recombine your pieces together and generate a new POSCAR.
• Matlab: "make_pos(<xyz>,<unitcell>, '<elements listed in order of appearance>, 1)"
8. Double check by reading your POSCAR. Look carefully for gaps and/or atoms overlapping.
• Matlab: Open "script". Set the super cell variable "sc". Set the filepath variable "p". And the "type" variable (either "type = 'p' " or "type = 'c' " for POSCAR and CONTCAR, respectively). Run "script".

WAVECAR

Quick Description: This huge file is great to use as a bridge between one job and another, because "...the WAVECAR file which stores the final wave functions. To speed up calculations, VASP usually tries to read this file upon startup. At the end of calculations, the file is written (or if it exists overwritten)." - Penn State's Materials Simulations Tutorial

Prerequisites: See "Submitting Jobs".

Notes: To write the WAVECAR, the following INCAR settings are necessary...

ISTART = 0
LWAVE=.TRUE.
LPARD=.FALSE.
LCHARG=.FALSE.

 Then you can go on to use this to produce your CHGCAR, PARCHG, carry on an optimization without regenerating this file.... etc...