Compiling Software For Personal Use
If you need software that is not available as an environmental module on the cluster (current modules can be seen with the command module avail) you can request this software be compiled as a module by reaching out to orcd-help-engaging@mit.edu. If you are the only user who will be utilizing this software, or if you need a special inhouse version of software for your job, you will need to compile the software yourself for personal use.
Software compilation workflows will vary based on the software and it’s dependencies, but the general workflow for compiling software is as follows. Always refer to the documentation of the software you wish to compile before following these steps, as these are very loosely strutured and the actual installation process of the software you wish to compile may vary greatly.
Please note that this process assumes you have downloaded the source code for the software you wish to compile, and that source code is located at /home/$USER on the engaging cluster:
1. Allocate a node for compilation
To allocate a compute node for compilation, use the command:
salloc -N1 -n4
This will allocate 1 node with 4 CPUs for compiling your software. You will recieve output that your request for allocation has been submitted, and when a node has been allocated for you to use, it will say “[Node_name] are ready for job”.
Then you will want to run the command:
ssh [Node_name]
To ssh into the node to use it for your compilation.
2. Load a compiler module.
Next you will need to load a software compiler. Common compilers include GCC and CMake, both of which are available to be loaded with the commands:
module load gcc/[version]
module load cmake/[version]
After loading a compiler, you will need to load any other dependencies that your software requires. This will vary based on the software you are trying to compile. Please refer to the documentation of the software to review these dependencies. Common dependencies include openmpi or python.
3. Configure software
Next you will want to configure the software’s makefile to install in the correct location, which will be your home directory. You can do this by changing into the directory of your software’s source code and running the command:
./configure --prefix=/home/$USER/[install_directory]
where ‘install_directory’ is the directory where you would like the software to be installed to. If you plan to install multiple pieces of personal software, we recommend making a folder entitled “software” in your home directory and installing software there.
3. Make software
Next you will run the ‘make’ command to prepare the software for installation. Note that this command will vary based on the software you are compiling. Some software comes with multiple packages and you will need to specify which package you intend to make, where as other softwares will only contain one package. Most software can be made with ‘make’ or ‘make all’.
When running this make command, you will want to specify that you want to utilize the 4 CPUs you had allocated for compilation. You can do this by using the command:
make -j 4
If you are making a specific module, you will also specify it in this command. For example, to compile the openmpi module for a software, the command would look like:
make -j 4 openmpi
Or, if you are making all modules included in the software, you will specify it such as:
make -j 4 all
4. Install Software
Once your make command finishes successfully, it is time to install the software. This is done simply with the command:
make install
This process should take longer than the previous make command, and that is why it is important to run this on a compute node so that it has more resources available for it to install faster.
5. Confirm Software Installation Was Successful
At this point, you should be able to run the software you have compiled if compiled successfully. Some software requires sourcing an environmental variable file located in the source code’s directory to properly use software in the command line without specifying the full path of the software’s binaries. Again, please refer to the documentation of the software you are compiling to confirm that you have included every step necessary for compilation of the software.
Example
This is an example of the process followed to install a softwared called Plumed.
1. Allocate Node for Compilation
salloc -N1 -n4
2. SSH into node
ssh node101
3. Load GCC Compiler and dependency modules
module load gcc/6.2.0
module load openmpi/2.1.1
4. Set an environment variable noted from the software’s documentation
export CRAYPE_LINK_TYPE=dynamic
5. Configure software with correct installation path
./configure --prefix=/home/[user]/Software/plumed
6. Make software
make -j 4
7. Install software
make install
8. Source environment variable file noted from the software’s documentation
source sourceme.sh
9. Test software
plumed -h
If you have any questions or problems when compiling software for personal use, please email orcd-help-engaging@mit.edu