Installing Your Own Software

Need new software or a more recent version of an installed application? Not a problem.

Building Software from Scratch

Much software is available as source code and can be configured to target a local directory for installation. The process, after downloading the source archive and unpacking it into a new subdirectory, usually involves some combination of the configure and make commands.

rice03:~$ ./configure --prefix=$HOME/.local
rice03:~$ make
rice03:~$ make test
rice03:~$ make install

Detailed instructions are often included in a README or INSTALL file, so check the source directory before you begin!

The prefix directory is where build products will end up, organized in subdirectories (e.g., bin, lib, and include). In the example above newly built software is installed in a (hidden) subdirectory of your home directory, ~/.local, but you can choose any subdirectory you like. You may want to modify your shell environment so new software can be easily found. At the least this means adding the bin subdirectory to your $PATH (e.g., in ~/.bashrc).

export PATH=$HOME/.local/bin:$PATH

While this method works well for simple applications and libraries, more complex software may require you to apply patches or build any dependencies that are not already installed. Luckily, there are a number of tools that can automate this process!

Using a Package Manager

Package managers provide simple, single-command installation of software from a central repository, usually with little or no additional configuration required. This is the most convenient method for most users, and you'll probably want to fall back on a manual build only when a package is not available in your preferred package manager.

Linuxbrew

Linuxbrew is a port of the Homebrew package manager for macOS. Installation is simple!

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install)"

By default, both Linuxbrew itself and all managed software are installed in ~/.linuxbrew, so you’ll want to add $HOME/.linuxbrew/bin to your $PATH.

The brew command is used to search the repository, get information on software, and install packages.

rice03:~$ brew search tmux
rice03:~$ brew info tmux
rice03:~$ brew install tmux

Linuxbrew can also update itself and the list of available packages, and upgrade already installed software when new versions become available.

rice03:~$ brew update
rice03:~$ brew upgrade tmux

Software is divided into a number of repositories. The default repositories contain thousands of packages, but if you’re looking for a particular scientific or numeric application you may want to add the homebrew/science repository.

brew tap homebrew/science

Run brew help or man brew for more information on how to use the command.

Spack

Spack is a package manager targeted at scientific computing. It has support for fewer packages than Linuxbrew and is a bit more difficult to install, configure, and use, but it provides far more extensive control over versions, dependencies, and build-time options when installing software. Spack also supports installing more than one version of a single package, or multiple installations of the same version with different build options, and integrates with FarmShare's existing module system, allowing you to select from among those versions at runtime.

To install it, just clone the Spack git repository.

git clone https://github.com/llnl/spack.git .spack

In this example Spack and managed software is installed in ~/.spack, but you can choose another directory if you like. You should probably add $HOME/.spack/bin to your $PATH but, unlike Linuxbrew, installed software is not linked in ~/.spack/bin. Instead, Spack automatically creates modules that manage the environment for each package. You’ll need to enable shell integration by running source $HOME/.spack/share/spack/setup-env.sh so you can load the modules.

The spack command can be used to search the software repository and get information on packages.

rice03:~$ spack list python
rice03:~$ spack info python

When building a package you can control a number of options, including the specific version to install (using @), what "variant" to build (+ and -), the exact versions of any dependencies that will be installed (^), and even which compiler to use (%). The list of options provided is called a "spec;" you can list installed packages and load a particular version according to its spec.

rice03:~$ spack install python @3.5.2 +ucs4 %clang
rice03:~$ spack find python
rice03:~$ spack load python@3.5.2

Spack can’t update itself, but because it is a git repository you can check out releases as they are tagged or use git pull to stay in sync with the latest changes. It also won't upgrade installed packages automatically, but you can use spack install to build a new version whenever one becomes available, and choose which version to load by spec.

Run spack help for details on syntax and usage, and read the excellent Spack manual for more information.

Using Containers