Using Perlbrew for custom Perl environments
Perlbrew lets you have your own complete Perl installation, while leveraging the system's libraries and toolchain.
Perlbrew is a Perl module that orchestrates the building of Perls within your home directory, and modifying your session's environment so that you can use your local Perl automatically. Almost all major versions of Perl can be built using Perlbrew, and Perl code run by your local Perl does not know (or care) that you are running from a local Perl (as opposed to the system Perl).
There are two major advantages to using Perlbrew:
- You can have the latest released version of Perl, without having to wait for SRCC to build and install it.
- You can install whatever modules you want, without having to rely on SRCC to build and deploy them to a system Perl (which might not be doable easily) or a Perl module.
Perlbrew is by far the best option when you are doing any sort of Perl development, because it is so easy to blow away and rebuild the environment when something goes wrong. However, there are also disadvantages to using Perlbrew:
- Only you have access to your Perlbrew. Since other people can't access it directly, they have to get access to a system that has the necessary bits (Perl plus modules) installed already, or they have to make their own Perlbrew.
- Each Perlbrew is not portable. You can not take a Perl that you built on FarmShare and copy it to Sherlock, because different libary versions are in use. Instead, you would have to build fresh in each environment.
- Each Perl takes space in your home directory. For example, Perl 5.24 with a number of modules takes ~725 MB of space.
It is up to you to decide if Perlbrew is the right way to go, or it SRCC should build it for you as a module. Another alternative, if you just want a module and are OK with using the system Perl, is to use local::lib.
This part is really easy! You don't need to install anything (we did that already), but you do need to initialize Perlbrew. You only have to do this once, and it's easy:
akkornel@rice05:~$ perlbrew init perlbrew root (~/perl5/perlbrew) is initialized. Append the following piece of code to the end of your ~/.bash_profile and start a new shell, perlbrew should be up and fully functional from there: source ~/perl5/perlbrew/etc/bashrc Simply run `perlbrew` for usage details. Happy brewing!
You might be told to add something to your
~/.bash_profile file, or you might be told to edit a different file (it depends on what shell you are using). Once that is done, log out and log back in.
For reference, Perlbrew installs all of its software at the path
~/perl5/perlbrew. If you really want to get rid of everything, simply delete that directory, undo the change to your shell profile, log out, and log back in.
Building a Perl
Once Perlbrew has been configured, you can go ahead and install Perl! Perlbrew has Perls going back as far as 5.4.5; new Perls become available once they are released.
First, find out what Perl versions are available:
akkornel@rice05:~$ perlbrew available perl-5.24.0 perl-5.22.2 perl-5.20.3 ... there are lots more!
Once you have chosen the Perl you want to install, simply run
perlbrew install with the Perl version you want. For example,
perlbrew install perl-5.24.0. Or, to get the latest stable Perl, you can run
perlbrew install stable, like so:
akkornel@rice05:~$ perlbrew --notest install stable Installing /home/akkornel/perl5/perlbrew/build/perl-5.24.0 into ~/perl5/perlbrew/perls/perl-5.24.0 This could take a while. You can run the following command on another shell to track the status: tail -f ~/perl5/perlbrew/build.perl-5.24.0.log perl-5.24.0 is successfully installed.
--notest option is used to disable all of the post-build tests. This is needed because one of the tests (the filesystem access-time test) fails (because we don't track last-accessed time on network storage). This is expected by us, but Perlbrew expects all tests to pass, so
--notest is used to disable post-build testing.
You should expect the entire build process to take about half an hour. Once built, your Perl will remain available until you change or delete it.
Using Your Perl
Once Perl is installed, you can use it on any of the systems in the cluster. To see what versions are available, run
akkornel@rice05:~$ perlbrew list perl-5.24.0
To actually use a particular Perl, the command is
akkornel@rice05:~$ perl -v | grep version This is perl 5, version 22, subversion 1 (v5.22.1) built for x86_64-linux-gnu-thread-multi akkornel@rice05:~$ perlbrew list perl-5.24.0 akkornel@rice05:~$ perlbrew use 5.24 ERROR: The installation "5.24" is unknown. akkornel@rice05:~$ perlbrew use 5.24.0 akkornel@rice05:~$ perl -v | grep version This is perl 5, version 24, subversion 0 (v5.24.0) built for x86_64-linux
perlbrew use command can also be placed into scripts, so that your batch jobs can use the Perl you built. You can also leave out the "perl-" prefix when you run
Software Module compatibility
NOTE: In this section, we are talking about the software modules that are loaded using the
module load command. This section does not refer to Perl modules, unless it explciitly says "Perl module".
Your local Perls might conflict with software modules loaded using the
module load command. There are several ways that conflicts can occur:
- If you run
perlbrew use, and then use
module loadto load a module, and that module uses Perl, then that module might end up using your local Perl. This is normally OK, as long as your local Perl has all of the Perl modules that the software wants to use.
- If you load a GCC or clang module, before you run
perlbrew install, then your Perl will be built with that newer compiler. That should be OK, but you will want to test your newly-built Perl without the GCC/clang module loaded.
- if you load a software module to get access to a shared library, and then build a Perl module which uses that shared library; then every time you want to use that Perl module, you will need to load that software module first (so that the Perl module is able to find the shared library).
The important thing to remember is, in most cases, the later software is loaded, the higher its precedence. If you run
module load perl/5.24.0, and then run
perlbrew use 5.22.2, then you will probably be using Perl 5.22 (running
perl -v will tell you for sure).
Installing Perl Modules
Now that you have a Perl built, you probably want to install modules. Installing modules is easy enough with Perlbrew: First you run
perlbrew use to load the Perl version that should get the modules, then you run
cpan install Module_Name, which will find the latest version of the module, build it, test it, and (assuming all those steps passed) install it:
akkornel@rice05:~$ perlbrew list perl-5.24.0 akkornel@rice05:~$ perlbrew use 5.24.0 akkornel@rice05:~$ cpan install Crypt::SSLeay Loading internal null logger. Install Log::Log4perl for logging messages Reading '/home/akkornel/.cpan/Metadata' Database was generated on Tue, 18 Oct 2016 16:29:02 GMT Running install for module 'Crypt::SSLeay' ... many, many lines of output skipped here ... Appending installation info to /home/akkornel/perl5/perlbrew/perls/perl-5.24.0/lib/5.24.0/x86_64-linux/perllocal.pod NANIS/Crypt-SSLeay-0.72.tar.gz /usr/bin/make install -- OK
in the above example, the Crypt::SSLeay module had several required packages that were not installed, so
cpan downloaded and installed those as well. The system's compiler, and system libraries, were used for the build, but all Perl code was executed by your local Perl (not system Perl), and the modules you build will only be used by your local Perl (specifically, the one you're using to run the
One issue you may encounter, when building your own modules, is missing header files. For example, the first time
cpan install Crypt::SSLeay was run on FarmShare 2, it failed to build:
akkornel@rice05:~$ perlbrew use 5.24.0 akkornel@rice05:~$ cpan install Crypt::SSLeay Loading internal null logger. Install Log::Log4perl for logging messages Reading '/home/akkornel/.cpan/Metadata' Database was generated on Tue, 18 Oct 2016 16:29:02 GMT Running install for module 'Crypt::SSLeay' ... many, many lines of output skipped here ... openssl-version.c:2:30: fatal error: openssl/opensslv.h: No such file or directory compilation terminated. Failed to build and link a simple executable using OpenSSL No 'Makefile' created NANIS/Crypt-SSLeay-0.72.tar.gz /home/akkornel/perl5/perlbrew/perls/perl-5.24.0/bin/perl Makefile.PL -- NOT OK akkornel@rice05:~$
This failure happened because a package was missing: Crypt::SSLeay uses OpenSSL to do the actual work; although the OpenSSL libraries were installed, the OpenSSL header files were not. Header files are only needed when you are building software, so the lack of header files caused the build to fail. In this case, installing the libssl-dev package was enough to solve the problem.
If you are trying to use
cpan in your own Perl, and
cpan fails, it may be due to missing libraries or missing header files. Feel free to contact us for help!
In addition to the commands we've explored so far, there are other helpful commands available:
perlbrew offturns off Perlbrew for the current shell. In other words, this switches you back to the Perl version that is installed on the system.
perlbrew upgrade-perlwill take an already-installed Perl, and upgrade it. For example, if you used
perlbrew install perl-5.22.1to build and install Perl 5.22.1, you could upgrade that by running
perlbrew use 5.22.1, followed by
perlbrew switchmakes a specific Perl the defaut one for all future sessions. So, if you have already built Perl 5.24.0 (using
perlbrew install perl-5.24.0), then running
perlbrew switch 5.24.0will make Perl 5.24.0 available every time you start a new session. To un-do this command, run
SRCC is happy to help you! To get support, all you need to do is email us. When you email us, be sure to include the following information:
- The name of the system you were on when you had problems.
- The output of the
It would be nice if you could attach the output of those commands as a separate file, instead of pasting it directly in the email!
- What you are trying to do.
- The complete output that you got when you ran your scripts/commands.
It would be nice if you could attach this as a separate file, instead of pasting it directly in the email!
If the problem is a missing library or missing header files, we can normally address that quickly. If the problem is an out-of-date library, that will take longer to address.