FAQ: Installing your own applications on SuperMUG-NG (internet access from SuperMUC-NG)

To install your own software packages from the internet by using git/svn, tarballs, or via an installation script that fetches files via http or ftp, please consider the following options. SuperMUC-NG has rather tight security restrictions, and thus a firewall, which prevents any outgoing connections.

Copy all needed installation files to supermuc

The easiest way to install a software package is to copy all needed files into a directory on SuperMUC-NG, unpack them and run the configure script. Be sure that you have resolved all the dependencies and then you can compile the software and install it in a directory in your home. Most of the time this will include an option like this:

sng> ./configure --prefix=/home/<group>/<account>/mydir
sng> make
sng> make install

Before, load the environment modules as needed. Strive for a clean built environment. Specifically, unload the Intel modules when you want to build your software with GCC.

Mount a directory of SuperMUC-NG on your local machine

Another way is to mirror the directory on the supermuc to your local machine (when you are running SUSE SLES11 or another compatible linux) and install the software on your local machine.

This can be done by creating a directory on supermuc in your HOME directory e.g. $HOME/mydir and a directory at your local machine $HOME/mydir and then mounting it. You need sshfs in order to do the mapping on your local machine.

local> mkdir sng_mnt
local> sshfs <account>@skx.supermuc.lrz.de:mydir ./sng_mnt

Then install the software on your local machine in the directory sng_mnt. It will be automatically mirrored to the supermuc directory and can then be used there. Once finished, use fusermount -u sng_mnt to unmount the directory again. (Warning: Latency due to slow network connection is possible.)

Create a reverse SSH tunnel for internet access from SuperMUC-NG login node

In case the last method fails you can create a tunnel via ssh from your local machine to SuperMUC-NG and use a proxy at your local machine to re-direct internet connections from supermuc to the internet.

Get a proxy software for http connections and unpack it on your local machine (pip3 install --upgrade --user proxy.py   for python3).

Then start the local proxy using:

local> python TinyHTTPProxy.py -p 1234

or

local> .local/bin/proxy --port 1234

for the pip installed proxy for python3.

Then create a tunnel to supermuc via:

local> ssh -l <kennung> -R 1234:localhost:1234 skx.supermuc.lrz.de

On SuperMUC-NG, you must set http_proxy, HTTP_PROXY, https_proxy and HTTPS_PROXY to localhost:1234 (adapt ports as needed), and have full access to the http protocoll. (e.g. firefox etc).

sng> export http_proxy="localhost:1234"                          
sng> export https_proxy="localhost:1234"                         
sng> export HTTP_PROXY="localhost:1234"                          # for conda, "http://" may need to be prepended 
sng> export HTTPS_PROXY="localhost:1234"                         # for conda, "https://" may need to be prepended 

Also user_spack can be possibly used with these settings. Otherwise, check for ftp, git etc. whether there are similar proxy settings needed!

Trouble Shooting: This workflow was tested to work also under local Windows environment with either Virtual Box or WSL -- Debian or Ubuntu. For Ubuntu, you probably must change in the /etc/hosts a line to match

 ::1  localhost  ip6-localhost ip6-loopback

(Warning: Latency due to slow network connection is possible.)

Spack Mirrors

https://spack-tutorial.readthedocs.io/en/latest/tutorial_binary_cache.html

The idea is to bundle source code or binary packages with all dependencies together, move the bundle to the SuperMUC–NG, and unroll/install it there. This is also possible with conda and pip.