Web Resources

Manual Walkthrough Demo

  1. We assume that you have prepared your PC as described in the text How to prepare for the Singularity Workshop

  2. Connect your PC to the Wifi.

  3. Start VirtualBox and the uibk-singularity-workshop virtual machine (VM). All subsequent steps "on your PC" refer to your VM.

  4. On your PC, start a terminal window.

    Prepare your LCC2 account - necessary steps or the commands below will fail.

    Log on to LCC2 using the login data provided.

    ssh cb0110xx@lcc2
    (enter password)
    ln -s /scratch/$USER Scratch 
    (set new convenient password)
  5. On your PC, install Singularity. First make sure you have logged off from LCC2.

    cd $HOME
    wget https://github.com/singularityware/singularity/releases/download/$VERSION/singularity-$VERSION.tar.gz
    tar xvf singularity-$VERSION.tar.gz
    cd singularity-$VERSION
    ./configure --prefix=/usr/local
    sudo make install
    cd $HOME
  6. On your PC, create working directory for first exercise.

    cd $HOME
    ls -la             # note contents of your $HOME
    mkdir sing-walk1
    cd sing-walk1
    wget https://www.uibk.ac.at/downloads/c102mf/singularity-2018/outside.c
       # outside.c used for demonstration purposes only

    Create new container in sandbox directory.

    sudo singularity build --sandbox walk1 docker://ubuntu:latest

    Inspect contents of sandbox

    ls -la walk1
  7. Note the appearance of your shell prompt:

    singularity@uibk-singularity:~/sing-walk1$ ▮

    Start container writable shell as root...

    sudo singularity shell --writable walk1

    Note the different shell prompt inside the container:

    Singularity walk1:~> ▮

    Look around (check contents of current working directory, /, $HOME, /etc/os-release, output of hostname command etc.)

    pwd               # where am I
    ls -la /          # container's root directory
    echo $HOME        # note that this is root's HOME in host system
    ls -la $HOME
    cat /etc/os-release

    then start configuring your container: update OS, install OS utilities...

    export LANG=C LC_ALL=C
    apt-get update
    apt-get -y upgrade
    apt-get -y install nano vim less gcc make wget
    apt-get -y autoremove

    copy demo programs from web, compile and test in /data. mycat is a simple variant of cat(1) which adds line numbers, and hello echoes all components of argv.

    mkdir /data /scratch
    cd /data
    mkdir src bin
    cd /data/src
    wget https://www.uibk.ac.at/downloads/c102mf/singularity-2018/hello.c
    wget https://www.uibk.ac.at/downloads/c102mf/singularity-2018/mycat.c
    cc -o ../bin/hello hello.c
    cc -o ../bin/mycat mycat.c
  8. Still in the container shell, test your programs

    export PATH=/data/bin:$PATH
    hello a b c
    mycat *.c | less
    ls -la / | mycat | less
  9. Make sure you have exited your container shell.

    Convert container to squashfs. Start shell with the read-only squashfs image of the container (this time non-root)

    sudo singularity build walk1.simg walk1
    singularity shell walk1.simg
    pwd              # where am I
    ls -la           # what is here
    ls -la /         # what is in my root directory
    echo $HOME       # which $HOME is this
    ls -la $HOME     # what is in $HOME
    cd $HOME/sing-walk1
    ls -la
    mycat outside.c
    cp outside.c junk1.c  # yes we have full access to our host's HOME

    Note that now you are in your HOME directory, not ROOT's. From within the container you can read and write files in your $HOME. Try more things and exit container when done.

  10. Make sure you have exited your container shell.

    Transfer container image to LCC2 and log on there:

    scp walk1.simg cb0110xx@lcc2.uibk.ac.at:Scratch
    ssh cb0110xx@lcc2.uibk.ac.at
    cd Scratch
    module load singularity/2.x
    singularity shell walk1.simg
    cd /data/src

    and try similar tests as above (step 9). Note this time you are accessing your LCC2 $HOME.

    Also inspect mounted file systems and note which parts of the host's directory tree are visible from within the container.

    df -a

    When done...


    from container shell and...

  11. ... demonstrate use of container as part of pipeline

    cat $HOME/.bashrc | singularity exec walk1.simg /data/bin/mycat | less

Exercise: Build Container using Build File

Prerequisites: you have successfully installed Singularity.

  1. On your PC, prepare working directory for new container. This time we will copy data from the working directory into the container.

    cd $HOME
    mkdir sing-automate
    cd sing-automate
    wget https://www.uibk.ac.at/downloads/c102mf/singularity-2018/hello.c
    wget https://www.uibk.ac.at/downloads/c102mf/singularity-2018/mycat.c
    wget https://www.uibk.ac.at/downloads/c102mf/singularity-2018/automate.build
  2. Carefully inspect the build file. All steps we had to do manually at build and execution time are now automated.

    less automate.build
  3. Build your container directly into a squashfs image and observe the output

    sudo true  # make sure build is not interrupted by password prompt
    sudo singularity build automate.simg automate.build > build.out 2>&1 &
    less build.out
    <SHIFT-F>         # to view output file as it grows
    <CTRL-C>          # to revert to normal less mode
  4. Repeat above experiments locally or on LCC2. Demonstrate that a container can process files in current working directory:

    singularity exec automate.simg mycat *.c

Exercise: Use MPI with Singularity

Prerequisites: you have successfully installed Singularity.

  1. On your PC, prepare working directory for new container. This time we will copy data from the working directory into the container.

    cd $HOME
    mkdir sing-mpitest
    cd sing-mpitest
    wget https://www.uibk.ac.at/downloads/c102mf/singularity-2018/mpitest.build
    wget https://www.uibk.ac.at/downloads/c102mf/singularity-2018/mpitest.sge
  2. Carefully inspect the build file. Note the download and compilation of the OSU MPI benchmark.

    less mpitest.build
  3. Build your MPI test container directly into a squashfs image

    sudo true  # make sure build is not interrupted by password prompt
    sudo singularity build mpitest.simg mpitest.build > build.out 2>&1 &
    less build.out
  4. Copy your container to LCC2 and prepare target directory.

    scp mpitest.simg mpitest.sge cb0110xx@lcc2.uibk.ac.at:Scratch
    ssh cb0110xx@lcc2.uibk.ac.at
    module load singularity/2.x openmpi/2.1.1
    cd Scratch
    mkdir sing-mpitest
    mv mpitest.* sing-mpitest
    cd sing-mpitest
  5. Run the benchmark program locally on login node and note the output.

    mpirun -np 2 singularity exec mpitest.simg osu_bw
  6. Inspect the batch script mpitest.sge, then submit it to the cluster and watch.

    less mpitest.sge
    qsub mpitest.sge
    watch qstat -u $USER
    ls -l

    Inspect the output file and try to explain the different performance.

Nach oben scrollen