Jupyter notebook

It is possible to connect to the Hoffman2 cluster opening a Jupyter Notebook. or a Jupyter Lab user interface.

While you can connect to Hoffman2 with X11-Forwarding and run a jupyter notebook on a web-browser instance opened on the Hoffman2 cluster, we find that running jupyter notebook or jupyter lab on your local browser is a preferred strategy that can eliminate substantial latency.

To run an jupyter notebook or jupyter lab on the Hoffman2 cluster while displaying it on your local browser please follow the steps outlined here.

Prerequisites

Software

On your local machine (the client) you will need:

  1. a working python installation (version 2.6 or greater)
  2. a web browser
  3. an ssh client
    • Linux/Mac OSX/MacOS
      • A terminal and an ssh client installations which can be found in your $PATH. The ssh client program needs to be able to be invoked with the command name ssh
    • Windows
  4. the script h2jupynb (see below)

Setting up passwordless connection to localhost (on Hoffman2)

First time only: log onto the Hoffman2 cluster to make sure that you can passwordlessly connect to localhost. You will not need to repeat this operation unless you remove the $HOME/.ssh directory from your account on Hoffman2

At the Hoffman2 shell command prompt issue the following commands:

ssh localhost

if you are prompted for a password interrupt the command. Which you can do by pressing “Ctrl-c” on your keyboard (press the key “c” while pressing down the key “Ctrl” or “control”):

and generate ssh keys with the command:

ssh-keygen

add the newly generated public key:

$HOME/.ssh/id_rsa.pub

to the bottom of the file:

$HOME/.ssh/authorized_keys

with the command:

cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

Note: unless you remove the file $HOME/.ssh/authorized_keys or the directory $HOME/.ssh or remove the key from the $HOME/.ssh/authorized_keys file you will not need to repeat this procedure again.

Also note: Mind the permissions on your $HOME and $HOME/.ssh directories on the Hoffman2 cluster. $HOME cannot be group (or world) writable. Your $HOME/.ssh should be 700 (for more info please see for example: https://help.ubuntu.com/community/SSH/OpenSSH/Keys).

Starting a Jupyter Notebook on the Hoffman2 cluster & displaying it on your local web-browser

On your local computer:

  1. Download the script, h2jupynb, available at: https://gitlab.idre.ucla.edu/dauria/jupyter-notebook.git (this script is adapted from:  https://github.com/pyHPC/ipynbhpc)
  2. Execute the script by issuing:
    python h2jupynb -u username
    
    where: username is your Hoffman2 username
    
    alternatively issue (just one time):
    
    chmod u+x h2jupynb
    
    and invoke the notebook with:

    ./h2jupynb -u username

  3. As soon as the interactive session is allocated on the Hoffman2 cluster a jupyter notebook will be opened on your default web browser, please notice:
    • unless you have set up passwordless authentication with the Hoffman2 cluster you will be prompted to enter your Hoffman2 password twice
    • the default settings are so that the remote jupyter notebook will be allowed to use up to 1GB of memory, one core and will run for up to 2 hours

To learn how to change the default options run:

./h2jupynb -h
Usage:
h2jupynb [-u <Hoffman2 user name>] [-t <time in hours>] [-m <memory in GB>] [-s <number of slots>] [-v <python-version>] [-p <port>] [-g <yes/no>] [-c <gpu-card-type>] [-l <cuda-version>] [-o <yes/no>] [-d <dir>]

If no arguments are given to this script it is assumed that:
your Hoffman2 user name is the same as on your client machine
the time duration for your session is of 2 hours
the memory per slot for your session is of 1GB
the number of slots for your session is of 1
the python version for your notebook is 2.7.13
the port on which the server is started is 8789
use GPU? default is no
GPU type default is P4 (if -g = yes)
CUDA version 9.1 (if -g = yes)
not running on owned nodes
the starting directory on Hoffman2 is your $HOME

python versions currently available are: 2.7.13, 3.6.1 and 3.7.0
python versions: 2.7.3 or 3.4 are available but deprecated.

cuda versions currently available are: 7.0, 7.5, 8.0 and 9.1

Available kernels

As of 2017/05/03 Jupiter Notebook can be run on the Hoffman2 cluster with the following programming languages (kernels):

    • Python 2
    • Python 3
    • R (version 3.2.3)
    • R/3.5.1
    • Julia/0.6.4
    • Julia 0.7 and more recent need to be installed on the user side (instructions are given below)

Installing kernels in your user account

Installing Julia 0.7 or later

As of 2018/12/10 the most recent version of julia installed on the cluster is 1.0.1, instructions here are given for this specific version. It is also assumed that you are using the 3.7.0 version of python.

Open a terminal (you can do so from a jupyter session following selecting “Terminal” in the scroll-down menu under “New”) on Hoffman2 and issue:

module load python/3.7.0
module load julia/1.0.1
which jupyter

the last command should return:

/u/local/apps/python/3.7.0/bin/jupyter

make note of this value as you will use it in the next step;

Start julia:

julia

At the julia command prompt issue:

using Pkg
Pkg.add(“IJulia”)

Start (or re-start) a jupyter session on Hoffman2 you should now be seeing both julia version 0.6.4 and julia version 1.0.1 it the pull-down “New” menu .

Running Jupyter Lab

Please download the latest version of the h2jupynb script  and start the notebook requesting python version 3.7.0:

./h2jupynb -v 3.7.0 ...

(where … is any other option that you need to pass).

When you get to the jupyter session on your browser, go to the address bar and modify it so that instead of localhost:PORT/tree it will have localhost:PORT/lab. For example, should your address will look like:

http://localhost:9394/tree

replace tree with lab as in:

http://localhost:9394/lab

A jupyter lab session should start.

Notebook Extensions

To be able to use jupyter notebook extensions you will need to log on the hoffman2 cluster and issue:

module load python/<version>
jupyter contrib nbextension install --user

Report problems at hpc@ucla.edu

Report Typos and Errors
UCLA OIT

© 2016 UC REGENTS TERMS OF USE & PRIVACY POLICY