While you can connect to Hoffman2 with X11-Forwarding and run a jupyter notebook on a web-browser instance opened on the Hoffman2 cluster, we find that running jupyter notebook or jupyter lab on your local browser is a preferred strategy that can eliminate substantial latency.
To run an jupyter notebook or jupyter lab on the Hoffman2 cluster while displaying it on your local browser please follow the steps outlined here.
On your local machine (the client) you will need:
- a working python installation (version 2.6 or greater)
- a web browser
- an ssh client
- Linux/Mac OSX/MacOS
- A terminal and an ssh client installations which can be found in your $PATH. The ssh client program needs to be able to be invoked with the command name ssh
- Linux/Mac OSX/MacOS
- the script h2jupynb (see below)
Setting up passwordless connection to localhost (on Hoffman2)
First time only: log onto the Hoffman2 cluster to make sure that you can passwordlessly connect to localhost. You will not need to repeat this operation unless you remove the $HOME/.ssh directory from your account on Hoffman2
At the Hoffman2 shell command prompt issue the following commands:
if you are prompted for a password interrupt the command. Which you can do by pressing “Ctrl-c” on your keyboard (press the key “c” while pressing down the key “Ctrl” or “control”):
and generate ssh keys with the command:
add the newly generated public key:
to the bottom of the file:
with the command:
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
Note: unless you remove the file $HOME/.ssh/authorized_keys or the directory $HOME/.ssh or remove the key from the $HOME/.ssh/authorized_keys file you will not need to repeat this procedure again.
Also note: Mind the permissions on your $HOME and $HOME/.ssh directories on the Hoffman2 cluster. $HOME cannot be group (or world) writable. Your $HOME/.ssh should be 700 (for more info please see for example: https://help.ubuntu.com/community/SSH/OpenSSH/Keys).
Starting a Jupyter Notebook on the Hoffman2 cluster & displaying it on your local web-browser
On your local computer:
- Download the script, h2jupynb, available at: https://gitlab.idre.ucla.edu/dauria/jupyter-notebook.git (this script is adapted from: https://github.com/pyHPC/ipynbhpc)
- Execute the script by issuing:
python h2jupynb -u username where: username is your Hoffman2 username alternatively issue (just one time): chmod u+x h2jupynb and invoke the notebook with:
./h2jupynb -u username
- As soon as the interactive session is allocated on the Hoffman2 cluster a jupyter notebook will be opened on your default web browser, please notice:
- unless you have set up passwordless authentication with the Hoffman2 cluster you will be prompted to enter your Hoffman2 password twice
- the default settings are so that the remote jupyter notebook will be allowed to use up to 1GB of memory, one core and will run for up to 2 hours
To learn how to change the default options run:
./h2jupynb -h Usage:
h2jupynb [-u <Hoffman2 user name>] [-t <time in hours>] [-m <memory in GB>] [-s <number of slots>] [-v <python-version>] [-p <port>] [-g <yes/no>] [-c <gpu-card-type>] [-l <cuda-version>] [-o <yes/no>] [-d <dir>] If no arguments are given to this script it is assumed that: your Hoffman2 user name is the same as on your client machine the time duration for your session is of 2 hours the memory per slot for your session is of 1GB the number of slots for your session is of 1 the python version for your notebook is 2.7.13 the port on which the server is started is 8789 use GPU? default is no GPU type default is P4 (if -g = yes) CUDA version 9.1 (if -g = yes) not running on owned nodes the starting directory on Hoffman2 is your $HOME python versions currently available are: 2.7.13, 3.6.1 and 3.7.0 python versions: 2.7.3 or 3.4 are available but deprecated. cuda versions currently available are: 7.0, 7.5, 8.0 and 9.1
As of 2017/05/03 Jupiter Notebook can be run on the Hoffman2 cluster with the following programming languages (kernels):
- Python 2
- Python 3
- R (version 3.2.3)
- Julia 0.7 and more recent need to be installed on the user side (instructions are given below)
Installing kernels in your user account
Installing Julia 0.7 or later
As of 2018/12/10 the most recent version of julia installed on the cluster is 1.0.1, instructions here are given for this specific version. It is also assumed that you are using the 3.7.0 version of python.
Open a terminal (you can do so from a jupyter session following selecting “Terminal” in the scroll-down menu under “New”) on Hoffman2 and issue:
module load python/3.7.0
module load julia/1.0.1
the last command should return:
make note of this value as you will use it in the next step;
At the julia command prompt issue:
Start (or re-start) a jupyter session on Hoffman2 you should now be seeing both julia version 0.6.4 and julia version 1.0.1 it the pull-down “New” menu .
Running Jupyter Lab
Please download the latest version of the h2jupynb script and start the notebook requesting python version 3.7.0:
./h2jupynb -v 3.7.0 ...
(where … is any other option that you need to pass).
When you get to the jupyter session on your browser, go to the address bar and modify it so that instead of localhost:PORT/tree it will have localhost:PORT/lab. For example, should your address will look like:
replace tree with lab as in:
A jupyter lab session should start.
To be able to use jupyter notebook extensions you will need to log on the hoffman2 cluster and issue:
module load python/<version> jupyter contrib nbextension install --user
Report problems at firstname.lastname@example.org