ResourcesMark

Computers

• Lychee (serves /home)
• Guava
• Melon
• Grape
• Kiwi
• Compbio
• CentosNodes
• CentosWorkstation
• ThinClients
• Mitchell Workstations
• Cisco Switches

Other Information

• Network
• Faults

What to do when the computers crash and an admin is not there

1. Shutdown workstations, nodes, lychee and guava
2. Turn on:
   1. guava
   2. lychee
   3. nodes
   4. workstations

How do I find out how to use a program

• Use the manual pages. Run: man PROGRAM_NAME

How do I transfer files from lychee without slowing down the network

• Use rsync with the --bwlimit option. Read up on using rsync in the manual pages

How to enable a computer to send automatic emails to non-UQ addresses?

• If postfix is used for SMTP, in /etc/postfix/main.cf make sure relayhost = smtp.uq.edu.au

What kinds of workstations are used in the lab?

• AMD 64-bit workstations, for floating-point consistency with the cluster, which also has AMD 64-bit processors. We also have only NVIDIA video cards for ease of maintenance.

How to reinstall/install an NVIDIA driver?

• An NVIDIA display driver depends on a specific version of the kernel, so if the kernel is updated it will not work any more (you will just get a blank screen). To install an NVIDIA display driver from the .run file downloaded from nvidia.com (e.g. cudadriver_2.3_linux_64_190.18.run - 190.18 is the driver version; linux_64 means it can be installed for any version of a 64-bit linux kernel), do the following as a user:
  • Login via ssh
  • mv /path/to/cudadriver_2.3_linux_64_190.18.run /tmp # this step ensures root can read the run file because its original location, e.g. in a nfs mounted directory, may not be readable by root.
  • sudo init 3 # this closes the X server
  • sudo sh /tmp/cudadriver_2.3_linux_64_190.18.run # this starts the driver
    • Say yes to everything
  • sudo init 5 # this restarts the X server

How to set up a VPN connection from a workstation?

• Example workstation: uqmd00.md.smms.uq.edu.au
• Example username: bob
• Example vpn server/Host: a.com, at 12.13.14.15, on network 12.13.14.00/255.255.255.0
• Example server of interest only accessible through vpn, a.a.com at 12.13.16.1, on network 12.13.16.0/255.255.255.0
• The Host, GroupName and GroupPwd can be read from CISCO .pcf file, if only enc_GroupPwd is found you can decrypt it using this site.
• Get ITS to allow incoming Cisco VPN connections from a.com to uqmd00.md.smms.uq.edu.au
• Once these are done, follow the following instructions to set up VPN:

1. sudo yum install vpnc # Install vpnc
2. sudo vim /etc/vpnc/vpnc.conf # Edit settings

IPSec gateway a.com
IPSec ID GroupName
IPSec secret GroupPwd

1. sudo /sbin/vpnc /etc/vpnc/vpnc.conf # Start vpn client. Warning: this screws up the routing and DNS settings. The following steps are to correct them.
2. sudo /sbin/route # Print route table to check status. All internet now goes via the vpn (tun0).

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
a.com           gateway-b195-12 255.255.255.255 UGH   0      0        0 eth0
152.98.195.128  *               255.255.255.192 U     0      0        0 eth0
12.13.14.00     *               255.255.255.0   U     0      0        0 tun0
default         *               0.0.0.0         U     0      0        0 tun0

1. sudo /sbin/route add -net 12.13.16.0 netmask 255.255.255.0 dev tun0 # Add route so traffic to a.a.com goes through tun0
2. sudo /sbin/route del default dev tun0 # Remove wrong default gateway. Warning: Internet traffic doesn't get routed anywhere
3. sudo /sbin/route add default gw 152.98.195.129 eth0 # Put right default gateway in. Internet traffic goes through the normal gateway for the MD VLAN (gateway-b195-128 = 152.98.195.129)
4. sudo /usr/bin/vim /etc/resolv.conf # Set DNS settings

domain md.smms.uq.edu.au
search md.smms.uq.edu.au
nameserver 130.102.128.43
nameserver 130.102.2.15

Endnote X on kiwi

• To set up Endnote X on kiwi for a new user, the files EN10Cwyw.dot, EN10Cwyw.WordXP.wll and ENWebCwyw.dot must be copied to the USERNAME's "C:\Documents and Settings\USERNAME\Application Data\Microsoft\Word\STARTUP" directory from "C:\Program Files\EndNote X".

Linux introduction for new users

• Please read Introduction_To_Computing

System Configuration (workstations, grape)

• System

Thin Client/workstation configuration

• ThinClients

System Configuration (lychee & mango*)

lychee system

• lychee sytem
• Network Booting and Imaging
• Batch system and scheduling
• Directory Server

System Configuration (Melon - Solaris ZFS)

• SolarisZfs
• NFS4permissions

NFS disk layout

• lychee:

        /home      500G      (for MDGroup/BMMG, quota for each user: soft 30G, hard 40G)
        /home1     250G      (for SBeatson)
        /home2     250G      (for KobeLab)
        /data5     500G      
        /data7     150G      (backups of old user accounts)

• guava:

        /data1     500G
        /data2     500G
        /data3     500G
        /data6     180G

• grape:

        /data8     400G
        /data9     400G
        /data10    400G

• melon:

        /melon1    8.1T       (data pool for MDGroup/BMMG, zfs quota for user 500G)
        /melon2    8.1T       (data pool for SBeatson)

Cluster Command & Control (c3) tool suite

c3 is install on lychee, the configure file is /etc/c3.conf . The binaries are installed in /opt/c3-4 .

To run a command to all nodes:

       sudo cexec <command>

To upload a file to all nodes:

       sudo cpush <file> /path/for/the/file

Alarm handling/Shutting down the rack

• Alarm Handling

Booting up the system

1. Boot up NFS servers: guava, grape, melon. melon may take a long time to boot, so be patient.

2. Boot up our main server: lychee

3. Boot up all cluster nodes and all workstations.

System Configuration (windows and Linux laptop)

• System windows
• System Linux

System Configuration (CCMS Blackhole)

• How To's

Grape queueing system

• Restarting the queue
• Administration
• Debugging Queue Errors
• Restart Nodes without jobs

Add new user (LDAP)

Please use the system "firstnamel", where l is the first letter of the last name, for new user names

Managing Fedora-DS

To get a untaken UID for the new accont, use nextUID.py placed in '/opt/scripts'. Note that the UID may be already in use on grape.

To add a group of new users with similar uid, use script ldapBatchAddUser.py placed in '/opt/scripts/'. You will have to run it as root, and keep in mind the following things.

       This program helps to create multiple user accounts on the fly. 
       In order to use it, you have to provide some basic information 
       of the new accounts. The new accounts will be named as "BASE$N",
       in which "BASE" is the base name, and $N is the count. You have
       to specify the starting number of $N. Users' home directory
       will be /home/BASE/BASE$N, and bear in mind that this program
       does not create /home/BASE automatically. The program will invoke
       "nextUID.py" to set UIDs for the new accounts, However, you still 
       have to take care of the GID issue yourself. All new accounts
       will use "BASE" as their passwords.

       Modify the program yourself to make it better suit your need.

       Use CTRL+C to interrupt the program when typing something wrong.

add user for kiwi

• go to administration tools in control panel, and create a user with an identical name as the one in LDAP.
• right click and set up properties of the user the same as the others.
• right click and set the passwd of the user.
• go to start menu -> Services for Unix administration, select "user name mapping" on the left panel, select maps on the right, then add the new mapping. If the user id does not appear in the unix user list: on lychee, do

      getent passwd > /nfs4exports/passwd.txt
      getent group > /nfs4exports/group.txt

copy those files to kiwi's c:/ , and refresh the unix user list.

Add new user (grape)

Create account in /etc/passwd

useradd -u NEWUID -g md NAME

Update NIS (for cluster authentication)

cd /var/yp/
make

Gromos Live (ubuntu based live DVD)

The latest DVD image can be found in /home/exercise/LiveDVD

The DVD is based on the one from IGC ETH with some minor modifications including:

• changing gnome background
• removing unnecessary language and game packages
• changing firefox homepage (/usr/share/ubuntu-artwork/home/index.html)
• changing license file in GROMOS directory, and moving GROMOS directory from /opt to /usr/local
• installing gromacs to /opt
• removing nfs-common, portmap and games packages

Modify the Live DVD

Guidelines on how to modify an ubuntu live cd can be found in the following urls. Note that ALL of the above modifications are made in squash file:

http://www.debuntu.org/how-to-customize-your-ubuntu-live-cd

http://www.atworkonline.it/~bibe/ubuntu/custom-livecd.htm

Here is a rough walkthrough:

• mount the original iso and unpack everything to a folder, here we choose /media/live as our working directory.

  sudo mount -t iso9660 -o loop /path/to/gromos-live.iso /mnt/dvd
  sudo cp -a /mnt/dvd /media/live/dvd

• mount the squashfs image and unpack everything

  sudo mount -t squashfs -o loop /media/live/dvd/casper/filesystem.squashfs \
       /mnt/squash
  sudo cp -a /mnt/squash /media/live/squash

• umount the original iso

  sudo umount /mnt/squash; sudo umount /mnt/dvd

• chroot to the squash folder and start to make modifications

  cd /media/live/squash
  cp /etc/resolv.conf etc/
  sudo chroot .
  mount -t proc none /proc
  mount -t sysfs none /sys
  export $HOME=/root

• make changes to the files in squash
  • to install new package or remove a package, use apt-get as in a normal ubuntu installation.
  • files can be put directly in to the directories and then accessed after boot, but make sure files in /home/ubuntu have owner and group id set to 999:999
• clean up before reimaging

  rm -rf /tmp/*
  apt-get clean
  umount /sys
  umount /proc
  rm /etc/resolv.conf
  exit

• regenerate squash image

  cd /media/live/dvd
  sudo chroot ../squash dpkg-query -W --showformat='${Package} ${Version}\n' \
       > casper/filesystem.manifest
  sudo rm casper/filesystem.squashfs
  sudo mksquashfs ../squash casper/filesystem.squashfs

• regenerate iso image

  cd /media/live/dvd
  sudo find . -type f -print0 |xargs -0 md5sum |sudo tee md5sum.txt
  cd ..
  sudo mkisofs \
    -o Gromos-Live.iso \
    -b isolinux/isolinux.bin \
    -c isolinux/boot.cat \
    -no-emul-boot \
    -boot-load-size 4 \
    -boot-info-table \
    -r \
    -V "Gromos Live" \
    -cache-inodes  \
    -J \
    -l \
    dvd

Port the live DVD to USB key drive

please refer to the following url. As Gromos Live is based on Ubuntu 7.10, we cannot use scripts written for other versions.

http://www.pendrivelinux.com/2007/09/28/usb-ubuntu-710-gutsy-gibbon-install/

Install the live DVD to a hard disk

• copy the unpacked /casper to the partition that you want to install on.
• run sudo grub-install --root-directory=/media/hda1 --force-lba --no-floppy /dev/hda to install grub to the hard drive.
• write the following content to /boot/grub/menu.lst, and modify it if needed.

timeout         10

color cyan/blue white/blue

title           Start Ubuntu in Persistent mode
root            (hd0,0)
kernel          /casper/vmlinuz boot=casper initrd=/casper/initrd.gz 
                 ramdisk_size=1048576 root=/dev/ram rw persistent
initrd          /casper/initrd.gz
boot

title           Live - Start or install Ubuntu
root            (hd0,0)
kernel          /casper/vmlinuz boot=casper initrd=/casper/initrd.gz 
                 ramdisk_size=1048576 root=/dev/ram rw
initrd          /casper/initrd.gz
boot

title           Start Ubuntu in safe graphics mode
root            (hd0,0)
kernel          /casper/vmlinuz boot=casper xforcevesa initrd=/casper/initrd.gz
                 ramdisk_size=1048576 root=/dev/ram rw
initrd          /casper/initrd.gz
boot

title           Check CD for defects
root            (hd0,0)
kernel          /casper/vmlinuz boot=casper integrity-check initrd=/casper/initrd.gz
                 ramdisk_size=1048576 root=/dev/ram rw
initrd          /casper/initrd.gz
boot

title           Memory test
root            (hd0,0)
kernel          /install/mt86plus
boot

How to compile GROMOS96 on APAC (ac)

In order to run parallel version of GROMOS96 on APAC (ac) the following changes should be made:

In ../src/promd_db.F change the line

    call system (
   $'cat /proc/cpuinfo |grep MHz| awk {print $4} > tmp.info.xyz')

to

    call system (
   $"cat /proc/cpuinfo |grep MHz| awk '{print $4}'> tmp.info.xyz")

Single quotes became a double quote and the pair of single quotes before and after the {} became just one single quote each.

You should make the same corrections a couple of lines later as well even though the code will not enter that conditional.

Then a bit further on change it to read

   call system (
   $"cat /proc/cpuinfo |grep model |
   $  awk -F: '{print $2}' > tmp.info.xyz")

We should thank Dr Margaret Kahn from APAC for this solution.

Revision Control System (git+gitosis+redmine)

Git is used as revision control system on lychee, as it makes merging branches easier. Gitosis is installed to provide a safer and easier way of publishing, and home directory of user "git" is /srv/git, in which all the source code is put. Redmine provides a friendly user interface for project management and code browsing, and can be accessed at http://lychee:3000 .

Configuration of Gitosis

Gitosis uses pure git commands to update its configurations.

To start, first add yourself to the admin group of gitosis.

   sudo -H -u gitosis gitosis-init < /path/to/your/sshkey/id_rsa.pub

Then you can download its configurations

   git clone git@lychee:gitosis-admin.git

modify gitosis.conf, to add a new project named "atb"

   [group atb]
   writable = atb
   members = roy root alpesh pramod

copy user roy, root, alpesh, pramod's id_rsa.pub to {{{keydir}}}, name them to <uid>.pub

Then push the updated version

   git commit -a -m "project atb added"
   git push

Configuration of redmine

• install rubygems, and use it to install right version of rails according to redmine's homepage.

• copy config/database.yml.example to config/database.yml and edit it to reflect the type of database in use. Here we use sqlite3.

• run

    rake db:migrate RAILS_ENV="production"
    rake redmine:load_default_data RAILS_ENV="production"

• now the page can be served by running

    ruby script/server webrick -e production

default login/passwd : admin/admin

• add {{{ruby script/server webrick -e production}}} to /etc/rc.local to make a daemon

git user configuration

To make git aware of who is the contributor and whom to blame, add the following to ~/.gitconfig, email address is important

  [user]
     name = <full name of the user>
     email = <email of the user>

clone the repository:

   git clone git@lychee:atb

Tips

Use rsync

An efficient way to transfer files or directory between computers:

  rsync -avzP --delete -e 'ssh -p20' username@old_server:/old_place/ \ 
    username@new_server:/new_place/

Important, the flag --delete will delete any file at the target which does not appear at the source. Also you should put at the port flag the port on our system (if you don't know, ask the one of the 'Guru's).

Submit jobs from any client machines

• You need to setup key based ssh login.
• Put the following in your .bashrc, then you can use qsub, qdel, showq, qnodes, qstat from everywhere.

  if [ "$HOSTNAME" != "lychee" ]; then
      function qsub()
          {
              ssh lychee "cd $PWD;
              /opt/bin/qsub $1";
          }
      for i in showq qdel qstat qnodes showstart; do
          alias $i="ssh lychee /opt/bin/$i"
      done
  fi

Mount your home directories on your linux laptops with sshfs

First you have to install fuse, and

          modeprobe fuse

to install the module. You may also want to add fuse to your /etc/modules.

Then, use the following line to mount your sshfs. You must also have public key based authentication setup correctly.

          sshfs -p 2023 <account>@lychee: /path/to/your/local/mount/point

For mac users, there is a sshfs port for mac users.

How to compile GROMOSXX 0.3.1 (Oct09) on NCI(APAC) Vayu and Xe clusters with openmpi

 Get the source code from:
 /marksw/gromos.ethz.ch_repos_gromosXXc++/Oct09/trunk/

 module unload intel-cc
 module unload intel-fc
 module load gsl/1.12
 module load fftw3/3.2.2

 cd gromosXX
 ./configure --prefix='give the path name'  --enable-openmp
 make
 make install

How to compile GROMOS++ 0.3.1 (Oct09) on NCI(APAC) Vayu and Xe clusters

 Get the source code from:
 /marksw/gromos.ethz.ch_repos_gromos++/latest/trunk

 module load gcc/4.3.3
 module load gsl/1.12

 cd gromos++
 ./Config.sh
 ./configure --prefix='give the path name' --enable-shared=No CC=gcc CXX=g++
 make
 make install

How to compile GROMOS96 (24June04) on NCI(APAC) Vayu and Xe clusters

 get the tarball from 
 /marksw/gromos96/24Jun04/SOURCE/gromos96_24Jun04.tar

 tar xvf gromos96_24Jun04.tar
 cd gromos96_24Jun04

Edit the config.dat file to select the Intel Fortran compiler

  OS       = LINUX
  COMPILER = IFC

  make gromos       to build the entire distribution & cleanup
  make distrib      to build the entire distribution
  make promd        to build several versions of promd only
  make promdopt     to build optimized versions of promd only

How to compile GROMOSXX on NCI(APAC) Vayu and Xe clusters

Make sure that lam is properly installed. To install your own version, download it from the web and configure it like this:

./configure --prefix=$PATH_TO_LAM/lam/install

Make sure that you have fftw3mpi installed. You can download it from
the web and compile it yourself. Configure like this:

./configure --enable-mpi CXX=$PATH_TO_LAM/lam/install/bin/mpiCC
CC=$PATH_TO_LAM/lam/install/bin/mpicc
F77=$PATH_TO_LAM/lam/install/bin/mpif77 --prefix=$PATH_TO_FFTW/lib/fftw

Now you can install md++. Configure like this (revisions that need
fftw, i.e. the ones that have p3m):

../configure --with-gsl=$GSL/lib/gsl --disable-debug --disable-shared
--enable-mpi --with-fftw=$LIB/lib/fftw/ CC=$LAM/lam/install/bin/mpicc
CXX=$LAM/lam/install/bin/mpiCC