Cluster Computing

MIT houses several computing clusters that are available for the lab to use. As of 2025, we use the Engaging cluster, though this may change in the future.

The most common use case in our lab is RNA-seq analysis because reading and aligning millions of transcripts is computationally intensive, as you can imagine. The general workflow (in detail below) is using a Snakefile to execute a list of commands for trimming, aligning, and/or analyzing transcripts. These commands may point to RNA-seq-related packages or to user-defined python scripts that run analysis. At a high level, you upload your raw reads and your project repo housing your Snakefile, run snakemake on the cluster, the cluster will compute, and then you will extract the data you need (usually gene counts and/or differentially expressed genes) to make plots locally.

First-time setup

0. Create an account

Following the instructions on the MIT ORCD docs page, log in to the Engaging cluster through the web portal using your Kerberos ID and password (instructions here). This will automatically trigger a new account to be created.

Note

There may be a delay of a day after creating your account before you can start any jobs. However, you should still be able to log in.

Confirm you can log in to Engaging via your terminal or PowerShell using ssh. Replace [your-kerberos] with your Kerberos ID:

$ ssh [your-kerberos]@orcd-login.mit.edu

This will prompt you for your Kerberos password and Duo authentication.

1. Add an ssh shortcut

Once you’ve confirmed that you can log in, create an ssh shortcut to the cluster.

On your computer (not in the cluster), add the following to your config file using nano ~/.ssh/config:

Host engaging
    HostName orcd-login.mit.edu
    User [your-kerberos]
    ForwardAgent yes

While you’re at it, add a shortcut to the BioMicro Center cluster. This is where they’ll temporarily store your sequencing data.

Host bmc
    HostName bmc-150.mit.edu
    User galloway_ill

Important

You can’t use nano on Windows. Instead, navigate to the folder directly in the File Explorer and edit your config file with a text editor:

  1. In PowerShell, run cd ~/.ssh

  2. Get the directory path by pwd

  3. Copy this path into “File Explorer”. This might look like C:\Users\ChemeGrad2025\.ssh

  4. Once you’ve located the hidden .ssh directory, edit the config file with “Notepad” (or “VSCode”, etc.) and add in the above.

See MIT ORCD docs SSH key setup for help.

Now, check to confirm that the shortcut runs:

$ ssh engaging

This should generate the same prompt for your Kerberos password and Duo authentication as above, just via a simpler command.

2. Set up an SSH key with forwarding

Based on MIT ORCD docs “SSH key setup” and GitHub docs “Using SSH agent forwarding”.

At a high level: SSH agent forwarding can be used to make deploying to a server simple. It allows you to use your local SSH keys instead of leaving keys (without passphrases!) sitting on remote servers, like the Engaging cluster.

You can set up ssh-agent for your local computer which runs in the background and keeps your SSH key loaded into memory so you don’t need to enter a passphrase every time you need to use the key. Then, you can give remote servers, like the Engaging cluster, access to your local ssh-agent as if they were running on the server. This is sort of like asking a friend to enter their password so that you can use their computer.

The end result basically means you get use git clone and other things without having to re-enter passphrases every time while on the Engaging cluster.

We’ll start with GitHub docs “Using SSH agent forwarding”. Check to see if your own SSH key is set up and working by entering ssh -T git@github.com in the terminal. If successful it will look like:

$ ssh -T git@github.com
# Attempt to SSH in to github
> Hi USERNAME! You've successfully authenticated, but GitHub does not provide shell access.

If not, next make sure your local computer has an SSH public key for GitHub.

  1. Check for an existing SSH key on your local computer: GitHub docs “Checking for existing SSH”

  2. If no key exists, then generate and add a new SSH key: GitHub docs “Generating a new SSH key and adding it to the ssh-agent”

  3. Now add the SSH key from your local computer to your Github account: GitHub docs “Adding a new SSH key to your GitHub account”

  4. Confirm that the SSH key works by entering ssh -T git@github.com in the terminal. You should see the message above.

Note

Your public key is likely id_ed25519.pub but may alternatively be id_rsa.pub or id_ecdsa.pub.

Now GitHub has your public key but you still need to let ssh-agent get access to your private key. This way, when a remote server with ForwardAgent true needs to sign something with your private key, the request gets funneled back to your ssh-agent which returns the signed request so the private key never leaves your local computer. By copying the public key onto remote systems—such as copy-pasting onto Github like we just did or using ssh-copy-id—your public key gets pre-loaded onto remote systems but you can still control access to your private keys for each individual remote server.

To make your key available to ssh-agent:

  1. Check that your key is visible to ssh-agent by running the following command on your local computer: ssh-add -L

  2. If the command says that no identity is available, you’ll need to add your key with the following command: ssh-add . This will add any “default” keys. You can also add a specific key. For NBW this looks like ssh-add ~/.ssh/id_rsa which is different than the public key, ~/.ssh/id_rsa.pub!

  3. On macOS, ssh-agent will “forget” this key, once it gets restarted during reboots. But you can import your SSH keys into Keychain using this command: ssh-add --apple-use-keychain YOUR-KEY

Great! You should be done now! The secret was in something we added before:

Host engaging
    HostName orcd-login.mit.edu
    User [your-kerberos]
    ForwardAgent yes

The ForwardAgent yes tells your ssh-agent to let the Engaging cluster use your local keys. This is known as “SSH agent forwarding”.

Check to make sure it’s set up correctly:

  1. Log in to the Engaging cluster using ssh engaging, enter your Kerberos password, and authenticate with Duo.

  2. On the Engaging cluster, test to see if the SSH key is set up and working with Github by entering ssh -T git@github.com in the terminal. It should show the same successful response as above.

If it’s not working, check GitHub docs “Using SSH agent forwarding: Troubleshooting SSH agent forward” for tips.

Per-project setup

TODO

Needs description of other files in the suggested repo layout. Pipeline templates are in progress.

After getting your ssh-agent set up as described above, you should clone your project repo into ~/katiegal_shared/projects/. This will let you edit your script files locally or on the server, and track changes. You will want to make a new directory to house all of your Engaging cluster files. You can either copy a cluster folder from someone else’s pipeline (CJ is working on an incoming template repo) or make a new one.

To clone your repo:

  1. Log in to the Engaging cluster via ssh engaging

  2. Navigate to the projects directory by cd katiegal_shared/projects

  3. Clone your project repo by using the ssh URL, which you can get from GitHub. This might look like:

    $ git clone git@github.com:GallowayLabMIT/your_repo.git

Next, make a new cluster directory in your your_repo (if not using an existing template) and symlink the raw reads:

$ mkdir ~/katiegal_shared/projects/your_repo/cluster
$ mkdir ~/katiegal_shared/projects/your_repo/cluster/data
$ ln -s ~/katiegal_shared/data/raw_reads/ ~/katiegal_shared/projects/your_repo/cluster/data/raw_reads

Ultimately, your cluster file structure should look something like this:

katiegal_shared/
├── data/
├── hpc_infra/
└── projects/
    └── your_repo/
        ├── ...             # everything else in your repo, like Python data analysis, figures, etc.
        └── cluster/
            ├── config      # TODO DESCRIPTION - Metadata for configuring
            ├── data/       # Data you don't want tracked, like genomes
            │   └── raw_reads
            ├── envs/       # TODO DESCRIPTION
            ├── inputs/     # Inputs that should be tracked, like transgenes or metadata
            ├── profiles/   # TODO DESCRIPTION
            ├── scripts/    # Scripts for analysis
            ├── .gitignore  # TODO DESCRIPTION
            └── Snakefile   # Runs pipeline