Uploading raw data to ArrayExpress

There are multiple ways to upload raw data to ArrayExpress, here is a short tutorial on how to do so from the command line with the FTP protocol. It is based on this official documentation with small amendments and explanations.

Manual

“Manual” basically means that you will be typing the commands below within your terminal. If your files are large and you do not want risking your upload process be broken due to disconnecting from the server, you can check the alternative section below.

1. Log into Annotare and create a new experiment submission.

2. cd into the directory where the data are stored.

3. Connect to ArrayExpress and enter username (annotare) password (annotare1) combination.

Terminal

ftp -p ftp-private.ebi.ac.uk

4. Within the ArrayExpress server, cd into the directory ArrayExpress assigned for your repo.

Terminal

cd /.../...

5. put files into the remote directory you just navigated to. If using mput (for multiple files instead of just one) over put, first enter prompt to turn the interactive mode off so that there will be no need to confirm transfers.

Terminal

prompt
mput *.fastq.gz

Automated

I have never experienced issues with “typical” bulk RNA-seq experiments where total data size is around 200-300 GB and distributed over 20-30 files/samples. If you would like to be able utilize nohup so that your upload process is not disturbed due to disconnecting from the server you can try the following:

1. Create a .netrc file within your home directory. This will enable bypassing username and password. ftp checks for this file and uses the username and password combination specified for the listed ftp servers within this file.

Terminal

vi ~/.netrc

2. Populate ~/.netrc with server name, user name and password. For ArrayExpress you can use the following:

~/.netrc

machine ftp-private.ebi.ac.uk  login annotare password annotare1

3. Change the permissions so that just the owner can read the file:

Terminal

chmod 0600 ~/.netrc

This does not matter if the only entry within ~/.netrc is ArrayExpress’ ftp server info, it is public anyway.

3. Navigate to the folder with the raw data to be uploaded to ArrayExpress, create a bash script and make it executable:

bash_script_for_ftp.sh

touch bash_script_for_ftp.sh
chmod +x bash_script_for_ftp.sh

Normally it is advised for executable bash scripts not to have any extension but I prefer to have the .sh extension for my bash scripts so that I know what I am executing.

bash_script_for_ftp.sh

#!/usr/bin/env bash

ftp -p ftp-private.ebi.ac.uk <<EOF
cd /prod/.../
prompt
mput *.fastq.gz
quit
EOF