Computing Resource@Purdue

Author: Masudur Rahman

Running on RCAC Machines

Storage (Gilbreth, Brown) Running code with large data requirements, the scratch filesystem can be used. The data can be at /scratch/USERNAME, and the code can be at /home/USERNAME. The data path /scratch/USERNAME/data can be used inside code in /home/USERNAME/code-implementation-dir.

ITAP removes (purges) data from scratch, which is older than 2 months. That means, if any data file is not modified for 2 months, it will be deleted. Usually, ITAP sends an email reminder a few days before the purge date. So, make sure you backup your data periodically.

Data Backup Fortress system can be used for the backup, where you can keep the data permanently. The good thing is that currently, there is no quota for Fortress disk use. To know details of Fortress, read here.

How to transfer data to Fortress? There are several ways; see here. I find Globus particularly more useful, and it can be accessed via Purdue account. Once you log in, transfer data between filesystems (e.g., Gilbreth to Fortress or Fortress to Gilbreth) will take a few clicks.

Tips: On scratch filesystem, if you open a file, it will be considered as “modified”; thus, the file age will start from that date. Not sure, it will be considered modified if you open/read a file from inside a script (e.g., python). But you can give it a try. Remember, if a file is modified, it will be removed from the purge list for 2 more months. I hope this will give you a workaround to keep files on scratch for longer. Try if that works for you. In any case, keep a backup on Fortress periodically.