Answer

There are large files like ibdata1, ib_logfile0 and ib_logfile1 under /var/lib/mysql. ibdata1 is MySQL InnoDB engine system data file and ib_logfile* are recovery log files for InnoDB tables and they cannot be removed. Removing them will cause MySQL table data corruption.

One solution is to move ibdata1, ib_logfile0 and ib_logfile1 to another partition and then create symbolic links of them under /var/lib/mysql.

A. The procedure will be stopping all the RTM related services in the following sequence:
1. service lsfpollerd stop
2. service licpollerd stop
3. service advocate stop
4. service crond stop
5. service httpd stop
6. service mysqld stop
Check and make sure mysql is fully stopped:
service mysqld status
ps -ef grep | mysql

B. Once mysqld is stopped, you can move files ibdata1, ib_logfile0 and ib_logfile1 to another partition with free space, and then create 3 symbolic links under /var/lib/mysql. Please make sure MySQL daemon user, by default 'mysql', has full permission to the new folder location and the three files.

# cd /var/lib/mysql
# mv ib_logfile0 <new_partition_location>/ib_logfile0
# mv ib_logfile1 <new_partition_location>/ib_logfile1
# mv ibdata1 <new_partition_location>/ibdata1
# ln -s <new_partition_location>/ib_logfile0 ib_logfile0
# ln -s <new_partition_location>/ib_logfile1 ib_logfile1
# ln -s <new_partition_location>/ibdata1 ibdata1

C. Please make sure the symbolic links are successfully created, and then we can start mysqld service.
# service mysqld start

Log in to make sure mysqld starts okay
# mysql cacti
mysql> show engines\G

We are expecting the InnoDB are shown as "YES":
...
*************************** 4. row ***************************
      Engine: InnoDB
     Support: YES
     Comment: Supports transactions, row-level locking, and foreign keys
Transactions: YES
          XA: YES
  Savepoints: YES
...


D. If they are okay, then we can start the rest of related services in the order below:
1. service httpd start
2. service crond start
3. service advocate start
4. service licpollerd stop
5. service lsfpollerd start


RTM should be online again and partition /var has disk space freed.