One thing you could do is only keep data in memory that is constantly being accessed.

If you're only referencing data every so often then the overhead of it being on the hard drive doesn't really matter.