Adobe Experience Manager 6 and Apache Sling using the TarMK with Apache Jackrabbit Oak can be wickedly fast. One trick that it does is to deserialize the repository memory. It is not a caching the repository in memory. The repository exists in memory and uses the tar files to serialize and persist the data.
It will deserialize the entire repository into memory if it has enough memory to use. AEM will curl up next to your chair and purr if you give it that much memory. Without enough memory, only part of the repository will be deserialized into memory. To optimize performance in the form of request speed, give AEM 6 all the memory it wants.
Even though AEM and Apache Sling run in a Java virtual machine, the memory they use for the repository is not the JVM heap memory. It uses “off-heap” memory, memory not used by Java. If Java is given 8GB to run AEM/Sling and the total memory on the machine is 16GB, it will use as much of the remaining memory memory that it can for the repository. That means the repository has around 6GB to use.
In my experience, I have found that AEM with Oak uses virtual memory much more aggressively than CQ5. The memory used for the repository may indeed be virtual memory. Limiting the amount of memory that can be used for the repository is important.
It is possible to limit the amount of memory outside the heap that AEM/Sling uses. The -XX:MaxDirectMemorySize argument for Java is the way to set that.
The TarMK will run with much less memory than what it takes to load the whole repository. Doing so comes with the expense of performance in the form of much slower responses. If faced with a situation in which there is not enough memory for the repository to load completely, slimming down the Java heap size can help provide more for the repository to use.
Ways to reduce the memory footprint of AEM/Sling varies from situation to situation. Any configuration must be tested under loads that mimic the stress of production. One way to gain memory is to set the Pool Size parameter of the Apache Sling Job Thread Pool to 1. This will limit the memory used by jobs at the expense of the speed of job completion when queues jobs get queued. An instance may have bundles that are not being used. Those unneeded bundles can be turned off.
In summary, with AEM 6 and TarMK, the heap memory is not the only factor drastically affecting performance.