The Server will potentially use a very large amount of virtual memory at startup when it is configured to use Java version 1.7. This is because the default behavior of the 1.7 JVM (64-bit JVM on Windows) is to reserve 1/4 of the total physical memory. This does not affect physical memory usage. When looking at the Windows Committed memory or Virtual memory usage, ctreesql may show a very large value at startup if the JVM is configured.
http://www.oracle.com/technetwork/java/javase/jdk7-relnotes-418459.html
The JVM heap size can be limited with the following Server configuration keyword:
; Limit JVM maximum heap to 256 MB
SETENV DH_JVM_OPTION_STRINGS=-Xmx256m
The maximum heap size should be tuned based on how the Stored procedures are written and used (256MB should be enough for simple usage). See below for details on Java Tuning.
http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html