job000089

norman_morse (Norman Morse)
Norman Morse created this job , modified by Paul Allen
Closed
CVS Conversion of 4GB repository in Import mode runs out of memory.

In a Java VM with 2GB of memory the conversion fails with an OutOfMemoryError:  GC overhead limit exceeded.

It failed in "Building a revision list" at 23%

Here is the stack trace:

java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.util.Arrays.copyOf(Arrays.java:2219)
    at java.util.ArrayList.toArray(ArrayList.java:358)
    at java.util.ArrayList.<init>(ArrayList.java:164)
    at com.perforce.cvs.RcsNavigator.getBranchId(RcsNavigator.java:181)
    at com.perforce.cvs.RcsNavigator.followCodeLine(RcsNavigator.java:90)
    at com.perforce.cvs.RcsNavigator.followCodeLine(RcsNavigator.java:132)
    at com.perforce.cvs.RcsNavigator.add(RcsNavigator.java:69)
    at com.perforce.cvs.process.CvsProcessChange.processChange(CvsProcessChange.java:107)
    at com.perforce.common.process.ProcessChange.runSingle(ProcessChange.java:86)
    at com.perforce.common.process.ProcessChange.call(ProcessChange.java:49)
    at com.perforce.common.process.ProcessChange.call(ProcessChange.java:19)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
Caught EXIT shutting down ...

The exception caused a heap dump to take place, and I analyzed the results with eclipse MAT
Class Name                                                           Shallow Heap | Retained Heap | Percentage
---------------------------------------------------------------------------------------------------------------
java.lang.Thread @ 0x1187a66a8  pool-1-thread-1 Native Stack, Thread |        104 | 1,887,101,856 |     99.91%
|- com.perforce.cvs.RevisionSorter @ 0x11a474f00                     |         24 | 1,822,811,856 |     96.51%
|  '- java.util.ArrayList @ 0x11a466240                              |         24 | 1,822,811,832 |     96.51%
|     '- java.lang.Object[4102267] @ 0x16528ff70                     | 16,409,088 | 1,822,811,808 |     96.51%
|        |- com.perforce.cvs.RevisionEntry @ 0x1305b7e98             |         72 |        90,560 |      0.00%
|        |- com.perforce.cvs.RevisionEntry @ 0x13054f828             |         72 |        90,552 |      0.00%
|        |- com.perforce.cvs.RevisionEntry @ 0x130512bc0             |         72 |        90,544 |      0.00%
|        |- com.perforce.cvs.RevisionEntry @ 0x1305ea188             |         72 |        90,544 |      0.00%
|      <more lines like the 4 above>
---------------------------------------------------------------------------------------------------------------

It looks like 16 million discrete revisions are causing retained memory to exceed 2GB.
Perhaps we need a way to store this information in a file or a Database.

I will email the complete results to paul seperately.
10921String.intern() null protection.
10920CVS intern Strings to help reduce memory.
(Use Java 7 or greater)
Status
Closed
Project
perforce-software-p4convert
Severity
A
Reported By
Norman Morse
Reported Date
Modified By
Paul Allen
Modified Date