[Mulgara-general] Mulgara size limits
Alex Hall
alexhall at revelytix.com
Thu Jul 3 07:01:05 PDT 2008
OK, I've been looking at this and am starting to narrow down the
problem. The AVLNode and Block entries at the top of the heap profile
indicate to me that something is building up the entire index structures
(or a very large portion of them) in memory. Clearly this isn't going
to work for such a large data set. However, I need a bit more
information to be able to fix this. Could you please run hprof again,
this time adding the flag "depth=6" to the -Xrunhprof (i.e.
"-Xrunhprof:heap=sites,depth=6") and send out the same info (heap
allocation table and stack traces for the top few entries, specifically
the AVLNode ones).
Thanks,
Alex
Chuck Borromeo wrote:
> It's no trouble at all. You guys have been great!
>
> I pulled these from the java.hprof.txt file:
>
> TRACE 308151:
> org.mulgara.store.xa.Block.<init>(Block.java:129)
> org.mulgara.store.xa.Block.newInstance(Block.java:161)
> org.mulgara.store.xa.MappedBlockFile.readBlock(MappedBlockFile.java:282)
> org.mulgara.store.xa.ManagedBlockFile$Phase.readBlock(ManagedBlockFile.java:413)
> TRACE 308249:
> org.mulgara.store.xa.AVLNode.<init>(AVLNode.java:147)
> org.mulgara.store.xa.AVLNode.newInstance(AVLNode.java:212)
> org.mulgara.store.xa.AVLNode.getChildNode_N(AVLNode.java:1171)
> org.mulgara.store.xa.AVLNode.getRightChildNode_N(AVLNode.java:1152)
> TRACE 308250:
> org.mulgara.store.xa.AVLNode.<init>(AVLNode.java:133)
> org.mulgara.store.xa.AVLNode.newInstance(AVLNode.java:212)
> org.mulgara.store.xa.AVLNode.getChildNode_N(AVLNode.java:1171)
> org.mulgara.store.xa.AVLNode.getRightChildNode_N(AVLNode.java:1152)
> TRACE 308158:
> org.mulgara.store.xa.AVLNode.<init>(AVLNode.java:147)
> org.mulgara.store.xa.AVLNode.newInstance(AVLNode.java:212)
> org.mulgara.store.xa.AVLNode.getChildNode_N(AVLNode.java:1171)
> org.mulgara.store.xa.AVLNode.getLeftChildNode_N(AVLNode.java:1126)
> TRACE 308159:
> org.mulgara.store.xa.AVLNode.<init>(AVLNode.java:133)
> org.mulgara.store.xa.AVLNode.newInstance(AVLNode.java:212)
> org.mulgara.store.xa.AVLNode.getChildNode_N(AVLNode.java:1171)
> org.mulgara.store.xa.AVLNode.getLeftChildNode_N(AVLNode.java:1126)
>
>
>
> --- On Wed, 7/2/08, Alex Hall <alexhall at revelytix.com> wrote:
>
>> From: Alex Hall <alexhall at revelytix.com>
>> Subject: Re: [Mulgara-general] Mulgara size limits
>> To: cborromeo3 at yahoo.com, "Mulgara General" <mulgara-general at mulgara.org>
>> Date: Wednesday, July 2, 2008, 10:44 AM
>> Thanks, this should be helpful in starting to track down the
>> problem.
>> If I could trouble you for one more piece of information,
>> could you
>> please send the stack traces associated with the top 5
>> entries in the
>> list (these are denoted by the stack trace ID's in the
>> second column
>> from the right).
>>
>> Alex
>>
>> Chuck Borromeo wrote:
>>> Hey Ronald,
>>> Thanks for the 1.6 tip. That did the trick. I will
>> include the stack trace below the hprof table. This was
>> generated using Java 1.6 and -Xmx1024:
>>> SITES BEGIN (ordered by live bytes) Wed Jul 2
>> 09:41:58 2008
>>> percent live alloc'ed
>> stack class
>>> rank self accum bytes objs bytes objs
>> trace name
>>> 1 43.70% 43.70% 450682944 4694614 450683616
>> 4694621 308151 org.mulgara.store.xa.Block
>>> 2 31.16% 74.86% 321304632 3651189 321305072
>> 3651194 308249 org.mulgara.store.xa.AVLNode
>>> 3 11.33% 86.19% 116838144 3651192 116838208
>> 3651194 308250 int[]
>>> 4 8.90% 95.10% 91819816 1043407 91820168
>> 1043411 308158 org.mulgara.store.xa.AVLNode
>>> 5 3.24% 98.34% 33389088 1043409 33389152
>> 1043411 308159 int[]
>>> 6 0.34% 98.68% 3545472 63312 6552000 117000
>> 317465 long[]
>>> 7 0.16% 98.83% 1600096 4 72004320 180
>> 308174 long[][]
>>> 8 0.04% 98.88% 447680 5596 450480 5631
>> 306720 java.nio.DirectByteBuffer
>>> 9 0.04% 98.92% 432160 5402 127100480 1588756
>> 306802 java.nio.DirectByteBufferR
>>> 10 0.04% 98.96% 402912 5596 405432 5631
>> 306723 sun.misc.Cleaner
>>> 11 0.03% 98.99% 358144 5596 360384 5631
>> 306727 java.nio.DirectLongBufferU
>>> 12 0.03% 99.03% 358144 5596 360384 5631
>> 306725 java.nio.DirectIntBufferU
>>> 13 0.03% 99.06% 279344 34 279344 34
>> 306730 java.nio.LongBuffer[]
>>> 14 0.03% 99.08% 279344 34 279344 34
>> 306729 java.nio.IntBuffer[]
>>> 15 0.03% 99.11% 279344 34 279344 34
>> 306728 java.nio.MappedByteBuffer[]
>>> 16 0.02% 99.13% 213616 26 213616 26
>> 306836 java.nio.MappedByteBuffer[]
>>> 17 0.02% 99.15% 213616 26 213616 26
>> 306838 java.nio.IntBuffer[]
>>> 18 0.02% 99.17% 213616 26 213616 26
>> 306839 java.nio.LongBuffer[]
>>> 19 0.02% 99.19% 213616 26 213616 26
>> 306837 java.nio.ByteBuffer[]
>>> 20 0.02% 99.21% 172864 5402 172864 5402
>> 306799 sun.nio.ch.FileChannelImpl$Unmapper
>>> 21 0.01% 99.22% 148008 1327 157136 1448
>> 300437 char[]
>>> 22 0.01% 99.24% 131096 1 131096 1
>> 316926 java.lang.Object[]
>>> 23 0.01% 99.25% 125440 1 125440 1
>> 317046 byte[]
>>> 24 0.01% 99.26% 109760 1960 241472 4312
>> 314229 org.enhydra.apache.xerces.dom.TextImpl
>>> SITES END
>>>
>>> =============STACK TRACE ========================
>>> Exception in thread "RMI
>> RenewClean-[192.168.2.43:54091]"
>> java.lang.OutOfMemoryError: Java heap space
>>> at java.lang.String.substring(String.java:1940)
>>> at java.lang.String.subSequence(String.java:1973)
>>> at java.util.regex.Pattern.split(Pattern.java:1002)
>>> at java.lang.String.split(String.java:2293)
>>> at
>> sun.net.util.IPAddressUtil.textToNumericFormatV4(IPAddressUtil.java:29)
>>> at
>> java.net.SocketPermission.init(SocketPermission.java:432)
>>> at
>> java.net.SocketPermission.<init>(SocketPermission.java:265)
>>> at
>> java.lang.SecurityManager.checkConnect(SecurityManager.java:1034)
>>> at java.net.Socket.connect(Socket.java:513)
>>> at java.net.Socket.connect(Socket.java:469)
>>> at java.net.Socket.<init>(Socket.java:366)
>>> at java.net.Socket.<init>(Socket.java:180)
>>> at
>> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
>>> at
>> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
>>> at
>> sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
>>> at
>> sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
>>> at
>> sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
>>> at
>> sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
>>> at sun.rmi.transport.DGCImpl_Stub.dirty(Unknown
>> Source)
>>> at
>> sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:342)
>>> at
>> sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:153)
>>> at
>> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:555)
>>> at java.lang.Thread.run(Thread.java:637)
>>> Jul 2, 2008 9:39:25 AM
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop
>> executeAcceptLoop
>>> WARNING: RMI TCP Accept-0: accept loop for
>> ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=54091]
>> throws
>>> java.lang.OutOfMemoryError: Java heap space
>>> at java.net.PlainSocketImpl.socketAccept(Native
>> Method)
>>> at
>> java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>> at
>> java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>> at
>> java.net.ServerSocket.accept(ServerSocket.java:421)
>>> at
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
>>> at
>> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
>>> at java.lang.Thread.run(Thread.java:637)
>>> 2008-07-02 09:40:15,594 ERROR TripleWriteThread -
>> Exception in TripleWriteThread
>>> java.lang.OutOfMemoryError: Java heap space
>>> at
>> org.mulgara.store.xa.AVLNode.newInstance(AVLNode.java:212)
>>> at
>> org.mulgara.store.xa.AVLNode.getChildNode_N(AVLNode.java:1171)
>>> at
>> org.mulgara.store.xa.AVLNode.getRightChildNode_N(AVLNode.java:1152)
>>> at
>> org.mulgara.store.xa.AVLNode.findDown(AVLNode.java:339)
>>> at
>> org.mulgara.store.xa.AVLNode.find(AVLNode.java:300)
>>> at
>> org.mulgara.store.statement.xa.TripleAVLFile$Phase.syncAddTriple(TripleAVLFile.java:794)
>>> at
>> org.mulgara.store.statement.xa.TripleAVLFile$Phase.syncAddTriples(TripleAVLFile.java:727)
>>> at
>> org.mulgara.store.statement.xa.TripleWriteThread.run(TripleWriteThread.java:158)
>>> 2008-07-02 09:40:15,601 WARN TripleWriteThread -
>> Exception during abort of TripleWriteThread:
>> /Applications/mulgara-2.0-alpha/dist/server1/xaStatementStore/xa.g_2013
>>> java.lang.OutOfMemoryError: Java heap space
>>> at
>> org.mulgara.store.xa.AVLNode.newInstance(AVLNode.java:212)
>>> at
>> org.mulgara.store.xa.AVLNode.getChildNode_N(AVLNode.java:1171)
>>> at
>> org.mulgara.store.xa.AVLNode.getRightChildNode_N(AVLNode.java:1152)
>>> at
>> org.mulgara.store.xa.AVLNode.findDown(AVLNode.java:339)
>>> at
>> org.mulgara.store.xa.AVLNode.find(AVLNode.java:300)
>>> at
>> org.mulgara.store.statement.xa.TripleAVLFile$Phase.syncAddTriple(TripleAVLFile.java:794)
>>> at
>> org.mulgara.store.statement.xa.TripleAVLFile$Phase.syncAddTriples(TripleAVLFile.java:727)
>>> at
>> org.mulgara.store.statement.xa.TripleWriteThread.run(TripleWriteThread.java:158)
>>>
>>> --- On Wed, 7/2/08, Life is hard, and then you die
>> <ronald at innovation.ch> wrote:
>>>> From: Life is hard, and then you die
>> <ronald at innovation.ch>
>>>> Subject: Re: [Mulgara-general] Mulgara size limits
>>>> To: "Mulgara General"
>> <mulgara-general at mulgara.org>
>>>> Date: Wednesday, July 2, 2008, 8:55 AM
>>>> On Wed, Jul 02, 2008 at 05:21:21AM -0700, Chuck
>> Borromeo
>>>> wrote:
>>>>> Hello,
>>>>> OK I'm going to be a pain in the neck
>> :). I
>>>> tried running Mulgara with the
>> -Xrunhprof:heap=sites
>>>> switch. The process threw a
>>>> java.security.AccessControlException: access
>> denied error
>>>> when I ran it using the -Xrunhprof switch. I have
>> never
>>>> run java with the dump file commands so I would
>> appreciate
>>>> anyone telling me how to fix this issue. I looked
>> up the
>>>> error and it said something about changing my
>> java.policy
>>>> and java.security files to allow the process to
>> see these
>>>> files. However, it didn't make much sense to
>> me. I am
>>>> running Mulgara with the sudo command so I should
>> have
>>>> permission to see everything. How do I fix this?
>>>> (I'll attached the stack trace at the end of
>> this
>>>> message).
>>>>
>>>> The java access-control stuff has nothing to do
>> with unix
>>>> permissions,
>>>> so being root or not doesn't change anything.
>> The odd
>>>> thing is,
>>>> mulgara sets a policy which should allow access to
>> all
>>>> files - see the
>>>> mulgara startup message
>>>>
>>>> INFO [main] (EmbeddedMulgaraServer.java:692) -
>>>> java.security.policy set to
>>>>
>> jar:file:/..../mulgara-2.0-alpha.jar!/conf/mulgara-rmi.policy
>>>> So I'm not sure what's going on either.
>>>>
>>>>> So...I tried running Mulgara with the
>>>>> -XX:+HeapDumpOnOutOfMemoryError switch. This
>> did
>>>> generate a .hprof
>>>>> file (about 1 GB in size!). I tried opening
>> this file
>>>> using a 1.6
>>>>> version of jhat (1.5's jhat can't open
>> a 64
>>>> bit .hprof file).
>>>>> However, I am getting a NullPointer exception
>> from
>>>> jhat when opening
>>>>> the file.
>>>> I'm not surprised - as I said before, I've
>> found
>>>> 1.5 to be unreliable
>>>> in this respect. If you can run under 1.6 I think
>>>> you'll have better
>>>> luck.
>>>>
>>>>
>>>> Cheers,
>>>>
>>>> Ronald
>>>>
>>>> _______________________________________________
>>>> Mulgara-general mailing list
>>>> Mulgara-general at mulgara.org
>>>>
>> http://mulgara.org/mailman/listinfo/mulgara-general
>>>
>>>
>>> _______________________________________________
>>> Mulgara-general mailing list
>>> Mulgara-general at mulgara.org
>>> http://mulgara.org/mailman/listinfo/mulgara-general
>
>
>
> _______________________________________________
> Mulgara-general mailing list
> Mulgara-general at mulgara.org
> http://mulgara.org/mailman/listinfo/mulgara-general
More information about the Mulgara-general
mailing list