[Mulgara-general] Mulgara size limits
Chuck Borromeo
cborromeo3 at yahoo.com
Tue Jul 8 06:07:48 PDT 2008
Hi Everyone,
I just wanted to check to see if anyone has figured out the issue I was encountering when I tried to load RDF/XML files larger than 98 MB. On a side note, am I loading the files correctly? This is my code:
// Create a factory, and connect to the server
factory = new ConnectionFactory();
connection = factory.newConnection(serverURI);
// execute a CREATE command, myData is a URI
connection.execute(new CreateGraph(myData));
// execute a LOAD command
connection.execute(new Load(dataFile, myData, true));
Do I need to run the "new CreateGraph(myData))" each time I load a file? Is there a method to check for the existence of a graph? Also are there transaction methods I should commit after loading a file?
Thanks,
Chuck
--- On Thu, 7/3/08, Alex Hall <alexhall at revelytix.com> wrote:
> From: Alex Hall <alexhall at revelytix.com>
> Subject: Re: [Mulgara-general] Mulgara size limits
> To: cborromeo3 at yahoo.com, "Mulgara General" <mulgara-general at mulgara.org>
> Date: Thursday, July 3, 2008, 10:01 AM
> OK, I've been looking at this and am starting to narrow
> down the
> problem. The AVLNode and Block entries at the top of the
> heap profile
> indicate to me that something is building up the entire
> index structures
> (or a very large portion of them) in memory. Clearly this
> isn't going
> to work for such a large data set. However, I need a bit
> more
> information to be able to fix this. Could you please run
> hprof again,
> this time adding the flag "depth=6" to the
> -Xrunhprof (i.e.
> "-Xrunhprof:heap=sites,depth=6") and send out the
> same info (heap
> allocation table and stack traces for the top few entries,
> specifically
> the AVLNode ones).
>
> Thanks,
> Alex
>
> Chuck Borromeo wrote:
> > It's no trouble at all. You guys have been great!
>
> >
> > I pulled these from the java.hprof.txt file:
> >
> > TRACE 308151:
> >
> org.mulgara.store.xa.Block.<init>(Block.java:129)
> >
> org.mulgara.store.xa.Block.newInstance(Block.java:161)
> >
> org.mulgara.store.xa.MappedBlockFile.readBlock(MappedBlockFile.java:282)
> >
> org.mulgara.store.xa.ManagedBlockFile$Phase.readBlock(ManagedBlockFile.java:413)
> > TRACE 308249:
> >
> org.mulgara.store.xa.AVLNode.<init>(AVLNode.java:147)
> >
> org.mulgara.store.xa.AVLNode.newInstance(AVLNode.java:212)
> >
> org.mulgara.store.xa.AVLNode.getChildNode_N(AVLNode.java:1171)
> >
> org.mulgara.store.xa.AVLNode.getRightChildNode_N(AVLNode.java:1152)
> > TRACE 308250:
> >
> org.mulgara.store.xa.AVLNode.<init>(AVLNode.java:133)
> >
> org.mulgara.store.xa.AVLNode.newInstance(AVLNode.java:212)
> >
> org.mulgara.store.xa.AVLNode.getChildNode_N(AVLNode.java:1171)
> >
> org.mulgara.store.xa.AVLNode.getRightChildNode_N(AVLNode.java:1152)
> > TRACE 308158:
> >
> org.mulgara.store.xa.AVLNode.<init>(AVLNode.java:147)
> >
> org.mulgara.store.xa.AVLNode.newInstance(AVLNode.java:212)
> >
> org.mulgara.store.xa.AVLNode.getChildNode_N(AVLNode.java:1171)
> >
> org.mulgara.store.xa.AVLNode.getLeftChildNode_N(AVLNode.java:1126)
> > TRACE 308159:
> >
> org.mulgara.store.xa.AVLNode.<init>(AVLNode.java:133)
> >
> org.mulgara.store.xa.AVLNode.newInstance(AVLNode.java:212)
> >
> org.mulgara.store.xa.AVLNode.getChildNode_N(AVLNode.java:1171)
> >
> org.mulgara.store.xa.AVLNode.getLeftChildNode_N(AVLNode.java:1126)
> >
> >
> >
> > --- On Wed, 7/2/08, Alex Hall
> <alexhall at revelytix.com> wrote:
> >
> >> From: Alex Hall <alexhall at revelytix.com>
> >> Subject: Re: [Mulgara-general] Mulgara size limits
> >> To: cborromeo3 at yahoo.com, "Mulgara
> General" <mulgara-general at mulgara.org>
> >> Date: Wednesday, July 2, 2008, 10:44 AM
> >> Thanks, this should be helpful in starting to
> track down the
> >> problem.
> >> If I could trouble you for one more piece of
> information,
> >> could you
> >> please send the stack traces associated with the
> top 5
> >> entries in the
> >> list (these are denoted by the stack trace
> ID's in the
> >> second column
> >> from the right).
> >>
> >> Alex
> >>
> >> Chuck Borromeo wrote:
> >>> Hey Ronald,
> >>> Thanks for the 1.6 tip. That did the trick.
> I will
> >> include the stack trace below the hprof table.
> This was
> >> generated using Java 1.6 and -Xmx1024:
> >>> SITES BEGIN (ordered by live bytes) Wed Jul 2
> >> 09:41:58 2008
> >>> percent live
> alloc'ed
> >> stack class
> >>> rank self accum bytes objs bytes
> objs
> >> trace name
> >>> 1 43.70% 43.70% 450682944 4694614
> 450683616
> >> 4694621 308151 org.mulgara.store.xa.Block
> >>> 2 31.16% 74.86% 321304632 3651189
> 321305072
> >> 3651194 308249 org.mulgara.store.xa.AVLNode
> >>> 3 11.33% 86.19% 116838144 3651192
> 116838208
> >> 3651194 308250 int[]
> >>> 4 8.90% 95.10% 91819816 1043407
> 91820168
> >> 1043411 308158 org.mulgara.store.xa.AVLNode
> >>> 5 3.24% 98.34% 33389088 1043409
> 33389152
> >> 1043411 308159 int[]
> >>> 6 0.34% 98.68% 3545472 63312 6552000
> 117000
> >> 317465 long[]
> >>> 7 0.16% 98.83% 1600096 4 72004320
> 180
> >> 308174 long[][]
> >>> 8 0.04% 98.88% 447680 5596 450480
> 5631
> >> 306720 java.nio.DirectByteBuffer
> >>> 9 0.04% 98.92% 432160 5402 127100480
> 1588756
> >> 306802 java.nio.DirectByteBufferR
> >>> 10 0.04% 98.96% 402912 5596 405432
> 5631
> >> 306723 sun.misc.Cleaner
> >>> 11 0.03% 98.99% 358144 5596 360384
> 5631
> >> 306727 java.nio.DirectLongBufferU
> >>> 12 0.03% 99.03% 358144 5596 360384
> 5631
> >> 306725 java.nio.DirectIntBufferU
> >>> 13 0.03% 99.06% 279344 34 279344
> 34
> >> 306730 java.nio.LongBuffer[]
> >>> 14 0.03% 99.08% 279344 34 279344
> 34
> >> 306729 java.nio.IntBuffer[]
> >>> 15 0.03% 99.11% 279344 34 279344
> 34
> >> 306728 java.nio.MappedByteBuffer[]
> >>> 16 0.02% 99.13% 213616 26 213616
> 26
> >> 306836 java.nio.MappedByteBuffer[]
> >>> 17 0.02% 99.15% 213616 26 213616
> 26
> >> 306838 java.nio.IntBuffer[]
> >>> 18 0.02% 99.17% 213616 26 213616
> 26
> >> 306839 java.nio.LongBuffer[]
> >>> 19 0.02% 99.19% 213616 26 213616
> 26
> >> 306837 java.nio.ByteBuffer[]
> >>> 20 0.02% 99.21% 172864 5402 172864
> 5402
> >> 306799 sun.nio.ch.FileChannelImpl$Unmapper
> >>> 21 0.01% 99.22% 148008 1327 157136
> 1448
> >> 300437 char[]
> >>> 22 0.01% 99.24% 131096 1 131096
> 1
> >> 316926 java.lang.Object[]
> >>> 23 0.01% 99.25% 125440 1 125440
> 1
> >> 317046 byte[]
> >>> 24 0.01% 99.26% 109760 1960 241472
> 4312
> >> 314229 org.enhydra.apache.xerces.dom.TextImpl
> >>> SITES END
> >>>
> >>> =============STACK TRACE
> ========================
> >>> Exception in thread "RMI
> >> RenewClean-[192.168.2.43:54091]"
> >> java.lang.OutOfMemoryError: Java heap space
> >>> at
> java.lang.String.substring(String.java:1940)
> >>> at
> java.lang.String.subSequence(String.java:1973)
> >>> at
> java.util.regex.Pattern.split(Pattern.java:1002)
> >>> at java.lang.String.split(String.java:2293)
> >>> at
> >>
> sun.net.util.IPAddressUtil.textToNumericFormatV4(IPAddressUtil.java:29)
> >>> at
> >>
> java.net.SocketPermission.init(SocketPermission.java:432)
> >>> at
> >>
> java.net.SocketPermission.<init>(SocketPermission.java:265)
> >>> at
> >>
> java.lang.SecurityManager.checkConnect(SecurityManager.java:1034)
> >>> at java.net.Socket.connect(Socket.java:513)
> >>> at java.net.Socket.connect(Socket.java:469)
> >>> at
> java.net.Socket.<init>(Socket.java:366)
> >>> at
> java.net.Socket.<init>(Socket.java:180)
> >>> at
> >>
> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
> >>> at
> >>
> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
> >>> at
> >>
> sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
> >>> at
> >>
> sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
> >>> at
> >>
> sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
> >>> at
> >>
> sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
> >>> at
> sun.rmi.transport.DGCImpl_Stub.dirty(Unknown
> >> Source)
> >>> at
> >>
> sun.rmi.transport.DGCClient$EndpointEntry.makeDirtyCall(DGCClient.java:342)
> >>> at
> >>
> sun.rmi.transport.DGCClient$EndpointEntry.access$1600(DGCClient.java:153)
> >>> at
> >>
> sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:555)
> >>> at java.lang.Thread.run(Thread.java:637)
> >>> Jul 2, 2008 9:39:25 AM
> >> sun.rmi.transport.tcp.TCPTransport$AcceptLoop
> >> executeAcceptLoop
> >>> WARNING: RMI TCP Accept-0: accept loop for
> >>
> ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=54091]
> >> throws
> >>> java.lang.OutOfMemoryError: Java heap space
> >>> at
> java.net.PlainSocketImpl.socketAccept(Native
> >> Method)
> >>> at
> >>
> java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
> >>> at
> >>
> java.net.ServerSocket.implAccept(ServerSocket.java:453)
> >>> at
> >>
> java.net.ServerSocket.accept(ServerSocket.java:421)
> >>> at
> >>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
> >>> at
> >>
> sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> >>> at java.lang.Thread.run(Thread.java:637)
> >>> 2008-07-02 09:40:15,594 ERROR
> TripleWriteThread -
> >> Exception in TripleWriteThread
> >>> java.lang.OutOfMemoryError: Java heap space
> >>> at
> >>
> org.mulgara.store.xa.AVLNode.newInstance(AVLNode.java:212)
> >>> at
> >>
> org.mulgara.store.xa.AVLNode.getChildNode_N(AVLNode.java:1171)
> >>> at
> >>
> org.mulgara.store.xa.AVLNode.getRightChildNode_N(AVLNode.java:1152)
> >>> at
> >>
> org.mulgara.store.xa.AVLNode.findDown(AVLNode.java:339)
> >>> at
> >>
> org.mulgara.store.xa.AVLNode.find(AVLNode.java:300)
> >>> at
> >>
> org.mulgara.store.statement.xa.TripleAVLFile$Phase.syncAddTriple(TripleAVLFile.java:794)
> >>> at
> >>
> org.mulgara.store.statement.xa.TripleAVLFile$Phase.syncAddTriples(TripleAVLFile.java:727)
> >>> at
> >>
> org.mulgara.store.statement.xa.TripleWriteThread.run(TripleWriteThread.java:158)
> >>> 2008-07-02 09:40:15,601 WARN
> TripleWriteThread -
> >> Exception during abort of TripleWriteThread:
> >>
> /Applications/mulgara-2.0-alpha/dist/server1/xaStatementStore/xa.g_2013
> >>> java.lang.OutOfMemoryError: Java heap space
> >>> at
> >>
> org.mulgara.store.xa.AVLNode.newInstance(AVLNode.java:212)
> >>> at
> >>
> org.mulgara.store.xa.AVLNode.getChildNode_N(AVLNode.java:1171)
> >>> at
> >>
> org.mulgara.store.xa.AVLNode.getRightChildNode_N(AVLNode.java:1152)
> >>> at
> >>
> org.mulgara.store.xa.AVLNode.findDown(AVLNode.java:339)
> >>> at
> >>
> org.mulgara.store.xa.AVLNode.find(AVLNode.java:300)
> >>> at
> >>
> org.mulgara.store.statement.xa.TripleAVLFile$Phase.syncAddTriple(TripleAVLFile.java:794)
> >>> at
> >>
> org.mulgara.store.statement.xa.TripleAVLFile$Phase.syncAddTriples(TripleAVLFile.java:727)
> >>> at
> >>
> org.mulgara.store.statement.xa.TripleWriteThread.run(TripleWriteThread.java:158)
> >>>
> >>> --- On Wed, 7/2/08, Life is hard, and then you
> die
> >> <ronald at innovation.ch> wrote:
> >>>> From: Life is hard, and then you die
> >> <ronald at innovation.ch>
> >>>> Subject: Re: [Mulgara-general] Mulgara
> size limits
> >>>> To: "Mulgara General"
> >> <mulgara-general at mulgara.org>
> >>>> Date: Wednesday, July 2, 2008, 8:55 AM
> >>>> On Wed, Jul 02, 2008 at 05:21:21AM -0700,
> Chuck
> >> Borromeo
> >>>> wrote:
> >>>>> Hello,
> >>>>> OK I'm going to be a pain in the
> neck
> >> :). I
> >>>> tried running Mulgara with the
> >> -Xrunhprof:heap=sites
> >>>> switch. The process threw a
> >>>> java.security.AccessControlException:
> access
> >> denied error
> >>>> when I ran it using the -Xrunhprof switch.
> I have
> >> never
> >>>> run java with the dump file commands so I
> would
> >> appreciate
> >>>> anyone telling me how to fix this issue.
> I looked
> >> up the
> >>>> error and it said something about changing
> my
> >> java.policy
> >>>> and java.security files to allow the
> process to
> >> see these
> >>>> files. However, it didn't make much
> sense to
> >> me. I am
> >>>> running Mulgara with the sudo command so I
> should
> >> have
> >>>> permission to see everything. How do I
> fix this?
> >>>> (I'll attached the stack trace at the
> end of
> >> this
> >>>> message).
> >>>>
> >>>> The java access-control stuff has nothing
> to do
> >> with unix
> >>>> permissions,
> >>>> so being root or not doesn't change
> anything.
> >> The odd
> >>>> thing is,
> >>>> mulgara sets a policy which should allow
> access to
> >> all
> >>>> files - see the
> >>>> mulgara startup message
> >>>>
> >>>> INFO [main]
> (EmbeddedMulgaraServer.java:692) -
> >>>> java.security.policy set to
> >>>>
> >>
> jar:file:/..../mulgara-2.0-alpha.jar!/conf/mulgara-rmi.policy
> >>>> So I'm not sure what's going on
> either.
> >>>>
> >>>>> So...I tried running Mulgara with the
> >>>>> -XX:+HeapDumpOnOutOfMemoryError
> switch. This
> >> did
> >>>> generate a .hprof
> >>>>> file (about 1 GB in size!). I tried
> opening
> >> this file
> >>>> using a 1.6
> >>>>> version of jhat (1.5's jhat
> can't open
> >> a 64
> >>>> bit .hprof file).
> >>>>> However, I am getting a NullPointer
> exception
> >> from
> >>>> jhat when opening
> >>>>> the file.
> >>>> I'm not surprised - as I said before,
> I've
> >> found
> >>>> 1.5 to be unreliable
> >>>> in this respect. If you can run under 1.6
> I think
> >>>> you'll have better
> >>>> luck.
> >>>>
> >>>>
> >>>> Cheers,
> >>>>
> >>>> Ronald
> >>>>
> >>>>
> _______________________________________________
> >>>> Mulgara-general mailing list
> >>>> Mulgara-general at mulgara.org
> >>>>
> >>
> http://mulgara.org/mailman/listinfo/mulgara-general
> >>>
> >>>
> >>>
> _______________________________________________
> >>> Mulgara-general mailing list
> >>> Mulgara-general at mulgara.org
> >>>
> http://mulgara.org/mailman/listinfo/mulgara-general
> >
> >
> >
> > _______________________________________________
> > Mulgara-general mailing list
> > Mulgara-general at mulgara.org
> > http://mulgara.org/mailman/listinfo/mulgara-general
More information about the Mulgara-general
mailing list