I am loading Wikidata - the big version, not the truthy one. (I had to make two patches to allow non-terrestrial geo-literals to be loaded.) Because the file appears to not have labelled blank nodes and has blank lines between objects, I have split it into ten pieces and am now loading them in one rdf_loader_run() call.
I’m doing this on a four-core (and four-thread) machine with 32GB of main memory, running Fedora 29. Having concurrent loads doesn’t make sense on this machine, I think. One load is sufficient to use most of the available computing power.
However virtuoso still increases in memory size and doesn’t shrink down when a file is completely loaded. It is on track to run out of memory sometime tonight.
Here is the output of status();. I cut down on the number of buffers in the hope that this would allow completion of the load, but that didn’t work.
OpenLink Virtuoso Server
Version 07.20.3230-pthreads for Linux as of Nov 16 2018
Started on: 2018-11-19 13:12 GMT-5
Database Status:
File size 115343360, 15742720 pages, 3695763 free.
1360000 buffers, 1350569 used, 982956 dirty 20530 wired down, repl age 20277301 277 w. io 0 w/crsr.
Disk Usage: 164040987 reads avg 0 msec, 0% r 0% w last 0 s, 170668406 writes flush 259.1 MB,
4070219 read ahead, batch = 30. Autocompact 15806277 in 12270280 out, 22% saved col ac: 94862572 in 9% saved.
Gate: 5272543 2nd in reads, 0 gate write waits, 0 in while read 0 busy scrap.
Log = /home/virtuoso/var/lib/virtuoso/db/virtuoso.trx, 484860187 bytes
8280467 pages have been changed since last backup (in checkpoint state)
Current backup timestamp: 0x0000-0x00-0x00
Last backup date: unknown
Clients: 3 connects, max 3 concurrent
RPC: 35 calls, 2 pending, 2 max until now, 0 queued, 2 burst reads (5%), 0 second 394M large, 2293M max
Checkpoint Remap 0 pages, 0 mapped back. 363 s atomic time.
DB master 15742720 total 3697208 free 0 remap 0 mapped back
temp 256 total 251 free
Lock Status: 0 deadlocks of which 0 2r1w, 0 waits,
Currently 4 threads running 0 threads waiting 0 threads in vdb.
Pending:
Client 1111:2: Account: dba, 1294 bytes in, 11197 bytes out, 1 stmts.
PID: 16852, OS: unix, Application: unknown, IP#: 127.0.0.1
Transaction status: PENDING, 0 threads.
Locks:
Client 1111:3: Account: dba, 206 bytes in, 289 bytes out, 1 stmts.
PID: 24434, OS: unix, Application: unknown, IP#: 127.0.0.1
Transaction status: PENDING, 1 threads.
Locks:
Client 1111:1: Account: dba, 969 bytes in, 3179 bytes out, 1 stmts.
PID: 9111, OS: unix, Application: unknown, IP#: 127.0.0.1
Transaction status: PENDING, 1 threads.
Locks:
Running Statements:
Time (msec) Text
22921077 rdf_loader_run()
1126 status()
Hash indexes