Bulk Loading RDF Source Files into one or more Graph IRIs

[hwilliams]

Bulk Loading RDF Source Files into one or more Graph IRIs

This document details how large RDF data set files can be bulk loaded into Virtuoso. The data sets may consist of multiple files, which may be loaded into one or several graphs.

• Prerequisites
• Bulk loading process
  • Running multiple Loaders
• Stopping the bulk load process
• Checking bulk load status
• Cluster-specific details
• Faceted Browser “Entity Label Lookups”
• Related

Prerequisites

• The Virtuoso Bulk Loader functions must be present. They are pre-loaded starting with commercial version 06.02.3129 and open source version 6.1.3, but must be manually loaded into older versions.
• The directory containing the data set files must be included in the **DirsAllowed**parameter defined in the virtuoso INI file, after which the Virtuoso server must be restarted.
• The Virtuoso Server should be appropriately configured to use sufficient memory and other system resources as detailed in the Virtuoso RDF Performance Tuning Guide, or the load may take an unacceptably long time, approaching forever.
• The file formats and file name extensions of the files to be loaded must be among the following, which the rdf_loader_run() function understands. Any of these may be compressed in gzip (i.e., with the additional .gz file name extension), bz2 or xz compression formats to save space; in such case, they will be automatically expanded by the bulk loader.

.grdf	Geospatial RDF
.nq	N-Quads
.nt	N-Triples
.owl	OWL
.rdf	RDF/XML
.trig	TriG
.ttl	Turtle
.xml	RDF/XML
.jsonld	JSON-LD

Bulk loading process

1. The name of the RDF graph into which the data set(s) should be loaded can be specified through a text file placed in the same source directory as the source data files. This file’s contents will override any options specified in the ld_dir() or ld_dir_all() function call. The content of a file with the same name as a data file plus the .graph filename extension will be used for that data file (e.g., my_data.n3.graph will be used with my_data.n3). The content of a file named global.graph will be used for any and all other data files in that directory.

2. Note: if the third parameter (graph_iri) of ld_dir() or ld_dir_all() is NULL, any data files that do not have a corresponding .graph file will not be loaded.*. If datasets are NQUAD or TRIG then the graph name specified in such dataset files in quad format will always be used.

<source-file>.<ext> 
<source-file>.<ext>
.graph global.graph

— e.g., —

myfile.n3          ;; RDF data
myfile.n3.graph    ;; Contains Graph IRI name into which RDF data from myfile.n3 will be loaded
global.graph       ;; Contains Graph IRI name into which RDF data from any files that do not have a specific graph name file will be loaded

3. Place the graph IRI, , e.g., http://dbpedia.org , in the *.graph file.
4. Use isql to register the file(s) to be loaded by running the appropriate function, e.g. –

SQL> ld_dir ('/path/to/files', '*.n3', 'http://dbpedia.org');

• **ld_dir() **to load only from the specified directory, excluding any subdirectories –

SQL> ld_dir ('<source-filename-or-directory>', '<file name pattern>', 'graph iri');

• **ld_dir_all() **to load from the specified directory, including any and all subdirectories –

SQL> ld_dir_all ('<source-filename-or-directory>', '<file name pattern>', 'graph iri');

5. The table DB.DBA.load_list can be used to check the list of data sets registered for loading, and the graph IRIs into which they will be or have been loaded. The **ll_state**field can have three values: 0 indicating the data set is to be loaded; 1 the data set load is in progress; or 2 the data set load is complete:

SQL> select * from DB.DBA.load_list;
ll_file               ll_graph        ll_state    ll_started              ll_done                 ll_host    ll_work_time    ll_error
VARCHAR NOT NULL      VARCHAR         INTEGER     TIMESTAMP               TIMESTAMP               INTEGER    INTEGER         VARCHAR
_____________________________________________________________________________________________________________________________________

./dump/d1/file1.n3    http://file1    2           2010.10.20 9:21.18 0    2010.10.20 9:21.18 0    0          NULL            NULL
./dump/d2/file2.n3    http://file2    2           2010.10.20 9:21.18 0    2010.10.20 9:21.18 0    0          NULL            NULL
./dump/file.n3        http://file     2           2010.10.20 9:21.18 0    2010.10.20 9:21.18 0    0          NULL            NULL

3 Rows. -- 1 msec.
SQL>

6. Finally, perform the bulk load of all data by executing the rdf_loader_run() function:

SQL> rdf_loader_run();

• Note: the rdf_loader_run() function prototype is:

rdf_loader_run 
  (  IN  max_files   INTEGER := NULL
  ,  IN  log_enable  INT     := 2
  )

One of the side effects of the default log_enable = 2 setting is that triggers are disabled, to speed the loading of data. If triggers are required (e.g., for RDF Graph replication between nodes), then the log_enable mode should be set to 3 when calling the rdf_loader_run() function as follows:

rdf_loader_run (log_enable=>3);

7. Note at the end of the Bulk Load process the checkpoint command MUST be run to commit the bulk loaded data to the Virtuoso database file. This is because the default log_enable = 2 mode used by the bulk loader turns off transaction logging, thus if the database is shutdown before a checkpoint is run all the bulk loaded data would be lost. The bulk loader also disables checkpointing and the scheduler, which also need to be re-enabled post bulk load.

checkpoint; 
checkpoint_interval(N); 
scheduler_interval(M);

Running multiple Loaders

On a multi-core machine, it is recommended that data sets be split into multiple files, and that these be registered in the DB.DBA.load_list table with the ld_dir() function. Once registered for load, the rdf_loader_run() function can be run multiple times, it is recommended a maximum of no cores / 2.5, to optimally parallelize the data load and hence maximize load speed. A typical script that can be run from command line (e.g., bulk_load.sh) might look like –

. /opt/openlink/virtuoso/virtuoso-enterprise.sh
isql 1111 dba dba exec="rdf_loader_run();" & 
isql 1111 dba dba exec="rdf_loader_run();" & 
isql 1111 dba dba exec="rdf_loader_run();" & 
isql 1111 dba dba exec="rdf_loader_run();" & 
isql 1111 dba dba exec="rdf_loader_run();" & 
isql 1111 dba dba exec="rdf_loader_run();" & 
isql 1111 dba dba exec="rdf_loader_run();" &  
wait 
isql 1111 dba dba exec="checkpoint;" 

and be run with the command:

sh /opt/openlink/virtuoso/bin/bulk_load.sh

Stopping the bulk load process

1. All RDF loader threads can be stopped using the command rdf_load_stop(), at which point all currently running threads will be allowed to complete and then exit:

SQL> rdf_load_stop();

Checking bulk load status

1. Once the rdf_loader_run() is complete, you can check the DB.DBA.load_list to confirm all data sets were loaded successfully. This is indicated by an ll_state value of 2 and an ll_error value of NULL.
   2.Note:* The following query can be used to check if any errors occurred when loading datasets file during the bulk load:

select * from DB.DBA.LOAD_LIST where ll_error IS NOT NULL

Cluster-specific details

1. On a Virtuoso Clustered Server the “cl_exec('rdf_ld_srv(log_enable)')” commands (where log_enable is 2 or 3, as with the rdf_loader_run() function) can be used to invoke a single “rdf_loader_run()” on each node of the cluster:

SQL> cl_exec('rdf_ld_srv()'); 

Done. -- 265956 msec.
SQL>

Faceted Browser “Entity Label Lookups”

If the Virtuoso Faceted Browser is install its “Entity Label Lookups” table can be automatically updated when new “rdfs:label” entries are added to the database, during bulk load and other insert operations, by adding the new LabelInferenceName = facets parameter in the “[SPARQL]” section of the virtuoso.ini config file and restarting the database for it to be picked up.

Related

• Example of single file load
• Example of multiple file load
• Example of Dbpedia datasets load
• Virtuoso RDF Bulk Update “with_delete” option
• How can I determine the time taken to load datasets with RDF Bulk Loader

[yayamamo]

Hi, when trying to bulk-load JSON-LD files, we got an error issued by a turtle parser.
We use VOS 7.2.9.
Could you tell me if there is any way to work around this?

SQL> ld_dir('/mnt/.../datasets/test', '*.jsonld', 'http://example.com/test');

Done. -- 2 msec.
SQL> rdf_loader_run (log_enable=>2);

Done. -- 4 msec.

SQL> select * from dba.load_list;
ll_file                                                                           ll_graph                                                                          ll_state    ll_started           ll_done              ll_host     ll_work_time  ll_error
VARCHAR NOT NULL                                                                  VARCHAR                                                                           INTEGER     TIMESTAMP            TIMESTAMP            INTEGER     INTEGER     VARCHAR
_______________________________________________________________________________

/mnt/.../datasets/test/test.jsonld                                    http://example.com/test                                                           2           2023.12.25 16:45.49 40293000  2023.12.25 16:45.49 42606000  0           NULL        37000 [Vectorized Turtle loader] TURTLE RDF loader, line 2: SP029: TURTLE RDF loader, line 2: Default namespace prefix is not defined, so standalone ':' can not be used as an identifier.

1 Rows. -- 0 msec.

Thanks.

[hwilliams]

As per the Virtuoso Open Source 7.2.10 Release Notes support for loading JSON-LD files was added in this release, and it not supported the 7.2.9 release you indicate to be running. Thus you need to upgrade to the latest Virtuoso Open Source 7.2.11 Release , to get all the latest fixes including support for loading JSON-LD files.

Note also HowTo – Load JSON-LD files into Virtuoso.