Virtuoso 8.3 Infinite Optimization Loop

I am currently developing a piece of software that uses rules from property shapes (namely SHACL constraints) to generate forms which have their information inserted into triplestores over a SPARQL endpoint. The main use cases are for users entering data about a new entity, and users filling in missing data for existing entities in a knowledge graph. Until recently I have been running version 7.2 of virtuoso on my system. I have recently tested some of the queries (an example of such a query is below) on version 8.3 of Virtuoso and discovered that many queries that were working prior are now returning the error

‘Virtuoso 37000 Error SP031: SPARQL compiler: Internal error: sparp_rewrite_qm_optloop(): Infinite optimization loop?’.

It appears that these errors are occurring when running a sub query on a named graph which differs from the named graph of the ambient query. I should also note I have also noticed found that the behaviour of zero or one paths as part of an alternate path (e.g. rdf:rest*/rdf:first) have caused what appears to be unexpected errors both in version 7.2 and 8.3 of virtuoso but I do not believe this to be the cause as I never saw infinite optimization loops in virtuoso 7.2.

The below query is designed to extract all first order properties that we know apply to a target in a knowledge graph. It first queries the knowledge graph to get all predicates attached to the target and all classes that the target belongs to (including implicit classes). It then uses sh:targetNode, sh:targetSubjectsOf, sh:targetObjectsOf and sh:targetClass labels in the shacl/shapes graph to establish the applicable shacls and then returning a filtered dictionary of property shapes.

I have highlighted the portion of the query that appears to be causing the error (I have run some tests using only this section of code).

PREFIX sh: <http://www.w3.org/ns/shacl#>
SELECT DISTINCT (concat("{",group_concat(distinct ?tproperties; separator="},{"),"}") AS ?r) {
{SELECT DISTINCT ?t (concat("'",group_concat(distinct ?properties; separator="},{'")) AS ?x)
{SELECT DISTINCT ?t (group_concat(distinct ?prop; separator=",'") AS ?properties)
FROM <http://shacl>
WHERE {
{?s sh:targetNode ?t}
UNION {?s sh:targetSubjectsOf ?p}
UNION {?s sh:targetObjectsOf ?pi}
UNION {?s  sh:targetClass ?class}
{SELECT DISTINCT ?s ?n ?pa (group_concat(distinct ?obs; separator="','") AS ?ob)
WHERE {?s sh:property ?n . ?n ?pa ?oa OPTIONAL {?oa (rdf:rest*/rdf:first)? ?obs}
FILTER (!isBlank(?obs) && ?obs != rdf:nil)}}
{SELECT DISTINCT ?t ?class ?p ?pi
FROM <some_knowledge_graph>
WHERE {
{SELECT DISTINCT ?t ?class ?p WHERE {?t ?p ?os OPTIONAL {?p rdfs:subPropertyOf*/rdfs:domain+/rdfs:subClassOf* ?class}}}
UNION {SELECT DISTINCT ?t ?class ?pi WHERE {?s ?pi ?t OPTIONAL {?pi rdfs:subPropertyOf*/rdfs:range+/rdfs:subClassOf* ?class}}}
UNION {SELECT DISTINCT ?t ?class WHERE {?t a/rdfs:subClassOf* ?class}}
FILTER (?p != rdf:type && ?p != rdfs:subPropertyOf && ?pi != rdf:type && ?pi != rdfs:subPropertyOf)}}
VALUES (?t) {(<some_target>)}
FILTER (NOT EXISTS { SELECT ?n { ?n ?q ?w . VALUES (?w) {(sh:Info)}}} && EXISTS {SELECT ?o { ?o sh:severity ?x}})
BIND(REPLACE(STR(?pa), sh:, "") AS ?pa2)
BIND(IF(?pa = sh:severity, REPLACE(STR(?ob), sh:, ""),?ob) AS ?ob2)
BIND(CONCAT(IF(?pa2 = 'path' || ?pa2='inversePath', CONCAT("pathType':'", ?pa2, "','path"), ?pa2), IF(regex(?ob, ",", "i") && !regex(?pa2, "message", "i"),"':['","':'"), ?ob2, IF(regex(?ob, ",", "i") ,"']","'")) AS ?prop)}
GROUP BY ?t ?n}
GROUP BY ?t}
BIND (CONCAT("'", ?t, "':{", ?x, "}") AS ?tproperties)}

I can get the first variant of what I assume is a query to run, but the other two with the UNION clause fail giving error Virtuoso 37000 Error SP030: SPARQL compiler, line 35: syntax error at 'UNION' before '{'

So not sure I am reconstructing the queries as expected or if something might have been lost in the post ?

Are all 3 variant suppose to give the Virtuoso 37000 Error SP031: SPARQL compiler: Internal error: sparp_rewrite_qm_optloop(): Infinite optimization loop? error (on your platform with relevant data) ?

What is the actual definition of the sh: prefix as I created a dummy one PREFIX sh: <sh> to get the first query variant to run ?

Do you have a live externally accessible instance where the issue can be seen ?

It would be easiest if you just provide the 3 query variants in entirety in verbatim text such that they can be run as is, rather then trying to reconstruct them …

Many apologies, it is a single query which I had accidentally broken up when trying to highlight the section in which I thought the infinite loop may be occurring. I have fixed the post to reflect that it is a single query. I have also added the appropriate sh: prefix for http://www.w3.org/ns/shacl#.

Unfortunately I am not currently running a live externally accessible instance, however, the data on the server should not affect the output as the error occurs in the compilation stage before the graph is queried. On a 7.2 virtuoso instance, without the appropriate datasets (for instance dbpedia), I have confirmed that the query still returns an empty set of data (not an error) which in JSON format is

{ “head”: { “link”: [], “vars”: [“r”] },
“results”: { “distinct”: false, “ordered”: true, “bindings”: [
{ “r”: { “type”: “literal”, “value”: “{}” }} ] } }

whilst an 8.3 server returns the Virtuoso 37000 Error SP031: SPARQL compiler: Internal error: sparp_rewrite_qm_optloop(): Infinite optimization loop?.

My machine is currently running on Microsoft Windows 10.

I had also opened a new topic relating a 7.2 error I found. I initially withdrew that topic initially because my API was getting conflicting error messages to when I made the query manually over a sparql-auth endpoint. I have now obtained the same error over the sparql-auth endpoint with a manually entered statement. The cause appears to be the use of ‘COUNT’ within the ‘IF’ statement.

DETAILS
Server: OpenLink Virtuoso VDB Server
Platform: Win64
Version: 07.20.3219

ERROR
Virtuoso 37000 Error SQ156: Internal Optimized compiler error : Bad dfe in sqlo_place_exp in \opl_port\pkg\virtuoso7.git\libsrc\Wi\sqldf.c:2317.
Please report the statement compiled.

SPARQL query:

#output-format:text/html
define sql:signal-void-variables 1 PREFIX sh: <http://www.w3.org/ns/shacl#>
PREFIX ex: <http://example.org/>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX rdfs: <http://www.w3.org/ns/shacl#>
PREFIX ex2: <http://example.org>
SELECT DISTINCT (concat('[',group_concat(distinct ?properties; separator=','), ']') AS ?r)
    {SELECT DISTINCT (concat('{"', group_concat(distinct ?prop; separator=',"'), '}') AS ?properties)
    FROM <http://shacl>
    WHERE {
    {SELECT DISTINCT ?s ?n ?pa ((IF(COUNT(distinct ?obs) > 1, group_concat(distinct ?obs; separator='","'), concat( '["', group_concat(distinct ?obs; separator='","')   ,'"]'  ))) AS ?ob)
    WHERE {?s sh:property ?n . ?n ?pa ?oa OPTIONAL {?oa (rdf:rest*/rdf:first)? ?obs
    FILTER(IF(NOT EXISTS {SELECT ?r WHERE {?n ?p ?r FILTER(lang(?r) = 'undefined')}}, lang(?oa) = '', lang(?oa) = 'undefined'))
    }

    FILTER (!isBlank(?obs) && ?obs != rdf:nil)}}
    VALUES (?s) {(<http://example.org/NewZealandLanguagesShape>)}
    FILTER (EXISTS {SELECT ?n {?n sh:severity ?w . VALUES (?w) {(sh:Warning)(sh:Violation)}}} || NOT EXISTS { SELECT ?n WHERE {?n sh:severity ?w}} )
    BIND(REPLACE(STR(?pa), sh:, '') AS ?pa2)
    BIND(IF(?pa = sh:severity, REPLACE(STR(?ob), sh:, ''),?ob) AS ?ob2)
    BIND(CONCAT(IF(?pa2 = 'path' || ?pa2='inversePath' || ?pa2='alternativePath' || ?pa2='zeroOrMorePath' || ?pa2='oneOrMorePath' || ?pa2='zeroOrOnePath', CONCAT('pathType":"', ?pa2, '","path'), ?pa2), '":"', ?ob2,'"') AS ?prop)}
    GROUP BY ?n}

Running the query above against the latest Virtuoso open source 07.20.3230 build the query executes without error, see live link

Running against a latest Virtuoso 08.03.3316 commercial binary it fails with error Virtuoso 37000 Error SP031: SPARQL compiler: Internal error: ssg_tmpl_X_of_short(): bad mode needed , see live link , does this run against your 8.3 instance ?

Running the query above against a latest Virtuoso 08.03.3316 commercial binary it fails with error Virtuoso 37000 Error SP031: SPARQL compiler: Internal error: ssg_print_builtin_expn(): bad native type for isBLANK() , see live link

Running the query above against the latest Virtuoso open source 07.20.3230 build the query executes without error, see live link

Note the 07.20.3219 Virtuoso opensource binary you are using is rather old and the latest available 07.20.3229 prebuilt binary can be downloaded from here

I shall report the Virtuoso 8.3 error to development to look into and fix …