If the query returns results against isql presumably connecting as the dba user, but does not return results against the ~/sparql endpoint which runs as the special internal SPARQL user , what happens if you connect to the available ~/sparq-auth endpoint of your installation, which required authentication enabling you to connect as the dba user (as with isql) and run the query to see if it then returns results as isql does ?
Should results be returned via ~/sparql-auth and not ~/sparql then please select the Generate SPARQL compilation report (instead of executing the query) checkbox option to enable Generation of Query Compilation Reports against both endpoints for comparison.
Also, you indicate that at one point the query was returning results when running against the ~/sparql endpoint, in which case when did it stop returning results and did anything change on the server at the time ie configuration , new data load etc ?
The param tn_max_memory can be set in the [Flags] section of the INI file or with the __dbf_set()
function to control the transitive memory size.