I don't know what to tell you, on my box NextCloud works perfectly.
I just had to modify open_basedir as follows:
and a couple of other options:
The rest is absolutely standard…
NextCloud has his own log file with EVERYTHING written. If there's a problem it will be written there...
Thank you very much for the tip, it wasn't open_basedir, it was the shell_exec function which stopped the OnlyOffice from working, in case someone else runs into this problem, just disable all disable_functions and the connection between NextCloud and OnlyOffice will work.