Added possibility to link spreadsheet for automatic submission export in multiple formats

[Koc]

Hello everyone!

I've added possibility to link form with spreadsheet in csv/ods/xlsx format and automatically export it once new submission added. Inspired from #1403 but with more formats. Also we edit already existent file, which give us possibility to add additional columns with notes/statuses/etc.

Here you can find few video/screenshots:

nextcloud-forms-export-03.mp4

Exported file

What is out of scope right now but can be added in upcoming PRs:

• handling of responses deletion
• handling of adding/removal of the questions

Todo:

• fix old tests and add new

[Chartman123]

@Koc thank you very much for your contribution! We'll have a look at it 🙂

@nextcloud/designers Could you please do a quick review of the screenshots? I'm not really convinced of the UI in the menu. Looks a bit crowded...

@hamza221 fine for you to close your PR and keep working on this one?

[susnux]

Thank you very much! Nice work!

Some first thought about your code, besides there are quite a lot of linter issues to fix.
I think we do not need to implement this as a breaking change API wise but can make this compatible with the current API by just extending.

Moreover I do not know if it makes much sense to store the file path in the database. We probably should just work with the file id.

[Koc]

Mates, any idea how to fix that?

Error: lib/Service/SubmissionService.php:175:12: MissingDependency: OCP\Files\IRootFolder depends on class or interface oc\hooks\emitter that does not exist (see https://psalm.dev/157)

I haven't touched \OCP\Files\IRootFolder $storage dependency at all

Also I'm trying to run Psalm locally and got 27 errors which is completely unrelated to my changes and not present in pipeline 😕

Also pipeline not starting for my commits due to lack of approvals 🤷‍♂️

[Koc]

@susnux

Moreover I do not know if it makes much sense to store the file path in the database. We probably should just work with the file id.

Yes, it's doable but little bit tricky because we can choose file or folder. If it is a blocker for merge - just say and I will rework. Because now it sounds like you are not 100% sure :)

[susnux]

I haven't touched \OCP\Files\IRootFolder $storage dependency at all

Yes that error is unrelated, needs to be fixed on the server, so you can ignore it.
But the eslint issues still need to be fixed :)

Yes, it's doable but little bit tricky because we can choose file or folder.

Folders also have a file id. The reason why I would prefer to only use the ID rather then hard coding the path is:

1. If you rename on of the folders the path would be broken
2. If we (in future) support collaboration of forms the same folder will or can have a different path for different users. (depending if shared etc)

So it would make things more robust and safe for the future if we already stick with the ID.

[nimishavijay]

Super nice! :)
I have some questions:

• Is the spreadsheet automatically updated when new responses are added?
• What does "re-export spreadsheet" do? There are several actions on the screen that are closely related (create spreadsheet, save, download, re-export) so it could get a bit confusing to someone new

I do agree that the action menu looks a bit crowded, we can easily fix that :)

• Instead of disabling, we could hide the "Re-export" and "Unlink" actions if a spreadsheet has not been created yet
• For "Save" and "Download" we could use a second level action menu, similar to the send later action in mail

Recording.2023-10-24.190452.mp4

What do you think?

[susnux]

Error: lib/Service/SubmissionService.php:175:12: MissingDependency: OCP\Files\IRootFolder depends on class or interface oc\hooks\emitter that does not exist (see https://psalm.dev/157)

Can be silenced like this:
https://github.com/nextcloud/spreed/blob/55e5fa00ffcdafe3eb3a94bc03b25e6f2ee9982d/psalm.xml#L91

[Koc]

@nimishavijay yes, we are automatically update spreadsheet once new response added. But sometimes we can get into trouble with a race conditions. Imagine that Employee edits file with submissions in a browser and overwrites changes that exported after new response added. In this case we should have mechanism to force re-export responses to file after Employee finishes his work.

[Koc]

@nimishavijay I've added nested menu, you can find a video record for it in the PR description. Also disabling of non-active buttons has been replaced with hidding.

@susnux I've reworked PR, now we are storing fileId, fileFormat only, without filePath.

Also I've fixed all pipelines (I hope 🤞 ). Can we enable full pipeline for an every push commit for this PR?

[Chartman123]

@Koc The workflows will run automatically once you have contributed to this repository. Until then we have to manually start it :)

[codecov]

Codecov Report

Attention: 65 lines in your changes are missing coverage. Please review.

Comparison is base (a6a4264) 44.61% compared to head (d6f5cb6) 44.79%.
Report is 7 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1758      +/-   ##
============================================
+ Coverage     44.61%   44.79%   +0.17%     
- Complexity      662      686      +24     
============================================
  Files            58       59       +1     
  Lines          2600     2697      +97     
============================================
+ Hits           1160     1208      +48     
- Misses         1440     1489      +49     

[Koc]

wow, we have completely green pipeline for the very first time! 🎉

[Chartman123]

I had a look at the code, but not everything in-depth. Added some comments :) I will run the code in the next few days and see how it works.

[susnux]

Nice work! Just fix the rest of the routes (API version) then this is good to go 🎉

[susnux]

@Koc Thank you very much for this fantastic work! 🚀
I really like the feature and I am very happy you kept on working on this, I hope we can release it with 4.1.0 soon!

[Koc]

very happy to hear that, thanx! It was hard 😃

[Chartman123]

Yes, thank you very much @Koc :) We'd be glad if you continue contributing here 🙂

[jancborchardt]

Awesome work @Koc! Do you already have a guest account on our instance? We could add you there so you can join the Forms community Talk channel. :)
(I just need an email address to add you with)

[Koc]

@jancborchardt no, I haven't. You can find my email address in copyrights 😃

[Gamechiefx]

Having this error when attempting to create a spreadsheet:

"no app in context","method":"POST","url":"/ocs/v2.php/apps/forms/api/v2.4/form/link/ods","message":"Exception thrown: OCP\Files\NotFoundException","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36","version":"28.0.2.5","exception":{"Exception":"OCP\Files\NotFoundException","Message":"/admin/files/Documents/folder/TCC \nData Usage Assessment Form (responses).ods","Code":0,"Trace":[{"file":"/var/www/html/lib/private/Files/Node/Folder.php","line":135,"function":"get","class":"OC\Files\Node\Root","type":"->","args":["/TCC/files/Documents/folder/TCC \nData Usage Assessment Form (responses).ods"]},{"file":"/var/www/html/custom_apps/forms/lib/Service/SubmissionService.php","line":216,"function":"get","class":"OC\Files\Node\Folder","type":"->","args":["TCC \nData Usage Assessment Form (responses).ods"]},{"file":"/var/www/html/custom_apps/forms/lib/Controller/ApiController.php","line":1324,"function":"writeFileToCloud","class":"OCA\Forms\Service\SubmissionService","type":"->","args":[["OCA\Forms\Db\Form",1],"/Documents/folder","ods"]},{"file":"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php","line":230,"function":"linkFile","class":"OCA\Forms\Controller\ApiController","type":"->","args":["LZyTtsgDfqtFWbNo","/Documents/folder","ods"]},{"file":"/var/www/html/lib/private/AppFramework/Http/Dispatcher.php","line":137,"function":"executeController","class":"OC\AppFramework\Http\Dispatcher","type":"->","args":[["OCA\Forms\Controller\ApiController"],"linkFile"]},{"file":"/var/www/html/lib/private/AppFramework/App.php","line":184,"function":"dispatch","class":"OC\AppFramework\Http\Dispatcher","type":"->","args":[["OCA\Forms\Controller\ApiController"],"linkFile"]},{"file":"/var/www/html/lib/private/Route/Router.php","line":315,"function":"main","class":"OC\AppFramework\App","type":"::","args":["OCA\Forms\Controller\ApiController","linkFile",["OC\AppFramework\DependencyInjection\DIContainer"],["v2.4","ods","ocs.forms.api.linkFile"]]},{"file":"/var/www/html/ocs/v1.php","line":65,"function":"match","class":"OC\Route\Router","type":"->","args":["/ocsapp/apps/forms/api/v2.4/form/link/ods"]},{"file":"/var/www/html/ocs/v2.php","line":23,"args":["/var/www/html/ocs/v1.php"],"function":"require_once"}],"File":"/var/www/html/lib/private/Files/Node/Root.php","Line":207,"CustomMessage":"Exception thrown: OCP\Files\NotFoundException"},"id":"65c302dad2cd8"}

Chat GPT Findings:

The error log you've provided outlines a NotFoundException occurring within a Nextcloud instance, specifically when trying to access or link to an .ods file located in a user's documents folder. Here's a detailed breakdown of what happened, the probable cause, and how to address it:

Error Summary

Exception Type: OCP\Files\NotFoundException
Affected Component: The Nextcloud Forms app (custom_apps/forms), particularly its SubmissionService and ApiController.
User Action: Attempting to link an .ods file ("TCC Data Usage Assessment Form (responses).ods") to a form submission.
System Response: The system was unable to find the specified .ods file in the path provided.

Probable Cause

The error was triggered during a POST request to link an .ods file to a form submission. The path to the file includes a line break (\n) which might be unintentional or misinterpreted by the system as part of the file path. File paths typically do not include line breaks, and their presence can cause the system to search for a file in a non-existent location.

Troubleshooting Actions

Attempted to remove and reinstall the app

Version

Latest Nextcloud 28

[Chartman123]

@Gamechiefx please create a new issue for that and use the template for bug reports. Thank you!

[matthijskooijman]

You're other proposals will definitely not make it into this PR anymore. So please open separate feature request issues for them. We can then discuss them there.

About the technical name: the little error should be indeed for saving the values and this function is available since a few versions already. So as you said not related. Please also add a new issue for that.

Done, I've opened #1970, #1971, #1972 and #1973 for these issues.

[JocelynDelalande]

Hello, I think this feature is what I look for, but I don't understand how to use it. Is it documented somewhere ? Thanks by advance.

[Chartman123]

@JocelynDelalande Just have a look at the video in the OP. :)

[JocelynDelalande]

@Chartman123 Thanks for your answer, and nevermind: I was just using a version of NC that was too old for forms 4.1. Can't wait to test the new Forms :)

[sbernard31]

Hi,

I understand that file should be automatically added when new form entry are submitted.

Is the spreadsheet automatically updated when new responses are added?

yes, we are automatically update spreadsheet once new response added.

I tested recently with nextcloud (29.0.7) and Forms (4.3.13)
And this is not done automatically, I need to click manually on "Re-Export" button.
Is it normal ? or should it work ? maybe it is available on most recent version ? 🙏

Another question.
Style and formula seems to be lost after a re-export when using ods.
With xlsx there are not lost. same questions. 🙏

[Chartman123]

@sbernard31 the export is done by a background job, so it will take some time until the linked spreadsheet will be updated automatically.

Style and formula should also get lost with other formulas as we don't only add new responses but do a full re-export of all responses. So all changes to the linked files will be lost after the next export.