How to handle tests that are not unit tests?

[cwollenhaupt]

Most of my tests are unit tests. They test a single class or a few couple classes in an isolated way that doesn't depend on a particular setup in my environment or database. Typically, all of these tests of a project are loaded in FoxUnit. I execute them all when I'm done with a ticket or change to make sure that everything still works. These tests execute relatively fast.

However, I also have other tests that are not independent. There are tests that communicate with live web services and ensure that the WSDL file hasn't changed or that the API is still working. These tests take considerably more time.

Currently I'm working on classes that use a document scanner, so my tests even include required hardware, a UI (select the scanner) and require specific setups (put two pages into the document feeder). These tests are great for development, because I don't have to run the application to test my code and I don't have to setup some test environment. However, they are totally unsuitable to run with my unit tests. Instead I call them on a case by case basis when I make changes or need to validate certain functionality.

I've considered defining a text file in the project folder that lists different Test folders with "Tests" being the default selected, the default when no text file exists and the default when tests are run automatically. However, when the text file exists and includes a line that is not "Tests", FoxUnit would display a dialog when launched and let the user pick a folder. This would impact the DataPath property in the FXU instance.

So basically, the Tests folder would contain all unit tests. But I could also create a folder "Integration Tests" or "Hardware Tests" with a separate set of loaded test classes.

Does that sound reasonable?

[eselje]

If I understand, these tests be run in the FoxUnit UI, giving you the green/red, but actually be your own scripts that you're running instead? Or would they be FoxUnit tests that are dependent on the previous results?

…

On Tue, Jan 14, 2020 at 5:28 AM Christof Wollenhaupt < ***@***.***> wrote: Most of my tests are unit tests. They test a single class or a few couple classes in an isolated way that doesn't depend on a particular setup in my environment or database. Typically, all of these tests of a project are loaded in FoxUnit. I execute them all when I'm done with a ticket or change to make sure that everything still works. These tests execute relatively fast. However, I also have other tests that are not independent. There are tests that communicate with live web services and ensure that the WSDL file hasn't changed or that the API is still working. These tests take considerably more time. Currently I'm working on classes that use a document scanner, so my tests even include required hardware, a UI (select the scanner) and require specific setups (put two pages into the document feeder). These tests are great for development, because I don't have to run the application to test my code and I don't have to setup some test environment. However, they are totally unsuitable to run with my unit tests. Instead I call them on a case by case basis when I make changes or need to validate certain functionality. I've considered defining a text file in the project folder that lists different Test folders with "Tests" being the default selected, the default when no text file exists and the default when tests are run automatically. However, when the text file exists and includes a line that is not "Tests", FoxUnit would display a dialog when launched and let the user pick a folder. This would impact the DataPath property in the FXU instance. So basically, the Tests folder would contain all unit tests. But I could also create a folder "Integration Tests" or "Hardware Tests" with a separate set of loaded test classes. Does that sound reasonable? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#25?email_source=notifications&email_token=AB6LYTGUTI6TSZSS35QLWK3Q5WOUTA5CNFSM4KGRNLR2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IGAZGMQ>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB6LYTGUHTH3XW5ULGFXAQ3Q5WOUTANCNFSM4KGRNLRQ> .

[cwollenhaupt]

It's a FoxUnit test that looks like this:

Procedure Test_Scan
	Local loRef, laPages[1], lnCount
	loRef = NewObject("cUiWindowsImageAcquisition", ;
		"Klassen/cuiWindowsImageAcquisition.prg")
	lnCount = loRef.Scan (@laPages)
	This.AssertEquals ("C", Vartype (laPages[2]))
	This.AssertEquals (2, m.lnCount)

However, the Scan method will bring up several Windows WIA dialogs in the course of scanning and the test requires that there is a document feeder with two sheets of paper. So this isn't a test that I can run automatically whenever I commit a change in the app or whenever I build a release.

Currently I have to add the test to my list when I work on the program and remove it when I'm testing everything. That is a manual process and somewhat difficult to automate.

[eselje]

I wonder if there's a way to annotate a test as [Integration Test] (of course VFP doesn't have that but we could fake it somehow) so it knows not to run that test under certain conditions. Does the double assertion currently work properly? I'd think you'd only get the last result. Eric

…

On Tue, Jan 14, 2020 at 9:21 AM Christof Wollenhaupt < ***@***.***> wrote: It's a FoxUnit test that looks like this: Procedure Test_Scan Local loRef, laPages[1], lnCount loRef = NewObject("cUiWindowsImageAcquisition", ; "Klassen/cuiWindowsImageAcquisition.prg") lnCount = loRef.Scan ***@***.***) This.AssertEquals ("C", Vartype (laPages[2])) This.AssertEquals (2, m.lnCount) However, the Scan method will bring up several Windows WIA dialogs in the course of scanning and the test requires that there is a document feeder with two sheets of paper. So this isn't a test that I can run automatically whenever I commit a change in the app or whenever I build a release. Currently I have to add the test to my list when I work on the program and remove it when I'm testing everything. That is a manual process and somewhat difficult to automate. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#25?email_source=notifications&email_token=AB6LYTCCUOLBU3XBEHXME6TQ5XJ7DA5CNFSM4KGRNLR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEI47ZKY#issuecomment-574225579>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB6LYTHJE5SEKWJPCYISWATQ5XJ7DANCNFSM4KGRNLRQ> .

[cwollenhaupt]

Yes, the assertions work just fine. Annotation is what .NET would do, I agree. In VFP that might be the test method name. I don't know how everyone else is naming their tests, but mine are all named Test_something. The filter would work, if I name the other tests IntegrationTest_something. That would work for interactive tests with one extra step. I have to think about how to handle automated tests without the FXU UI.