×

INDI Library v2.0.6 is Released (02 Feb 2024)

Bi-monthly release with minor bug fixes and improvements

INDI Testing Framework

  • Posts: 193
  • Thank you received: 46

Replied by Gerry Rozema on topic INDI Testing Framework


This shows one of the issues / problems having a conversation by typing into web browsers, sometimes what one feels is clear because 'its obvious to me' the point one is trying to get across, doesn't come across on the other end the way it was intended, results in misunderstanding, often aggrivated by folks typing in a hurry.

This is the point I was originally trying to get across. With the drivers that include vendor provided binaries, it's very difficult, and beyond the scope of most folks involved in this project, to create a test jig that includes the vendor provided binary code and introduces the test points _under_ that layer. for the most part, in those cases it's done by 'hooking up the hardware and see what it does'. But all of the layers above that, are prime candidates for testing setups. This was why I originally started writing device simulators when I first tackled updates to the indi project. At least with the simulators, we have what should be a 'known good' piece of code pretending to react like a physical hardware device, and allows for end to end testing of the system from client thru to indi server and down thru base classes, but excluding the various hardware specific layers. Ie, using a camera simulator with a correctly install gsc catalog, one can test code in a client application, and that code will receive data that includes correct star fields, including mount periodic error if so configured, etc. In essence, the simulator devices _are_ the unit test jigs for all of the layers above that. Maybe not yet fleshed out as far as you are suggesting, but they are the basic framework in this case.

To do a proper job of setting up a test environment for indi, yes, it's feasible to introduce the unit testing concepts for _most_ of the code beginning with client connection and following the data paths on thru to devices, but some of the devices themselves are only suitable for black box style of testing where inputs are provided, and one looks to the outputs to see what happens because we dont have access to the internals of those components. Using the camera examples above as such candidates, I guess I _could_ spend an inordinate amount of time on a libusb hook set to test inputs and outputs for one of those cameras, but I have niether the time nor inclination to take on that magnitude of a project, I'd rather spend what limited time I have for astro software on client analysis stuff.

As mentioned earlier in the thread, I have another new driver set on the go here, this one for a commercially produced dome that is new to the market, manufactured by a company not far from here. I have a unique opportunity with this one in that not only do I control the code in the driver, I control the firmware in the dome, and it WAS written in a manner as you describe, test jig on the bench and each and every interaction with hardware had code written specifically to test that function, then the final all up integration for the firmware includes individual functional tests that can be triggered from the host. But I'll be the first to admit, the test set is not as rigorous as the ones I used to write when I was working in the DO-178B style of development environment, there is no need for that level of rigor in this case. After we had the hardware all working correctly the next step was to build environment specific drivers, one of which was an ascom driver for windows, the other an indi driver. The other unique aspect of this project, all of the drivers AND the firmware for the device are destined to sit on a github account, so it'll all be open source. This is a vendor that 'gets it' in this respect, they are very clear on one aspect of this project, the company expertise is in plastics and mechanicals, they dont view the software etc as 'secret sauce' and see tremendous value in having an implementation that hobby folks can tinker with. I ended up with this project because I wanted a couple of domes, and when I contacted them regarding the state of development for automation back in January, I was asked if I'd be interested in doing a fully open solution for them.

In the process of doing this project, one thing I came across is the ascom conform test suite, far from perfect, it attempts to reach some form of testing. It appears to be a client program that will exercise a device testing for inputs and expected outputs. It is essentially a black box form of testing. I think indi could benefit greatly from something along those lines as a starting point for automated tests. what I would envision for that, is something along this line.

a- Client program starts, then spawns an indi server with a pipe for driver startup
b- Walk thru the entire list of drivers, telling the server to load, then unload each one. This will catch any number of errors, even without physical hardware in place.
c- Test the indi server components by handing the server a list of simulator devices to load and then connect to each of them
d- Walk the simulators thru a full set of tests on the 'required' entry points

That would be a starting point that gives a full reference set, with the side effect of stressing the indi layers in the process. Once that starting point is in place, then the environment can be expanded to start testing individual drivers for function. Some of the drivers can only be fully tested with hardware in place, but others could be outfitted with test points. Again, using the dome example, I'll pick on Maxdome in this case. It's a proprietary binary protocol partially reverse engineered. Yes, it's possible to write an underlying test jig for it, but, there are a very limited number of folks with that dome anyways, and since the person who wrote the driver has one to test against physical hardare, and is today probably the _only_ person using that driver, I'm thinking the time/effort spent on a test jig for it is better spent on properly outfitting the basic indilib environment with better testing / functionality. It's simply a case of allocating limited resources to the spots where they give the best bang for the effort.

The way the ascom folks do it, when I run the conform tests on the dome driver, final output is a test sheet showing pass/fail on each of the tests, along with a hash for the driver that was tested. When I submit the driver for inclusion in the downloads, it must contain the conform report and the hash in that report must match the driver submitted. That's a workable way for folks dealing with binary only distrbutions, but, doesn't fit well with a source distribution.

BUT, there is a way we could solve some of the issues that originally triggering this thread, and with a little more thought, we could fully automate the process. I'm envisioning a 3 tree setup similar to the way debian does it.

Unstable - code builds in the test environment, drivers load - no functional tests completed (no hardware present for testing), ie it has passed the first set of automated tests in terms of building and loading.

Testing - code builds in the test environment, drivers load and function correctly against test jigs for those that have test jigs.

Stable - code builds in the test environment, and has been tested against physical hardware at a location that has the hardware.

We have recently moved to git for hosting the indi project, so, my first thought is, each of these trees gets packages tagged with a specific get revision. So using the example of a driver where I am the maintainer, and I have the physical hardware.

1 - automated build initially populates the unstable tree with a package that successfully built and loaded. this happens on every run of the automated build system.
2 - For a driver with test jig points included, it will bump up to the testing tree after an automated run passes the test jig setup. this probably doesn't happen as often, and, each driver bumping up to this level is tagged with the git revision used to build / test.
3 - Driver bumps up to the stable tree when it's been built, and tested against physical hardware, and a test report is generated that includes the git revision tag at which it was built.

For this setup, somebody with hardware in the field would keep the repositories pointed at 'stable' to ensure an apt-get update doesn't bring in broken / untested stuff.

Within git, we can run 3 separate branches, trunk, testing and stable, then as various pieces pass various parts of testing, that code can be merged from trunk to testing and later from testing into stable.

And yes, this is a very ambitious addition to the project as a whole, but, it would vault us up into a much higher level of professionalism in the development process. The real question then becomes, does anybody have the time / inclination to follow thru, it's a big task. but there is a fairly strait forward direction to accomplish it incrementally, first we get core indilib set up for automated testing with just simulator drivers under test. The next phase would be to start tackling drivers on an individual bases, define how each could be rigged for automated tests, then implement.

I think one major outcome from all of this, with an automated set of client tests that generate reports from start to finish we make the tasks for folks writing client software much easier, because we introduce a consistent set of expected responses. As an example, today, different drivers of the same class can and do present significantly different responses at times, mostly related to timing, order of events, and some of the state conditions. A lot of that would start to 'go away' if we have a definitive set of tests that get run, and they trigger all of these conditions.

I offer one example from my experience doing the ascom dome driver. There is a status for 'slewing', and my original interpretation was, that meant the dome was turning. But when running the conform tests, I found an error, both in my interpretation of the state, and in the way conform tests it. Turns out, the test really only considers dome 'open or closed', but this one has the ability to control shutter altitude as well. To make timing and order of events issues go away, I had to include either dome rotation, or shutter moving in the 'slewing' state. Without doing that, the test program would issue a movement command to the shutter, and immediately proceed as 'completed' even tho the shutter was still in motion, then it issued the next shutter movement command. Conform would test that shutter was open, and test slewing, and expect the shutter to be in it's final state when open was true and slewing was false. it never once queried shutter altitude to see if shutter was indeed at the requested altitude.

My guess is, if we start doing that kind of automated testing against indi drivers, we will be amazed at how much inconsistency there is from driver to driver in terms of how some of these states are returned during different phases of operation.
The following user(s) said Thank You: Jasem Mutlaq, Andy Kirkham
7 years 9 months ago #8704
The topic has been locked.
  • Posts: 24
  • Thank you received: 6

Replied by Andy Kirkham on topic INDI Testing Framework

Will respond later, need to walk my dog. Great write up though and thanks for sharing. We are getting closer to a realisation of direction :) Please remember, I'm new so much of the experience you are sharing is all new to me. I am coming from a different angle but I believe we are on the same page.
7 years 9 months ago #8708
The topic has been locked.
  • Posts: 24
  • Thank you received: 6

Replied by Andy Kirkham on topic INDI Testing Framework


The point about Unit Testing is you exclude the vendor provided binary. Once again, we are testing your code, not the binary.

For example, a binary has an exported function MoveTo(double RA, double DEC); Now, your driver at some point in it's life (in some function you write) will call that driver function. What we are testing is your driver makes that call with the correct RA and DEC arguments as expected given the inputs and the state. We do not call the binary at all, we don't even load it.

It's not impossible for most people to create what I am talking about. It's actually much more simple than you might think. But yes, if you have never seen this before it looks alien first time you see it. But that's just learning, we all learn as we move along.

Using simulators can test the clients, it can test the base class and the derived indidriver class makes the correct callback but it cannot test your driver. For example:-

indi/libindi/drivers/dome
  • baader_dome.cpp
  • dome_simulator.cpp
  • roll_off.cpp
In your model we test indi Core, Indi derived driver subclass and we finally test dome_simulator.cpp
If we were to run code coverage analysis which two files pop out from that list as untested? That's right the two we actually want to test and the one we cover we don't care about because it's only a simulator that will never drive anything other than a client integration test.

That's the difference here, you describe system wide integration tests (and not actually testing the driver unless you own the hardware). So in your world this type of testing falls short because the code you wrote in those drivers is not excerised by your test suite.

In order to test those missing "real" drivers you need to introduce two things, dependency injection which pulls in the second thing, mockable abstraction. You stated earlier these "test jigs" are "very hard and beyond most folk". They are not once you understand the concept and see it in action, it's actually pretty simple.

Simulators are only useful for those without the hardware to "get going" until the hardware they ordered arrives or for client developers who don't have the hardware. For driver writers they are useless for automated testing because we don't want to test the code in the simulator, we want to test your real code in your real driver.

You may believe that that code is really complex and hard to test. I disagree. Yes, the driver logic may well be complex itself. But when it calls IDSet*() in core that just spits out "something" to stdout. That's all you need, the ability to capture that output so you can analyse it against an expectation of what should be being sent to stdout for a given test.

Not so, I know for a fact I can mock both Indi's hid_device AND libusb in an evening. Job done, and done once, reuse MANY times for all test. The key that takes the time is refactoring your to use the dependency injection in the drivers and then retrospectively designing suitable tests for legacy code. It's those tests for the legacy existing code that will take time for sure. But what's the rush? Introducing the DI won't break anything, the driver will still function perfectly without a single test. But it doesn't have to be done in one go. Do it one driver at a time and one driver API function at a time.

This is what I am advocating. The difference is rather than one big huge client that tests the entire world you break it up into unit tests that exercise API functions. So here is my question. You say "testing for inputs and outputs". That's what I am saying. You control teh inputs, that's easy. Exactly how are you going to get those outputs? In my world that's where the DI and mocks come into play. It makes it simple. But essentially we are talking about the same functionality here, just now discussing how to actually do it. Just trying to redirect stdout will only take you so far. Mocks provide the ability to return values which in other ways you would need to try and use stdin. stdin/stdout redirection is way to messy. Mocks are perfect, they were designed for it, specifically Google Mocks in Google's GTEST suite.

That's exactly what Google Test will do for a pass. For a fail however, you get a ton of extra information about the nature of the failure. Not just a "fail, game over, you lose" which is pretty much all integration tests can tell you.

This is pretty much standard in the open source world, it's normally call, in git dev, "release tags" (stable), master branch (pre-stable), develop (unstable), other branches are down to developers and they should merge to develop when ready.

It is ambitious yes. But we already made a start with this discussion. There are two types of testing:-

1. Unit tests. These run on every commit of code, they are automated and hold your contract to account for APIs.
2. Regression tests. These normally happen when you form a new branch from master into a release branch. Once you have a release branch you regression test it and run any automated integration tests you have (usually requires some human input). You may get bug fixes here but they have to be very minor. Major bugs cause release process abort as you are not ready. Once you release, you tag and then merge back to master and the branch can be deleted. New cycle starts.

I hope this is understandable. The key point is to test as much code as is possible. I think simulators are only useful for client and integration tests. But they don't reach the real complex driver code base at all. That requires DI and mocking.
7 years 9 months ago #8714
The topic has been locked.
  • Posts: 193
  • Thank you received: 46

Replied by Gerry Rozema on topic INDI Testing Framework


Actually no, that's the really easy part. But, since we know NOTHING of the propietary protocol happening over that usb connection, once you have the connection hooked, the hard part is just beginning. Now we have to figure out what kind of responses are expected by the vendor provided binary on the usb link. Without generating the correct responses in the libusb hooks, we still haven't tested it. To generate the correct responses from the libusb underlying test setup involves reverse engineering the entire camera protocols. If we did that, then I'd take it one step farther, throw away the vendor provided binary completely and write our own code to replace it, we now know how to talk to the device anyways.

In fact, I have considered attempting this for a couple of gadgets, but it just becomes a cost / benefit equation. the task of reverse engineering the protocols for a couple of the cameras we have here is bordering on monumental, but could be done with appropriate hooks in libusb. I know if i set my mind to it, and was willing to dedicate enough time to the project, I could figure out all the pieces of how to talk to the ST-10XME camera I have sitting here, along with it's filter wheel. But then again, I could skip all that by simply selling this camera, and buying a QSI camera with the same sensor. It'll cost me a few thousand dollars, and very little time / effort in that case.

If I went the other way, fully reverse engineered the SBIG camera, assuming I eventually got all the details correct, I would have the same capability, less cost in dollars, far more cost in time. There would be an added benefit, more folks could use the stable driver, and it would be 'fixable'. But a side effect of that, which IMHO is a seriously BAD side effect, then the vendor will potentially sell more cameras because they work well, and I'm not sure I want to put that kind of effort into supporting an open source hostile vendor. In fact, I dont want to do that. In all honesty, I'd rather spend the money to support an open source friendly vendor, and let the open source hostile vendors sink or swim on their own efforts, or lack thereof.

The maxdome driver was done exactly this way, but it's an easy example since it's a serial connection with a very limited set of functions to figure out. Just hook a null modem cable up to a windows computer, other end too the linux computer, and then the cable out to the dome. Put a program in place that echo's data between the two serial ports on the linux box, and logs data on the way thru. In this case, it's a high value target, because the owner already has the dome in place, and it's not practical to swap it out for one that has good open source support, so we end up with a high value target accomplished with a relatively small reverse engineering effort. And now that the protocol is basically understood, it's kind of trivial to write something that would respond correctly thru a series of hooks to test the maxdome driver, I have done exactly that over the last few months, but using a null modem cable instead of software hooks.

But lets look now at the extreme opposite end of the spectrum. A camera with integrated filter wheel, cooling, shutter, etc. it's all driven by proprietary protocols on the usb link, for which we know virtually nothing about the protocols. The value proposition is reversed, a camera is an easy thing to swap out, for far less expense than the dome would be, yet the reverse engineering component of a camera project is two orders of magnitude larger than it was for the dome. it's essentially a comparatively low value target, with a large expense in terms of time and effort to support properly in this case.

The techie in me says 'ya, a real meaty project, i wanna do that', but the businessman in me says, stop the bus, lets not go down this road. The end result of this path is a well supported camera which will only increase sales for a vendor that's openly hostile to open source solutions, at the expense of sales for a vendor that embraces the open source solutions. I frankly dont want to invest my time and effort into that end goal, and would discourage the project as a whole from going down that road. there are vendors out there that provide specifications and/or code, SX FLI and QSI come to mind. Why should we put a huge effort into fixing problems with SBIG and/or QHY cameras when there is already a viable alternative that doesn't require the huge effort ?

Another big gripe of mine regarding the open source hostile vendor mentality, they provide pre-compiled binaries for specific architectures. I have a couple of very capable boards here that I would prefer to use in the observatory over x86 or arm based stuff. One is based on a mips processor, another based on the octeon processor. to use those boards in the observatory excludes the use of some of the camera hardware we have, as there are no binaries for those processors. And in the long run, even those available are married to specific glibc versions, so everything is going to break when upstream linux distributions switch out the libraries.

So this is where part of my philosophy on all this comes from. If indi ends up with an automated testing environment, I believe those tests should terminate at the entry point to vendor proprietary stuff, and if a vendor wants to fix and/or make it work correctly, fill your boots. But I believe building up tests to go beyond that point and fix issues black box style that happen after the transition from indi to vendor code is beyond what we should, or even want to do. this is partly philisophical on my part, I dont think we should expend ANY effort into fixing broken vendor code if it's not released and open. If a test environment is built, it should test the data paths up to those entry points, then let the vendors deal with it beyond that. BUT, if we have a good / repeatable set of tests that will run up to that point, maybe some of the problematic ones will actually get fixed, because now the developers there who are very unfamiliar with the environment will have a tool to run which tells them things work as expected, or dont, something more detailed than 'hook up the camera, take a picture, call it done'.
7 years 9 months ago #8716
The topic has been locked.
  • Posts: 24
  • Thank you received: 6

Replied by Andy Kirkham on topic INDI Testing Framework

Gerry

Why do you keep bring up the vendor binary? I keep saying we shouldn't even be loading their binary let alone testing it!

What I am saying is THE CODE YOU WRITE should be tested.

If your driver needs to do some math to move a scope from point A on the sky to point B on the sky I want to know your math is correct and the result of your calculation is a slew command.

Given point A and point B I EXPECT your driver to issue COMMAND foo{...} to the scope. Test complete. That's it.

Now, there maybe bugs in the driver that mean it does something un-expected and you need to adjust the command that is sent to compensate for it. Now that's all this reverse engineering you keep going on about. Personally I would write to the manufacture to get the bug fixed upstream. Failure to fix means no further development and a big public notice on the Indi Website telling people to avoid the buggy product.

Whatever "punishment is deemed fit", that's just an example. So people may prefer the hard life of fixing these buggy vendor binaries, that's entirely down to you. I personally probably wouldn't, I would just do my homework and buy something recommended and known to work in the first place.

When it comes to unit testing I just want to know you wrote the code to the specification you say you are working to, even if that's one adjusted for the failings of manufacturers.

I am in no way advocating fixing commercial stuff in open software, quite the opposite in fact, name and shame them into compliance would be my moto.

What I do expect is OUR CODE to work as we expect. I don't care about vendor binaries. If you do, carry on sweating over it by all means. It's your time.

Unit testing is all about OUR CODE. No one elses.

I really don't understand why you keep ing this issue about vendor binaries at all. They are totally irrelevant to unit testing our own code. Imagine for one second if I manufactured domes and I implemented "DROP TROUSERS" as the serial command to open the roof what I want your driver to emit is that text. All I have to test is the driver emits that. I don't have to stand there waiting to see if the roof opens. I know for a fact my driver is correct and compliant.

If however my driver does nothing despite I said it would open for that command I fully expect an email from you complaining. No response from me gets talked about in social media, in forums, endlessly visible for decades to come to a wealth of future buyers. My reputation plummets and I sell nothing.

But so long as your driver emits the command correctly your unit test will pass. YOU and your code are compliant. That's what unit testing is all about.

This constant talk about having to reverse engineer broken drivers is totally off topic and nothing to do with testing our own code. If they are that bad just don't support them.
7 years 9 months ago #8717
The topic has been locked.

Replied by Jasem Mutlaq on topic INDI Testing Framework

Okay folks, I think everyone made their point. So let's move forward and get some initial testing framework developed.
7 years 9 months ago #8719
The topic has been locked.
Time to create page: 0.421 seconds