Right on Bazzar. If it weren't for Eric Payne on our team, we would be in deep trouble. As an engineer, he is very methodical and has a format/checklist we developed to put things through their paces. His primary focus and strength is on the panels and gauges, but gives the overall aircraft a good workout. Tom and I work the FM, I work the models, mapping and textures with Damian, and we all look at the sounds with no strong expertise there.
You need fresh eyes with various expertise and backgrounds for the most part and that's why using testers outside the team works best. But, I agree, they must be willing to research data, pictures, read pilot reports, and after becoming fluent with the aircraft performance and nuances, be capable of flying by the numbers, and finding the boundaries. All of this is for nothing without a great test procedure and a formatted reporting structure. Beta testers need to understand what's expected of them, what to test, how to test, and how and when to report back. Without setting expectations, process, and proper reporting structure, things are half-done, reported poorly, and basically a waste of everyone's time.
So I would recommend that a Beta Test Package be assembled to include all the specs, pilot's reports, testing procedures, and the test results checklist for each aspect of the project (Exterior/Interior/panels/gauges/models/textures/sounds/documentation/checklists accuracy and useability/etc), with a rating system that has ratings for day and night, readability, realism, packaging ease of installation/use. etc. Developers have certainly had enough feedback to understand where their issues have been, repeated issues, project after project, typical gripes, etc. to build this on. Most testing should be done on the ground before the plane ever gets off the ground. Once that is done, then we can go flying as the final tests. With this structure, you can parse out the testing in phases and get most issues resolved before the full package beta is released.
You need fresh eyes with various expertise and backgrounds for the most part and that's why using testers outside the team works best. But, I agree, they must be willing to research data, pictures, read pilot reports, and after becoming fluent with the aircraft performance and nuances, be capable of flying by the numbers, and finding the boundaries. All of this is for nothing without a great test procedure and a formatted reporting structure. Beta testers need to understand what's expected of them, what to test, how to test, and how and when to report back. Without setting expectations, process, and proper reporting structure, things are half-done, reported poorly, and basically a waste of everyone's time.
So I would recommend that a Beta Test Package be assembled to include all the specs, pilot's reports, testing procedures, and the test results checklist for each aspect of the project (Exterior/Interior/panels/gauges/models/textures/sounds/documentation/checklists accuracy and useability/etc), with a rating system that has ratings for day and night, readability, realism, packaging ease of installation/use. etc. Developers have certainly had enough feedback to understand where their issues have been, repeated issues, project after project, typical gripes, etc. to build this on. Most testing should be done on the ground before the plane ever gets off the ground. Once that is done, then we can go flying as the final tests. With this structure, you can parse out the testing in phases and get most issues resolved before the full package beta is released.
