The #1 test that I can come up with is to find a zero-day exploit in their code (usually not very difficult with Magento extensions), report only the resulting damage of a mock exploit to their security team (giving no indication of which part of the code is vulnerable), and start the stop-watch -- because this is exactly what is going to happen when your site gets hacked. When their support staff asks for global FTP and mysql access, politely decline stating that it is in violation of the PCI-DSS and offer to let them have read-only access to your source code repository.
The #2 test that I perform is to call the vendor up and catch them off guard. Ask them what kind of behavioral / unit testing that they do, what source-control system they use, which versions of PHP they test on, which versions of Magento are tested, which web-servers are tested on, whether or not they use browser-stack for testing front-end components, etc... If the vendor does not know what you're talking about, goes silent or wants to "get an expert to email you back", run like hell because they most likely use numbered zip files for "version control" and only fix bugs 3 months after their customers report them.
Speaking of the PCI-DSS, all system modifications are also required to have a reversion strategy. With modules that add non-nullable columns to core tables, this becomes nigh impossible while still maintaining a reversion strategy that would pass an audit. Run like hell from any modules that will cause issues (read: SQL Errors) when disabled.
18.104.22.168 Back-out procedures.
Verify that back-out procedures are prepared for each sampled change.
For each change, there should be documented back-out procedures in case the change fails or adversely affects the security of an application or system, to allow the system to be restored back to its previous state
This, in addition to the other answers. IMO there should be a wall of shame for some of the dangerous crap that has been spawned on this platform.