MECHANICSBURG, Pa. – A new report undertaken by ICSA Labs to mark its’ twentieth anniversary contains some dispiriting news for security vendors, and some alarming news for customers. Nearly 80 percent of security products fail to perform as intended when first tested and generally require two or more cycles of testing before achieving certification.
The number one reason why a product fails during initial testing is that it doesn’t adequately perform as intended. Across seven product categories, core product functionality accounted for 78 percent of initial test failures. This is a huge problem for a couple of reasons. First, problems with core functionality means the product basically does not work properly. And second, many of these products that failed went to market anyway.
“Vendors are under a lot of pressure out there,” said George Japak, managing director, ICSA Labs and a co-author of the report. “They need to get the new versions out with improved functionality, and some of the things that take a back seat are quality control. Instead they attempt to do fixes after they get to the market.”
Second of the list of failure reasons was a product not completely and accurately logging data. Incomplete or inaccurate logging of who did what and when accounted for 58 percent of initial failures.
“Historically vendors have not placed a lot of attention on logging,” Japak said. “After the introduction of compliance laws, we have seen improvement, but there are still gaps. It can be little things like time-date stamping for logs.” The report found that logging is often considered a nuisance and undervalued. Almost every network firewall (97 percent) or Web application firewall (80 percent) tested has experienced at least one logging problem.
The third problem was the startling finding that 44 percent of security products had inherent security problems, from vulnerabilities that compromise the confidentiality or integrity of the system to problems in scripting and random behavior that affects product availability.
“Security vendors often leave their guard down to security around their own device,” Japak said, noting the irony with this problem.
Patching is another issue, notably the failure of some vendors to roll patches forward in updated versions, something one would think would be routinely done, but is not.
“We constantly have to be diligent for vendors making fixes and patches publicly available,” Japak said. “We see numerous occasions in updates where the fixes and patches don’t get rolled forward.” Approximately 20 percent of products also struggle to accept updates correctly.
It is rare that a product earns certifications on its first round of testing – only 4 percent of products tested attained certification during the first ICSA Labs’ testing cycle. 82 percent of products resubmitted for testing eventually earn ICSA Labs certification. Vendors do not necessarily wait for that certification before going to market however.
“Vendors will bring products to market one thousand percent of the time, even if they fail,” Japak said.
Once a vendor earns certification, products are required to undergo ongoing testing to maintain certification. Japak said it’s common for a product to pass, and then fail to retain later because the vendor didn’t properly maintain the product.
“Enterprises need to make sure that products retain their certification,” he said. “There are lots of problems with legacy products, because it’s often not cost-effective for the vendor to keep them staying up – especially since vendors want customers to migrate to new products anyway.”
Part of the problem is getting the customer to demand better quality from vendors, Japak said.
“You get these CIOs who are fixed on the HIPAA box, whose only concern is getting it checked off and not being as concerned with how well the product works,” he said. “They are less interested in the actual security, or take it for granted that the security is there. They are much more concerned about saying regulatory compliance is met. But these products have to meet core functionality, and too often, they don’t.”
This column was written by Mark Cox of ConnectIT, an IntegratedMarCompany
a>>
