When selecting the right cybersecurity vendor to protect their operations, small- and medium-sized businesses (SMBs) can lean on several third-party research organizations that analyze which cybersecurity products can best prevent, detect, and clean up various types of cyberattacks today.

But these tests can sometimes assume a level of end-user complexity—and funding and staffing—that the average SMB might lack. Without a full-time security team, or even a single full-time internal IT hire, an SMB could unwittingly purchase a cybersecurity product that, while effective, requires a level of expertise they simply do not have.

This is where one third-party research team, in particular, can help.

MRG-Effitas, which produces quarterly reports about cybersecurity products that publicly participate in evaluations, focuses its analyses on “real world” malware attacks and detection capabilities. Not only do the researchers test malware samples that are currently infecting endpoints across the world, but the researchers also stress the importance of simple, effective notifications that will help the average user respond to any detected cyberthreat.

“Simulating normal user behaviour means that we pay special attention to all alerts given by security applications,” wrote the researchers in their most recent quarterly report for their program, the “360° Assessment & Certification.”

The 360° Assessment & Certification combines several tests that are then grouped into four separate certifications. Based on how a cybersecurity product performed in certain tests, that product will either earn a certificate or not. This almost-binary representation of a product’s performance is simple and effective, and it can help to quickly inform an SMB about whether a certain product is right for their company.

At the core of the MRG-Effitas certification process—which tests how products respond to known exploits, ransomware, botnets, adware, and more—is the user.

“A pass is given only when alerts are straightforward, and clearly suggest that the malicious action should be blocked,” the report said. “With this in mind, it is very important to note that the best choice for an average user is to keep things as simple as possible and not to overwhelm them with cryptic pop-ups, alerts or questions.”

Testing and certification

The 360° Assessment & Certification by MRG-Effitas involves the following nine rounds of testing:

• In the Wild/Full Spectrum Test
• PUA/Adware Test
• Exploit/Fileless Test
• Real Botnet Test
• Banking Simulator Test
• Ransomware Simulator Test
• False Positive Ransomware Test
• False Positive Test
• Performance Test

Each test has a specific purpose, from testing how cybersecurity products respond to an end-user visiting a malicious URL that delivers malware, to the detection of non-malicious but meddlesome applications such as adware, to even testing how a product responds to live ransomware samples observed in real world applications, and to simulated ransomware samples developed by MRG-Effitas. Importantly, MRG-Effitas also tests the performance load of each cybersecurity product, analyzing how much time it takes to perform certain tasks on devices that have the cybersecurity product installed.

While MRG-Effitas performs testing in the above nine categories, it only awards certificates in four categories: The 360° Assessment, the 360° Exploit Degree, the 360° Online Banking Degree, and the 360° Ransomware Degree.

For the 360° Assessment, MRG-Effitas assigns two levels of certification—Level 1 and Level 2—depending on how successfully a cybersecurity product detected the cyberthreats that were launched at it during testing. A vendor only receives Level 1 certification if it detected all threats on “first exposure or via behaviour protection,” the report said, and it passed the Real Botnet Test.

The malware load used during the 360° Assessment is significant. In the most recent round, it involved 360 “In The Wild” samples that included: “20 trojans, 54 backdoors, 50 financial malware samples, 53 ransomware, 49 spyware, 84 malicious documents, [and] 50 malicious script files.”

Just four products publicly received a Level 1 certification in the recent 360° Assessment: Malwarebytes Endpoint Protection, Bitdefender Endpoint Security, Microsoft Windows Defender, and Symantec Endpoint Protection.

A similar test deploys 50 financial malware samples against the detection and protection capabilities of the cybersecurity products, along with simulated banking malware. Five products publicly received the 360° Online Banking Certification: Malwarebytes Endpoint Protection, Avira Antivirus Pro, Bitdefender Endpoint Security, ESET Endpoint Security, and Symantec Endpoint Protection.

Ransomware simulations

In just the past decade, ransomware has evolved tremendously. Developers of the infamous family of malware have gone from asking for measly sums of money from individuals to creating entire business models in which they license out their ransomware tool to other threat actors. When those threat actors successfully hit a business—which they could have purchased access to from other threat actors—the original ransomware developers take a cut of whatever eventual payment is made. To make matters worse, threat actors have also begun deploying ransomware that not only encrypts a company’s files, but it also first exfiltrates any sensitive data, which the threat actors then use as a second point of leverage: Pay up or your data will be published for everyone to see.

The researchers at MRG-Effitas, recognizing this rapid pace of ransomware evolution, have, for years, tested cybersecurity products against ransomware samples developed in-house that could represent where ransomware development is headed in just months or years.

In the most recent 360° Assessment & Certification, MRG-Effitas deployed 53 ransomware samples against the cybersecurity products, and an additional four simulated ransomware samples. To achieve the 360° Ransomware Certification, a product must have protected a device from the 53 ransomware samples and 4 simulated ransomware simulated samples, and it must have passed the false positive ransomware test.

In the most recent round of testing, all nine publicly-evaluated cybersecurity products achieved ransomware certification.

Performance

Understanding whether a cybersecurity product works well is, obviously, important. But of similar importance to SMBs is understanding what impact a cybersecurity product will have on a suite of endpoints. Without large budgets that could allow for constantly refreshed, new devices to be purchased, SMBs should consider how much a cybersecurity product could slow down their organizations’ devices.

Thankfully, MRG-Effitas analyzes cybersecurity products based on their impact on performing simple operations, like downloading a file, opening a Microsoft Office program, or opening a website. The analysis also measures the time spent performing a security software update and the CPU usage during the update process.

Unlike the certificates offered by MRG-Effitas for other categories, there is no certificate or “pass/fail” result when testing performance. Instead, SMBs can look at the performance measurements for each product in the latest 360° Assessment & Certification.

Less “interpretation,” quicker answers

The simplicity of MRG-Effitas’ 360° Assessment & Certification gives SMBs a quick guide into what cybersecurity products could be the right fit for them. Without having to dive into countless interpretive reports from each cybersecurity vendor, SMBs can instead look at the most recent 360° Assessment & Certification and ask themselves: Which of these products received certification and which did not?

Knowing that MRG-Effitas hews its testing ideology to the user—only offering certifications for products that clearly notify and warn users about how to respond to a threat—SMBs can be sure that whatever tool they choose will, at the very least, be easy to use on their end.