• Legal
  • Gen Safe Deployment Practices

Gen Safe Deployment Practices

Overview

As a leading provider of cybersecurity solutions, Gen recognizes the critical importance of delivering timely and reliable updates to our customers. Our products are deeply integrated into our customers' environments, and it is essential that updates and code modifications are seamlessly integrated without disrupting their operations.

To achieve this balance between speed and stability, Gen has implemented robust quality processes that ensure the integrity of our product updates. These currently apply to Avast, AVG, and Norton. This excludes legacy Norton products which continue to follow a similarly rigorous process as we migrate endpoints to this new system.

Safe Deployment Phases

Development

The sections below cover some of the practices we follow to support safe development.

Code review

Code is reviewed for every pull request by peers before merging into the main branch. For critical components 2 reviews are required.

During the code review, constructive feedback is provided to improve code quality.

After code review, pre-build is executed, and only compilable code is merged to the main branch.

Unit testing

Every new component is required to be covered by unit tests. We aim for high code coverage for thorough testing of methods, conditions, and statements.

Static testing

Static testing is seamlessly integrated into our build pipeline, ensuring regular code analysis.

We conduct weekly static testing builds, prioritizing the immediate resolution of high/critical errors.

Kernel risk mitigation

While security software must operate in the kernel to protect users, Gen proactively looks for ways to limit kernel usage to essential components. Code is moved to user mode where possible to limit the potential for system crashes.

Driver Certification

All of our drivers undergo rigorous testing and certification through the Microsoft Hardware Lab Kit (HLK) framework. This rigorous process guarantees that our drivers meet the stringent standards set by Microsoft, ensuring optimal performance and reliability on Windows systems.

Driver updates are typically introduced during the first week of our four-week beta cycles, followed by a thorough monitoring period to ensure driver stability. We avoid deploying new drivers through virus definition updates.

Internal testing

Every product build undergoes a comprehensive internal testing process to ensure quality and stability before release. The follow sections list some of our testing stages.
Build-time testing

Functional smoke tests as a part of the standard build pipeline. Smoke test failures stop the build pipeline.

Functional testing

Automated functional tests verify that all features and functionalities work as intended. These test components at multiple levels of depth for comprehensive coverage.

Performance testing

Automated performance tests track build-over-build performance trends across various user scenarios including file copying and site load time.

Compatibility testing

Automated tests are run across multiple configurations to ensure compatibility with different hardware and operating systems. These include x86 and x64 configurations of Windows 7  Windows 11, along with ARM64 for Windows 11.

SUVP testing and other Windows pre-release testing is also performed.

Pre-release testing

We perform rigorous pre-release testing of our beta and public candidate builds on physical machines to replicate real-world usage scenarios. Test case completion and status is recorded on a per-release basis.

Public beta  

Our Beta Community offers a unique opportunity to actively participate in the development of Gen’s products and help make them even better.

Every beta and new release change is transparently communicated to our community, ensuring open dialogue and collaboration.

Beta cycles span four weeks. New features are introduced during the initial week, followed by three weeks of stabilization and testing. Over 10K users participate in a beta cycle.

Programs:

Staged rollout

Upon successfully completing the beta community phase, the latest version enters a staged rollout process. To ensure controlled deployment, automatic updates are initially disabled, requiring users to manually initiate the installation.

Following a comprehensive telemetry evaluation, the update is gradually released to a larger portion of the user base. Throughout this process, the new version is closely monitored to identify and address any potential issues, ensuring a seamless and controlled transition for all users.

Typical rollout timeline:

  • Release day: Only new installations, no update forced
  • Week 1: 25% user base migration starts
  • Week 2: 50% user base migration starts
  • Week 3: 100% user base migration forced

Monitoring

We maintain a robust monitoring system that continuously tracks key performance indicators and system behavior. This comprehensive monitoring enables us to proactively identify and address potential issues. By closely monitoring various aspects of our product's performance, we optimize its stability, reliability, and overall user satisfaction.

We collect crash dumps, product logs, performance traces, and product telemetry. Collected crash dumps are analyzed automatically with private symbols to group them into buckets, determining the problem’s prevalence based on the occurrence counts. The system provides trend lines to help spot raising issues.

In addition to error reporting, we also monitor feature engagement through release transitions to make sure the product is still working as intended even when other quality metrics aren’t showing issues.

We also carefully watch the number of contacts through our support channels, specifically separating technical issues from all other contact reasons.

Recovery mechanism

μUpdate channel

An independent mechanism, separate from the standard update process, enables the patching of user-mode binaries and kernel drivers. This update mechanism provides a robust recovery option for the product. Additionally, these updates can be applied during the installation phase, ensuring timely fixes.

Virus database forward rollback

Concurrently with new virus definition updates, a backup of the last known stable configuration is prepared. In the event of false positives or other critical issues, the backup virus definition database can be promptly released to users to mitigate potential widespread problems.

Communication Channels

Our brands all have dedicated support pages with live support contact details and community forums:

Avira SDK

This section outlines how the endpoint protection components of the Avira product and partner products (later referenced as EPP) are deployed to ensure high quality, low risk, and fast reaction times in case of an incident.

Introduction

EPP is a unique product, targeting Avira's own consumer product as well as enabling partners to build an endpoint security solution. EPP brings its own services and drivers which are installed and updated on the end user's system. Our Safe Deployment Practices (SDP) are designed to avoid compatibility issues and failures on the end users' machines. However, we also use it to monitor, detect and fix problems as fast and as early as possible. Very fast release cycles (release ready every day) are an essential aspect to achieve fast reaction times.

EPP's content can be grouped into four updateable groups:

  1. Services and driver files
  2. Remote definition files
  3. Security intelligence and detection logic files
  4. EPP libraries for Avira's own product and partner applications
  5. The deployment of the first three groups is part of EPP's update cycle. The fourth is at the responsibility of Avira / partner's deployment.

We provide two different approaches on how updates can be shipped to end users. Either the standard EPP update cycle is used or partners host their own update infrastructure, mirrored from Avira's EPP update servers. The latter is out of scope of this document as the partner controls that deployment cycle.

Service and driver update

Once the quality gates on EPP side have passed (blue) a step-by-step rollout (green) is conducted. Constant monitoring (yellow) on telemetry and partner feedback is taken as a metric if the next rollout step is possible. Each finding during this workflow is evaluated carefully and considered in the decision. Possible actions are:

  • Pause the rollout step: Indication of an issue detected, stop for further investigations
  • Start a new rollout with a fix: Preferred way of quickly fixing a non-fatal issue without the need of rolling back
  • Rollback to previous version: Very critical issue found that cannot be fixed with a new rollout

Rollout monitoring and decisions are made by a council of technical experts who are familiar with the code base so that sophisticated, fast conclusions can be drawn.

The partner vendor can decide if, and when, participation in the rollout takes place. That is done by controlling the mirroring timepoint from our update servers. This approach is recommended if their deployment practices differ too much from ours.

Remote definition updates

This update type is considered less risky and meant to be deployed and rolled out faster. The purpose is to configure or toggle product features fast, if required.

The deployment flow is very close to the one described in "Service and driver update". The differences are that Microsoft performance testing is skipped, and the different rollout phases are conducted much faster (Sometimes within the same day).

Security intelligence and detection logic updates

These update types are usually provided several times a day and are considered lightweight, low risk, and fast to deploy. The difference of this deployment compared to the one described in "Service and driver update" is that the rollout is much faster, and no partial rollout steps are available. That is required because of the fast update interval. Monitoring and emergency actions are available like "rollback" or "Start a new rollout".

Any undesired detection events that might occur through such an update can also be mitigated by the live connection to Avira's backend services.

EPP libraries for the partner application

As mentioned above, the shipping of these libraries is part of the partner vendor's deployment workflow. The files are packaged and available to our partners via nuget.org and the Avira OEM portal. We deploy new versions immediately once the full rollout of the EPP version is done.

Gen's Dedication to Excellence

Gen's layered quality process is designed to proactively identify and address issues before they impact our customers. While no system can prevent all failures, we’re committed to continuously improving this process to minimize any potential impact to our users.