Author at Avast Engineering

Our team invests countless hours in the exploration, discovery and investigation of emerging trends such as AI and futuristic technology. Stay informed about the evolving landscape of cyberthreats through our diligent efforts and insights.

Security Evangelist at Gen

Threat Operations Analyst at Gen

Research

A fake invoice in your email is easy to ignore. A fake invoice inside your order history feels different.&nbsp;Norton customers have reported fake Norton invoices appearing inside the Shop app, the shopping and order-tracking app from Shopify. Public reports suggest the same technique is not limited to Norton. Similar suspicious Shop app notifications have used McAfee, Apple gift cards, iPhones, PayPal-style payment claims and other high-value purchases as bait. The impersonated brand may change, but the mechanics are familiar: make the user believe they have been charged, then give them a phone number to call.&nbsp;That phone number is where the scam really begins.&nbsp;<h4>Why the Shop app makes this more believable&nbsp;</h4>Shop is a legitimate app. People use it to track packages, check receipts and receive order updates. According to Shop’s own documentation, the app can automatically track orders from several sources, including Gmail, Outlook, the email address linked to the user’s Shop account and orders paid through Shop Pay. If Gmail or Outlook is connected, Shop says it scans recent emails for keywords such as “tracking number” and “track your package” to populate deliveries in the Orders tab.&nbsp;

Shop also supports push notifications for delivery updates, payments and favorite products. So the first alert the victim sees may not be an email, an SMS or a calendar invite. It may be a notification from an app they already use for real purchases.&nbsp;That context matters. The scam text is not especially polished. Some of the examples we reviewed contain awkward wording and obvious signs of fraud. But when the message appears inside an order-tracking app, surrounded by real receipts and shipping updates, users may give it more credibility than it deserves.&nbsp;The screenshots reported to us show fake orders under generic seller names such as “My Store.” The receipts claim that a Norton or Norton LifeLock subscription has been activated or renewed, usually for several hundred dollars. The phone number appears in the product description, the receipt body or even the shipping address field.&nbsp;That is not how a normal receipt should look.&nbsp;<h4>The brand is interchangeable&nbsp;</h4>Norton is the example we saw through customer reports. Norton support now warns that scammers may abuse Android and iOS shopping platforms or applications to generate fake order notifications that appear to be legitimate Norton purchases. Those notifications may include fake order confirmations, fake charges, trusted third-party app delivery and fake support phone numbers embedded in order details.&nbsp;Public user reports point to a wider pattern. One Reddit user described receiving a Shop.app message about a transaction they did not recognize, with no follow-up email confirmation, and a message telling them to call support if the order was not placed by them. Another thread includes reports of unrecognized iPhone purchases under “My Store,” suspicious contact emails and users saying they saw no matching charges on their bank cards. Other reports mention fictitious Apple gift card purchases and iPhone orders using the same kind of wording: “If Order Not Placed By You,” followed by a phone number.&nbsp;The choice of brand is part of the social engineering. A fake security subscription charge plays on two fears at once: being billed for something you did not buy, and being left unprotected if you ignore it. Apple products and gift cards create high-value purchase panic. PayPal-style wording makes users worry about payment account abuse. In each case, the fake order is there to push the victim toward a call.&nbsp;<h4>What we know, and what we do not&nbsp;</h4>We have not found any evidence that any of the impersonated brands have been compromised.&nbsp;&nbsp;We also should not claim that Shop or Shopify was breached based on the evidence we have. The exact abuse path still needs confirmation. The fake order may be entering the app through a merchant/order workflow, through email parsing, through account association, or through another legitimate mechanism that scammers have found a way to misuse. Shop’s own documentation shows that orders can populate from multiple sources, so assuming one route would be premature.&nbsp;&nbsp;What we can say is narrower and stronger: scammers are placing fake purchase claims inside shopping app order experiences, where users are used to seeing real receipts and order updates. Norton’s own support page now describes this class of shopping app scam, and public reports show similar lures using other brands and products.&nbsp;&nbsp;That is enough to warn users without overstating the technical evidence.&nbsp;<h4>How the scam works&nbsp;</h4>The user receives a notification or sees a new order in the Shop app. The order looks like a receipt for something expensive: a security subscription, an Apple gift card, an iPhone, a MacBook or another high-value purchase.

The receipt says the charge has been processed, the subscription has renewed, or the order is being prepared. Somewhere in the order details, the scammer inserts a phone number, email address or support instruction.&nbsp;The wording is often clumsy. We have seen phrases like “If Order Not Place By You” and “If Need Help.” Those errors may look obvious when reviewing a screenshot. They are easier to miss when a phone notification tells you that hundreds of dollars have just been charged.&nbsp;If the victim calls, the scam moves off-platform. The person answering may claim to be billing support, Norton support, PayPal support or a cancellation department. The script can vary, but the request usually moves toward credentials, payment data, one-time codes or remote access software.&nbsp;By that point, the fake receipt has done its job. It has created enough urgency for the victim to enter a conversation controlled by the scammer.&nbsp;<h4>Why this is harder for users to judge&nbsp;</h4>Most people understand that email can be spam. They may still fall for phishing, but at least the inbox is a place where scams are expected.&nbsp;Order-tracking apps are different. Their purpose is to collect receipts, shipping updates and purchase information in one place. A notification from that environment can feel as if it has already passed through some layer of trust, even when the content inside the order is fraudulent.&nbsp;We have seen the same logic with calendar invite scams. The message itself is not always convincing, but the delivery channel changes how people read it. A fake invoice in an inbox is one thing. A fake invoice inside a calendar reminder or shopping app receipt can feel closer to a real event.&nbsp;For security products, this also creates a detection problem. The first visible alert may not be an email, SMS or malicious website. It may be a legitimate app notification carrying fraudulent text. The scam still depends on social engineering, but the delivery surface is harder to classify with the usual rules.&nbsp;<h4>What to do if you see one&nbsp;</h4>Do not call the number in the receipt.&nbsp;Check the charge directly. Open your bank or credit card app. Open the official account for the brand being impersonated. If it claims to be Norton, go to your Norton account or official Norton support. If it claims to involve PayPal, open PayPal directly. Do not use the phone number, email address or link inside the suspicious order.&nbsp;If there is no matching charge, treat the receipt as a scam attempt.&nbsp;Report the suspicious store or message in the Shop app if the option is available. Shop’s help pages say suspicious Shop stores can be reported from the store page through the app, and Shopify also tells users to forward phishing messages to its phishing inbox.&nbsp;&nbsp;Norton users can forward suspicious messages to Norton and should verify unexpected charges only through their Norton account or official Norton support channels.&nbsp;&nbsp;Opening the receipt does not automatically mean your device was compromised. The risk increases if you call the number, share information, install software or follow instructions from the person on the phone.&nbsp;<h4>What to do if you have already called&nbsp;</h4>End the call.&nbsp;If you shared payment details, contact your bank or card provider immediately using the number on your card or inside your banking app. If you gave away a password, change it from a different device if possible. Start with your email account, Apple ID or Google account, banking account and any account connected to the fake invoice.&nbsp;If you installed remote access software, disconnect the device from the internet and remove the software. Run a security scan before using the device again for banking, email or shopping.&nbsp;If you gave a one-time code to the caller, treat the related account as compromised. Change the password, review recovery options and check recent login activity.&nbsp;<h4>The lesson&nbsp;</h4>Scammers keep testing new places to put old tricks. For years, fake invoices were mostly an email problem. Then similar subscription notices appeared in calendar invites. Now we are seeing fake receipts inside shopping app order histories and push notification flows.&nbsp;The safest response is to separate the alert from the action. If an app, email, text message or calendar invite says you have been charged, do not use the contact details inside that alert. Go directly to the account or payment provider yourself. Check whether the charge exists. Use official support channels.&nbsp;The fake receipt, the fake support number and the fake emergency are all part of the same controlled environment. The moment you verify the charge somewhere else, the scam loses most of its power.&nbsp;

Scammers are using order-tracking apps to place fake receipts where users expect to see real purchases, then pushing them to call fake support numbers. 

Fake invoices are moving from inboxes to shopping apps 

Threat Researcher at Gen

<h2>Introduction&nbsp;</h2>Infostealers are constantly evolving, and so are the techniques they use to bypass <a href="https://security.googleblog.com/2024/07/improving-security-of-chrome-cookies-on.html" target="_blank" rel="noreferrer noopener">Application-Bound Encryption (ABE)</a>. In recent weeks, Vidar has been among the most actively developed stealers and, apart from <a href="https://x.com/GenThreatLabs/status/2064836539952361976" target="_blank" rel="noreferrer noopener">multiple updates to its string obfuscation</a> and <a href="https://x.com/GenThreatLabs/status/2061462524604649840" target="_blank" rel="noreferrer noopener">a reworked approach to protecting its configuration</a>, it has also introduced a novel technique for bypassing ABE. And while there have been many other changes in Vidar lately, with new versions dropping every week, in this blog post we focus solely on the ABE bypass and its technical aspects.&nbsp; &nbsp;<h2>The ABE Bypass&nbsp;</h2>From a high-level perspective, Vidar’s approach to bypassing ABE is somewhat similar to that of Remus/Lumma, in that both try to extract the <code>v20_master_key</code> directly from the browser’s memory. What sets them apart, however, is how they achieve it. Since we’ve already covered the inner workings of ABE in earlier posts (<a href="https://www.gendigital.com/blog/insights/research/remus-64bit-variant-of-lumma-stealer" target="_blank" rel="noreferrer noopener">Remus</a>, <a href="https://www.gendigital.com/blog/insights/research/voidstealer-abe-bypass" target="_blank" rel="noreferrer noopener">VoidStealer</a>), we don’t revisit them here and focus instead on the bypass itself. For readers interested in the broader background, we encourage checking those earlier posts.&nbsp;&nbsp;In short, the end goal of infostealers is to obtain the <code>v20_master_key</code>, as it alone is sufficient for decrypting any ABE-protected data tied to a specific application. This key can be obtained in one of two ways: either from disk, where it is protected by two layers of <code>CryptProtectData</code> (one applied in the context of the logged-in user and one as <code>NT AUTHORITY\SYSTEM</code>), or from the browser’s memory – the direction Vidar took.&nbsp;In memory, however, the key is kept protected with <code>CryptProtectMemory</code> using the <code>CRYPTPROTECTMEMORY_SAME_PROCESS</code> flag. This ensures that even if an attacker manages to read the encrypted form of the key from memory, they cannot simply decrypt it (they would need to invoke the complementary <code>CryptUnprotectMemory</code> function directly from within the browser process). That leaves attackers with two problems to solve: how to find the encrypted key in memory, and how to get it decrypted.&nbsp;<h2>Locating the v20_master_key in Browser’s Memory&nbsp;</h2>Vidar’s approach to locating the encrypted <code>v20_master_key</code> in memory involves scanning the entire memory of a running browser for a specific pattern. But that first requires a browser process to scan.&nbsp;When the target browser is already running, Vidar reuses the existing process. However, it doesn’t read its memory directly. Instead, it creates a fork of it using <code>NtCreateProcessEx</code> with <code>SectionHandle</code> set to <code>NULL</code>, instructing the kernel to clone the address space of the process specified by the <code>ParentProcess</code> as a copy-on-write mapping. The resulting process has no threads and is never resumed – it exists purely as a static memory snapshot.&nbsp;&nbsp;To perform the fork, Vidar first opens a handle to the target browser via <code>OpenProcess</code> with <code>PROCESS_QUERY_INFORMATION | PROCESS_CREATE_PROCESS | PROCESS_VM_READ</code>, falling back to <code>PROCESS_CREATE_PROCESS</code> if the initial call fails. It then passes this handle as the <code>ParentProcess</code> argument to <code>NtCreateProcessEx</code>, requesting <code>PROCESS_ALL_ACCESS</code> on the resulting child. If the fork fails, Vidar falls back to scanning the live browser process directly.&nbsp;

If no target browser process is found running, Vidar creates a new isolated desktop via <code>CreateDesktopA</code>, assigns it a name of the form <code>v20_%d</code> (where <code>%d</code> is replaced with the browser’s index in Vidar’s configuration), and launches the target browser in there. In the case of Vidar version 2.1, the browser is launched with the command line <code>--no-first-run --disable-gpu about:blank</code>, although the exact command line varies across versions. Since modern browsers use a multi-process architecture, there are typically multiple processes matching a given browser. Vidar enumerates all of them (up to 64) and applies the fork-and-scan procedure to each one independently.&nbsp;

With a handle to the forked process in hand, Vidar proceeds to enumerate its virtual memory regions using <code>NtQueryVirtualMemory</code>. It iterates through the entire browser’s address space, collecting regions that match a specific profile: committed (<code>MEM_COMMIT</code>), private (<code>MEM_PRIVATE</code>), and either readable (<code>PAGE_READONLY</code>) or read-write (<code>PAGE_READWRITE</code>). Vidar records up to 4096 memory regions matching these requirements, along with their base addresses and sizes. The actual pattern scanning is then performed in parallel with up to 64 worker threads, depending on the system.&nbsp;&nbsp;

Each thread scans its assigned regions using <code>NtReadVirtualMemory</code>, searching for a 32-byte pattern. When a match is found, the thread reads 8 bytes at offset +32 from the match, interprets them as a pointer, and validates that it points to committed, private, and readable memory (the expected memory state for the encrypted <code>v20_master_key</code>).&nbsp;&nbsp;

This naturally raises the question of what pattern Vidar searches for. The answer is a 32-byte signature: <code>76 32 30 00 ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 03 00 00 00 00 01 ?? ?? ??</code>, where <code>??</code> denotes a wildcard (Vidar uses the byte <code>AA</code> as the wildcard marker in its internal representation). Notably, the first four bytes correspond to the ASCII <code>v20\x00</code>, which is a common prefix used for ABE-related data.&nbsp;&nbsp;

We assess with high confidence that this pattern is intended to match a node structure within Chromium’s <code>Encryptor::KeyRing</code>, the structure that holds the keys used by the <code>Encryptor</code> class. Now, this part is admittedly a bit tricky as it relies on several assumptions we made about the internal memory layout, as well as on implementation-specific representations of various C++ standard library types <code>(std::map</code>, <code>std::string</code>, <code>std::vector</code>, <code>std::optional</code>), which is why we frame this as a high-confidence rather than a certainty.&nbsp;&nbsp;Our reasoning is as follows: <code>KeyRing</code> is <a href="https://github.com/chromium/chromium/blob/cf5d502bcaedb22927e3829a18be1ff6d8b6f598/components/os_crypt/async/common/encryptor.h#L132" target="_blank" rel="noreferrer noopener">defined</a> as <code>std::map&lt;/*tag=*/std::string, std::optional&lt;Key&gt;&gt;</code>, and <code>std::map</code> is typically implemented as a red-black tree, where each tree node stores three pointers (left child, parent, and right child) followed by the key-value pair. In this case, the pair is in the form of <code>std::pair&lt;/*tag=*/std::string, std::optional&lt;Key&gt;&gt;</code>, meaning that the <code>std::string</code> and the <code>Key</code> object might be laid out consecutively in memory within each node. For a <code>v20_master_key</code> entry, the tag corresponds to the string <code>v20</code>, which at 4 bytes including the null terminator fits within the small string optimization (SSO) buffer and is therefore stored inline rather than heap-allocated. The <code>Key</code> class that immediately follows <a href="https://github.com/chromium/chromium/blob/cf5d502bcaedb22927e3829a18be1ff6d8b6f598/components/os_crypt/async/common/encryptor.h#L93-L98" target="_blank" rel="noreferrer noopener">contains three member variables</a>: <code>std::optional&lt;mojom::Algorithm&gt; algorithm_</code>, <code>std::vector&lt;uint8_t&gt; key_</code>, and <code>bool encrypted_ = false</code>.&nbsp;Putting it all together and applying it to a concrete example from a browser memory dump, we get the following layout:&nbsp;

Please note that while the exact layout may contain minor inaccuracies on our part, the overall structure should be roughly correct. For easier orientation, the following diagram illustrates how the individual structures are nested within each other:&nbsp;

At this point, it should be clear how the scan pattern maps onto the searched structure. The 32-byte match covers the <code>std::string</code> tag <code>v20\x00</code> and the beginning of the <code>Key</code> object up to and including <code>algorithm_</code>. The pointer at offset +32 from the start of the match then lands precisely on <code>key_.begin</code>, the first member of <code>std::vector&lt;uint8_t&gt;</code> that holds the encrypted <code>v20_master_key</code>.&nbsp;Matches from all threads are collected into a shared buffer and subjected to a majority-voting scheme: candidates consisting mostly of zeroes are discarded, the rest are pairwise-compared, and each is scored by the number of identical copies found. This voting is performed independently for each browser process, producing one best candidate per process.&nbsp;<h2>Decrypting the v20_master_key via APC Injections&nbsp;</h2>For each browser process that produced a match, Vidar now holds a candidate address pointing to a potential encrypted <code>v20_master_key</code>. However, as already noted, since the key is protected with <code>CryptProtectMemory</code> using <code>CRYPTPROTECTMEMORY_SAME_PROCESS</code>, it can only be decrypted by invoking <code>CryptUnprotectMemory</code> from within the browser process itself. Vidar solves this by injecting an Asynchronous Procedure Call (APC) into the live browser process.&nbsp;Vidar supports two APC injection methods, selected based on whether ESET or Bitdefender is detected on the system. If either of these products is present, Vidar uses the first method, which it refers to as “classic” (based on decrypted strings) – it creates a suspended thread in the target browser via <code>CreateRemoteThread</code> (with <code>NtTestAlert</code> as the thread start address and the <code>CREATE_SUSPENDED</code> flag), queues an APC using <code>NtQueueApcThread</code>, resumes the thread, and then waits for it to complete via <code>WaitForSingleObject</code> with a randomized timeout. When the thread resumes, it calls <code>NtTestAlert</code>, which flushes the APC queue and executes the injected routine.&nbsp;

If neither product is detected, it uses the “special” method – it finds an existing thread in the target browser using a combination of <code>CreateToolhelp32Snapshot</code> and <code>Thread32First/Thread32Next</code>, opens it via <code>NtOpenThread</code>, and calls <code>NtQueueApcThreadEx2</code> with the <code>QUEUE_USER_APC_FLAGS_SPECIAL_USER_APC</code> flag. Unlike regular APCs, special user APCs execute immediately. Therefore, the target thread does not need to be in an alertable wait state.&nbsp;

In both cases, the APC routine to be executed is <code>CryptUnprotectMemory</code>, called with the following three arguments: the address of the encrypted <code>v20_master_key</code> candidate, its size (32 bytes), and <code>CRYPTPROTECTMEMORY_SAME_PROCESS</code> for the <code>flags</code> parameter.&nbsp;&nbsp;Since <code>CryptUnprotectMemory</code> decrypts the data in place, once the APC executes, the key is decrypted directly in the live browser's memory. To read it, Vidar creates a second fork of the browser process and reads the key from the same address using <code>NtReadVirtualMemory</code>. It then compares the result against a snapshot of the encrypted key taken from the first fork before the APC call – if the data is unchanged, decryption failed, and Vidar moves on to the next browser process’s candidate.&nbsp;If the data differs, Vidar uses the same second fork to perform an additional verification: it scans the forked process’s memory for the byte sequence <code>v20</code> and, treating each match as a potentially ABE-encrypted entry (cookie, password), attempts AES-256-GCM decryption with the key candidate using <code>BCryptDecrypt</code> (successful GCM tag verification confirms that the key is correct).&nbsp;Whenever the data has changed (indicating that decryption of the key took place), Vidar re-encrypts the key in the browser’s memory by injecting another APC with <code>CryptProtectMemory</code> as the routine, preserving the browser’s state. If no candidate produced a successful decryption (i.e., the data remained unchanged after every APC call), Vidar takes a more aggressive approach: it terminates all browser instances, restarts the browser, and repeats the entire scan-decrypt-verify cycle.&nbsp;By abusing APC injections instead of the more traditional approach of injecting code via <code>NtWriteVirtualMemory</code> that performs the decryption, Vidar trades one suspicious behavior for another. APC injections are arguably less common and may thus slip past some detection systems, though they are still well-known techniques monitored by AV and EDR solutions. But that is the nature of ABE bypasses – each has its own advantages and disadvantages, and that’s exactly why infostealers keep inventing new ones. &nbsp;<h2>Indicator of Compromise (IoC)&nbsp;</h2>459daa809751e73f60fbbe4384a7d1653c36bb06945e4eb3635270924241100a (Vidar 2.1)&nbsp;

A Technical Walkthrough of How Vidar Defeats Application-Bound Encryption

Inside Vidar’s ABE Bypass: From Memory Scanning to APC Injections

You have a summer flight booked. You have seen the headlines about jet fuel shortages, cancelled routes and higher travel costs. Then a text message arrives: your flight has been affected, and you need to click a link to confirm your refund or rebook before the offer expires.&nbsp;That message feels plausible because the situation around it is real. In April, <a href="https://www.iata.org/en/pressroom/2026-releases/2026-04-17-01/" target="_blank" rel="noreferrer noopener">IATA warned</a> that Europe could start seeing flight cancellations by the end of May if jet fuel shortages materialized. Airlines and governments have since been trying to reassure travelers, while also publishing guidance on passenger rights, refunds and re-routing.&nbsp;&nbsp;That is exactly the kind of confusion scammers like. They do not need to cause the disruption, they only need to arrive at the right moment, with the right brand logo and the right sense of urgency.&nbsp;<h4>Why travel disruption works so well as a scam hook&nbsp;</h4>Most scam messages fail because they arrive out of context. A random refund email from an airline you never use is easy to ignore. A refund message from the airline you are flying with next week, during a period of public travel disruption, is different.&nbsp;The scammer’s job is to make you think: “This could be real.”&nbsp;That can happen in several ways. You might receive a fake cancellation email that links to a refund form. You might search for an airline’s customer service number and land on a fake support page. You might post publicly on social media asking an airline for help, then receive a reply from an account that looks official. Or you might get a WhatsApp message that knows your hotel name, travel dates or booking reference.&nbsp;Each version uses the same pressure point: you are worried about losing money, losing your flight or being stranded.&nbsp;Criminals are already using disruption this way. In March, <a href="https://www.theguardian.com/money/2026/mar/15/travel-scam-airline-accounts-fake-refunds-iran-war-flight-disruption" target="_blank" rel="noreferrer noopener">The Guardian reported</a> that fake airline support accounts were targeting travelers affected by Middle East flight disruption. Some victims were looking for help or refunds, then received replies from accounts pretending to represent airlines. The scam moved into direct messages, where victims were asked for details and sent links that led to money being taken instead of refunded.&nbsp;&nbsp;The jet fuel story gives scammers another pretext. A message can claim that your flight was cancelled, your fare changed, your fuel surcharge failed, or your refund must be claimed within a few hours. None of those claims needs to be true to be effective.&nbsp;<h4>The scam may start before the message arrives&nbsp;</h4>A fake travel message becomes much harder to spot when it includes real information.&nbsp;This is the idea behind <a href="https://www.gendigital.com/blog/insights/research/reservation-hijack-scam" target="_blank" rel="noreferrer noopener">Reservation Hijack scams</a>, a form of targeted phishing where cybercriminals use real booking details to make a message look legitimate. In some cases, attackers may use information exposed through breaches or compromised travel systems. In other cases, they collect details from emails, social media posts, previous leaks or fake customer service interactions. Gen has previously described Reservation Hijack scams as scams that use real hotel booking details to make fraudulent payment or verification requests feel authentic.&nbsp;&nbsp;That distinction matters. A scam message does not need your credit card number to fool you. Knowing your name, your destination, your travel dates and your booking platform may be enough.&nbsp;<h4>What these scams can look like&nbsp;</h4>A travel disruption scam usually does not begin with a strange offer. It begins with something boring and administrative. That is part of the trick.&nbsp;You may see messages like:&nbsp;“Your flight has been cancelled due to fuel supply issues. Claim your refund here.”&nbsp;“Your rebooking is pending. Confirm passenger details within two hours.”&nbsp;“Additional airport tax or fuel surcharge is required to keep your booking.”&nbsp;“Your hotel reservation will be cancelled unless payment is verified.”&nbsp;“Due to high call volume, please continue this request on WhatsApp.”&nbsp;The exact wording changes, but the structure is familiar. There is a problem, a deadline and a link. The message may also move you away from the official channel into SMS, WhatsApp, Telegram, direct messages or a phone call. Once that happens, the scammer controls the conversation.&nbsp;There is another detail worth watching: some messages do not ask for money immediately. They start by asking for your booking reference, last name, phone number or one-time password. That can still be dangerous. Those details can help scammers access a booking, impersonate you to customer support, reset an account or make the next message more convincing.&nbsp;<h4>Your real passenger rights can be used against you&nbsp;</h4>If an airline cancels your flight, you may have rights to a refund, re-routing or assistance, depending on where you are travelling and the rules that apply. The European Commission said passengers affected by cancellations continue to have rights to reimbursement, re-routing or return, airport assistance and compensation for last-minute cancellations in relevant circumstances. It also said airlines cannot retroactively add fuel surcharges to already published final ticket prices.&nbsp;&nbsp;That information is useful for travelers, but it is also useful for scammers.&nbsp;A fake refund message becomes more believable when people already know refunds are possible. A fake rebooking link becomes more believable when airlines are genuinely changing schedules. A fake “fuel surcharge” demand becomes more dangerous when fuel prices are in the news, even though legitimate retroactive charges may not be allowed in many cases.&nbsp;This is why the safest route is not to judge the message only by whether the story sounds plausible. During travel disruption, many scam stories will sound plausible. Judge the channel instead.&nbsp;<h4>How to check before you click&nbsp;</h4>Start from the airline or booking platform’s official app or website. Do not use the link in the message. Open the app yourself, type the website address yourself, or use the contact details listed on your original booking confirmation.&nbsp;Check whether the same cancellation, refund or payment request appears inside your account. If the message is real, the airline or hotel should be able to confirm it through an official channel.&nbsp;Be careful with social media support. If an account replies to your public complaint, check whether it is linked from the airline’s official website. Look at the handle carefully. Fake support accounts often copy logos and names, but have small differences in spelling, low follower counts or recently created profiles. The Guardian reported that fraudsters often scan social media for passengers trying to contact legitimate airline accounts and reply to them first, which makes the scam harder to spot.&nbsp;&nbsp;Do not share one-time passwords, payment card details or login credentials with anyone who contacts you unexpectedly. A real airline or hotel does not need your banking OTP to send you a refund.&nbsp;Be especially suspicious of pressure. “You have 30 minutes to claim your refund” is not customer service. It is a tactic.&nbsp;<h4>What to do if you already clicked&nbsp;</h4>If you entered payment details, contact your bank or card provider immediately. Ask whether the transaction can be blocked or reversed, and request a new card if needed.&nbsp;If you shared login details for an airline, hotel or booking platform, change the password from the official site or app. Use a unique password, not one you use elsewhere. Turn on two-factor authentication where it is available.&nbsp;If you shared a one-time password or authentication code, assume the account may have been accessed. Check your booking details, payment methods and recent account activity.&nbsp;Report the scam to the airline, hotel or booking platform being impersonated. If the scam came through social media, report the account as impersonation. If money was taken, report it to the relevant fraud authority in your country.&nbsp;<h4>The real lesson for summer travel&nbsp;</h4>Travel disruption changes how people behave. They search for support numbers in a hurry. They post publicly because customer service queues are long. They accept strange instructions because they are worried about losing a flight or refund.&nbsp;Scammers build around that behavior.&nbsp;The safest assumption is simple: if a message about your flight, hotel or refund creates urgency and gives you a link, pause. Go back to the official source. A real cancellation will still be there when you check through the airline or booking platform. A scammer needs you to act before you do.&nbsp;

Travel disruption gives scammers the one thing they need most: a believable reason to rush you

Your flight was cancelled. Is the refund message real? 

A fake job interview page is easier to trust than a fake ticket shop.In samples reviewed by Gen threat researchers, scammers used branded hiring pages that looked like ordinary recruitment flows: a company logo, a recruiter profile, a 30-minute meeting, and a button to continue with Google or Facebook. There was no attachment to open and no software to install. The credential theft attempt sat inside a step many job applicants already expect: signing in to schedule a call.FIFA is an obvious lure ahead of the 2026 World Cup. On May 22, ESET reported fake FIFA-themed websites that copied ticket and merchandise flows to steal money and personal data. Our researchers later found another FIFA-themed domain, hxxps://fifa-mg[.]shop, which followed a naming pattern not mentioned in that report. Domains like these can be short-lived, often appearing for hours or days before they are blocked, abandoned, or taken down.Our researchers also found a different kind of FIFA lure: hxxps://fifahiring[.]com, a fake hiring page that encouraged users to schedule a call with an alleged FIFA recruiter. When a user tried to register with a personal Gmail address, the page asked for a business email instead. The page was steering victims toward credentials with more value: accounts that could provide access to a company environment.

<h4>The FIFA hiring page did not point to FIFA</h4>The fake page looked believable at first glance.

It used FIFA branding, presented a recruiter identity, and copied parts of the experience users see on legitimate recruiting and scheduling platforms. The recruiter’s name and photo appeared to match a real LinkedIn profile, which made the page feel more credible. For publication, we should keep that person’s identity redacted.The domain did not match FIFA’s real hiring infrastructure. FIFA’s legitimate careers pages are jobs.fifa.com and fifa.pinpointhq.com. The fake page used neither. It also loaded the favicon from the real FIFA recruitment site hosted at pinpointhq.com, giving the page a small visual cue borrowed from the legitimate service.Other signals pointed in the same direction. The domain and certificate were newly created, from May 23. The site used commodity hosting infrastructure, including Vercel and Render, rather than a corporate environment. Those signs are not enough on their own, but they fit the rest of the evidence.The login flow was the clearest part. Credentials were posted to:fifeq2026eqbackeq[.]onrender[.]com/api/loginThe Google sign-in prompt was not a real Google authentication window. It was rendered inside the page itself. The visible controls and links did nothing, and the page did not send the user into a genuine Google sign-in flow. According to the analysis, the only Google asset loaded was the favicon, apparently to make the fake window look familiar.A page can look like it opened a Google login without ever handing control to Google.Researchers also observed hxxps://careers-fifahiring[.]com, which later became inactive, and hxxps://fifajobs[.]com, which showed the same content and was indexed by Google.&nbsp;<h4>The recruitment lure went beyond FIFA</h4>The FIFA page was one version of a wider recruitment phishing pattern. The brand can change. The flow stays familiar: recruiter profile, meeting prompt, business email, fake sign-in.Researchers found similar fake hiring pages abusing other major brands, including Heineken, Hilton, Coca-Cola, Netflix, PepsiCo, Delta,&nbsp;and Spotify. In the Heineken case, the page appeared to use the identity of someone who, according to LinkedIn, really works as a recruiter at the company.

Additional hunting found two other recruitment-themed kits:hxxps://aquent-talent-jobs[.]com hxxps://hays-talent-opportunities[.]comThe Hays-themed version targeted Facebook credentials instead of Google credentials. Researchers assessed that the Aquent and Hays pages shared code, although they differed from the earlier FIFA examples.

This flexibility makes the campaign harder to frame as a single World Cup scam. FIFA provides the timely hook, but the mechanism is recruitment phishing. A scammer can swap the logo, recruiter name, and sign-in provider while keeping the same basic path to the victim’s credentials.<h4>Why fake recruitment pages work</h4>Recruitment gives attackers a clean excuse to ask for things that might otherwise raise suspicion.A candidate expects to click a scheduling link. They expect a recruiter profile. They expect a calendar prompt. In some cases, they may also expect to use a work email, especially for corporate, contractor, or partner-facing roles.The fake login window turns that ordinary flow into credential theft. Users are used to “Continue with Google” and “Continue with Facebook” buttons, but the button itself proves nothing. A genuine authentication flow should take the user to the provider’s domain and controls. A fake page can draw a convincing copy while keeping the victim inside the attacker-controlled site.The business email prompt is also a warning sign. A page that rejects a personal email and asks for a work address may be trying to raise the value of the stolen account.<h4>What job seekers should check before signing in</h4>Start from the company’s official careers page, not from a link in a message, ad, search result, or forwarded post. For FIFA, that means checking jobs.fifa.com or fifa.pinpointhq.com, not trusting a domain because it contains “fifa,” “career,” “hiring,” or “jobs.”Look closely at the domain before entering credentials. Scam pages often add plausible hiring words around a brand name: career, talent, jobs, hiring, portal, or connect. A polished page on a new domain is still a new domain.Treat the sign-in step as the real checkpoint. If a page offers Google or Facebook login, check whether the flow actually opens a real provider-controlled authentication page. A fake window embedded inside the original site, broken controls, inactive links, or a provider logo used only as decoration should stop the process.Use security software that includes web and phishing protection. These tools can block known malicious pages before they load, which is especially useful when scam domains look polished, appear in search results, or are shared through otherwise legitimate channels.<h4>What companies can do</h4>Recruitment pages are public by design, which makes them easy to copy.Security teams should monitor for newly registered domains combining their brand with hiring-related terms. They should also watch for pages that reuse favicons, recruiter photos, job descriptions, and calendar-booking language from legitimate hiring flows.HR and recruiting teams should make the official hiring path easy to verify. Candidates should not have to guess whether a domain is legitimate. A clear careers page, consistent recruiter communication, and warnings about unofficial scheduling links can remove some of the ambiguity attackers rely on.<h4>Conclusion</h4>The fake FIFA hiring page worked because it did not feel unusual. It looked like a normal step in a hiring process.Attackers do not need to invent new behavior when they can copy one people already trust. A branded scheduling page, a recruiter profile, and a familiar sign-in button can be enough to move a victim from interest to credential entry.The decision point is the login. Before entering a work email or password, check the domain, check where the authentication flow really goes, and verify the role through the company’s official careers site. Security software with web and phishing protection adds another layer, especially against short-lived scam pages that may only stay online for hours or days.IOCs:hxxps://fifahiring[.]com hxxps://careers-fifahiring[.]com hxxps://fifajobs[.]com hxxps://fifa-mg[.]shop hxxps://heineken-careers[.]com hxxps://aquent-talent-jobs[.]com hxxps://hays-talent-opportunities[.]com fifeq2026eqbackeq[.]onrender[.]com/api/login

Scammers are copying recruitment and calendar-booking flows to make fake Google and Facebook sign-ins look routine. 

Fake hiring pages abuse FIFA and other major brands to steal work credentials 

Threat Research Team Lead

Since January 2026, we have detected 7.4 million malicious incidents tied to residential proxy traffic, affecting 572,000 users in our telemetry. In this model, the household whose connection is being used is often not the buyer of the traffic. It is the exit point.That is the uncomfortable part of residential proxy networks. A home router, laptop, phone, smart TV, or other always-on device can quietly become part of a commercial network that routes strangers’ internet traffic through ordinary consumer IP addresses. To the websites on the other end, that traffic looks as if it came from a real household, because it did.The technology itself is not inherently malicious. Businesses use residential proxies for market research, ad verification, brand protection, security testing, and services that need to see the web as a real user would. But the same feature that makes these networks useful also makes them attractive for abuse: traffic from a household IP address is hard to block without risking false positives against legitimate users.The problem for consumers is simpler. Many people do not realize their device or connection can be enrolled into this kind of network at all. Some users opt in through pay-for-bandwidth apps. Others are enrolled through “free” VPNs, bundled software development kits inside unrelated apps, or unwanted software that installs proxy components without meaningful consent.Here is how the market works, who pays for it, and what it can cost the household that unknowingly hosts it.<h2>What is a residential proxy</h2>A residential proxy is an internet connection – typically a home laptop, phone, smart TV, or router – that has been enrolled into a third-party network and is being used to route someone else's traffic. To the rest of the internet, that traffic looks like it is coming from an ordinary household IP address assigned by a consumer ISP, not from a datacenter or&nbsp;VPN provider.Residential proxy networks are sold commercially. The model is two-sided: some people pay to tunnel their traffic out through someone else's connection (much like a VPN, except the exit point is a real home), while others get paid to provide those exit endpoints. Some are structured as paid peer-to-peer networks, where any participant's connection can serve as an exit for any other.Buyers pay for access to pools of millions of these "real user" IPs because they are extremely difficult to block, or rather, blocking them carries real consequences: the traffic originates from genuine households, so any block risks shutting out a legitimate user – effectively a false positive.Common buyers include market research firms, price-comparison services, brand-protection vendors, security teams testing their own defenses, and consumer privacy tools. At the abusive end of the market, the same infrastructure used by operators running automated purchasing bots, scrapping campaigns, ad fraud, credential stuffing, phishing and other attacks where hiding the true origin of the traffic matters. Demand is also being shaped by large-scale web scraping, including&nbsp;<a href="https://blog.includesecurity.com/2026/06/the-smart-tv-in-your-livingroom-is-a-node-in-the-aiscraping-economy/">scraping for AI training data</a>. As anti-bot systems become better at identifying traffic from datacenters and known VPN providers, residential IP addresses become more attractive because they look like ordinary users. That does not make every residential proxy user malicious. It does mean consumers may be carrying traffic for businesses and actors they have never heard of.<h2>How consumer devices end up in these networks</h2>There are several ways a home device can become part of a residential proxy network. Some are explicit. Others are easy to miss.The most obvious route is the pay-for-bandwidth model. Apps such as Honeygain, IPRoyal Pawns, and PacketStream pay users to share their internet connection. The user opts in, but they may not understand who will buy access to that connection later, what kind of traffic will pass through it, or what happens if that traffic triggers abuse reports.A second route is the “free” VPN or proxy app. Instead of charging a subscription, the app monetizes the user’s connection by reselling bandwidth. In practice, the user may think they are getting a privacy tool while also becoming infrastructure for other people’s traffic.A third route is through bundled software development kits, or SDKs. A developer adds a third-party component to an otherwise unrelated app, such as a flashlight, wallpaper, game, or utility. The developer gets paid when the app is active. The user sees the host app, not the proxy network behind it.The fourth route is malware or potentially unwanted software. In those cases, the proxy component is installed silently or deceptively, with no meaningful user consent at all.<h2>Why the "consent" is rarely meaningful</h2>Even when consent technically exists, it is often not the kind of consent most people would recognize.The disclosure may be buried in a long terms-of-service document. The wording may be vague, with phrases such as “share your unused bandwidth,” “support the app,” or “join our network.” These phrases do not clearly tell the user that arbitrary third parties may route traffic through the user’s home IP address.There is also the question of downstream use. Users are rarely told who buys access to the network or what those buyers do with it. Their connection could be used for benign market research. It could also be used for scraping, automated account creation, ad fraud, credential stuffing, or phishing infrastructure.The problem gets worse when the proxy component sits inside another app. The host app’s privacy policy may mention a partner SDK by brand name, but that still does not make it clear to a normal user that their device can become a proxy endpoint.Uninstalling the visible app may not always resolve the issue cleanly either, especially when proxy components are installed separately or bundled with other unwanted software.<h2>The risks consumers actually carry</h2>When a household device becomes a residential proxy node, the user takes on risks created by someone else’s traffic.The first risk is attribution. Abusive, fraudulent, or illegal activity routed through the device appears to come from the household’s IP address. ISP abuse notices, platform bans, fraud checks, and law enforcement requests point first to the subscriber, not to the person who bought the proxy traffic.The second risk is performance. Continuous third-party traffic can consume bandwidth, slow the home network, hit data caps, or violate ISP terms of service. The user may only notice that the internet feels slower, a bill is higher, or a service suddenly blocks access.The third risk is security exposure. Proxy clients run persistent background processes and maintain network connections so they can relay traffic. That adds always-on networking software to the device, often with limited visibility for the user.The fourth risk is reputation. A home IP address used for abuse can end up on blocklists used by banks, retailers, streaming services, and online platforms. The result can be account friction, failed logins, extra verification steps, or outright blocking from services the household actually uses.There is also a family angle. A child or another household member may install a free game, VPN, or utility without understanding the consequences. The legal, financial, and reputational risk still lands on the account holder.<h2>What our telemetry shows</h2>Since the beginning of 2026, we have detected 7.4 million malicious traffic incidents associated with residential proxy traffic, affecting 572,000 unique users in our user base.&nbsp; This is not a measurement of the entire residential proxy market. It reflects what we observe in our telemetry: malicious traffic, detections, and related activity affecting protected users. The real footprint is far larger, because residential proxy networks span major consumer platforms and many devices outside our visibility.The activity is global, but it is concentrated in a smaller set of countries. India leads by a wide margin, with roughly 83,000 affected users, followed by Vietnam with 61,000 and Brazil with 39,000. The Philippines (23,000), Poland (18,000), Indonesia (18,000) and Thailand (14,000) form a second tier, with Egypt, Spain, Ukraine, Argentina, Mexico and Italy each accounting for 10,000 to 13,000. The spread across South and Southeast Asia, Latin America, and Eastern and Southern Europe reflects where the host apps that carry this software are most widely installed.That geographic spread likely reflects a combination of app distribution, local install patterns, user appetite for free or pay-for-bandwidth apps, and the economics of bandwidth resale. It should not be read as a simple ranking of where residential proxy abuse originates.The chart below breaks down the threats we observed in this proxy-related traffic. Two categories dominate: phishing and malicious advertising. Behind them sits a long tail of consumer-facing fraud, including e-shop scams, generic scams, financial scams, and fake-tutorial pages, alongside outright malware such as trojans and other malicious traffic.&nbsp;

In the malicious proxy traffic we observed, scams and malware dominate. The important point for consumers is not only that their connection may be used by someone else. It is that, when abuse happens, their household IP address can be the return address.<h2>What consumers can do</h2>Most people will not identify a residential proxy component by name. But there are practical warning signs.Review “free” VPNs, proxy apps, bandwidth-sharing apps, and utilities from unknown publishers. Check whether they mention bandwidth sharing, peer-to-peer networking, traffic routing, or network participation in their terms. If an app offers payment for leaving your device online, assume your connection may be resold unless the provider explains exactly how traffic is vetted and who can buy access.On phones and computers, remove apps you do not recognize or no longer use. On home routers, check the list of connected devices and look for anything unfamiliar. If your ISP sends abuse notices, websites start blocking you, or your home IP address suddenly fails fraud checks, a proxy component is one possible explanation.Security software can also help by detecting unwanted proxy clients, bundled components, and related malicious activity. But this should not be a guessing game for consumers. Software that turns a household connection into a commercial exit node should be named clearly and made easy to remove.<h2>The takeaway&nbsp;</h2>With residential proxiesthe user’s connection becomes an infrastructure for someone else’s business.The issue is not that residential proxy technology exists. It has legitimate uses. The issue is whether people know when their devices are part of it, understand what kind of traffic may pass through their connection, and have a real choice to opt out.The fix is not to pretend consumers will read every EULA. This software should be named clearly, shown plainly, and made easy to remove before a household connection becomes someone else’s infrastructure.

Your home connection could be routing traffic for strangers. Here's how residential proxy networks work, how devices get enrolled and what our telemetry reveals about the risks for consumers.

Who's Really Using Your Home Internet Connection?

This blog post is based on my paper<a href="https://ieeexplore.ieee.org/document/9410267"> Pattern Matching in YARA: Improved Aho-Corasick Algorithm</a> and a <a href="https://github.com/VirusTotal/yara/pull/1379">pull request</a> that I opened on the upstream version of <a href="https://github.com/VirusTotal/yara">Yara</a>. My main goal is to describe the changes from a more practical point of view and also provide some updates and changes that were made from the initial design proposed in the paper, as practical usage in Avast showed me some additional ways how to improve my code.Yara is a pattern-matching tool used mainly to classify and identify malware. It is also an open-source project with an active community that is working hard to improve this tool every day. In this post, I will not go into details about what Yara is and how it can be used. For that, you can visit my <a href="https://regeciova-dominika.medium.com/botconf-2022-388109f43387">personal blog</a> or watch my talk about Yara at <a href="https://www.youtube.com/watch?v=3G0xaJkIE3M&amp;list=PL8fFmUArVzKjQ6A1OLfN2nnob4KIz8aT-&amp;index=10">Botconf 2022</a>.<h3>Motivation</h3>We use the Yara tool every day at Avast, and it is a critical help for our malware analysts. We are also trying to give back and bring as many of our improvements and additional features into the upstream version as possible.The changes I am proposing address a problem faced by our analysts: the “slowing down” warning, where rules can generate the following:

Besides the fact that we should be aware of every warning, not just straight errors, this negatively influences the usability of rules. For example, the VT Hunting platform does not accept rules with warnings.Most notably, one of our analysts was trying to create a rule that would detect Bitcoin addresses, as in the rule below, and had to combat this warning constantly.

However, we would like to use some of these rules, so what is the solution? And what exactly is the reason that Yara is warning about low performance? Can this be changed or improved?<h3>Problem analysis</h3>After analyzing the problem, I found the potential issue that decreased the number of primarily regular expressions that could be used in Yara rules without warnings.To understand that, I will briefly explain how Yara works internally. For more details, you can visit an excellent summary in <a href="https://github.com/Neo23x0/YARA-Performance-Guidelines">Performance Guidelines</a>, where I also added a few of my notes, or&nbsp; look directly at the documentation and the code itself.Yara firstly searches for potential matches with the help of the Aho-Corasick machine. In this step, it finds all possible places in input, where the whole strings can be found, and additionally, where rules can hit. Just after that, the bytecode engine is run to verify the matches, and then the conditions of the rules are also checked.For this first step, short substrings, up to 4 bytes, are selected from strings. A series of heuristics are used to choose the unique sequences.This is wanted as we do not want to search for the <code>00 00 00 00</code> sequence as it would hit too many times.The main problem is that the heuristics strongly prefer text strings and cannot work with regular expressions very effectively.In our case, the Bitcoin addresses let to zero-length atom. What does that mean? The bytecode engine must check every byte from the input because every position could be a potential match. And this is a truly time-consuming task, thus the warnings.But what if we could help Yara better understand the regular expressions and help it narrow down the number of potential matches? Would the speed of scanning improve?After creating the proof of concept, I can say yes, we can do it.

<h3>Proposed Changes</h3>You can read about all the details and formal theory behind the changes in my paper, but I will try to describe the changes in a more practical way here.First, I extended the definition of atoms from text strings to regular expressions. That means that they are able to represent the subclass of these languages. I kept the length of 4 bytes or rather a sequence of four classes. The class is a set of characters. It can be one character, that would be a literal, a collection of characters, like in <code>[a-d]</code> representing the set <code>{a, b, c, d}</code>. The last case is noted as a dot in regular expression as any character (sometimes without the new line, depending on the implementation).In this version, the Bitcoin address would be represented as <code>[13][a-km-zA-HJ-NP-Z1-9]{2}</code>, noting the repetition of the second class. It also could be <code>[13][a-km-zA-HJ-NP-Z1-9]{3}</code>. Still, experiments showed that the additional class does not improve the speed as significantly as the repetition does not provide a substantial decrease in the number of potential matches.The heuristics for selections of atoms were broadened and changed to prefer still simple text literals where they can be found and include more complex ones, as in our case at the same time.&nbsp;This part underwent several changes, and in the end, I tried to keep the changes as minimal as possible. My goal is not only to improve the scanning of the regular expressions but also scanning of other cases the same in the worst scenario. Even a tiny change in heuristics can dramatically affect the number of potential matches and thus the scanning speed, so this had to be handled carefully.The class itself is represented as a bitmap, an array of integers where individual characters are coded with a bit value set to 1 if the character is set, 0 otherwise. For example, for character <code>a</code>, the bit on position <code>bitmap[12][1]</code> is used, as its ASCII code (97) is divided by 8, and the resulting number is 12. The presence of the number is indicated by a bit with the value one on the position given by operation modulo 8.The selected atoms are then inserted into the Aho-Corasick automaton, a prefix tree machine. It is an extended version of a general finite automaton, where we are adding strings from the root state and detecting the common prefixes as we create states only when necessary. In my proposed version, it can also accept the classes instead of just simple text strings. In this step, the same classes are detected in the common prefix in a similar fashion as before.This leads to a problem because the Aho-Corasick contains so-called failure links that allow several matches (and even interleaving ones) to be detected at one pass from the input. This requires that the machine is deterministic even with failure links that return to previous states. My change, however, broke this determinism. Additional changes had to be made, and the machine was transformed into a deterministic one even for failure links.With these changes, we detected a subset of potential matches again, and the bytecode engine is run less often, even with the use of regular expressions. Also, the Bitcoin address can be detected without warnings about the slowing down scanning with my proposed changes.But what about the actual speed of the scanning? Was it improved?

<h3>Tests</h3>For tests, I run both versions on the server with the Debian operating system in release 4.9.168 on a 64-bit architecture with a processor Intel Xeon 5–2697 v4 with a clock rate of 2.30GHz. The memory clock rate was 2.40GHz. As a compiler, the GCC in version 8.3.0 was used.I selected two rules, one with the Bitcoin address and one from unit tests, to provide additional information on the overall effects of the proposed changes. More tests can be found in my paper.To measure speed, I scanned two larger folders with various subfolders and files with 15 GB and 760 GB of data in total.<h5>BTC Address Rule</h5>

<h5>Hex Value Rule</h5> &nbsp;

<h3>Conclusion</h3>Yara is a fascinating tool for pattern matching, and I am happy that there is an online community that is very active and helpful.At Avast, we want to give back as much as possible, and for that reason, I also published the changes that we have been using in the company for a while now.Note that there will be a discussion with the authors of Yara about this commit, and there can be some additional updates and changes, so the updated version is the most suitable for general usage.

Enhancing Yara's Regular Expression Performance

Yara: In search of regular expressions

Following our guide about <a href="https://engineering.avast.io/know-your-yara-rules-series-5-everything-you-need-to-know-about-regular-expressions-in-yara">regular expressions</a>, we present a new unique tool that can help you with a creation of such expressions, mainly for those used in the YARA Cuckoo module.&nbsp;&nbsp;&nbsp;To fully understand the benefits of our new open-source project, we first expand our knowledge about regular expressions in the Cuckoo module, share resources that can come in handy, and explain how to use GenRex for the best results.&nbsp;&nbsp;&nbsp;<h3>Cuckoo module&nbsp;</h3>Dynamic analysis is an essential part of our toolkit for malware detection. For that reason, we use the Cuckoo module often and contribute to it, as well as tools supporting it for practical usage. If you have never used the Cuckoo module, it can be confusing at first because it works a little bit differently than the basic YARA rules, but it is an efficient module, and it is worth learning about.&nbsp;&nbsp;Let us look at a simple example of YARA rule using the Cuckoo module:&nbsp;

To use the following rule, we need the samples we want to scan, but also reports from dynamic analysis from these samples represented as JSON files (an example of such report available <a href="https://github.com/avast/avast-ctu-cape-dataset/blob/a6b4d754d29f4cd29b60ac17d534da83c5c53830/cape_min/yara/0000a65749f5902c4d82ffa701198038f0b4870b00a27cfca109f8f933476d82.json">here</a>):&nbsp; &nbsp;

YARA does not run the samples; it uses the collected data about the sample’s behavior. By utilizing both cuckoo functions, strings, and other modules, we can combine multiple characteristics of malware to capture it more precisely.&nbsp;&nbsp;You can find more information about other functions in this module in <a href="https://yara.readthedocs.io/en/stable/modules/cuckoo.html">the official documentation</a>. We are also planning a separate blog post about the modules to provide even more tips on using them effectively. Stay tuned for more information.&nbsp;<h3>Dataset&nbsp;</h3>We recommend checking out our report’s dataset containing CAPEv2 reports from labeled malware and cleanware for experiments. For malware families, we have examples of trojans, worms, spyware, and bots. The latest version can be found on <a href="https://github.com/avast/avast-ctu-cape-dataset/pull/1">the GitHub page</a>.&nbsp;&nbsp;

We also provide the list of clean named objects that can be used for filtering and an extension of the YARA Cuckoo module to detect API calls, atoms, resolved APIs, and semaphores. On top of that, you can also compare how many matches were found in the report, such as the following example:&nbsp; &nbsp;

GenRex is a Python library with a Poetry package manager. To run GenRex, you need Python 3.10 or newer.&nbsp;&nbsp;&nbsp;You can either use the GenRex directly from the code available on <a href="https://github.com/avast/genrex">the GitHub page</a> or install it with the pip tool:&nbsp; &nbsp;

On the main page of our repository, there are two elementary examples of how to use GenRex. You can use GenRex as part of your pipeline, but the most straightforward way to get results is the following:&nbsp; &nbsp;

For each regular expression created from the input (accessible as a result[“regex”]), several additional characteristics are also generated, such as statistical information about how prevalent the strings creating the regular expression were in the samples. Based on that information, you can write rules that specify that the family makes a set of 3 atoms covered by specific regular expression.&nbsp; &nbsp;

Even though GenRex is a project still in development, it was successfully integrated into our systems and produced good results. Thanks to this experience, we have a few tips that can help you achieve the best results possible with GenRex:&nbsp;<ul><li>Use input type when possible. GenRex has several heuristics used for specific input types, such as mutexes, file paths, etc. If you provide this information, GenRex can generate better results, recognizing the pattern in the specific named object category. </li><li>The empty result list is also a valid result. GenRex should not create regular expressions for all costs but rather notice interesting characteristics in the input set. If GenRex does not provide sufficient results, try to inspect the input – it is possible that it is too random, and common aspects are to be detected. Filtering the input (for example, with the set of the clean named objects) can help.&nbsp; </li><li>You can join all input strings into one list; however, you usually get better results when providing GenRex with information about the source of these strings. You also get better statistical information that can be used for YARA rules.</li><li>The goal of the GenRex is to generate a good quality regular expression. However, we still recommend checking them against clean samples to test if they do not cause false positives.&nbsp;&nbsp;&nbsp;</li></ul><h3>Conclusion&nbsp;</h3>We are excited to open-source and share this new project with the YARA community. We hope you find GenRex interesting and useful. We look forward to your feedback, and if you have any ideas or encounter any issues, don’t hesitate to contact us. Also, feel free to open issues or pull requests — we appreciate it.&nbsp;&nbsp;If you are interested in GenRex and want to know more about it and its research, the paper GenRex: Leveraging Regular Expressions for Dynamic Malware Detection will be published in the following months as a part of IEEE TrustCom 2023 proceedings. You can also request preprinting through <a href="https://www.researchgate.net/publication/377438553_GenRex_Leveraging_Regular_Expressions_for_Dynamic_Malware_Detection">ResearchGate </a>or contact Dominika Regéciová on <a href="https://twitter.com/regeciovad">Twitter</a>.&nbsp;&nbsp;&nbsp;As a very fresh update while writing this post, we are thrilled to announce that our proposal for the <a href="https://www.botconf.eu/">Botconf conference</a>, titled GenRex Demonstration: Level Up Your Regex Game, has been accepted. This exciting opportunity allows us to showcase practical examples and real-world applications of our GenRex tool. Join us from the 24th to the 26th of April in Nice, France at the conference, where we’ll delve into the intricacies of GenRex and demonstrate how it can significantly enhance your approach to regular expressions. We look forward to sharing insights, exchanging ideas, and engaging with the vibrant community at Botconf.&nbsp;&nbsp;To conclude this blog post, the main takeaways are:&nbsp;&nbsp;&nbsp;&nbsp;<ul><li>The Cuckoo module is a convenient tool for behavioral analysis, and it implements additional features for regular expression matching.&nbsp;&nbsp; </li><li>GenRex is an open-source project that can help you generate regular expressions for the Cuckoo module and other use cases. </li><li>The source of the project can be found on the GitHub page: <a href="https://github.com/avast/genrex">https://github.com/avast/genrex</a>&nbsp;&nbsp;</li></ul>And that is all for today! We wish you happy YARA rules writing!&nbsp;

A Tool for Advanced Regular Expression Generation

Know your YARA rules series: #6 we present GenRex – a generator of regular expressions 

For the fifth post of the <a href="https://engineering.avast.io/category/know-your-yara-rules/">Know Your YARA Rules</a> series, we want to create a comprehensive manual for regular expressions that would improve your YARA rules.&nbsp;<h3>Why is it so complicated?&nbsp;</h3>When discussing regular expressions, we must first address the elephant in the room. Regular expressions can get complicated rather quickly, mainly because they often look like winners in obfuscation games rather than something meaningful. It does not help that there are lots of sources on the Internet that sometimes contradict each other. Some documentation will tell you that regular expressions can match your case, while other sources will tell you that this cannot be done. So, where is the truth?&nbsp;The reason for this is that several implementations and standards of regular expressions, unfortunately, do not always produce the same results. The difference is mainly in extensions that allow us to describe a broader range of strings. However, some expressions you use in your JavaScript code do not necessarily work in YARA rules in the same way, and some constructs are not even supported in YARA.&nbsp;&nbsp;&nbsp;In this blogpost, we will first cover the basics that are the same for all implementations. Then, we will describe some additional features that can differ based on the implementation.&nbsp;<h3>The formal definition&nbsp;</h3>In formal theory, we define regular expressions as the way to describe a set of strings – a language. The descriptive power of them is useful but limited. Let’s say we use only the letters a and b in our language (for simplification). Of course, you can match only one letter, so a or b. Then, you can use the following operators to create more complex expressions:&nbsp;<ul><li>Concatenation: ab – a is followed by b, as in string “ab”</li><li>Alternation: a|b – we are matching a or b, as in strings “a”, or “b”</li><li>Repetition: a* – letter a is repeated zero to infinity times, such as in strings “”, “a”, “aa”, “aaa”, …</li></ul>And that is it. You can combine the operators, but there is a limit to what you can describe. For practical purposes, there are several extensions that make regular expressions more usable. There are two types of extensions. The first one creates a syntax sugar around the basic definition. There are flags that influence the whole pattern (such as i for case-insensitive matching), metacharacters, such as wildcard “.” that matches any character (in YARA, this excludes the new line character, unless used with flag s), or quantifiers such as {4,5}.&nbsp;&nbsp;&nbsp;The second type breaks the definition of the regular expressions so you can describe strings you cannot describe with the basic version.&nbsp;&nbsp;&nbsp;The most famous example of such strings is language described as a^nb^n, where the number of letters a is equal to the number of letters b and no letter b is in front of a. So, strings like ab, aaabbb, aaaaabbbbb,… With basic regular expressions, you can have a random number of letters a followed by some number of letters b, but there is no mechanism that would compare the occurrences of these two letters to ensure they are equal. The PCRE implementation, for example, supports so-called recursive patterns, creating surprisingly short expression: /^(a(?1)?b)$/.

You can imagine the expression (?1) as a recursive function call that creates a new level of recursion while reading the letter a. After all letters a are read, we return from the recursion while reading the letter b. Only if the numbers of these letters are equal will we end with a successful match.&nbsp;

If someone tells you that this example can be matched by regular expressions, it depends on what kind of regular expressions they are talking about.&nbsp;As YARA does not support these features, we will not include more details about the evaluation. If you want to learn more about this exciting topic, we recommend visiting <a href="http://www.regular-expressions.info/">www.regular-expressions.info</a> website.&nbsp;<h3>PCRE and other versions&nbsp;</h3>There are many versions of the regular expressions, and navigating through them can be confusing. Let’s focus on the most important of them to keep it relatively simple.&nbsp;&nbsp;If you use the grep tool, use the IEEE POSIX standard in the version of BRE (Basic Regular Expressions) or ERE (Extended Regular Expressions) when using the grep -E option. The only difference between these two is that in ERA, you do not need to escape metacharacters ( ) and { } in ERE, which are required in BRE.&nbsp;&nbsp;&nbsp;PCRE (Perl Compatible Regular Expressions) extends these versions and is also a Perl library written in C. If you used regular expressions in the past, the implementation was most likely derived from PCRE or very similar. This version is so popular that it is the most used implementation standard. That is the reason why even YARA is based on PCRE.&nbsp;&nbsp;Also note that there is the newest generation of the PCRE, PCRE2. There are some changes in syntax, but because YARA still uses PCRE, we will not go into details.&nbsp;<h3>YARA world&nbsp;</h3>As said before, YARA has an engine compatible with PCRE but with some limitations. The goal was to focus on the solution’s effectiveness, and for this reason, some extensions are not supported because they would slow down the rules.&nbsp; For the following summaries, we used <a href="https://yara.readthedocs.io/en/stable/writingrules.html#regular-expressions">the official documentation</a> as source, which we recommend checking out as well.&nbsp;<h6>Modifiers&nbsp;</h6>For regular expressions, you can use modifiers nocase, ascii, wide, and fullword after their definition, changing the matching to case-insensitive, ascii form, wide form, and conditional delimitation by non-alphanumeric characters, respectively.&nbsp;&nbsp;

<h6>Flags</h6>YARA supports only two flags – //i for case-insensitive matching and //s for matching, where the dot symbols also cover new lines. Other flags are not supported, at least for now.&nbsp;<h6>Metacharacters&nbsp;</h6>

<h6>Classes&nbsp;</h6>Classes, or character sets, are a listing of characters we want to match written in square brackets. We can write them individually, such as [abcd], or as a range, such as [a-d]. Note that [d-a] is an invalid range and leads to error. [d\-a] matches {a, d, -}, as well as [-ad] or [ad-] (if – is not in the middle of the class, it does not have to be escaped).&nbsp;&nbsp;You can also combine notations, like in the example [0-9a-fxA-FX].&nbsp;Sometimes, the list of characters you do not want to match is shorter and more readable. In that case, use the caret symbol (^) at the beginning of the class, like in example [^a-d] matching all characters except {a, b, c, d}.&nbsp;Also, be careful while mixing upper and lower case characters. For example, [a-Z] is not a valid range as upper-case characters have lower ASCII values. You can use [A-z], but generally, it is better to write them as two ranges for more readable expressions: [A-Za-z].&nbsp;&nbsp;<h6>Greedy quantifiers&nbsp;</h6>With greedy quantifiers, we are trying to match the longest match possible.&nbsp;

<h6>Non-greedy quantifiers&nbsp;</h6>With non-greedy (lazy) quantifiers, we are trying to match the shortest match possible.&nbsp;

<h6>Escaped sequences&nbsp;</h6> &nbsp;

Word boundaries (in the form of \bpattern\b) allow us to match only “whole words”, meaning the match is surrounded by non-word characters or the start or end of the file. The negated version, \B, matches the pattern surrounded by word characters.&nbsp;<h3>Tricky cases&nbsp;</h3><h6>. vs \.&nbsp;</h6>A typical mistake is forgetting to escape metacharacters such as “(“, “)” or “.”, changing the meaning of the expressions. Be careful about these mistakes, as they can lead to false positives.&nbsp;&nbsp;&nbsp;The situation is even more complicated when using square brackets to define the class you want (or don’t want) to match. In this scenario, the metacharacters are not usually escaped, but you need to escape a special character “-” if you are not using it as range and it is in the middle of the class.&nbsp;&nbsp;&nbsp;<h6>* also means nothing&nbsp;</h6>Sometimes, we need to express that the part of the strings is various lengths, and we use the star symbol for that. However, this also includes the empty string, which can lead to matching everything, such as an example /(malware|nonclean)*/.&nbsp;<h6>.* vs .*?&nbsp;&nbsp;</h6>When matching expressions without maximal length, there are two evaluation methods. The first one is called greedy, where we match the longest string possible. The second one is called non-greedy (or lazy) evaluation; in this case, we match the shortest string. For input Tomxxxx, /Tomx*/ will return Tomxxxx as greedy evaluation, while very similar regular expression /Tomx*?/ will return only Tom. In general, non-greedy evaluation is faster than greedy evaluation, but it is even better if you can specify the concrete numbers (such as {2,42}).&nbsp;&nbsp;&nbsp;However, be careful when using these quantifiers in YARA. If the expression starts with them, YARA will match all versions of such strings. For example, for input xxxxTom and regular expression /.*Tom/ (or /.*?Tom/), YARA will find five matches: xxxxTom, xxxTom, xxTom, xTom, and Tom. That can easily lead to a warning about “too many matches” which invalidates the results. This behavior is YARA-specific because it uses the Aho-Corasick automaton. Other implementations, even if based on PCRE, may not have the same behavior and may return the longest match (xxxxTom) instead.&nbsp;<h6>Not all quantifiers are treated equally&nbsp;</h6>As we mentioned earlier, using a specific range in quantifiers (like {2,5}) rather than more general versions * and + is better. &nbsp;There are two reasons why using format {x,y} is a good idea. Firstly, when you specify the minimal number of repetitions, YARA can use it for the Aho-Corasick automaton, which can improve the scanning speed (more information can be found in our first post [https://engineering.avast.io/know-your-yara-rules-series-1-know-your-yara/]). The second reason is that you are forcing a limit to the match and how many matches you can get from YARA. Using the examples from previous points, x{2,5}Tom will return three matches (xxxxTom, xxxTom, xxTom), while Tomx{2,5} will return only the longest match, Tomxxxx.&nbsp;<h6>Match what you need, not what you want&nbsp;</h6>It can be tempting to create complex regular expressions to match every possible variant of the search strings. However, be aware of this trap. In most cases, if you struggle to describe everything in one expression, it is a signal to take a step back and think about what needs to be matched. If some parts are too random, there is always an option to skip them. You can still match the string partially, maybe even with a simple text string. Regular expressions are not miracle remedies; sometimes, the simplest solution is the best.&nbsp;<h6>Issues with non-ASCII characters&nbsp;</h6>The situation with non-ASCII characters is complicated by itself, and in YARA, it is even more complicated. In the official documentation, you can find an explanation that YARA does not support non-ASCII characters in strings [https://github.com/VirusTotal/yara/wiki/Unicode-characters-in-YARA]. However, this is not entirely true because this decision was later partially reverted. You can now use either hexadecimal notation (such as \xD0\xAF for Я) or non-ASCII characters in strings and regular expressions, except classes. The classes do not currently support any form of non-ASCII characters.&nbsp;

<h3>Additional sources&nbsp;</h3><h6>Regular-expressions.info&nbsp;</h6>This fantastic website contains all the information about regular expressions you may want to know about. In addition, for great explanations about each (im)possible feature out there, there are also recommendations for other tools, tutorials, and books. &nbsp;<h6>Regex 101&nbsp;</h6>The website <a href="https://regex101.com/">regex101.com</a> is excellent for experimenting with regular expressions. You can write the regular expression and input you want to match, and it will highlight what was matched and why. You can also use so–called flavors – a type of implementation. We recommend using flavor PCRE for YARA rules and the preferred language for other cases.&nbsp;&nbsp;&nbsp;There is also the Explanation section, where you can describe each part of your expression and what it does. This can be very helpful, mainly when debugging.&nbsp;<h3>Conclusion&nbsp;</h3>In this post, we investigated the slightly chaotic world of regular expressions. Hopefully, after reading this, you can navigate this world better and evade the traps across the road.&nbsp;&nbsp;The main takeaways are:&nbsp;&nbsp;&nbsp;<ul><li>YARA’s regular expression engine is based on PCRE implementation; however, it does not contain special features like caption groups. </li><li>Matching can differ based on the implementation and extension features. Be careful when copying regular expressions from the web or other applications because you can get different results. </li><li>If you need clarification on what regular expression is describing (and believe me, you are not alone lost in the sea of bracelets and metacharacters!), try tools for visualization of matching, such as regex101.&nbsp;&nbsp;</li></ul>And that is all for today! We wish you happy YARA rules writing!&nbsp;

Mastering Regular Expressions in YARA

Know your YARA rules series: #5 everything you need to know about regular expressions in YARA

In the third post of the <a href="https://engineering.avast.io/category/know-your-yara-rules/">Know Your YARA Rules</a> series, we will look at the last release of the YARA tool and reasons why it is a good idea to be up to date about new versions of this open-source project. &nbsp;We will tell you about the interesting new features, important fixes, and extra tips on how to keep up with all the changes.&nbsp;&nbsp;&nbsp;<h3>Motivation&nbsp;</h3>Recently, a <a href="https://github.com/VirusTotal/yara/releases/tag/v4.4.0">new release candidate</a> (pre-release) was announced with an exciting list of changes in the YARA tool. We decided to investigate this more closely and explain why you may find the YARA releases and pre-releases valuable to your work. &nbsp;Note that even though this is not an official release yet, the changes can appear on VirusTotal in advance.&nbsp;

<h3>Bug fixes&nbsp;</h3>As with any software, YARA has occasional bugs and unsolved issues. Luckily, an active community is working strenuously to fix every problem that arises as soon as possible. Because the project is open source; you can check every change and fix and test it before updating your YARA. Every release has a transparent list of changes, so you know what to expect with each update. &nbsp;In the last release candidate, a few changes can change the evaluation of your rules. Namely, there were <a href="https://github.com/VirusTotal/yara/issues/1933">fixes for matching of regexes marked as “ascii wide”</a>, matching rules with <a href="https://github.com/VirusTotal/yara/commit/4de3d574bae5973c711095c1c755166c07dec322">– -fast-scan option</a>, and computation of <a href="https://github.com/VirusTotal/yara/pull/1944">the imphash function in the PE module</a>.&nbsp;&nbsp;&nbsp;It is good that these issues were fixed but be prepared that your rules can behave differently based on the version of YARA you are using.&nbsp;<h3>New features&nbsp;</h3>New features are always exciting. At Gen, we use rules that others have created, and we are also constantly working on providing them to the community. In this pre-release, we have two changes from our team. &nbsp;<a href="https://github.com/TommYDeeee">@TommYDeeee</a> exposed the RVA function in its raw form via pe.export_details[] before it is turned into offset via <a href="https://github.com/VirusTotal/yara/pull/1882">RVA-&gt;offset calculation</a>. You can use such as:&nbsp;

<a href="https://github.com/regeciovad">@regeciovad</a> also added a new warning about the performance issues while scanning that can help you improve your <a href="https://github.com/VirusTotal/yara/pull/1921">rules’ performance</a>.&nbsp; &nbsp;

Our goal was to create a deterministic way to detect potentially slow scanning due to a lower quality of rules. We came up with additional checks that indicate that the rule uses very general or common strings, leading to slow scanning. For more tips, visit our previous post about <a href="https://engineering.avast.io/category/know-your-yara-rules/">YARA’s performance</a>.&nbsp;<h3>LNK module&nbsp;</h3>A new module for Windows Shell Link (LNK) file format was part of the candidate release created by <a href="https://github.com/BitsOfBinary">@BitsOfBinary</a>. This change was reverted due to some open issues (see the discussion here for more details: <a href="https://github.com/VirusTotal/yara/pull/1957">https://github.com/VirusTotal/yara/pull/1957</a>). Even though we need to wait a little longer for this module, we welcome the initiative as we believe the modules are the future of YARA. We plan more posts about the existing modules (and possibly even planned ones), so stay tuned.&nbsp;<h3>The improved performance&nbsp;</h3>The second feature we are excited about was created by user <a href="https://github.com/secDre4mer">@secDre4mer</a>. While using a large ruleset, they noticed that the condition evaluation takes up a lot of time (the bytecode evaluation) and came up with a helpful optimalization. &nbsp;If the rule requires a string match and YARA does not find any match in the specific input, the condition evaluation part can be skipped entirely because the rule will always be false in the end.&nbsp;&nbsp;We are currently testing this improvement and we are excited to use it in our internal version of YARA.&nbsp;<h3>Keep up with the changes&nbsp;</h3>Life can get busy, and we understand that keeping up with the news about YARA can be challenging. If you are reading this blog post, you are most likely well-informed. However, you may still benefit from checking these extra resources that provide great updates and information about YARA rules. Twitter (or X) is still a good resource, and we recommend following these profiles:&nbsp;<ul><li><a href="https://twitter.com/plusvic">Victor M. Alvarez </a></li><li><a href="https://twitter.com/wxs">Wesley Shields</a></li><li><a href="https://twitter.com/cyb3rops">Florian Roth&nbsp;</a></li></ul><h3>Contribute&nbsp;</h3>We already emphasized the benefit of YARA being an open-source project, but we also encourage you to contribute to the YARA project. Reporting issues, discussing ideas, or even creating your own changes moves the whole community forward.&nbsp;&nbsp;There is a very promising new project <a href="https://github.com/VirusTotal/yara-x">YARA-X</a> that can change the YARA world completely. We plan to dedicate a whole post of our series to this new project. For now, if you are interested in YARA development, you should start to watch this project and learn something about Rust if you are not familiar with that programming language. &nbsp;Stay tuned for more updates about this project and more topics.&nbsp;<h3>Conclusion&nbsp;</h3>YARA is an open project with a very active community that continually improves the tool and fixes issues as soon as possible. From time to time, we recommend checking the latest changes and evaluating if you would benefit from updating your version of this tool (even though, in most cases, the answer is yes).&nbsp;&nbsp;As we looked at the last pre-release, there are several interesting changes, and even more exciting things are coming.&nbsp;&nbsp;And that is all for today! We wish you happy YARA rules writing!&nbsp;

Staying Current with YARA: Embracing Releases and Updates

Know your YARA rules series: #3 watch out for releases

In the second post of the <a href="https://engineering.avast.io/category/know-your-yara-rules/">Know Your YARA Rules</a> series, we will continue to explore more hurdles that keep you from achieving the maximum performance from your YARA rules.Today, we will demonstrate several examples where even a tiny change in your YARA rules can drastically improve the scanning performance.Before we start, note that YARA is a live project that changes over time. What we recommend today might not be the best available solution next year, so we recommend keeping up with the current version and recommendations as the project changes. For that, you can, for example, follow this blog site, as we will continue to write about news from the YARA world.<h3>A race between strings and condition</h3>In the <a href="https://engineering.avast.io/know-your-yara-rules-series-1-know-your-yara/">first post</a> from our series, we discussed that we must be careful with the string section because it is always evaluated before the condition section and cannot be skipped with methods like filesize. This section will discuss the special case where you can use it to your advantage. Let us say we want to match a file with specific bytes at the beginning of the file with a filesize limit, as shown below:&nbsp;

There are three reasons for slowing down the scanning. We have already learned that the filesize condition will not help us limit which files will be scanned for string $h00 and which will be skipped. The second reason is that not even limitation to position 0 would help us, as the position of the match will also be checked after we have called all matches of the bytes sequence in files. The last reason is too general nature of the strings. It can cause many false positives during the evaluation phase and slow the whole process down. But what can we do to improve the situation?If we know the exact position we want to match, we strongly recommend using intXY and uintXY functions. You can match uint8(0) or unit32(3), where the number in the function name represents how many bits you want to match starting with the offset or virtual address described as an argument. Be aware that both 16 and 32-bit integers are little-endian. For big-endian, you can use functions intXYbe() or uintXYbe(). The rewritten rule then looks like the following:

Notice that the string section is completely gone. This is good because now, we first check the size of the file, and if the condition is not met, the file is skipped for the rest of the rule evaluation. If the size is correct, we check the position 0 for the value 0x42. Only when this is fulfilled is the second position in the file checked.Note that there is a fast scan mode that changes the way the rules are evaluated. But to keep things simple, we will skip the details and leave it for future posts in this series.<h3>Help YARA figure out your strings&nbsp;</h3>YARA is generally very smart when scanning with your rules, but it has its limits. For this reason, there are ways to rewrite your rules so YARA can understand them better, focusing on short strings.The most typical example of this is alternations, mainly in short strings.

It can be tempting to write these short strings into one. The problem is that YARA does not parse this notation that well. Yes, YARA can work with alternations during the atom selection process. For instance, in a case such as /(hey|bye)\.\w*/, it would prefer two atoms hey and bye over searching just for a dot (\w cannot be part of the atom). However, Yara does not concatenate the strings together, even if it means the atoms would be longer. In this case, the alternation does not provide enough information, and the problem is even bigger because even values outside the alternations are just one character in length. We want to match four or more bytes for effective matching, but sometimes, it is not very easy. In this case, we want or need to work with these strings. So, how can we write this string, so YARA understands them better?

Rewriting the string into two will improve the scanning speed as your rules speak to YARA more clearly.<h3>Aim for the string you really need&nbsp;</h3>Sometimes, our rules are slow because we match more strings than we actually need to. In some cases, it is acceptable to scan for a more general version of the string and check other parts of the condition after that. However, in most cases, you could struggle with some performance issues.Let us say we want to detect IPv6 addresses. You could match a general format of the addresses, as in the following rule:

The question is, do you absolutely need to check for this general format? What are you trying to match? In our case, when assisting malware analysts, we discovered that we need only global unicast addresses starting with the prefix 2001. That is fantastic news for YARA and us because it means that we can simplify our regular expression.

Not only is this faster, in our tests, it was about 50% faster, but it does not falsely match strings that are not unicast IPv6 addresses. And that should always be our goal.A similar case is when we want to find a string with a filename ending on extension exe. It is tempting to write something like /*.exe/, or /[0-9a-z].exe/. The fastest variant is, surprisingly, for some, the text string “.exe”. In this situation, the number of matches will probably not change that much, but you are speeding your YARA rules by 40%.<h3>Conclusion&nbsp;</h3>In this post, we explained more examples of improving the speed and quality of your YARA rules.The main takeaways are:<ul><li>Think twice about the strings you want to match. Are you describing exactly what you need, or are you creating a more general version? Your rules will likely be prolonged if you count on a condition to sort out the real matches.</li><li>If you know the position of the match, help YARA and use (u)intXY functions. You can skip the scanning of large files and optimize the use of short-circuit evaluation.</li><li>Help YARA understand what strings you are trying to match. Sometimes, with slight changes, YARA will surprise you with better scanning speed. In future posts, we will show even more examples.</li></ul>And that is all for today! We wish you happy YARA rules writing!

Optimizing YARA Rules: Rewriting for Performance

Know your YARA rules series: #2 rewrite your rules

With this article, we are starting a new series on the Engineering blog – Know Your YARA Rules. In this series, we would like to share tips and tricks we learned from using YARA daily. &nbsp;We aim to pick fewer known facts about YARA and how it works, so you can write even better YARA rules that are fast and precise in detection. Because no one likes slow rules with false positives. This first post will investigate how YARA works internally and how it leads to common misconceptions, slowing down rules and problems with too many matches.<h3>Motivation</h3>Creating a good rule that correctly detects a described family and does not cause false positives requires skill and experience. Even after all the necessary static and behavioral information is extracted and described in the rule, another problem can arise – the scanning performance. YARA has several mechanisms to detect potential slow scanning and generates warnings, such as:

These warnings are based on heuristics that evaluate the quality of the rules. They inform us that the rule could be written more effectively. They often lack information on what change is needed to improve performance. What makes the issue worse, the rules with warnings cannot be used in some systems like VirusTotal Hunting (<a href="https://www.virustotal.com/gui/hunting-overview">https://www.virustotal.com/gui/hunting-overview</a>).For these reasons, it is a good idea to understand what the warnings and errors are trying to tell us. However, it is not always so simple. Sometimes, even experienced YARA rules writers wonder what is wrong with their rules and how they can fix them.In this series, we will go through some problematic cases that we have encountered at Gen, and we will provide the solutions we came up with.<h2>YARA Internals&nbsp;</h2>To understand more about our tips and tricks from this and the following posts, we need to look up YARA under the hood and go through the process of the rule’s evaluation. For this purpose, let us consider a simple YARA rule:

In this rule, we have three expressions where at least one must be true to match a sample. Either a text string delimited by non-alphanumeric characters is found in the sample (while ignoring the case of the characters), or regular expressions are found or a hexadecimal string in starting position 0 (at the beginning of the file). If a sample meets the condition, Yara will match it and report it as a result.The entire process of evaluation of this rule can be split into four steps: atoms selection from strings, creation of the Aho-Corasick automaton, bytecode engine run, and evaluation of conditions.<h4>Atoms Selection from Strings&nbsp;</h4>From all the strings from all rules, so-called atoms are selected. The atoms are substrings with lengths from zero to four bytes. Yara has several heuristics for choosing the most unique and, thus, most effective atoms. In our example, from a regular expression /abcd[x-z]/, Yara will select the part abcd.&nbsp;&nbsp;The heuristics, however, have their limits, and for strings that are too general, such as /w*/, even zero-length atoms can be chosen. This is problematic because a later, much slower state does all the matching, and the input is searched byte by byte. This will lead to a warning about slowed scanning and limiting the rule’s usability in systems like VirusTotal Hunting.&nbsp;<h4>Aho-Corasick Automaton&nbsp;</h4>All selected atoms are used to build a prefix tree called the Aho-Corasick automaton. This automaton works as a sifter – it quickly scans through every input and finds all potential matches for the atoms. This is a crucial step that influences how fast or how slow the scanning is. If we have too many general atoms, many — even each very byte, is selected as a potential match and must be inspected further. We want to limit the set of potential matches, so the rest of the evaluation is much faster. Below is an example of the Aho-Corasick automaton for atom abcd.&nbsp;

The basic idea behind using the Aho-Corasick automaton is that it effectively matches every occurrence of multiple strings simultaneously, even when the matches are interleaving. Starting in the root state, the automaton reads one symbol from the input at a time and changes states accordingly. The atom is found in the input if the final state is visited.&nbsp;&nbsp;We can find multiple matches in the same position because we have so-called failure functions. The failure function is used when there is no transition based on a combination of the current state and input symbol. The default action is to go to the root state and try again. However, if the prefix of the other string was partially matched, we can go there instead. All failure functions point to the root in our example because there are different characters, but we can choose another group of strings to demonstrate this situation.&nbsp;&nbsp;Let us say we are looking for the words she and his in the text shisa (Chinese guardian lions – <a href="https://en.wikipedia.org/wiki/Shisa">https://en.wikipedia.org/wiki/Shisa</a>). We started matching sh in the keyword she, but now, we need e, but we have i instead. We could go to the root again or try to match his instead because we know we already read h. And yes, we can find his in the rest of the text.<h4>Bytecode engine&nbsp;</h4>The bytecode engine gets the list of potential matches and verifies which are true positives based on the full definition of strings (including modifiers like wide, nocase, and others). This process takes time, so as we discussed before, making a list as accurate as possible is a clever idea. Yara also reports the position in the input if a match is found. In our example, there is a match in position 0x0.&nbsp;

Evaluating the condition for the whole rule is the last step. It is important to remember that conditions are evaluated after matching the strings. If you want to limit the file size that you want to match with the filesize function, it will not prevent the previous steps from being executed when your rules match static strings. The good news is there is short-circuit evaluation, so changing the order of conditions can positively impact performance.&nbsp;&nbsp;&nbsp;In our example, we found the match in position 0x0, and the condition is evaluated as True.<h4>Rulesets&nbsp;</h4>Very often, we work not only with a single rule but several rules simultaneously. YARA can operate with them effectively and use them for scanning at once rather than one rule at a time. However, this approach has its disadvantages. Scanning with rulesets is generally the same as with one rule. This means that if you have a set of fast rules and add one slow string to one rule, the whole ruleset will be slowed down.&nbsp;&nbsp;&nbsp;<h2>The filesize doesn’t rule them all&nbsp;</h2>Now, let us use the knowledge of YARA evaluation in the practical example. This is a widespread mistake that can make your rules really slow. Let’s say you want to use the following rule:&nbsp;

As we explained in the previous section, the condition part is evaluated AFTER the strings.&nbsp;It would be nice if YARA evaluated the filesize first, and then it would search the text string and regular expression afterward. However, it is not the case. There is an ongoing discussion with the main author of YARA (<a href="https://github.com/VirusTotal/yara/discussions/1781#discussioncomment-3522238">https://github.com/VirusTotal/yara/discussions/1781#discussioncomment-3522238</a>) about a new approach to this problem, so there is hope for the future version of YARA, but only time will show.&nbsp;Currently and in this rule, both regular expression and text string are evaluated first. Atoms from them are selected, Aho-Corasick automaton is created and run, returning all possible matches. That means the slow regular expression will be searched in every file even if the filesize part of the condition is False. One slow string can slow down rules or even ruleset, so be careful what is matching. In the next post, we will bring more recommendations on how to optimize the matching of specific strings.&nbsp;&nbsp;This does not mean the filesize cannot improve your rules. If you have more complex conditions that work with slower modules (cuckoo, magic, …), the limitation of the filesize as the first part of the condition is a good idea. If the filesize part of the condition is not True, the short-circuit evaluation occurs, and the rest of the evaluation is stopped.&nbsp;&nbsp;&nbsp;<h2>Conclusion&nbsp;</h2>In this post, we explained how YARA works internally and one of the most common misunderstandings of rules evaluation.&nbsp;&nbsp;The main takeaways are: &nbsp;<ul><li>The condition part of the rules is evaluated after the strings section.</li><li>One slow string can slow down a whole rule or even a ruleset. </li><li>Filtering input data using filesize will only work if you want to skip evaluating the rest of the condition. It will not skip the strings matching part.&nbsp;&nbsp;</li></ul>And that is all for today! We wish you happy YARA rules writing!&nbsp;

Understanding YARA Rule Evaluation: Internals and Optimization

Know your YARA rules series: #1 know your YARA  

This blog post is based on some testing that I did some time ago. In my team at Avast, we are using Yara to its fullest potential, and even though we are satisfied with this tool overall, we’re constantly working on additional improvements (as mentioned in the previous posts, such as <a href="https://engineering.avast.io/making-yara-better-authenticode-net-telfhash/">Making YARA better: Authenticode, .NET, Telfhash</a>).&nbsp;One aspect that we are trying to improve is the scanning speed, as we believe that there is room for improvement. For these reasons, we decided to test some alternatives to reveal different, faster solutions that could be used in a similar fashion as Yara.&nbsp;There are many pattern-matching engines for regular expressions (for example, a quite nice list of them can be found on <a href="https://en.wikipedia.org/wiki/Comparison_of_regular_expression_engines">Wikipedia</a>). However, not all of them were created for very fast matching or matching of multiple patterns at once. This doesn’t mean that they aren’t useful, but there are differences when matching one pattern in one or few strings and when matching hundreds and thousands of strings in petabytes of data.In this blog post, we will be taking a look at HyperScan, as it is very fast and can match multiple patterns simultaneously. To be clear, it is not a direct substitute for Yara, but the promising results initiated a project that will be announced very soon. Stay tuned for more information!<h3>Regex Performance Tool</h3>I am far from the first to compare pattern-matching engines – there are plenty of papers and articles online. Still, the one I would like to mention the most is the inspirational article, <a href="https://rust-leipzig.github.io/regex/2017/03/28/comparison-of-regex-engines/">A comparison of regex engines</a>, written by Sascha Grunert in 2017. He also created a <a href="https://github.com/rust-leipzig/regex-performance">comparison tool</a> that has been extended and updated since then.For this blog post, I created a <a href="https://github.com/rust-leipzig/regex-performance/pull/15">PR</a> that is adding Yara to the tests, as I believe that Yara can be easily used outside of the world of malware analyses as well.I will describe that HyperScan is faster in most cases, but Yara also has quite a nice speed compared to other engines such as Tre, Boost, and others.<h5>Yara</h5>For Yara, the newest version from the <a href="https://github.com/VirusTotal/yara">upstream</a> repository, version 4.2.0, was used. Yara is probably well known to the readers of this blog, but in short, Yara is a tool written in C and used mostly for malware analyses. So-called rules are used: a description consisting of a set of strings and a boolean expression determining its logic. One example of the rule is the following:

This rule searches for any of the well-known names from Mark Twain’s Huckleberry Finn. The strings can be defined as text strings, regular expressions, and hexadecimal sequences. Condition is a boolean expression, and Yara provides functions for supporting functions, such as searching for mutexes or the number of matches, for a specific string.&nbsp;For matching, the Aho-Corasick automaton is used to search for substrings, and the bytecode engine then confirms the whole match. Additional information can be found in the official <a href="https://yara.readthedocs.io/en/stable/">documentation</a>.<h5>HyperScan</h5>HyperScan is a project by Intel written in C/C++. Also, as in the case of Yara, the newest version of HyperScan (5.4.0) from the official <a href="https://github.com/intel/hyperscan">repository</a> was used.The project was presented at the 16th USENIX Symposium on Networked Systems Design and Implementation in 2019, where the motivation and the main building blocks of HyperScan were described in detail. The recording is available on the conference <a href="https://www.usenix.org/conference/nsdi19/presentation/wang-xiang">website</a>.&nbsp;HyperScan was created with the goal of being able to match large numbers of regular expressions simultaneously while maintaining high performance. It uses hybrid automata techniques and a series of additional optimizations that allow for very fast matching. There are differences between HyperScan and Yara, as I will demonstrate in specific test cases. Additional information can be found in the official <a href="http://intel.github.io/hyperscan/dev-reference/">documentation</a>.<h3>Specifications</h3>I run tests locally on my VM Ubuntu 20.04 with 4GB of RAM. That means that overall numbers can be much lower using the better conditions, but I aimed to test the engines on limited resources purposely as opposed to the usage in a different situation than what is presented on the blog post and GitHub repository. For that reason, take the times with a grain of salt, and I will also mostly comment on the differences between HyperScan and Yara, rather than whether the numbers are high or low by themselves.&nbsp;The main idea of the tests remains the same as in the original post. The regular expressions are firstly parsed by engines and compiled into internal representation in both Yara and HyperScan. This step is not measured; we are focusing on the following stage – the matching itself. This step is run five times, after which the lowest time is selected. The tool provides the input file – a copy of Mark Twain’s&nbsp;Huckleberry Finn&nbsp;book.I run the regular expressions from the original implementation with expectations of the two last expressions. Yara does not support the regular expression <code>\p{Sm}</code> for any mathematical symbol, and the last regular expression <code>(.*?,){13}z</code> is also not supported because Yara does not allow the use of <code>.*?</code>, as these are too general of expressions. Yara is, in general, more cautious about matching general expressions – and it has a good reason for it, as we will see in a moment.The overall results are the following:

Based on these numbers alone, HyperScan was notably faster. However, I would like to go through some specific cases and discuss more differences and reasons behind these numbers.<h5>.{0,2}Tom vs Tom.{0,2}</h5>One of the most notable differences in results is the number of matches that are returned by our engines in the cases where the prefix or suffix does not have a fixed length.&nbsp;In the first case, <code>.{0,2}(Tom|Sawyer|Huckleberry|Finn)</code>, if we have input <code>xxTom</code>, Yara will find all interleaving matches: <code>Tom</code>, <code>xTom</code>, <code>xxTom</code>. This is caused by Aho-Corasick, which will report three potential starts of matches that are all confirmed. HyperScan, on the other hand, will report only the longest match, <code>xxTom</code>, as it is more focusing on the last byte of the match rather than the first one as Yara.This is also visible in the second example, where the suffix has a variable length. For the regular expression&nbsp;<code> (Tom|Sawyer|Huckleberry|Finn).{0,2}</code> and input <code>Tomxx</code>, Yara would match only the longest match <code>Tomxx</code>, but HyperScan would report all three matches: <code>Tom</code>, <code>Tomx</code>, <code>Tomxx</code>.&nbsp;These can be seen as minor differences, but it is good to keep in mind when using these tools that what’s fast in one tool can be slower in another due to differing evaluations.<h5>Too General Regular Expressions</h5>Yara also has a known problem when regular expressions are too general. In cases like&nbsp; <code>["'][^"']{0,30}[?!\.]["']</code>, it is not able to generate substrings for the first phase using the Aho-Corasick, which results in the checking every byte in the input files with a slower regex engine.

HyperScan purposely avoids this phase, as it views it as ineffective to match first a substring followed by the whole string again to confirm a match. Instead, they chose a different approach, which is one of the key elements where HyperScan truly shines. Rather than firstly matching a substring, it translates regular expressions into a series of strings and finite automata. With this change, the authors eliminate the redundancy in matching parts of the string twice, and they are also able to create smaller (and thus faster deterministic) automata. The second aspect of improving the scanning speed is scanning accelerations using SIMD operations that leverage the CPU’s compute capability on data parallelism. Both tactics are very effective, as the tests show how the results are vastly different.<h5>Yara can be fast as well</h5>

To be fair to Yara, there are two cases in which it was faster than HyperScan. In both of them (<code>\b\w+nn\b</code>, and <code>[a-q][^u-z]{13}x</code>), Yara used the first phase of matching to its advantage. It was firstly looking for <code>nn</code> for the first regular expression and <code>x</code> for the second, and after that, it checked if the rest of the expression also matches. HyperScan has a visible struggle to find all matches. Both of these regular expressions are very commonly present in the input file. While other cases took HyperScan just a few milliseconds, here, it took much longer to go through all of them. But in general, the HyperScan provides impressive speed in most cases.<h3>Conclusion</h3>There are several pattern-matching engines, many of which have much potential for fast scanning. Of course, many of them profile for a specific use and not all of them can match multiple patterns simultaneously or scan behavioral reports.To be clear, this post is not about criticizing Yara or any other engine. It is rather about testing available options and thinking about the possibilities for how to improve what we already have.As I mentioned before, my team is planning to release some exciting projects in the near future that were inspired by similar tests, so stay tuned for more.

Dominika Regéciová

Posts by Dominika Regéciová