Author at Avast Engineering

Our team invests countless hours in the exploration, discovery and investigation of emerging trends such as AI and futuristic technology. Stay informed about the evolving landscape of cyberthreats through our diligent efforts and insights.

Security Evangelist at Gen

Threat Operations Analyst at Gen

Research

A fake invoice in your email is easy to ignore. A fake invoice inside your order history feels different.&nbsp;Norton customers have reported fake Norton invoices appearing inside the Shop app, the shopping and order-tracking app from Shopify. Public reports suggest the same technique is not limited to Norton. Similar suspicious Shop app notifications have used McAfee, Apple gift cards, iPhones, PayPal-style payment claims and other high-value purchases as bait. The impersonated brand may change, but the mechanics are familiar: make the user believe they have been charged, then give them a phone number to call.&nbsp;That phone number is where the scam really begins.&nbsp;<h4>Why the Shop app makes this more believable&nbsp;</h4>Shop is a legitimate app. People use it to track packages, check receipts and receive order updates. According to Shop’s own documentation, the app can automatically track orders from several sources, including Gmail, Outlook, the email address linked to the user’s Shop account and orders paid through Shop Pay. If Gmail or Outlook is connected, Shop says it scans recent emails for keywords such as “tracking number” and “track your package” to populate deliveries in the Orders tab.&nbsp;

Shop also supports push notifications for delivery updates, payments and favorite products. So the first alert the victim sees may not be an email, an SMS or a calendar invite. It may be a notification from an app they already use for real purchases.&nbsp;That context matters. The scam text is not especially polished. Some of the examples we reviewed contain awkward wording and obvious signs of fraud. But when the message appears inside an order-tracking app, surrounded by real receipts and shipping updates, users may give it more credibility than it deserves.&nbsp;The screenshots reported to us show fake orders under generic seller names such as “My Store.” The receipts claim that a Norton or Norton LifeLock subscription has been activated or renewed, usually for several hundred dollars. The phone number appears in the product description, the receipt body or even the shipping address field.&nbsp;That is not how a normal receipt should look.&nbsp;<h4>The brand is interchangeable&nbsp;</h4>Norton is the example we saw through customer reports. Norton support now warns that scammers may abuse Android and iOS shopping platforms or applications to generate fake order notifications that appear to be legitimate Norton purchases. Those notifications may include fake order confirmations, fake charges, trusted third-party app delivery and fake support phone numbers embedded in order details.&nbsp;Public user reports point to a wider pattern. One Reddit user described receiving a Shop.app message about a transaction they did not recognize, with no follow-up email confirmation, and a message telling them to call support if the order was not placed by them. Another thread includes reports of unrecognized iPhone purchases under “My Store,” suspicious contact emails and users saying they saw no matching charges on their bank cards. Other reports mention fictitious Apple gift card purchases and iPhone orders using the same kind of wording: “If Order Not Placed By You,” followed by a phone number.&nbsp;The choice of brand is part of the social engineering. A fake security subscription charge plays on two fears at once: being billed for something you did not buy, and being left unprotected if you ignore it. Apple products and gift cards create high-value purchase panic. PayPal-style wording makes users worry about payment account abuse. In each case, the fake order is there to push the victim toward a call.&nbsp;<h4>What we know, and what we do not&nbsp;</h4>We have not found any evidence that any of the impersonated brands have been compromised.&nbsp;&nbsp;We also should not claim that Shop or Shopify was breached based on the evidence we have. The exact abuse path still needs confirmation. The fake order may be entering the app through a merchant/order workflow, through email parsing, through account association, or through another legitimate mechanism that scammers have found a way to misuse. Shop’s own documentation shows that orders can populate from multiple sources, so assuming one route would be premature.&nbsp;&nbsp;What we can say is narrower and stronger: scammers are placing fake purchase claims inside shopping app order experiences, where users are used to seeing real receipts and order updates. Norton’s own support page now describes this class of shopping app scam, and public reports show similar lures using other brands and products.&nbsp;&nbsp;That is enough to warn users without overstating the technical evidence.&nbsp;<h4>How the scam works&nbsp;</h4>The user receives a notification or sees a new order in the Shop app. The order looks like a receipt for something expensive: a security subscription, an Apple gift card, an iPhone, a MacBook or another high-value purchase.

The receipt says the charge has been processed, the subscription has renewed, or the order is being prepared. Somewhere in the order details, the scammer inserts a phone number, email address or support instruction.&nbsp;The wording is often clumsy. We have seen phrases like “If Order Not Place By You” and “If Need Help.” Those errors may look obvious when reviewing a screenshot. They are easier to miss when a phone notification tells you that hundreds of dollars have just been charged.&nbsp;If the victim calls, the scam moves off-platform. The person answering may claim to be billing support, Norton support, PayPal support or a cancellation department. The script can vary, but the request usually moves toward credentials, payment data, one-time codes or remote access software.&nbsp;By that point, the fake receipt has done its job. It has created enough urgency for the victim to enter a conversation controlled by the scammer.&nbsp;<h4>Why this is harder for users to judge&nbsp;</h4>Most people understand that email can be spam. They may still fall for phishing, but at least the inbox is a place where scams are expected.&nbsp;Order-tracking apps are different. Their purpose is to collect receipts, shipping updates and purchase information in one place. A notification from that environment can feel as if it has already passed through some layer of trust, even when the content inside the order is fraudulent.&nbsp;We have seen the same logic with calendar invite scams. The message itself is not always convincing, but the delivery channel changes how people read it. A fake invoice in an inbox is one thing. A fake invoice inside a calendar reminder or shopping app receipt can feel closer to a real event.&nbsp;For security products, this also creates a detection problem. The first visible alert may not be an email, SMS or malicious website. It may be a legitimate app notification carrying fraudulent text. The scam still depends on social engineering, but the delivery surface is harder to classify with the usual rules.&nbsp;<h4>What to do if you see one&nbsp;</h4>Do not call the number in the receipt.&nbsp;Check the charge directly. Open your bank or credit card app. Open the official account for the brand being impersonated. If it claims to be Norton, go to your Norton account or official Norton support. If it claims to involve PayPal, open PayPal directly. Do not use the phone number, email address or link inside the suspicious order.&nbsp;If there is no matching charge, treat the receipt as a scam attempt.&nbsp;Report the suspicious store or message in the Shop app if the option is available. Shop’s help pages say suspicious Shop stores can be reported from the store page through the app, and Shopify also tells users to forward phishing messages to its phishing inbox.&nbsp;&nbsp;Norton users can forward suspicious messages to Norton and should verify unexpected charges only through their Norton account or official Norton support channels.&nbsp;&nbsp;Opening the receipt does not automatically mean your device was compromised. The risk increases if you call the number, share information, install software or follow instructions from the person on the phone.&nbsp;<h4>What to do if you have already called&nbsp;</h4>End the call.&nbsp;If you shared payment details, contact your bank or card provider immediately using the number on your card or inside your banking app. If you gave away a password, change it from a different device if possible. Start with your email account, Apple ID or Google account, banking account and any account connected to the fake invoice.&nbsp;If you installed remote access software, disconnect the device from the internet and remove the software. Run a security scan before using the device again for banking, email or shopping.&nbsp;If you gave a one-time code to the caller, treat the related account as compromised. Change the password, review recovery options and check recent login activity.&nbsp;<h4>The lesson&nbsp;</h4>Scammers keep testing new places to put old tricks. For years, fake invoices were mostly an email problem. Then similar subscription notices appeared in calendar invites. Now we are seeing fake receipts inside shopping app order histories and push notification flows.&nbsp;The safest response is to separate the alert from the action. If an app, email, text message or calendar invite says you have been charged, do not use the contact details inside that alert. Go directly to the account or payment provider yourself. Check whether the charge exists. Use official support channels.&nbsp;The fake receipt, the fake support number and the fake emergency are all part of the same controlled environment. The moment you verify the charge somewhere else, the scam loses most of its power.&nbsp;

Scammers are using order-tracking apps to place fake receipts where users expect to see real purchases, then pushing them to call fake support numbers. 

Fake invoices are moving from inboxes to shopping apps 

Threat Researcher at Gen

<h2>Introduction&nbsp;</h2>Infostealers are constantly evolving, and so are the techniques they use to bypass <a href="https://security.googleblog.com/2024/07/improving-security-of-chrome-cookies-on.html" target="_blank" rel="noreferrer noopener">Application-Bound Encryption (ABE)</a>. In recent weeks, Vidar has been among the most actively developed stealers and, apart from <a href="https://x.com/GenThreatLabs/status/2064836539952361976" target="_blank" rel="noreferrer noopener">multiple updates to its string obfuscation</a> and <a href="https://x.com/GenThreatLabs/status/2061462524604649840" target="_blank" rel="noreferrer noopener">a reworked approach to protecting its configuration</a>, it has also introduced a novel technique for bypassing ABE. And while there have been many other changes in Vidar lately, with new versions dropping every week, in this blog post we focus solely on the ABE bypass and its technical aspects.&nbsp; &nbsp;<h2>The ABE Bypass&nbsp;</h2>From a high-level perspective, Vidar’s approach to bypassing ABE is somewhat similar to that of Remus/Lumma, in that both try to extract the <code>v20_master_key</code> directly from the browser’s memory. What sets them apart, however, is how they achieve it. Since we’ve already covered the inner workings of ABE in earlier posts (<a href="https://www.gendigital.com/blog/insights/research/remus-64bit-variant-of-lumma-stealer" target="_blank" rel="noreferrer noopener">Remus</a>, <a href="https://www.gendigital.com/blog/insights/research/voidstealer-abe-bypass" target="_blank" rel="noreferrer noopener">VoidStealer</a>), we don’t revisit them here and focus instead on the bypass itself. For readers interested in the broader background, we encourage checking those earlier posts.&nbsp;&nbsp;In short, the end goal of infostealers is to obtain the <code>v20_master_key</code>, as it alone is sufficient for decrypting any ABE-protected data tied to a specific application. This key can be obtained in one of two ways: either from disk, where it is protected by two layers of <code>CryptProtectData</code> (one applied in the context of the logged-in user and one as <code>NT AUTHORITY\SYSTEM</code>), or from the browser’s memory – the direction Vidar took.&nbsp;In memory, however, the key is kept protected with <code>CryptProtectMemory</code> using the <code>CRYPTPROTECTMEMORY_SAME_PROCESS</code> flag. This ensures that even if an attacker manages to read the encrypted form of the key from memory, they cannot simply decrypt it (they would need to invoke the complementary <code>CryptUnprotectMemory</code> function directly from within the browser process). That leaves attackers with two problems to solve: how to find the encrypted key in memory, and how to get it decrypted.&nbsp;<h2>Locating the v20_master_key in Browser’s Memory&nbsp;</h2>Vidar’s approach to locating the encrypted <code>v20_master_key</code> in memory involves scanning the entire memory of a running browser for a specific pattern. But that first requires a browser process to scan.&nbsp;When the target browser is already running, Vidar reuses the existing process. However, it doesn’t read its memory directly. Instead, it creates a fork of it using <code>NtCreateProcessEx</code> with <code>SectionHandle</code> set to <code>NULL</code>, instructing the kernel to clone the address space of the process specified by the <code>ParentProcess</code> as a copy-on-write mapping. The resulting process has no threads and is never resumed – it exists purely as a static memory snapshot.&nbsp;&nbsp;To perform the fork, Vidar first opens a handle to the target browser via <code>OpenProcess</code> with <code>PROCESS_QUERY_INFORMATION | PROCESS_CREATE_PROCESS | PROCESS_VM_READ</code>, falling back to <code>PROCESS_CREATE_PROCESS</code> if the initial call fails. It then passes this handle as the <code>ParentProcess</code> argument to <code>NtCreateProcessEx</code>, requesting <code>PROCESS_ALL_ACCESS</code> on the resulting child. If the fork fails, Vidar falls back to scanning the live browser process directly.&nbsp;

If no target browser process is found running, Vidar creates a new isolated desktop via <code>CreateDesktopA</code>, assigns it a name of the form <code>v20_%d</code> (where <code>%d</code> is replaced with the browser’s index in Vidar’s configuration), and launches the target browser in there. In the case of Vidar version 2.1, the browser is launched with the command line <code>--no-first-run --disable-gpu about:blank</code>, although the exact command line varies across versions. Since modern browsers use a multi-process architecture, there are typically multiple processes matching a given browser. Vidar enumerates all of them (up to 64) and applies the fork-and-scan procedure to each one independently.&nbsp;

With a handle to the forked process in hand, Vidar proceeds to enumerate its virtual memory regions using <code>NtQueryVirtualMemory</code>. It iterates through the entire browser’s address space, collecting regions that match a specific profile: committed (<code>MEM_COMMIT</code>), private (<code>MEM_PRIVATE</code>), and either readable (<code>PAGE_READONLY</code>) or read-write (<code>PAGE_READWRITE</code>). Vidar records up to 4096 memory regions matching these requirements, along with their base addresses and sizes. The actual pattern scanning is then performed in parallel with up to 64 worker threads, depending on the system.&nbsp;&nbsp;

Each thread scans its assigned regions using <code>NtReadVirtualMemory</code>, searching for a 32-byte pattern. When a match is found, the thread reads 8 bytes at offset +32 from the match, interprets them as a pointer, and validates that it points to committed, private, and readable memory (the expected memory state for the encrypted <code>v20_master_key</code>).&nbsp;&nbsp;

This naturally raises the question of what pattern Vidar searches for. The answer is a 32-byte signature: <code>76 32 30 00 ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 03 00 00 00 00 01 ?? ?? ??</code>, where <code>??</code> denotes a wildcard (Vidar uses the byte <code>AA</code> as the wildcard marker in its internal representation). Notably, the first four bytes correspond to the ASCII <code>v20\x00</code>, which is a common prefix used for ABE-related data.&nbsp;&nbsp;

We assess with high confidence that this pattern is intended to match a node structure within Chromium’s <code>Encryptor::KeyRing</code>, the structure that holds the keys used by the <code>Encryptor</code> class. Now, this part is admittedly a bit tricky as it relies on several assumptions we made about the internal memory layout, as well as on implementation-specific representations of various C++ standard library types <code>(std::map</code>, <code>std::string</code>, <code>std::vector</code>, <code>std::optional</code>), which is why we frame this as a high-confidence rather than a certainty.&nbsp;&nbsp;Our reasoning is as follows: <code>KeyRing</code> is <a href="https://github.com/chromium/chromium/blob/cf5d502bcaedb22927e3829a18be1ff6d8b6f598/components/os_crypt/async/common/encryptor.h#L132" target="_blank" rel="noreferrer noopener">defined</a> as <code>std::map&lt;/*tag=*/std::string, std::optional&lt;Key&gt;&gt;</code>, and <code>std::map</code> is typically implemented as a red-black tree, where each tree node stores three pointers (left child, parent, and right child) followed by the key-value pair. In this case, the pair is in the form of <code>std::pair&lt;/*tag=*/std::string, std::optional&lt;Key&gt;&gt;</code>, meaning that the <code>std::string</code> and the <code>Key</code> object might be laid out consecutively in memory within each node. For a <code>v20_master_key</code> entry, the tag corresponds to the string <code>v20</code>, which at 4 bytes including the null terminator fits within the small string optimization (SSO) buffer and is therefore stored inline rather than heap-allocated. The <code>Key</code> class that immediately follows <a href="https://github.com/chromium/chromium/blob/cf5d502bcaedb22927e3829a18be1ff6d8b6f598/components/os_crypt/async/common/encryptor.h#L93-L98" target="_blank" rel="noreferrer noopener">contains three member variables</a>: <code>std::optional&lt;mojom::Algorithm&gt; algorithm_</code>, <code>std::vector&lt;uint8_t&gt; key_</code>, and <code>bool encrypted_ = false</code>.&nbsp;Putting it all together and applying it to a concrete example from a browser memory dump, we get the following layout:&nbsp;

Please note that while the exact layout may contain minor inaccuracies on our part, the overall structure should be roughly correct. For easier orientation, the following diagram illustrates how the individual structures are nested within each other:&nbsp;

At this point, it should be clear how the scan pattern maps onto the searched structure. The 32-byte match covers the <code>std::string</code> tag <code>v20\x00</code> and the beginning of the <code>Key</code> object up to and including <code>algorithm_</code>. The pointer at offset +32 from the start of the match then lands precisely on <code>key_.begin</code>, the first member of <code>std::vector&lt;uint8_t&gt;</code> that holds the encrypted <code>v20_master_key</code>.&nbsp;Matches from all threads are collected into a shared buffer and subjected to a majority-voting scheme: candidates consisting mostly of zeroes are discarded, the rest are pairwise-compared, and each is scored by the number of identical copies found. This voting is performed independently for each browser process, producing one best candidate per process.&nbsp;<h2>Decrypting the v20_master_key via APC Injections&nbsp;</h2>For each browser process that produced a match, Vidar now holds a candidate address pointing to a potential encrypted <code>v20_master_key</code>. However, as already noted, since the key is protected with <code>CryptProtectMemory</code> using <code>CRYPTPROTECTMEMORY_SAME_PROCESS</code>, it can only be decrypted by invoking <code>CryptUnprotectMemory</code> from within the browser process itself. Vidar solves this by injecting an Asynchronous Procedure Call (APC) into the live browser process.&nbsp;Vidar supports two APC injection methods, selected based on whether ESET or Bitdefender is detected on the system. If either of these products is present, Vidar uses the first method, which it refers to as “classic” (based on decrypted strings) – it creates a suspended thread in the target browser via <code>CreateRemoteThread</code> (with <code>NtTestAlert</code> as the thread start address and the <code>CREATE_SUSPENDED</code> flag), queues an APC using <code>NtQueueApcThread</code>, resumes the thread, and then waits for it to complete via <code>WaitForSingleObject</code> with a randomized timeout. When the thread resumes, it calls <code>NtTestAlert</code>, which flushes the APC queue and executes the injected routine.&nbsp;

If neither product is detected, it uses the “special” method – it finds an existing thread in the target browser using a combination of <code>CreateToolhelp32Snapshot</code> and <code>Thread32First/Thread32Next</code>, opens it via <code>NtOpenThread</code>, and calls <code>NtQueueApcThreadEx2</code> with the <code>QUEUE_USER_APC_FLAGS_SPECIAL_USER_APC</code> flag. Unlike regular APCs, special user APCs execute immediately. Therefore, the target thread does not need to be in an alertable wait state.&nbsp;

In both cases, the APC routine to be executed is <code>CryptUnprotectMemory</code>, called with the following three arguments: the address of the encrypted <code>v20_master_key</code> candidate, its size (32 bytes), and <code>CRYPTPROTECTMEMORY_SAME_PROCESS</code> for the <code>flags</code> parameter.&nbsp;&nbsp;Since <code>CryptUnprotectMemory</code> decrypts the data in place, once the APC executes, the key is decrypted directly in the live browser's memory. To read it, Vidar creates a second fork of the browser process and reads the key from the same address using <code>NtReadVirtualMemory</code>. It then compares the result against a snapshot of the encrypted key taken from the first fork before the APC call – if the data is unchanged, decryption failed, and Vidar moves on to the next browser process’s candidate.&nbsp;If the data differs, Vidar uses the same second fork to perform an additional verification: it scans the forked process’s memory for the byte sequence <code>v20</code> and, treating each match as a potentially ABE-encrypted entry (cookie, password), attempts AES-256-GCM decryption with the key candidate using <code>BCryptDecrypt</code> (successful GCM tag verification confirms that the key is correct).&nbsp;Whenever the data has changed (indicating that decryption of the key took place), Vidar re-encrypts the key in the browser’s memory by injecting another APC with <code>CryptProtectMemory</code> as the routine, preserving the browser’s state. If no candidate produced a successful decryption (i.e., the data remained unchanged after every APC call), Vidar takes a more aggressive approach: it terminates all browser instances, restarts the browser, and repeats the entire scan-decrypt-verify cycle.&nbsp;By abusing APC injections instead of the more traditional approach of injecting code via <code>NtWriteVirtualMemory</code> that performs the decryption, Vidar trades one suspicious behavior for another. APC injections are arguably less common and may thus slip past some detection systems, though they are still well-known techniques monitored by AV and EDR solutions. But that is the nature of ABE bypasses – each has its own advantages and disadvantages, and that’s exactly why infostealers keep inventing new ones. &nbsp;<h2>Indicator of Compromise (IoC)&nbsp;</h2>459daa809751e73f60fbbe4384a7d1653c36bb06945e4eb3635270924241100a (Vidar 2.1)&nbsp;

A Technical Walkthrough of How Vidar Defeats Application-Bound Encryption

Inside Vidar’s ABE Bypass: From Memory Scanning to APC Injections

You have a summer flight booked. You have seen the headlines about jet fuel shortages, cancelled routes and higher travel costs. Then a text message arrives: your flight has been affected, and you need to click a link to confirm your refund or rebook before the offer expires.&nbsp;That message feels plausible because the situation around it is real. In April, <a href="https://www.iata.org/en/pressroom/2026-releases/2026-04-17-01/" target="_blank" rel="noreferrer noopener">IATA warned</a> that Europe could start seeing flight cancellations by the end of May if jet fuel shortages materialized. Airlines and governments have since been trying to reassure travelers, while also publishing guidance on passenger rights, refunds and re-routing.&nbsp;&nbsp;That is exactly the kind of confusion scammers like. They do not need to cause the disruption, they only need to arrive at the right moment, with the right brand logo and the right sense of urgency.&nbsp;<h4>Why travel disruption works so well as a scam hook&nbsp;</h4>Most scam messages fail because they arrive out of context. A random refund email from an airline you never use is easy to ignore. A refund message from the airline you are flying with next week, during a period of public travel disruption, is different.&nbsp;The scammer’s job is to make you think: “This could be real.”&nbsp;That can happen in several ways. You might receive a fake cancellation email that links to a refund form. You might search for an airline’s customer service number and land on a fake support page. You might post publicly on social media asking an airline for help, then receive a reply from an account that looks official. Or you might get a WhatsApp message that knows your hotel name, travel dates or booking reference.&nbsp;Each version uses the same pressure point: you are worried about losing money, losing your flight or being stranded.&nbsp;Criminals are already using disruption this way. In March, <a href="https://www.theguardian.com/money/2026/mar/15/travel-scam-airline-accounts-fake-refunds-iran-war-flight-disruption" target="_blank" rel="noreferrer noopener">The Guardian reported</a> that fake airline support accounts were targeting travelers affected by Middle East flight disruption. Some victims were looking for help or refunds, then received replies from accounts pretending to represent airlines. The scam moved into direct messages, where victims were asked for details and sent links that led to money being taken instead of refunded.&nbsp;&nbsp;The jet fuel story gives scammers another pretext. A message can claim that your flight was cancelled, your fare changed, your fuel surcharge failed, or your refund must be claimed within a few hours. None of those claims needs to be true to be effective.&nbsp;<h4>The scam may start before the message arrives&nbsp;</h4>A fake travel message becomes much harder to spot when it includes real information.&nbsp;This is the idea behind <a href="https://www.gendigital.com/blog/insights/research/reservation-hijack-scam" target="_blank" rel="noreferrer noopener">Reservation Hijack scams</a>, a form of targeted phishing where cybercriminals use real booking details to make a message look legitimate. In some cases, attackers may use information exposed through breaches or compromised travel systems. In other cases, they collect details from emails, social media posts, previous leaks or fake customer service interactions. Gen has previously described Reservation Hijack scams as scams that use real hotel booking details to make fraudulent payment or verification requests feel authentic.&nbsp;&nbsp;That distinction matters. A scam message does not need your credit card number to fool you. Knowing your name, your destination, your travel dates and your booking platform may be enough.&nbsp;<h4>What these scams can look like&nbsp;</h4>A travel disruption scam usually does not begin with a strange offer. It begins with something boring and administrative. That is part of the trick.&nbsp;You may see messages like:&nbsp;“Your flight has been cancelled due to fuel supply issues. Claim your refund here.”&nbsp;“Your rebooking is pending. Confirm passenger details within two hours.”&nbsp;“Additional airport tax or fuel surcharge is required to keep your booking.”&nbsp;“Your hotel reservation will be cancelled unless payment is verified.”&nbsp;“Due to high call volume, please continue this request on WhatsApp.”&nbsp;The exact wording changes, but the structure is familiar. There is a problem, a deadline and a link. The message may also move you away from the official channel into SMS, WhatsApp, Telegram, direct messages or a phone call. Once that happens, the scammer controls the conversation.&nbsp;There is another detail worth watching: some messages do not ask for money immediately. They start by asking for your booking reference, last name, phone number or one-time password. That can still be dangerous. Those details can help scammers access a booking, impersonate you to customer support, reset an account or make the next message more convincing.&nbsp;<h4>Your real passenger rights can be used against you&nbsp;</h4>If an airline cancels your flight, you may have rights to a refund, re-routing or assistance, depending on where you are travelling and the rules that apply. The European Commission said passengers affected by cancellations continue to have rights to reimbursement, re-routing or return, airport assistance and compensation for last-minute cancellations in relevant circumstances. It also said airlines cannot retroactively add fuel surcharges to already published final ticket prices.&nbsp;&nbsp;That information is useful for travelers, but it is also useful for scammers.&nbsp;A fake refund message becomes more believable when people already know refunds are possible. A fake rebooking link becomes more believable when airlines are genuinely changing schedules. A fake “fuel surcharge” demand becomes more dangerous when fuel prices are in the news, even though legitimate retroactive charges may not be allowed in many cases.&nbsp;This is why the safest route is not to judge the message only by whether the story sounds plausible. During travel disruption, many scam stories will sound plausible. Judge the channel instead.&nbsp;<h4>How to check before you click&nbsp;</h4>Start from the airline or booking platform’s official app or website. Do not use the link in the message. Open the app yourself, type the website address yourself, or use the contact details listed on your original booking confirmation.&nbsp;Check whether the same cancellation, refund or payment request appears inside your account. If the message is real, the airline or hotel should be able to confirm it through an official channel.&nbsp;Be careful with social media support. If an account replies to your public complaint, check whether it is linked from the airline’s official website. Look at the handle carefully. Fake support accounts often copy logos and names, but have small differences in spelling, low follower counts or recently created profiles. The Guardian reported that fraudsters often scan social media for passengers trying to contact legitimate airline accounts and reply to them first, which makes the scam harder to spot.&nbsp;&nbsp;Do not share one-time passwords, payment card details or login credentials with anyone who contacts you unexpectedly. A real airline or hotel does not need your banking OTP to send you a refund.&nbsp;Be especially suspicious of pressure. “You have 30 minutes to claim your refund” is not customer service. It is a tactic.&nbsp;<h4>What to do if you already clicked&nbsp;</h4>If you entered payment details, contact your bank or card provider immediately. Ask whether the transaction can be blocked or reversed, and request a new card if needed.&nbsp;If you shared login details for an airline, hotel or booking platform, change the password from the official site or app. Use a unique password, not one you use elsewhere. Turn on two-factor authentication where it is available.&nbsp;If you shared a one-time password or authentication code, assume the account may have been accessed. Check your booking details, payment methods and recent account activity.&nbsp;Report the scam to the airline, hotel or booking platform being impersonated. If the scam came through social media, report the account as impersonation. If money was taken, report it to the relevant fraud authority in your country.&nbsp;<h4>The real lesson for summer travel&nbsp;</h4>Travel disruption changes how people behave. They search for support numbers in a hurry. They post publicly because customer service queues are long. They accept strange instructions because they are worried about losing a flight or refund.&nbsp;Scammers build around that behavior.&nbsp;The safest assumption is simple: if a message about your flight, hotel or refund creates urgency and gives you a link, pause. Go back to the official source. A real cancellation will still be there when you check through the airline or booking platform. A scammer needs you to act before you do.&nbsp;

Travel disruption gives scammers the one thing they need most: a believable reason to rush you

Your flight was cancelled. Is the refund message real? 

A fake job interview page is easier to trust than a fake ticket shop.In samples reviewed by Gen threat researchers, scammers used branded hiring pages that looked like ordinary recruitment flows: a company logo, a recruiter profile, a 30-minute meeting, and a button to continue with Google or Facebook. There was no attachment to open and no software to install. The credential theft attempt sat inside a step many job applicants already expect: signing in to schedule a call.FIFA is an obvious lure ahead of the 2026 World Cup. On May 22, ESET reported fake FIFA-themed websites that copied ticket and merchandise flows to steal money and personal data. Our researchers later found another FIFA-themed domain, hxxps://fifa-mg[.]shop, which followed a naming pattern not mentioned in that report. Domains like these can be short-lived, often appearing for hours or days before they are blocked, abandoned, or taken down.Our researchers also found a different kind of FIFA lure: hxxps://fifahiring[.]com, a fake hiring page that encouraged users to schedule a call with an alleged FIFA recruiter. When a user tried to register with a personal Gmail address, the page asked for a business email instead. The page was steering victims toward credentials with more value: accounts that could provide access to a company environment.

<h4>The FIFA hiring page did not point to FIFA</h4>The fake page looked believable at first glance.

It used FIFA branding, presented a recruiter identity, and copied parts of the experience users see on legitimate recruiting and scheduling platforms. The recruiter’s name and photo appeared to match a real LinkedIn profile, which made the page feel more credible. For publication, we should keep that person’s identity redacted.The domain did not match FIFA’s real hiring infrastructure. FIFA’s legitimate careers pages are jobs.fifa.com and fifa.pinpointhq.com. The fake page used neither. It also loaded the favicon from the real FIFA recruitment site hosted at pinpointhq.com, giving the page a small visual cue borrowed from the legitimate service.Other signals pointed in the same direction. The domain and certificate were newly created, from May 23. The site used commodity hosting infrastructure, including Vercel and Render, rather than a corporate environment. Those signs are not enough on their own, but they fit the rest of the evidence.The login flow was the clearest part. Credentials were posted to:fifeq2026eqbackeq[.]onrender[.]com/api/loginThe Google sign-in prompt was not a real Google authentication window. It was rendered inside the page itself. The visible controls and links did nothing, and the page did not send the user into a genuine Google sign-in flow. According to the analysis, the only Google asset loaded was the favicon, apparently to make the fake window look familiar.A page can look like it opened a Google login without ever handing control to Google.Researchers also observed hxxps://careers-fifahiring[.]com, which later became inactive, and hxxps://fifajobs[.]com, which showed the same content and was indexed by Google.&nbsp;<h4>The recruitment lure went beyond FIFA</h4>The FIFA page was one version of a wider recruitment phishing pattern. The brand can change. The flow stays familiar: recruiter profile, meeting prompt, business email, fake sign-in.Researchers found similar fake hiring pages abusing other major brands, including Heineken, Hilton, Coca-Cola, Netflix, PepsiCo, Delta,&nbsp;and Spotify. In the Heineken case, the page appeared to use the identity of someone who, according to LinkedIn, really works as a recruiter at the company.

Additional hunting found two other recruitment-themed kits:hxxps://aquent-talent-jobs[.]com hxxps://hays-talent-opportunities[.]comThe Hays-themed version targeted Facebook credentials instead of Google credentials. Researchers assessed that the Aquent and Hays pages shared code, although they differed from the earlier FIFA examples.

This flexibility makes the campaign harder to frame as a single World Cup scam. FIFA provides the timely hook, but the mechanism is recruitment phishing. A scammer can swap the logo, recruiter name, and sign-in provider while keeping the same basic path to the victim’s credentials.<h4>Why fake recruitment pages work</h4>Recruitment gives attackers a clean excuse to ask for things that might otherwise raise suspicion.A candidate expects to click a scheduling link. They expect a recruiter profile. They expect a calendar prompt. In some cases, they may also expect to use a work email, especially for corporate, contractor, or partner-facing roles.The fake login window turns that ordinary flow into credential theft. Users are used to “Continue with Google” and “Continue with Facebook” buttons, but the button itself proves nothing. A genuine authentication flow should take the user to the provider’s domain and controls. A fake page can draw a convincing copy while keeping the victim inside the attacker-controlled site.The business email prompt is also a warning sign. A page that rejects a personal email and asks for a work address may be trying to raise the value of the stolen account.<h4>What job seekers should check before signing in</h4>Start from the company’s official careers page, not from a link in a message, ad, search result, or forwarded post. For FIFA, that means checking jobs.fifa.com or fifa.pinpointhq.com, not trusting a domain because it contains “fifa,” “career,” “hiring,” or “jobs.”Look closely at the domain before entering credentials. Scam pages often add plausible hiring words around a brand name: career, talent, jobs, hiring, portal, or connect. A polished page on a new domain is still a new domain.Treat the sign-in step as the real checkpoint. If a page offers Google or Facebook login, check whether the flow actually opens a real provider-controlled authentication page. A fake window embedded inside the original site, broken controls, inactive links, or a provider logo used only as decoration should stop the process.Use security software that includes web and phishing protection. These tools can block known malicious pages before they load, which is especially useful when scam domains look polished, appear in search results, or are shared through otherwise legitimate channels.<h4>What companies can do</h4>Recruitment pages are public by design, which makes them easy to copy.Security teams should monitor for newly registered domains combining their brand with hiring-related terms. They should also watch for pages that reuse favicons, recruiter photos, job descriptions, and calendar-booking language from legitimate hiring flows.HR and recruiting teams should make the official hiring path easy to verify. Candidates should not have to guess whether a domain is legitimate. A clear careers page, consistent recruiter communication, and warnings about unofficial scheduling links can remove some of the ambiguity attackers rely on.<h4>Conclusion</h4>The fake FIFA hiring page worked because it did not feel unusual. It looked like a normal step in a hiring process.Attackers do not need to invent new behavior when they can copy one people already trust. A branded scheduling page, a recruiter profile, and a familiar sign-in button can be enough to move a victim from interest to credential entry.The decision point is the login. Before entering a work email or password, check the domain, check where the authentication flow really goes, and verify the role through the company’s official careers site. Security software with web and phishing protection adds another layer, especially against short-lived scam pages that may only stay online for hours or days.IOCs:hxxps://fifahiring[.]com hxxps://careers-fifahiring[.]com hxxps://fifajobs[.]com hxxps://fifa-mg[.]shop hxxps://heineken-careers[.]com hxxps://aquent-talent-jobs[.]com hxxps://hays-talent-opportunities[.]com fifeq2026eqbackeq[.]onrender[.]com/api/login

Scammers are copying recruitment and calendar-booking flows to make fake Google and Facebook sign-ins look routine. 

Fake hiring pages abuse FIFA and other major brands to steal work credentials 

Threat Research Team Lead

Since January 2026, we have detected 7.4 million malicious incidents tied to residential proxy traffic, affecting 572,000 users in our telemetry. In this model, the household whose connection is being used is often not the buyer of the traffic. It is the exit point.That is the uncomfortable part of residential proxy networks. A home router, laptop, phone, smart TV, or other always-on device can quietly become part of a commercial network that routes strangers’ internet traffic through ordinary consumer IP addresses. To the websites on the other end, that traffic looks as if it came from a real household, because it did.The technology itself is not inherently malicious. Businesses use residential proxies for market research, ad verification, brand protection, security testing, and services that need to see the web as a real user would. But the same feature that makes these networks useful also makes them attractive for abuse: traffic from a household IP address is hard to block without risking false positives against legitimate users.The problem for consumers is simpler. Many people do not realize their device or connection can be enrolled into this kind of network at all. Some users opt in through pay-for-bandwidth apps. Others are enrolled through “free” VPNs, bundled software development kits inside unrelated apps, or unwanted software that installs proxy components without meaningful consent.Here is how the market works, who pays for it, and what it can cost the household that unknowingly hosts it.<h2>What is a residential proxy</h2>A residential proxy is an internet connection – typically a home laptop, phone, smart TV, or router – that has been enrolled into a third-party network and is being used to route someone else's traffic. To the rest of the internet, that traffic looks like it is coming from an ordinary household IP address assigned by a consumer ISP, not from a datacenter or&nbsp;VPN provider.Residential proxy networks are sold commercially. The model is two-sided: some people pay to tunnel their traffic out through someone else's connection (much like a VPN, except the exit point is a real home), while others get paid to provide those exit endpoints. Some are structured as paid peer-to-peer networks, where any participant's connection can serve as an exit for any other.Buyers pay for access to pools of millions of these "real user" IPs because they are extremely difficult to block, or rather, blocking them carries real consequences: the traffic originates from genuine households, so any block risks shutting out a legitimate user – effectively a false positive.Common buyers include market research firms, price-comparison services, brand-protection vendors, security teams testing their own defenses, and consumer privacy tools. At the abusive end of the market, the same infrastructure used by operators running automated purchasing bots, scrapping campaigns, ad fraud, credential stuffing, phishing and other attacks where hiding the true origin of the traffic matters. Demand is also being shaped by large-scale web scraping, including&nbsp;<a href="https://blog.includesecurity.com/2026/06/the-smart-tv-in-your-livingroom-is-a-node-in-the-aiscraping-economy/">scraping for AI training data</a>. As anti-bot systems become better at identifying traffic from datacenters and known VPN providers, residential IP addresses become more attractive because they look like ordinary users. That does not make every residential proxy user malicious. It does mean consumers may be carrying traffic for businesses and actors they have never heard of.<h2>How consumer devices end up in these networks</h2>There are several ways a home device can become part of a residential proxy network. Some are explicit. Others are easy to miss.The most obvious route is the pay-for-bandwidth model. Apps such as Honeygain, IPRoyal Pawns, and PacketStream pay users to share their internet connection. The user opts in, but they may not understand who will buy access to that connection later, what kind of traffic will pass through it, or what happens if that traffic triggers abuse reports.A second route is the “free” VPN or proxy app. Instead of charging a subscription, the app monetizes the user’s connection by reselling bandwidth. In practice, the user may think they are getting a privacy tool while also becoming infrastructure for other people’s traffic.A third route is through bundled software development kits, or SDKs. A developer adds a third-party component to an otherwise unrelated app, such as a flashlight, wallpaper, game, or utility. The developer gets paid when the app is active. The user sees the host app, not the proxy network behind it.The fourth route is malware or potentially unwanted software. In those cases, the proxy component is installed silently or deceptively, with no meaningful user consent at all.<h2>Why the "consent" is rarely meaningful</h2>Even when consent technically exists, it is often not the kind of consent most people would recognize.The disclosure may be buried in a long terms-of-service document. The wording may be vague, with phrases such as “share your unused bandwidth,” “support the app,” or “join our network.” These phrases do not clearly tell the user that arbitrary third parties may route traffic through the user’s home IP address.There is also the question of downstream use. Users are rarely told who buys access to the network or what those buyers do with it. Their connection could be used for benign market research. It could also be used for scraping, automated account creation, ad fraud, credential stuffing, or phishing infrastructure.The problem gets worse when the proxy component sits inside another app. The host app’s privacy policy may mention a partner SDK by brand name, but that still does not make it clear to a normal user that their device can become a proxy endpoint.Uninstalling the visible app may not always resolve the issue cleanly either, especially when proxy components are installed separately or bundled with other unwanted software.<h2>The risks consumers actually carry</h2>When a household device becomes a residential proxy node, the user takes on risks created by someone else’s traffic.The first risk is attribution. Abusive, fraudulent, or illegal activity routed through the device appears to come from the household’s IP address. ISP abuse notices, platform bans, fraud checks, and law enforcement requests point first to the subscriber, not to the person who bought the proxy traffic.The second risk is performance. Continuous third-party traffic can consume bandwidth, slow the home network, hit data caps, or violate ISP terms of service. The user may only notice that the internet feels slower, a bill is higher, or a service suddenly blocks access.The third risk is security exposure. Proxy clients run persistent background processes and maintain network connections so they can relay traffic. That adds always-on networking software to the device, often with limited visibility for the user.The fourth risk is reputation. A home IP address used for abuse can end up on blocklists used by banks, retailers, streaming services, and online platforms. The result can be account friction, failed logins, extra verification steps, or outright blocking from services the household actually uses.There is also a family angle. A child or another household member may install a free game, VPN, or utility without understanding the consequences. The legal, financial, and reputational risk still lands on the account holder.<h2>What our telemetry shows</h2>Since the beginning of 2026, we have detected 7.4 million malicious traffic incidents associated with residential proxy traffic, affecting 572,000 unique users in our user base.&nbsp; This is not a measurement of the entire residential proxy market. It reflects what we observe in our telemetry: malicious traffic, detections, and related activity affecting protected users. The real footprint is far larger, because residential proxy networks span major consumer platforms and many devices outside our visibility.The activity is global, but it is concentrated in a smaller set of countries. India leads by a wide margin, with roughly 83,000 affected users, followed by Vietnam with 61,000 and Brazil with 39,000. The Philippines (23,000), Poland (18,000), Indonesia (18,000) and Thailand (14,000) form a second tier, with Egypt, Spain, Ukraine, Argentina, Mexico and Italy each accounting for 10,000 to 13,000. The spread across South and Southeast Asia, Latin America, and Eastern and Southern Europe reflects where the host apps that carry this software are most widely installed.That geographic spread likely reflects a combination of app distribution, local install patterns, user appetite for free or pay-for-bandwidth apps, and the economics of bandwidth resale. It should not be read as a simple ranking of where residential proxy abuse originates.The chart below breaks down the threats we observed in this proxy-related traffic. Two categories dominate: phishing and malicious advertising. Behind them sits a long tail of consumer-facing fraud, including e-shop scams, generic scams, financial scams, and fake-tutorial pages, alongside outright malware such as trojans and other malicious traffic.&nbsp;

In the malicious proxy traffic we observed, scams and malware dominate. The important point for consumers is not only that their connection may be used by someone else. It is that, when abuse happens, their household IP address can be the return address.<h2>What consumers can do</h2>Most people will not identify a residential proxy component by name. But there are practical warning signs.Review “free” VPNs, proxy apps, bandwidth-sharing apps, and utilities from unknown publishers. Check whether they mention bandwidth sharing, peer-to-peer networking, traffic routing, or network participation in their terms. If an app offers payment for leaving your device online, assume your connection may be resold unless the provider explains exactly how traffic is vetted and who can buy access.On phones and computers, remove apps you do not recognize or no longer use. On home routers, check the list of connected devices and look for anything unfamiliar. If your ISP sends abuse notices, websites start blocking you, or your home IP address suddenly fails fraud checks, a proxy component is one possible explanation.Security software can also help by detecting unwanted proxy clients, bundled components, and related malicious activity. But this should not be a guessing game for consumers. Software that turns a household connection into a commercial exit node should be named clearly and made easy to remove.<h2>The takeaway&nbsp;</h2>With residential proxiesthe user’s connection becomes an infrastructure for someone else’s business.The issue is not that residential proxy technology exists. It has legitimate uses. The issue is whether people know when their devices are part of it, understand what kind of traffic may pass through their connection, and have a real choice to opt out.The fix is not to pretend consumers will read every EULA. This software should be named clearly, shown plainly, and made easy to remove before a household connection becomes someone else’s infrastructure.

Your home connection could be routing traffic for strangers. Here's how residential proxy networks work, how devices get enrolled and what our telemetry reveals about the risks for consumers.

Who's Really Using Your Home Internet Connection?

This is a small story about tracking down a production bug in a back-end service written in the Rust language. It probably doesn’t provide much learning opportunities (unless you happen to hit the exact same bug), but might give a glimpse to a day of few computer engineers and supply a bit of entertainment.<h2>A bit of backstory</h2>We have an application called <a href="https://engineering.avast.io/rust-in-avast/">urlite</a>. It serves as a backend to some other applications, provides them with a HTTP API. It’s in Rust because it is latency critical. Latencies of most requests are under a millisecond.It uses the <a href="https://hyper.rs/">hyper</a> and <a href="https://tokio.rs/">tokio</a> libraries to handle HTTP. Some time ago, the libraries released a new version, to incorporate support for Rust’s new shiny async-await syntax. We were a bit late to the updating to the newer versions. Part of it was that it was working fine and we had only one short asynchronous function around, so there wasn’t real motivation. The other part was, urlite uses the <a href="https://docs.rs/spirit">spirit</a> configuration layer (I guess the fact urlite uses it has something to do with me being author of spirit). The library takes a configuration file and makes sure all the other libraries for logging, HTTP, etc, are configured according to it.And migrating spirit to new hyper and tokio was a big task, as both hyper and tokio changed quite a bit between the versions and because spirit does some things that are not usual for ordinary applications.Eventually, it was decided keeping up to date with dependencies is important enough for me to invest some of the work time (as opposed to the usual free time I use to work on spirit) to do the update. It took about a week to migrate both spirit and urlite. It went through review, the automatic tests and we put it to the staging environment for a while, watching logs and graphs. Everything seemed fine, so after few days of everything looking fine, we pushed the button and put it to production.<h2>The increased latencies</h2>As it goes in these kinds of stories, by now you’re expecting to see what went wrong.The thing is, our own metrics and graphs were fine. But the latency on the downstream service querying us increased by 40ms. The deployment got reverted, and we started to dig into where these latencies come from.<h2>It was acting really weird</h2>There were several very suspicious things about that.<ul><li>Our own „internal“ latencies stayed the same. Our CPU usage also stayed the same.</li><li>The latency graph on the downstream side was flat 40ms constant.</li></ul>Now, if we introduced some performance regression in the query handling, we would expect our CPU consumption to rise. We would also expect the latency graph to be a bit spiky, not completely flat 40ms constant. It almost looked like there was a 40ms <code>sleep</code> somewhere. But why would anyone put a 40ms sleep anywhere?I’ve looked through documentation and didn’t see anything obvious. I’ve tried searching both our code and code of the dependencies for <code>40</code>, asked on Slack if that <code>40ms</code> value was familiar to anyone. Nothing.The working theory we started with was that there could be some kind of back-off sleep on some kind of failure. Maybe <code>hyper</code> would be closing inactive connections in the new version, forcing the application to reconnect (the graph was for 99th percentile, so if we happened to close each 100th connection and reconnection took this long…) and maybe try IPv6 first and we would be listening on IPv4 only or… (in other words, we didn’t have a clear clue).<h2>The benchmarks</h2>My colleague started to investigate in a more thorough way than just throwing ideas around on Slack. He run a <code>wrk</code> benchmark against the service. On his machine, the latencies were fine. So he commandeered one of the stage nodes to play with it and run the benchmark there. And every request had 40ms latency on that machine. The previous version of <code>urlite</code> was fine, with under one millisecond.Something was probably sleeping somewhere on the production servers, but not on the development machine. There probably was some difference in the OS settings, but definitely difference in the kernel version. The servers are running some well-tried Linux distribution, so they have a lot older version, while a developer is likely to run something much more on the edge.<h2>Configuration options</h2>The selling point of <a href="https://crates.io/crates/spirit"><code>spirit</code></a> is that most of the thing one can set through the APIs of various libraries now can be just put into a config file without any recompilation. If we did configuration by hand, we would expose only the things we thought would be useful, but with spirit, there’s everything. So, naturally, tweaking some of the config knobs was the next step (because it was easy to do).It was discovered that turning the <code>http1-writev</code> option off, which corresponds to <a href="https://docs.rs/hyper/0.13.8/hyper/server/struct.Builder.html#method.http1_writev">this method</a> in <code>hyper</code>, made the latencies go away.We now had a solution, but I wasn’t happy about not understanding why that helped, so I went to dig the rabbit hole and find the root cause. Turns out there were several little things in just the constellation to make the problem manifest.<h2>Overriding the defaults of <code>http1_writev</code></h2>The method controls the strategy in which way data are pushed into the socket. With vectored writes enabled, it sends two separate buffers (one with headers, the other with the body if it’s small enough) through a single <a href="https://linux.die.net/man/2/writev"><code>writev</code></a> syscall. If they are disabled, <code>hyper</code> copies all the bytes into a single buffer and sends that as a whole.It turns out that the configuration method takes a <code>bool</code>, but there are actually 3 states. The third (auto) is signaled by not calling the method and <code>spirit-hyper</code> was calling it always, with the default to turn the <a href="https://linux.die.net/man/2/writev"><code>writev</code></a> on. I don’t know if leaving it on auto would make the bug go away, but I’ve fixed the problem in the library anyway.<h2>Splitting vectored writes</h2>Spirit wants to support a bit of configuration on top of what the underlying libraries provide on their own. One of such things is limiting the number of concurrently accepted connections on a single listening socket. Users don’t have to take advantage of that (the types for the configuration can be composed together to either contain that bit or not and the administrator may leave the values for the limits unset in the configuration and then they won’t be enforced).Anyway, in case the support for the limits is opted in through using the type with the configuration fields, the connections themselves are wrapped in a <a href="https://docs.rs/spirit-tokio/0.7.*/spirit_tokio/net/limits/struct.Tracked.html"><code>Tracked</code></a> type. The type tries to be mostly transparent for use and can be used inside <code>hyper</code> (which is what the default configuration type alias in <code>spirit-hyper</code> does), but tracks how many connections there are, to not accept more if it runs out of the limit.When implementing the bunch of traits for the wrapper, I’ve overlooked the <a href="https://docs.rs/tokio/0.2.*/tokio/io/trait.AsyncWrite.html#method.poll_write_buf"><code>AsyncWrite::poll_write_buff</code></a>. It is a provided method, which means it has a default implementation. It is the one that abstracts the OS-level <code>writev</code>, it can write multiple buffers (the <a href="https://docs.rs/bytes/0.5.*/bytes/buf/trait.Buf.html"><code>Buf</code></a> represents a segmented buffer).The default implementation simply delegates to multiple calls to the ordinary write. Therefore, this omission combined with the default of enabling vectored writes results in calling into the kernel twice, each time with a small buffer, instead of once with two small buffers or once with a big buffer.That was definitely something to get fixed, because if nothing else, syscalls are expensive and calling more of them is not great. But I’ve finally felt like I’m on the right path, because:<h2>Nagle’s algorithm</h2>You probably know that TCP stream is built from packets going there and back. Optimizing how to split the stream into the packets and when to send them is a hard problem and the research in that area is still ongoing, because there are many conflicting requirements. One wants to deliver the data with low latency, utilize the whole bandwidth, but not overflow the capacity of the link (in which case the latencies would go up or packets would get lost and would have to be retransmitted), leave some bandwidth to other connections, etc.The <a href="https://en.wikipedia.org/wiki/Nagle%27s_algorithm">Nagle’s algorithm</a> is one of the older tricks up in TCP’s sleeve. The network doesn’t like small packets. It is better to send as large packets as the link allows, because each packet has certain overhead ‒ the headers take some space, but also routers spend computation power mostly based on number of packets and less on their size. If you start sending a lot of tiny packets, the performance will suffer. While the links are limited by the number of bytes that can pass through them, routers are more limited by the number of packets. If a router declares to be able to handle a gigabit connection, it actually means a gigabit if you use full-sized packets, but will get to few megabits if you split the data into tiny packets (at least for the cheap routers, there are some that are capable of handling even a gigabit of small ones).So it would be better to wait until the send buffer contains a packetfull of data before sending anything. But we can’t do that, because we don’t know there’ll ever be a full packet of data, or generating more data might wait for the other side to answer. We would never send anything and wait forever and the Internet would not work.Instead, the algorithm is willing to send one undersized packet and then it waits for an <code>ACK</code> from the other side until sending another undersized one. If it gets a packetfull of data to send in the meantime, that’s great and it sends it (it won’t get better by waiting longer), but it won’t ever have two undersized ones somewhere in flight, therefore won’t kill the network’s performance by them.This works, it’ll make progress eventually because once the <code>ACK</code> comes and all the data that accumulated until then is sent.But it also slows things down. Like in our case. What happens if we do it using single syscall, the whole HTTP response forms a single undersized packet (we have really small answers) and gets sent, no matter if it’s submitted to the kernel by one big or two small buffers.On the other hand, if we split the submission into two syscalls, this is what happens:<ul><li>We write the first part (headers). The kernel sends them out as an undersized packet.</li><li>We write the second part (the body). But the kernel shelves them into the send buffer and waits for sending them until it sees the <code>ACK</code>, because there’s one undersized packet somewhere in flight already. Here is our sleep.</li></ul>This theory got confirmed by toggling the <code>tcp-nodelay</code> option on (and leaving <code>writev</code> on too, without the fix in the library applied), which also solved our problem. This option turns the Nagle’s algorithm off and just sends all data out as soon as possible (considering other things like buffers in the network card, TCP windows, congestion algorithms…).<h2 id="block-48a3e020-0a2e-4e6f-ba01-3148d57a7ed1">Ok, but why 40ms?</h2>But wait, the other service lives on the same machine. It really can’t take that long for the packet to pass through the loopback interface and an <code>ACK</code> to return back. Hell, it should not take 40ms if it was a different datacenter on the same continent. Not speaking about this taking exactly 40ms. Even if we had really long and slow loopback, it would not turn out to be 40ms exactly each time.So there must be something more happening.And indeed, there is. It’s called delayed ACKs and it’s also an optimisation of TCP. One that got introduced about the same time as Nagle’s algorithm, but without knowing about each other or considering how they would interact.You see, one doesn’t want to send small packets (deja-vu?). A packet carrying only an <code>ACK</code> and no data is certainly small and we don’t want to send it if we can help that.As an <code>ACK</code> can be used to acknowledge multiple received packets at once and it can piggy-back on a packet with actual data in it, the receiver wants to wait a little bit. The idea is, when data is received, it often provokes the application to respond to it with some other data so we wait for that to happen, then send the data with the <code>ACK</code>. If not, we at last wait for a second packet, to acknowledge two at once. Only if we don’t get any response data nor more incoming packets we decide to send that <code>ACK</code> eventually. After all, the sender doesn’t really need that <code>ACK</code> fast, does it, because if it would have more data to send, it would send at least the whole TCP window, which is several packets long.Except that the other side might be waiting for it to send more data because of the Nagle’s algorithm. That sucks.And indeed, a little bit of searching through the Internet actually finds a <a href="https://lwn.net/Articles/502585/">kernel patch</a> talking about delayed ACKs and the value of 40ms. Bingo, this can’t be coincidence.I have reasons to believe this interaction got somehow solved in newer kernels (I don’t know how, but apparently, it no longer happens on our development machines, at least with loopback). And that 40ms timeout is something the client side chooses.<h2>Conclusion</h2>Well, sometimes the situation happens to be a really complex mess that results in some weird interaction (and make a good material for a blog post). The above combination of the TCP optimisations is described on the Internet, but I’d say it’s not widely known one. That getting triggered through defaulting to a less optimal but generally good enough implementation is even bigger chance. And that such a small detail can make your optimized service 40 times slower.Nevertheless, our takeaway, apart from fixing the discovered issues is:<ul><li>We started to watch not only our own graphs, but graphs of other services around and have alerts for them. Our own graphs just can’t contain delays introduced by packets sleeping somewhere in the kernel.</li><li>We now run with <code>tcp-nodelay</code> enabled. Even when the bug is now fixed, the interaction is still technically possible to hit in case the client uses pipelining ‒ the client could send two HTTP requests in a row without waiting for the first response in between, possibly in the same TCP packet. Then we would have written the first response and hit the problem after writing the second one.</li></ul>Rust is generally known to catch a lot of bugs during compile time. But no matter how good your language, tooling and tests are, some complex combination of chances still can cause bugs nobody foresees. Then it’s up to how well the rest of the system deals with it, how easy it is to roll back to previous versions and, as always, some luck and intuition while debugging (toggling the <code>writev</code> option was only a hunch at best, for example).

Originally published on the author’s private blog.

40 millisecond bug

I like the <a href="https://www.rust-lang.org/">Rust</a> programming language, and so I tried to to make it a popular language in Avast, where I work. For some time I gave presentations, made workshops and generally tied to raise the awareness of it here.Nevertheless, for the venue to be successful, I needed an actual project where Rust would be a good match and the management could be persuaded it was a good idea to try it on. Avast is open to new technologies, but only if they make sense, not just as an excuse to play with them.<h2>The project</h2>So there’s this service, called urlinfo. It’s an old piece of software. Every company that is a year old has a code base that’s two years old. Avast is 31 years old. Today, most of Avast’s backend software is written in Scala. And some Python and Perl and C# and… whatever else too. But mostly Scala. This thing is in Java, because it predates Scala. Well, probably not the language itself, but definitely its use in Avast. You can give it an URL and it’d look into its data and tell you if it’s hosting malware or phishing or if it’s interesting in some other way. This service is used by many of Avast’s solutions, such as the browser extension.My team’s <a href="https://www.avast.com/en-us/omni#omni">project, Avast Omni,</a> would become another (indirect) user of that service. We aim at ”bigger embedded“ ‒ home routers, raspberry-style devices and the like. It’s supposed to protect other devices on the network level. With clever tricks you can fit a lot of functionality into such a device, but there are limits and this URL database simply doesn’t fit. So for some things, the device asks the cloud (read: Avast servers) for assistance. That assistance includes other analyses than just consulting the large urlinfo blacklist, but we are not going to be talking about these right now.For our use case, latency is important. We want to impact the user’s browsing experience as little as possible, so waiting a long time for the server response and delaying the packets for that long is a no-go. If the device doesn’t get an answer from the server in time, it gives up waiting and decides based on its own limited best judgement. That internal timeout includes the network communication with the server and the devices will live in our customers’ homes, often with questionable connectivity. The time limit is not very generous.How do we deal with that short time? In several ways.The client keeps a persistent connection to the backend server. We don’t want to waste the precious time with TCP handshake and TLS negotiation when time’s ticking ‒ just one packet there and one back.The client actively seeks to find the closest server available, but for that to have some reasonable effect we need some of the servers to be physically close to each customer. We need servers scattered throughout the world, which means having a lot of them at different data centers.The server needs to answer fast.The urlinfo service was giving us trouble in these areas. It uses about 16GB of RAM. If you want to have a server close to just about anyone, you have to buy VM instances from various providers and you’re going to be billed based on how much RAM you want them to have. But that’s just money.More importantly, the garbage collector stops the application for several hundreds of milliseconds to scan that 16GB. That just plain out doesn’t fit in our limit. Of course we didn’t know until our end-to-end test began to fail and investigated where the delay come from.When I saw the reason for our test failures in a Slack conversation, I reacted with this little <code>:crab:</code> icon, followed by a question-mark icon (by that time it was enough to indicate a “would this be a good place to try out Rust on?“ from me). Partly in a half-joke, partly because it was part of the habit, except my boss took it seriously and gave it a thought.And it turned out not to be a bad idea. Some colleagues tried to tune the JVM GC in the old service, to limited effect. Slimming the memory consumption down significantly would be a rewrite of the whole thing anyway and nobody wanted to touch the old code base.It’s a back-end service, therefore it is easily fixable if anything goes wrong (unlike code running at people’s homes), so trying Rust out there is lower risk. Furthermore, we needed only part of the functionality of the old service, which was another way to slim it down.So the <code>urlite</code> project was born. We worked on it in a team of two (both of us having it as a secondary project), the other person a volunteer from the group of people who visit the workshops. We are trying to make more people interested, some other projects are watching it if it could be useful for them.<h2>Difficulties</h2>As I wrote earlier, all the infrastructure is expecting JVM applications. If you want to start a new project in Scala, you take a template, click two buttons in Teamcity (our CI server), magic happens, and you have the whole pipeline to compile it, test it, deploy it, etc. You just specify which servers it should be deployed to, give it a configuration file and everything is ready, including logs getting to the right places, metrics being available in the internal Graphana instance, the service being monitored with emails about its health routed to you, everything.We had to do most of that manually with Rust. The metrics format used in JVM applications is not available in Rust. The internal deployment tool creates <code>-noarch</code> packages (let’s hope nobody buys an Arm server any time soon) and runs the thing under JVM. Teamcity still tries to point out the ehm relevant places if tests fail thinking it should look like an output from the Java tests. Some colleagues still ask why another language is needed when everything else is in Scala.None of this was an impossible problem, nevertheless, it was a lot of work. Hopefully, next time, there’ll be less of it or we’ll at least know what to do. On the other hand, part of that work is public as part of various open-source libraries. We reported number of issues and sent several pull requests. Some of my own libraries got more testing and in case of <a href="https://docs.rs/spirit">spirit</a>, a bigger API overhaul as a result of that testing. While the core knowhow and the actual application needs to stay closed, doing the rest in public actually saves us future effort ‒ both with maintaining it and with being able to pull directly from <a href="https://crates.io/">crates.io</a> in our next project. General development in Rust feels slower than in Scala (according to the colleague, I don’t know Scala so I’d be definitely faster in Rust).<h2>The results</h2>It’s still not under heavy load yet, it’s being put into production slowly. But preliminary testing suggests the results are good. The processing times are very consistent and around 1ms. The memory usage is under 500MiB of RAM most of the time, slightly more during an update of the data sources (there’s a short time we have both the old and the new one loaded). Startup time to fully operational state is just a few seconds ‒ less than what the JVM takes to boot and load the <code>urlinfo</code> application without any data in it.

How is this possible? Part of the story is that Rust doesn’t have the garbage collector ‒ or, as pointed out, has a <a href="https://words.steveklabnik.com/borrow-checking-escape-analysis-and-the-generational-hypothesis">static compile-time garbage collector</a>. Garbage collectors waste some CPU (which is not the issue here), but also memory and introduce latencies that are really hard to control and compensate for.But there’s another part. Rust attracts people who like challenges. So while things like integrations into corporate systems (sentry, logging systems, metrics) feel immature, there are loads of libraries for tackling the hard problems. I’m not listing all the libraries we used, but having them available and ready to be used with minimal effort made the task not only possible, but actually enjoyable. I’ll point to just one of them, the <a href="https://docs.rs/fst/0.3.5/fst/">fst</a>, which allows us to have the data in RAM compressed while still searchable.Besides having the data in a format that is both compressed and searchable at the same time and having the hot-path lockfree (well, there might be a lock somewhere in the memory allocator, we haven’t made the hot path allocation free yet), we prepare the data images on a central server so all the virtual machines doing the serving don’t have to.When testing and deploying it, we have seen surprisingly small number of bugs ‒ as little of them that it makes me nervous some must be hiding somewhere we haven’t looked.Some other projects start to consider using the new <code>urlite</code> service too ‒ for example the VPN products couldn’t use the <code>urlinfo</code> for similar reasons as we but they think <code>urlite</code> would work for them, thanks to the lower memory consumption and consistent latencies.<h2>A little allocator mystery</h2>We were also investigating a little mystery. The usual processing latency (without external communication) is ~200μs. However, from time to time, there would be a latency spike of several milliseconds. It was happening even when not under any particular load ‒ few requests per second were enough. Not that it would be a serious problem for the use case. Such a spike happened few times a day. But it shouldn’t have been happening and the natural question was if it’s an indication of some more serious problem.It turned out to be an artifact of how Jemalloc works. During loading of the data, we allocate a lot (or, we did in the previous versions, now it happens on the central server). Then we free a large part of it, and also the previous version of data. Jemalloc keeps the unused memory for a while, but eventually decides it is not going to be used any time soon and returns it to the OS. This returning is supposed to happen gradually. However, because we had more worker threads than queries, the one thread unlucky to be responsible for freeing this memory was idle for large enough time and all that unused memory decayed (that’s the Jemalloc’s term for deciding it is unused for too long) during the inactivity time. Then, the one query that woke the thread was responsible for cleaning all that memory up, which consists of calling a lot of syscalls. Unlike the allocation which happens in a background performance non-critical thread, this hit the query processing.It also turns out Jemalloc has a knob for exactly this. It can start its own background threads to do these cleanups without disrupting the application. As can be seen in the image (the vertical dashed lines are deployments), this had the desired effect ‒ one line change in <code>Cargo.toml</code>, but discovering this took a long time and some experimentation.

<h2>Conclusion</h2>I can’t speak for other people around the project, but for what I see, Rust is less productive in the initial development phase than other languages ‒ part of it is the immature libraries and the internal systems not ready for a non-JVM language with different metrics and logs formats. But part is also that sometimes tweaking the code around the borrow checker just takes some effort, or that very strict types do get in the way (returning 3 alternatives in an <code>impl Trait</code> place, for example). I’m not even talking about the effort to learn it (on the other hand, my attempts to learn Scala didn’t end that well).On the other hand, it does allow fulfilling the more crazy needs. Pushing the RAM consumption and the latencies down may be just one of them. Being pretty confident about the code working correctly is another.So as always, choosing Rust as the language brings trade-offs. Sometimes the speed of development is the major factor and then there probably are better languages to pick. But when the problem being solved gets interesting, Rust may come to save the day.

Michal Vaner

Posts by Michal Vaner

40 millisecond bug

Rust in Avast