Senior Software Engineer @ ThreatLabs

Our team invests countless hours in the exploration, discovery and investigation of emerging trends such as AI and futuristic technology. Stay informed about the evolving landscape of cyberthreats through our diligent efforts and insights.

Threat Researcher at Gen

Research

<h2>Introduction&nbsp;</h2>Infostealers are constantly evolving, and so are the techniques they use to bypass <a href="https://security.googleblog.com/2024/07/improving-security-of-chrome-cookies-on.html" target="_blank" rel="noreferrer noopener">Application-Bound Encryption (ABE)</a>. In recent weeks, Vidar has been among the most actively developed stealers and, apart from <a href="https://x.com/GenThreatLabs/status/2064836539952361976" target="_blank" rel="noreferrer noopener">multiple updates to its string obfuscation</a> and <a href="https://x.com/GenThreatLabs/status/2061462524604649840" target="_blank" rel="noreferrer noopener">a reworked approach to protecting its configuration</a>, it has also introduced a novel technique for bypassing ABE. And while there have been many other changes in Vidar lately, with new versions dropping every week, in this blog post we focus solely on the ABE bypass and its technical aspects.&nbsp; &nbsp;<h2>The ABE Bypass&nbsp;</h2>From a high-level perspective, Vidar’s approach to bypassing ABE is somewhat similar to that of Remus/Lumma, in that both try to extract the <code>v20_master_key</code> directly from the browser’s memory. What sets them apart, however, is how they achieve it. Since we’ve already covered the inner workings of ABE in earlier posts (<a href="https://www.gendigital.com/blog/insights/research/remus-64bit-variant-of-lumma-stealer" target="_blank" rel="noreferrer noopener">Remus</a>, <a href="https://www.gendigital.com/blog/insights/research/voidstealer-abe-bypass" target="_blank" rel="noreferrer noopener">VoidStealer</a>), we don’t revisit them here and focus instead on the bypass itself. For readers interested in the broader background, we encourage checking those earlier posts.&nbsp;&nbsp;In short, the end goal of infostealers is to obtain the <code>v20_master_key</code>, as it alone is sufficient for decrypting any ABE-protected data tied to a specific application. This key can be obtained in one of two ways: either from disk, where it is protected by two layers of <code>CryptProtectData</code> (one applied in the context of the logged-in user and one as <code>NT AUTHORITY\SYSTEM</code>), or from the browser’s memory – the direction Vidar took.&nbsp;In memory, however, the key is kept protected with <code>CryptProtectMemory</code> using the <code>CRYPTPROTECTMEMORY_SAME_PROCESS</code> flag. This ensures that even if an attacker manages to read the encrypted form of the key from memory, they cannot simply decrypt it (they would need to invoke the complementary <code>CryptUnprotectMemory</code> function directly from within the browser process). That leaves attackers with two problems to solve: how to find the encrypted key in memory, and how to get it decrypted.&nbsp;<h2>Locating the v20_master_key in Browser’s Memory&nbsp;</h2>Vidar’s approach to locating the encrypted <code>v20_master_key</code> in memory involves scanning the entire memory of a running browser for a specific pattern. But that first requires a browser process to scan.&nbsp;When the target browser is already running, Vidar reuses the existing process. However, it doesn’t read its memory directly. Instead, it creates a fork of it using <code>NtCreateProcessEx</code> with <code>SectionHandle</code> set to <code>NULL</code>, instructing the kernel to clone the address space of the process specified by the <code>ParentProcess</code> as a copy-on-write mapping. The resulting process has no threads and is never resumed – it exists purely as a static memory snapshot.&nbsp;&nbsp;To perform the fork, Vidar first opens a handle to the target browser via <code>OpenProcess</code> with <code>PROCESS_QUERY_INFORMATION | PROCESS_CREATE_PROCESS | PROCESS_VM_READ</code>, falling back to <code>PROCESS_CREATE_PROCESS</code> if the initial call fails. It then passes this handle as the <code>ParentProcess</code> argument to <code>NtCreateProcessEx</code>, requesting <code>PROCESS_ALL_ACCESS</code> on the resulting child. If the fork fails, Vidar falls back to scanning the live browser process directly.&nbsp;

If no target browser process is found running, Vidar creates a new isolated desktop via <code>CreateDesktopA</code>, assigns it a name of the form <code>v20_%d</code> (where <code>%d</code> is replaced with the browser’s index in Vidar’s configuration), and launches the target browser in there. In the case of Vidar version 2.1, the browser is launched with the command line <code>--no-first-run --disable-gpu about:blank</code>, although the exact command line varies across versions. Since modern browsers use a multi-process architecture, there are typically multiple processes matching a given browser. Vidar enumerates all of them (up to 64) and applies the fork-and-scan procedure to each one independently.&nbsp;

With a handle to the forked process in hand, Vidar proceeds to enumerate its virtual memory regions using <code>NtQueryVirtualMemory</code>. It iterates through the entire browser’s address space, collecting regions that match a specific profile: committed (<code>MEM_COMMIT</code>), private (<code>MEM_PRIVATE</code>), and either readable (<code>PAGE_READONLY</code>) or read-write (<code>PAGE_READWRITE</code>). Vidar records up to 4096 memory regions matching these requirements, along with their base addresses and sizes. The actual pattern scanning is then performed in parallel with up to 64 worker threads, depending on the system.&nbsp;&nbsp;

Each thread scans its assigned regions using <code>NtReadVirtualMemory</code>, searching for a 32-byte pattern. When a match is found, the thread reads 8 bytes at offset +32 from the match, interprets them as a pointer, and validates that it points to committed, private, and readable memory (the expected memory state for the encrypted <code>v20_master_key</code>).&nbsp;&nbsp;

This naturally raises the question of what pattern Vidar searches for. The answer is a 32-byte signature: <code>76 32 30 00 ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 03 00 00 00 00 01 ?? ?? ??</code>, where <code>??</code> denotes a wildcard (Vidar uses the byte <code>AA</code> as the wildcard marker in its internal representation). Notably, the first four bytes correspond to the ASCII <code>v20\x00</code>, which is a common prefix used for ABE-related data.&nbsp;&nbsp;

We assess with high confidence that this pattern is intended to match a node structure within Chromium’s <code>Encryptor::KeyRing</code>, the structure that holds the keys used by the <code>Encryptor</code> class. Now, this part is admittedly a bit tricky as it relies on several assumptions we made about the internal memory layout, as well as on implementation-specific representations of various C++ standard library types <code>(std::map</code>, <code>std::string</code>, <code>std::vector</code>, <code>std::optional</code>), which is why we frame this as a high-confidence rather than a certainty.&nbsp;&nbsp;Our reasoning is as follows: <code>KeyRing</code> is <a href="https://github.com/chromium/chromium/blob/cf5d502bcaedb22927e3829a18be1ff6d8b6f598/components/os_crypt/async/common/encryptor.h#L132" target="_blank" rel="noreferrer noopener">defined</a> as <code>std::map&lt;/*tag=*/std::string, std::optional&lt;Key&gt;&gt;</code>, and <code>std::map</code> is typically implemented as a red-black tree, where each tree node stores three pointers (left child, parent, and right child) followed by the key-value pair. In this case, the pair is in the form of <code>std::pair&lt;/*tag=*/std::string, std::optional&lt;Key&gt;&gt;</code>, meaning that the <code>std::string</code> and the <code>Key</code> object might be laid out consecutively in memory within each node. For a <code>v20_master_key</code> entry, the tag corresponds to the string <code>v20</code>, which at 4 bytes including the null terminator fits within the small string optimization (SSO) buffer and is therefore stored inline rather than heap-allocated. The <code>Key</code> class that immediately follows <a href="https://github.com/chromium/chromium/blob/cf5d502bcaedb22927e3829a18be1ff6d8b6f598/components/os_crypt/async/common/encryptor.h#L93-L98" target="_blank" rel="noreferrer noopener">contains three member variables</a>: <code>std::optional&lt;mojom::Algorithm&gt; algorithm_</code>, <code>std::vector&lt;uint8_t&gt; key_</code>, and <code>bool encrypted_ = false</code>.&nbsp;Putting it all together and applying it to a concrete example from a browser memory dump, we get the following layout:&nbsp;

Please note that while the exact layout may contain minor inaccuracies on our part, the overall structure should be roughly correct. For easier orientation, the following diagram illustrates how the individual structures are nested within each other:&nbsp;

At this point, it should be clear how the scan pattern maps onto the searched structure. The 32-byte match covers the <code>std::string</code> tag <code>v20\x00</code> and the beginning of the <code>Key</code> object up to and including <code>algorithm_</code>. The pointer at offset +32 from the start of the match then lands precisely on <code>key_.begin</code>, the first member of <code>std::vector&lt;uint8_t&gt;</code> that holds the encrypted <code>v20_master_key</code>.&nbsp;Matches from all threads are collected into a shared buffer and subjected to a majority-voting scheme: candidates consisting mostly of zeroes are discarded, the rest are pairwise-compared, and each is scored by the number of identical copies found. This voting is performed independently for each browser process, producing one best candidate per process.&nbsp;<h2>Decrypting the v20_master_key via APC Injections&nbsp;</h2>For each browser process that produced a match, Vidar now holds a candidate address pointing to a potential encrypted <code>v20_master_key</code>. However, as already noted, since the key is protected with <code>CryptProtectMemory</code> using <code>CRYPTPROTECTMEMORY_SAME_PROCESS</code>, it can only be decrypted by invoking <code>CryptUnprotectMemory</code> from within the browser process itself. Vidar solves this by injecting an Asynchronous Procedure Call (APC) into the live browser process.&nbsp;Vidar supports two APC injection methods, selected based on whether ESET or Bitdefender is detected on the system. If either of these products is present, Vidar uses the first method, which it refers to as “classic” (based on decrypted strings) – it creates a suspended thread in the target browser via <code>CreateRemoteThread</code> (with <code>NtTestAlert</code> as the thread start address and the <code>CREATE_SUSPENDED</code> flag), queues an APC using <code>NtQueueApcThread</code>, resumes the thread, and then waits for it to complete via <code>WaitForSingleObject</code> with a randomized timeout. When the thread resumes, it calls <code>NtTestAlert</code>, which flushes the APC queue and executes the injected routine.&nbsp;

If neither product is detected, it uses the “special” method – it finds an existing thread in the target browser using a combination of <code>CreateToolhelp32Snapshot</code> and <code>Thread32First/Thread32Next</code>, opens it via <code>NtOpenThread</code>, and calls <code>NtQueueApcThreadEx2</code> with the <code>QUEUE_USER_APC_FLAGS_SPECIAL_USER_APC</code> flag. Unlike regular APCs, special user APCs execute immediately. Therefore, the target thread does not need to be in an alertable wait state.&nbsp;

In both cases, the APC routine to be executed is <code>CryptUnprotectMemory</code>, called with the following three arguments: the address of the encrypted <code>v20_master_key</code> candidate, its size (32 bytes), and <code>CRYPTPROTECTMEMORY_SAME_PROCESS</code> for the <code>flags</code> parameter.&nbsp;&nbsp;Since <code>CryptUnprotectMemory</code> decrypts the data in place, once the APC executes, the key is decrypted directly in the live browser's memory. To read it, Vidar creates a second fork of the browser process and reads the key from the same address using <code>NtReadVirtualMemory</code>. It then compares the result against a snapshot of the encrypted key taken from the first fork before the APC call – if the data is unchanged, decryption failed, and Vidar moves on to the next browser process’s candidate.&nbsp;If the data differs, Vidar uses the same second fork to perform an additional verification: it scans the forked process’s memory for the byte sequence <code>v20</code> and, treating each match as a potentially ABE-encrypted entry (cookie, password), attempts AES-256-GCM decryption with the key candidate using <code>BCryptDecrypt</code> (successful GCM tag verification confirms that the key is correct).&nbsp;Whenever the data has changed (indicating that decryption of the key took place), Vidar re-encrypts the key in the browser’s memory by injecting another APC with <code>CryptProtectMemory</code> as the routine, preserving the browser’s state. If no candidate produced a successful decryption (i.e., the data remained unchanged after every APC call), Vidar takes a more aggressive approach: it terminates all browser instances, restarts the browser, and repeats the entire scan-decrypt-verify cycle.&nbsp;By abusing APC injections instead of the more traditional approach of injecting code via <code>NtWriteVirtualMemory</code> that performs the decryption, Vidar trades one suspicious behavior for another. APC injections are arguably less common and may thus slip past some detection systems, though they are still well-known techniques monitored by AV and EDR solutions. But that is the nature of ABE bypasses – each has its own advantages and disadvantages, and that’s exactly why infostealers keep inventing new ones. &nbsp;<h2>Indicator of Compromise (IoC)&nbsp;</h2>459daa809751e73f60fbbe4384a7d1653c36bb06945e4eb3635270924241100a (Vidar 2.1)&nbsp;

A Technical Walkthrough of How Vidar Defeats Application-Bound Encryption

Inside Vidar’s ABE Bypass: From Memory Scanning to APC Injections

Security Evangelist at Gen

You have a summer flight booked. You have seen the headlines about jet fuel shortages, cancelled routes and higher travel costs. Then a text message arrives: your flight has been affected, and you need to click a link to confirm your refund or rebook before the offer expires.&nbsp;That message feels plausible because the situation around it is real. In April, <a href="https://www.iata.org/en/pressroom/2026-releases/2026-04-17-01/" target="_blank" rel="noreferrer noopener">IATA warned</a> that Europe could start seeing flight cancellations by the end of May if jet fuel shortages materialized. Airlines and governments have since been trying to reassure travelers, while also publishing guidance on passenger rights, refunds and re-routing.&nbsp;&nbsp;That is exactly the kind of confusion scammers like. They do not need to cause the disruption, they only need to arrive at the right moment, with the right brand logo and the right sense of urgency.&nbsp;<h4>Why travel disruption works so well as a scam hook&nbsp;</h4>Most scam messages fail because they arrive out of context. A random refund email from an airline you never use is easy to ignore. A refund message from the airline you are flying with next week, during a period of public travel disruption, is different.&nbsp;The scammer’s job is to make you think: “This could be real.”&nbsp;That can happen in several ways. You might receive a fake cancellation email that links to a refund form. You might search for an airline’s customer service number and land on a fake support page. You might post publicly on social media asking an airline for help, then receive a reply from an account that looks official. Or you might get a WhatsApp message that knows your hotel name, travel dates or booking reference.&nbsp;Each version uses the same pressure point: you are worried about losing money, losing your flight or being stranded.&nbsp;Criminals are already using disruption this way. In March, <a href="https://www.theguardian.com/money/2026/mar/15/travel-scam-airline-accounts-fake-refunds-iran-war-flight-disruption" target="_blank" rel="noreferrer noopener">The Guardian reported</a> that fake airline support accounts were targeting travelers affected by Middle East flight disruption. Some victims were looking for help or refunds, then received replies from accounts pretending to represent airlines. The scam moved into direct messages, where victims were asked for details and sent links that led to money being taken instead of refunded.&nbsp;&nbsp;The jet fuel story gives scammers another pretext. A message can claim that your flight was cancelled, your fare changed, your fuel surcharge failed, or your refund must be claimed within a few hours. None of those claims needs to be true to be effective.&nbsp;<h4>The scam may start before the message arrives&nbsp;</h4>A fake travel message becomes much harder to spot when it includes real information.&nbsp;This is the idea behind <a href="https://www.gendigital.com/blog/insights/research/reservation-hijack-scam" target="_blank" rel="noreferrer noopener">Reservation Hijack scams</a>, a form of targeted phishing where cybercriminals use real booking details to make a message look legitimate. In some cases, attackers may use information exposed through breaches or compromised travel systems. In other cases, they collect details from emails, social media posts, previous leaks or fake customer service interactions. Gen has previously described Reservation Hijack scams as scams that use real hotel booking details to make fraudulent payment or verification requests feel authentic.&nbsp;&nbsp;That distinction matters. A scam message does not need your credit card number to fool you. Knowing your name, your destination, your travel dates and your booking platform may be enough.&nbsp;<h4>What these scams can look like&nbsp;</h4>A travel disruption scam usually does not begin with a strange offer. It begins with something boring and administrative. That is part of the trick.&nbsp;You may see messages like:&nbsp;“Your flight has been cancelled due to fuel supply issues. Claim your refund here.”&nbsp;“Your rebooking is pending. Confirm passenger details within two hours.”&nbsp;“Additional airport tax or fuel surcharge is required to keep your booking.”&nbsp;“Your hotel reservation will be cancelled unless payment is verified.”&nbsp;“Due to high call volume, please continue this request on WhatsApp.”&nbsp;The exact wording changes, but the structure is familiar. There is a problem, a deadline and a link. The message may also move you away from the official channel into SMS, WhatsApp, Telegram, direct messages or a phone call. Once that happens, the scammer controls the conversation.&nbsp;There is another detail worth watching: some messages do not ask for money immediately. They start by asking for your booking reference, last name, phone number or one-time password. That can still be dangerous. Those details can help scammers access a booking, impersonate you to customer support, reset an account or make the next message more convincing.&nbsp;<h4>Your real passenger rights can be used against you&nbsp;</h4>If an airline cancels your flight, you may have rights to a refund, re-routing or assistance, depending on where you are travelling and the rules that apply. The European Commission said passengers affected by cancellations continue to have rights to reimbursement, re-routing or return, airport assistance and compensation for last-minute cancellations in relevant circumstances. It also said airlines cannot retroactively add fuel surcharges to already published final ticket prices.&nbsp;&nbsp;That information is useful for travelers, but it is also useful for scammers.&nbsp;A fake refund message becomes more believable when people already know refunds are possible. A fake rebooking link becomes more believable when airlines are genuinely changing schedules. A fake “fuel surcharge” demand becomes more dangerous when fuel prices are in the news, even though legitimate retroactive charges may not be allowed in many cases.&nbsp;This is why the safest route is not to judge the message only by whether the story sounds plausible. During travel disruption, many scam stories will sound plausible. Judge the channel instead.&nbsp;<h4>How to check before you click&nbsp;</h4>Start from the airline or booking platform’s official app or website. Do not use the link in the message. Open the app yourself, type the website address yourself, or use the contact details listed on your original booking confirmation.&nbsp;Check whether the same cancellation, refund or payment request appears inside your account. If the message is real, the airline or hotel should be able to confirm it through an official channel.&nbsp;Be careful with social media support. If an account replies to your public complaint, check whether it is linked from the airline’s official website. Look at the handle carefully. Fake support accounts often copy logos and names, but have small differences in spelling, low follower counts or recently created profiles. The Guardian reported that fraudsters often scan social media for passengers trying to contact legitimate airline accounts and reply to them first, which makes the scam harder to spot.&nbsp;&nbsp;Do not share one-time passwords, payment card details or login credentials with anyone who contacts you unexpectedly. A real airline or hotel does not need your banking OTP to send you a refund.&nbsp;Be especially suspicious of pressure. “You have 30 minutes to claim your refund” is not customer service. It is a tactic.&nbsp;<h4>What to do if you already clicked&nbsp;</h4>If you entered payment details, contact your bank or card provider immediately. Ask whether the transaction can be blocked or reversed, and request a new card if needed.&nbsp;If you shared login details for an airline, hotel or booking platform, change the password from the official site or app. Use a unique password, not one you use elsewhere. Turn on two-factor authentication where it is available.&nbsp;If you shared a one-time password or authentication code, assume the account may have been accessed. Check your booking details, payment methods and recent account activity.&nbsp;Report the scam to the airline, hotel or booking platform being impersonated. If the scam came through social media, report the account as impersonation. If money was taken, report it to the relevant fraud authority in your country.&nbsp;<h4>The real lesson for summer travel&nbsp;</h4>Travel disruption changes how people behave. They search for support numbers in a hurry. They post publicly because customer service queues are long. They accept strange instructions because they are worried about losing a flight or refund.&nbsp;Scammers build around that behavior.&nbsp;The safest assumption is simple: if a message about your flight, hotel or refund creates urgency and gives you a link, pause. Go back to the official source. A real cancellation will still be there when you check through the airline or booking platform. A scammer needs you to act before you do.&nbsp;

Travel disruption gives scammers the one thing they need most: a believable reason to rush you

Your flight was cancelled. Is the refund message real? 

A fake job interview page is easier to trust than a fake ticket shop.In samples reviewed by Gen threat researchers, scammers used branded hiring pages that looked like ordinary recruitment flows: a company logo, a recruiter profile, a 30-minute meeting, and a button to continue with Google or Facebook. There was no attachment to open and no software to install. The credential theft attempt sat inside a step many job applicants already expect: signing in to schedule a call.FIFA is an obvious lure ahead of the 2026 World Cup. On May 22, ESET reported fake FIFA-themed websites that copied ticket and merchandise flows to steal money and personal data. Our researchers later found another FIFA-themed domain, hxxps://fifa-mg[.]shop, which followed a naming pattern not mentioned in that report. Domains like these can be short-lived, often appearing for hours or days before they are blocked, abandoned, or taken down.Our researchers also found a different kind of FIFA lure: hxxps://fifahiring[.]com, a fake hiring page that encouraged users to schedule a call with an alleged FIFA recruiter. When a user tried to register with a personal Gmail address, the page asked for a business email instead. The page was steering victims toward credentials with more value: accounts that could provide access to a company environment.

<h4>The FIFA hiring page did not point to FIFA</h4>The fake page looked believable at first glance.

It used FIFA branding, presented a recruiter identity, and copied parts of the experience users see on legitimate recruiting and scheduling platforms. The recruiter’s name and photo appeared to match a real LinkedIn profile, which made the page feel more credible. For publication, we should keep that person’s identity redacted.The domain did not match FIFA’s real hiring infrastructure. FIFA’s legitimate careers pages are jobs.fifa.com and fifa.pinpointhq.com. The fake page used neither. It also loaded the favicon from the real FIFA recruitment site hosted at pinpointhq.com, giving the page a small visual cue borrowed from the legitimate service.Other signals pointed in the same direction. The domain and certificate were newly created, from May 23. The site used commodity hosting infrastructure, including Vercel and Render, rather than a corporate environment. Those signs are not enough on their own, but they fit the rest of the evidence.The login flow was the clearest part. Credentials were posted to:fifeq2026eqbackeq[.]onrender[.]com/api/loginThe Google sign-in prompt was not a real Google authentication window. It was rendered inside the page itself. The visible controls and links did nothing, and the page did not send the user into a genuine Google sign-in flow. According to the analysis, the only Google asset loaded was the favicon, apparently to make the fake window look familiar.A page can look like it opened a Google login without ever handing control to Google.Researchers also observed hxxps://careers-fifahiring[.]com, which later became inactive, and hxxps://fifajobs[.]com, which showed the same content and was indexed by Google.&nbsp;<h4>The recruitment lure went beyond FIFA</h4>The FIFA page was one version of a wider recruitment phishing pattern. The brand can change. The flow stays familiar: recruiter profile, meeting prompt, business email, fake sign-in.Researchers found similar fake hiring pages abusing other major brands, including Heineken, Hilton, Coca-Cola, Netflix, PepsiCo, Delta,&nbsp;and Spotify. In the Heineken case, the page appeared to use the identity of someone who, according to LinkedIn, really works as a recruiter at the company.

Additional hunting found two other recruitment-themed kits:hxxps://aquent-talent-jobs[.]com hxxps://hays-talent-opportunities[.]comThe Hays-themed version targeted Facebook credentials instead of Google credentials. Researchers assessed that the Aquent and Hays pages shared code, although they differed from the earlier FIFA examples.

This flexibility makes the campaign harder to frame as a single World Cup scam. FIFA provides the timely hook, but the mechanism is recruitment phishing. A scammer can swap the logo, recruiter name, and sign-in provider while keeping the same basic path to the victim’s credentials.<h4>Why fake recruitment pages work</h4>Recruitment gives attackers a clean excuse to ask for things that might otherwise raise suspicion.A candidate expects to click a scheduling link. They expect a recruiter profile. They expect a calendar prompt. In some cases, they may also expect to use a work email, especially for corporate, contractor, or partner-facing roles.The fake login window turns that ordinary flow into credential theft. Users are used to “Continue with Google” and “Continue with Facebook” buttons, but the button itself proves nothing. A genuine authentication flow should take the user to the provider’s domain and controls. A fake page can draw a convincing copy while keeping the victim inside the attacker-controlled site.The business email prompt is also a warning sign. A page that rejects a personal email and asks for a work address may be trying to raise the value of the stolen account.<h4>What job seekers should check before signing in</h4>Start from the company’s official careers page, not from a link in a message, ad, search result, or forwarded post. For FIFA, that means checking jobs.fifa.com or fifa.pinpointhq.com, not trusting a domain because it contains “fifa,” “career,” “hiring,” or “jobs.”Look closely at the domain before entering credentials. Scam pages often add plausible hiring words around a brand name: career, talent, jobs, hiring, portal, or connect. A polished page on a new domain is still a new domain.Treat the sign-in step as the real checkpoint. If a page offers Google or Facebook login, check whether the flow actually opens a real provider-controlled authentication page. A fake window embedded inside the original site, broken controls, inactive links, or a provider logo used only as decoration should stop the process.Use security software that includes web and phishing protection. These tools can block known malicious pages before they load, which is especially useful when scam domains look polished, appear in search results, or are shared through otherwise legitimate channels.<h4>What companies can do</h4>Recruitment pages are public by design, which makes them easy to copy.Security teams should monitor for newly registered domains combining their brand with hiring-related terms. They should also watch for pages that reuse favicons, recruiter photos, job descriptions, and calendar-booking language from legitimate hiring flows.HR and recruiting teams should make the official hiring path easy to verify. Candidates should not have to guess whether a domain is legitimate. A clear careers page, consistent recruiter communication, and warnings about unofficial scheduling links can remove some of the ambiguity attackers rely on.<h4>Conclusion</h4>The fake FIFA hiring page worked because it did not feel unusual. It looked like a normal step in a hiring process.Attackers do not need to invent new behavior when they can copy one people already trust. A branded scheduling page, a recruiter profile, and a familiar sign-in button can be enough to move a victim from interest to credential entry.The decision point is the login. Before entering a work email or password, check the domain, check where the authentication flow really goes, and verify the role through the company’s official careers site. Security software with web and phishing protection adds another layer, especially against short-lived scam pages that may only stay online for hours or days.IOCs:hxxps://fifahiring[.]com hxxps://careers-fifahiring[.]com hxxps://fifajobs[.]com hxxps://fifa-mg[.]shop hxxps://heineken-careers[.]com hxxps://aquent-talent-jobs[.]com hxxps://hays-talent-opportunities[.]com fifeq2026eqbackeq[.]onrender[.]com/api/login

Scammers are copying recruitment and calendar-booking flows to make fake Google and Facebook sign-ins look routine. 

Fake hiring pages abuse FIFA and other major brands to steal work credentials 

Threat Research Team Lead

Since January 2026, we have detected 7.4 million malicious incidents tied to residential proxy traffic, affecting 572,000 users in our telemetry. In this model, the household whose connection is being used is often not the buyer of the traffic. It is the exit point.That is the uncomfortable part of residential proxy networks. A home router, laptop, phone, smart TV, or other always-on device can quietly become part of a commercial network that routes strangers’ internet traffic through ordinary consumer IP addresses. To the websites on the other end, that traffic looks as if it came from a real household, because it did.The technology itself is not inherently malicious. Businesses use residential proxies for market research, ad verification, brand protection, security testing, and services that need to see the web as a real user would. But the same feature that makes these networks useful also makes them attractive for abuse: traffic from a household IP address is hard to block without risking false positives against legitimate users.The problem for consumers is simpler. Many people do not realize their device or connection can be enrolled into this kind of network at all. Some users opt in through pay-for-bandwidth apps. Others are enrolled through “free” VPNs, bundled software development kits inside unrelated apps, or unwanted software that installs proxy components without meaningful consent.Here is how the market works, who pays for it, and what it can cost the household that unknowingly hosts it.<h2>What is a residential proxy</h2>A residential proxy is an internet connection – typically a home laptop, phone, smart TV, or router – that has been enrolled into a third-party network and is being used to route someone else's traffic. To the rest of the internet, that traffic looks like it is coming from an ordinary household IP address assigned by a consumer ISP, not from a datacenter or&nbsp;VPN provider.Residential proxy networks are sold commercially. The model is two-sided: some people pay to tunnel their traffic out through someone else's connection (much like a VPN, except the exit point is a real home), while others get paid to provide those exit endpoints. Some are structured as paid peer-to-peer networks, where any participant's connection can serve as an exit for any other.Buyers pay for access to pools of millions of these "real user" IPs because they are extremely difficult to block, or rather, blocking them carries real consequences: the traffic originates from genuine households, so any block risks shutting out a legitimate user – effectively a false positive.Common buyers include market research firms, price-comparison services, brand-protection vendors, security teams testing their own defenses, and consumer privacy tools. At the abusive end of the market, the same infrastructure used by operators running automated purchasing bots, scrapping campaigns, ad fraud, credential stuffing, phishing and other attacks where hiding the true origin of the traffic matters. Demand is also being shaped by large-scale web scraping, including&nbsp;<a href="https://blog.includesecurity.com/2026/06/the-smart-tv-in-your-livingroom-is-a-node-in-the-aiscraping-economy/">scraping for AI training data</a>. As anti-bot systems become better at identifying traffic from datacenters and known VPN providers, residential IP addresses become more attractive because they look like ordinary users. That does not make every residential proxy user malicious. It does mean consumers may be carrying traffic for businesses and actors they have never heard of.<h2>How consumer devices end up in these networks</h2>There are several ways a home device can become part of a residential proxy network. Some are explicit. Others are easy to miss.The most obvious route is the pay-for-bandwidth model. Apps such as Honeygain, IPRoyal Pawns, and PacketStream pay users to share their internet connection. The user opts in, but they may not understand who will buy access to that connection later, what kind of traffic will pass through it, or what happens if that traffic triggers abuse reports.A second route is the “free” VPN or proxy app. Instead of charging a subscription, the app monetizes the user’s connection by reselling bandwidth. In practice, the user may think they are getting a privacy tool while also becoming infrastructure for other people’s traffic.A third route is through bundled software development kits, or SDKs. A developer adds a third-party component to an otherwise unrelated app, such as a flashlight, wallpaper, game, or utility. The developer gets paid when the app is active. The user sees the host app, not the proxy network behind it.The fourth route is malware or potentially unwanted software. In those cases, the proxy component is installed silently or deceptively, with no meaningful user consent at all.<h2>Why the "consent" is rarely meaningful</h2>Even when consent technically exists, it is often not the kind of consent most people would recognize.The disclosure may be buried in a long terms-of-service document. The wording may be vague, with phrases such as “share your unused bandwidth,” “support the app,” or “join our network.” These phrases do not clearly tell the user that arbitrary third parties may route traffic through the user’s home IP address.There is also the question of downstream use. Users are rarely told who buys access to the network or what those buyers do with it. Their connection could be used for benign market research. It could also be used for scraping, automated account creation, ad fraud, credential stuffing, or phishing infrastructure.The problem gets worse when the proxy component sits inside another app. The host app’s privacy policy may mention a partner SDK by brand name, but that still does not make it clear to a normal user that their device can become a proxy endpoint.Uninstalling the visible app may not always resolve the issue cleanly either, especially when proxy components are installed separately or bundled with other unwanted software.<h2>The risks consumers actually carry</h2>When a household device becomes a residential proxy node, the user takes on risks created by someone else’s traffic.The first risk is attribution. Abusive, fraudulent, or illegal activity routed through the device appears to come from the household’s IP address. ISP abuse notices, platform bans, fraud checks, and law enforcement requests point first to the subscriber, not to the person who bought the proxy traffic.The second risk is performance. Continuous third-party traffic can consume bandwidth, slow the home network, hit data caps, or violate ISP terms of service. The user may only notice that the internet feels slower, a bill is higher, or a service suddenly blocks access.The third risk is security exposure. Proxy clients run persistent background processes and maintain network connections so they can relay traffic. That adds always-on networking software to the device, often with limited visibility for the user.The fourth risk is reputation. A home IP address used for abuse can end up on blocklists used by banks, retailers, streaming services, and online platforms. The result can be account friction, failed logins, extra verification steps, or outright blocking from services the household actually uses.There is also a family angle. A child or another household member may install a free game, VPN, or utility without understanding the consequences. The legal, financial, and reputational risk still lands on the account holder.<h2>What our telemetry shows</h2>Since the beginning of 2026, we have detected 7.4 million malicious traffic incidents associated with residential proxy traffic, affecting 572,000 unique users in our user base.&nbsp; This is not a measurement of the entire residential proxy market. It reflects what we observe in our telemetry: malicious traffic, detections, and related activity affecting protected users. The real footprint is far larger, because residential proxy networks span major consumer platforms and many devices outside our visibility.The activity is global, but it is concentrated in a smaller set of countries. India leads by a wide margin, with roughly 83,000 affected users, followed by Vietnam with 61,000 and Brazil with 39,000. The Philippines (23,000), Poland (18,000), Indonesia (18,000) and Thailand (14,000) form a second tier, with Egypt, Spain, Ukraine, Argentina, Mexico and Italy each accounting for 10,000 to 13,000. The spread across South and Southeast Asia, Latin America, and Eastern and Southern Europe reflects where the host apps that carry this software are most widely installed.That geographic spread likely reflects a combination of app distribution, local install patterns, user appetite for free or pay-for-bandwidth apps, and the economics of bandwidth resale. It should not be read as a simple ranking of where residential proxy abuse originates.The chart below breaks down the threats we observed in this proxy-related traffic. Two categories dominate: phishing and malicious advertising. Behind them sits a long tail of consumer-facing fraud, including e-shop scams, generic scams, financial scams, and fake-tutorial pages, alongside outright malware such as trojans and other malicious traffic.&nbsp;

In the malicious proxy traffic we observed, scams and malware dominate. The important point for consumers is not only that their connection may be used by someone else. It is that, when abuse happens, their household IP address can be the return address.<h2>What consumers can do</h2>Most people will not identify a residential proxy component by name. But there are practical warning signs.Review “free” VPNs, proxy apps, bandwidth-sharing apps, and utilities from unknown publishers. Check whether they mention bandwidth sharing, peer-to-peer networking, traffic routing, or network participation in their terms. If an app offers payment for leaving your device online, assume your connection may be resold unless the provider explains exactly how traffic is vetted and who can buy access.On phones and computers, remove apps you do not recognize or no longer use. On home routers, check the list of connected devices and look for anything unfamiliar. If your ISP sends abuse notices, websites start blocking you, or your home IP address suddenly fails fraud checks, a proxy component is one possible explanation.Security software can also help by detecting unwanted proxy clients, bundled components, and related malicious activity. But this should not be a guessing game for consumers. Software that turns a household connection into a commercial exit node should be named clearly and made easy to remove.<h2>The takeaway&nbsp;</h2>With residential proxiesthe user’s connection becomes an infrastructure for someone else’s business.The issue is not that residential proxy technology exists. It has legitimate uses. The issue is whether people know when their devices are part of it, understand what kind of traffic may pass through their connection, and have a real choice to opt out.The fix is not to pretend consumers will read every EULA. This software should be named clearly, shown plainly, and made easy to remove before a household connection becomes someone else’s infrastructure.

Your home connection could be routing traffic for strangers. Here's how residential proxy networks work, how devices get enrolled and what our telemetry reveals about the risks for consumers.

Who's Really Using Your Home Internet Connection?

<h4>Key Points&nbsp;</h4><ul><li data-list-item-id="e43a143fd054561c50dba51071686adff">Gen Threat Labs has been tracking GoFlateLoader, a widespread Golang loader used to deliver multiple infostealers, including Amatera, Remus, Lumma, Vidar and StealC.&nbsp;</li><li data-list-item-id="e3d3b5d0814ea5b26aa75a8d3d6013a39">GoFlateLoader appears both in x86 (32-bit) and x86-64 (64-bit) variants, matching the bitness of the payload it is supposed to execute.&nbsp;</li><li data-list-item-id="e51ccafcfb8c418d894aeb685004a0ba5">The loader is designed for in-memory payload execution and is deliberately inflated with a massive PE overlay to hinder detection</li><li data-list-item-id="ec4bafef8a1113d24bc2b4b3a943228d1">The two main delivery paths are via supposedly cracked software and a <a href="https://research.checkpoint.com/2026/impersonation-click-hijacking-and-tds-inside-a-malware-distribution-ecosystem/" target="_blank" rel="noreferrer noopener">malicious TDS recently documented by Check Point Research</a>, which may redirect users to dedicated landing pages serving password-protected archives containing GoFlateLoader.&nbsp;</li></ul><h4>Introduction&nbsp;</h4>Not every threat that matters is technically sophisticated, and that is also the case with GoFlateLoader, which is a rather simple loader written in Go, whose sole purpose is to decode and execute the payload in memory. What stands out the most is not what the loader does but rather what it does not do – it comes without anti-debugging, anti-VM, or sandbox-evasion checks, and also lacks API hashing or CFG obfuscation, the kind of tricks that loaders almost always come with. Instead, GoFlateLoader relies on one of the simplest yet still effective tricks to stay under the radar – it appends a massive PE overlay at the end of the file, deliberately inflating the binary's size (hence the name GoFlateLoader).&nbsp;Nevertheless, it is important to understand that malware does not necessarily need to be elegant or stealthy. In the end, the only thing that matters to attackers is whether it can successfully deliver the final payload. And if that is the only metric, GoFlateLoader appears to be working fairly well for its operators, as we continue to see it widely distributed across our user base. Since the beginning of April, we have protected more than 33,000 unique users from it, with the most affected countries including Brazil, India, Argentina, Mexico, Turkey and Spain.&nbsp;

<h4>GoFlateLoader’s Execution Flow&nbsp;</h4>If we strip away the obfuscation noise that the Go compiler and the author’s decoy code add, the loader’s execution flow is short and linear:&nbsp;<ul type="disc"><li data-list-item-id="efa51af3b2859fd9c684efca0c82d3b66">Copy the encoded payload blob from the <code>.rdata</code> section onto the stack.&nbsp;</li><li data-list-item-id="e879bb970c6d1a81b8a42512222aceaa4">Decode the payload using a small, multi-stage, custom byte-level transformation into a valid PE.&nbsp;</li><li data-list-item-id="ebff4f340af15b75cbea0d682b26b7645">Parse the PE headers of the decoded payload (image base, image size, sections, data directories, etc.).&nbsp;</li><li data-list-item-id="e37fb26cad3d18a4a0c8bf9b2595d661f">Allocate an RWX memory region using <code>VirtualAlloc</code> with <code>MEM_COMMIT | MEM_RESERVE</code> and <code>PAGE_EXECUTE_READWRITE</code> flags, trying the payload’s preferred <code>ImageBase</code> first and falling back to an OS-assigned address if that fails.&nbsp;</li><li data-list-item-id="ebbde6d77cdc942c0a6bac79f4f8a6c28">Map the decoded payload into the allocated memory region by copying the PE headers first, then walking the section table and writing each section to its target virtual address.&nbsp;</li><li data-list-item-id="ed16738229e0f5c70a981c0c5f9b715f9">Apply relocations (only if preferred base mapping fails).&nbsp;</li><li data-list-item-id="e76cde4bae1be6c21b897c350c6fd15db">Resolve imports by walking the standard <code>IMAGE_IMPORT_DESCRIPTOR</code> table and rebuild the IAT in place via <code>LoadLibrary</code> and<code> GetProcAddress</code>.&nbsp;</li><li data-list-item-id="e786887a3d34fc7ce344ee8375944799e">Transfer execution to the final payload using Go’s <code>syscall.Syscall</code>.&nbsp;</li></ul>&nbsp;This is essentially a textbook example of manual PE loading with the payload reconstructed and executed entirely in memory so that it never touches the disk.&nbsp;&nbsp;

Notably, GoFlateLoader transfers execution to the final payload via <code>syscall.Syscall</code>. In Go, <code>syscall.Syscall</code> has the signature <code>Syscall(trap, nargs, a1, a2, a3)</code>, where <code>trap</code> is the function pointer to invoke, <code>nargs </code>is the argument count, and <code>a1</code>, <code>a2</code>, and <code>a3</code> are the first three arguments passed to the target function. Here, trap is set to point to the final payload’s entry point, <code>nargs</code> is set to <code>1</code>, and since the execution is transferred to that entry point, the remaining arguments serve only as filler values and are never used by the invoked payload. In other words, GoFlateLoader is not issuing a <code>syscall</code> at all, but abusing <code>syscall.Syscall</code> as a generic call gate to hand control to the payload.&nbsp;&nbsp;It is worth emphasizing that GoFlateLoader consistently sets these arguments to the hardcoded values <code>1</code>, <code>2</code>, <code>3</code>, and <code>4</code>, which is unusual enough to stand out as a potential detection opportunity (combined with other patterns).&nbsp;

The loader also carries substantial junk/decoy code, which varies across builds and serves no purpose other than padding the control flow and complicating static analysis.&nbsp;&nbsp;&nbsp;<h4>Inflated PE Overlay&nbsp;</h4>GoFlateLoader’s defining characteristic is its massive size (typically 700-950 MB), caused by an artificially inflated PE overlay appended to the original loader.&nbsp;

This is a deliberate design choice intended to bypass detection, as excessively large executable files may pose challenges for AVs and EDRs, primarily due to performance and resource constraints. As a result, to preserve performance and avoid noticeable impact to end users, these solutions often enforce practical file-size limits for deep scanning and/or emulation.&nbsp;The same logic extends to automated analysis pipelines – bloated files frequently cause timeouts during heuristic analysis and effectively prevent the sample from being uploaded to cloud-based sandboxes or threat intelligence platforms, which enforce their own strict upload limits to manage bandwidth and storage costs. A prime example is VirusTotal, which enforces a strict 650 MB upload limit. The fact that GoFlateLoader consistently sits just above this threshold strongly suggests that VirusTotal is one of the platforms it specifically aims to evade.&nbsp;&nbsp;In most cases, GoFlateLoader’s PE overlays are filled with null bytes, though we have also observed builds with random padding.&nbsp;&nbsp;

Crucially, adding such a PE overlay comes at virtually no distribution cost for attackers, as the inflated data compresses to a fraction of its original size (assuming constant padding).&nbsp;&nbsp;<h4>Delivered Final Payloads&nbsp;</h4>The final payloads distributed through GoFlateLoader are predominantly information stealers, with the most prevalent currently being Amatera, <a href="https://www.gendigital.com/blog/insights/research/remus-64bit-variant-of-lumma-stealer" target="_blank" rel="noreferrer noopener">Remus</a>, and Lumma. Beyond these, we have also observed it distributing Vidar, StealC, and SvitStealer.&nbsp;GoFlateLoader comes in both x86 and x86-64 variants, with each variant matched to the architecture of the payload it is intended to execute.&nbsp;&nbsp;As for how GoFlateLoader binaries are distributed, we have observed at least two notable delivery paths. The first is through supposedly cracked software. The second involves the payload being served via <a href="https://research.checkpoint.com/2026/impersonation-click-hijacking-and-tds-inside-a-malware-distribution-ecosystem/" target="_blank" rel="noreferrer noopener">the malicious TDS recently described by Check Point Research</a>, which redirects users to a landing page that offers a downloadable archive and separately displays the password required to extract it. This approach provides several advantages for the threat actor. First, compression significantly reduces the binary size due to the constant-padded PE overlay. Second, because the archive password is displayed separately on the landing page, it makes it harder for AV and EDR solutions to obtain it and decrypt the archive. As a result, the payload may remain concealed until the victim manually extracts it.&nbsp;&nbsp;&nbsp;<h4>Summary&nbsp;</h4>GoFlateLoader is a reminder that prevalence and sophistication are not the same thing. At its core, it is a rather simple manual in-memory PE loader written in Go, with no novel anti-analysis tricks under the hood. Whatever evasion it achieves comes not from obfuscation or other anti-analysis tricks but from its size – the appended PE overlay is built to slip past size-constrained scanners and automated analysis pipelines. Yet, despite its low level of sophistication, GoFlateLoader remains an active and relevant threat, used to deliver a wide range of prevalent information stealers.&nbsp;&nbsp;Gen products protect our users against both GoFlateLoader and the payloads it delivers.&nbsp;&nbsp;<h4>Indicators of Compromise (IoCs)&nbsp;</h4>Note: The files listed in the IoCs that are not archives all exceed 650 MB in size and are therefore not available on VirusTotal.&nbsp;b88c5744975d2abb447aecc6c090fee9f8580413f4612eecdc6ed1973e8a1739 (password-protected archive containing GoFlateLoader x64 variant loading Remus; pwd: 1234)&nbsp;ed5ae7f36453c5a23e9868a5729d67e0549a11f6dea54f5f52d654a8f51d4902 (archive containing GoFlateLoader x64 variant loading Remus)&nbsp;841c9297cb8a2e0ff89433d13c05bfc760eb2e98e251cb8fa785d2ad7cbac05f (archive containing GoFlateLoader x86 variant loading Amatera)&nbsp;ece7c48eb411b24f26762ede83badb4a644c41d5777129381ac2541804d64fc2 (archive containing GoFlateLoader x86 variant loading Lumma)&nbsp;421ce2d2f49c23bbe9f60ef3b9cd38d7eb912ce02e56a61837656210069bd9e2 (archive containing GoFlateLoader x64 variant loading Vidar)&nbsp;121c2dc793b3873f75a29ec02241f94136de19c049382a50a50d0d5b99507073 (GoFlateLoader x64 variant loading StealC)&nbsp;2415db5081cec9bfd14ad6da1a66169fd96f13a49010c319a73d1ed6fafd4efa (GoFlateLoader x64 variant loading Vidar)&nbsp;d9917ade3b4c125a95b5d3e6343cde26145dfbf569bd7e2a843fd0c6fc8ddc28 (GoFlateLoader x64 variant loading Remus)&nbsp;4cf6893756f441522b94b36f10e5de0e47aeed4743f95c51650746d1ecf97e3d (GoFlateLoader x64 variant loading SvitStealer)&nbsp;8b89d6c9152d3aab97aadd515ecb69ca72654db2f25425759ba4b646853d737d (GoFlateLoader x86 variant loading Lumma)&nbsp;90ce4ff9da23ac150da0a8e17930cab1e369aa349fdc1b65691b70369145664a (GoFlateLoader x86 variant loading Amatera)&nbsp;

 Intentionally Inflated Go Binaries That Use Massive PE Overlays to Bypass Detection

GoFlateLoader: A Widespread Golang Loader Delivering Multiple Infostealers

This is a post that I’ve been meaning to write already for some time. It’s based on work I did and a <a href="https://docs.google.com/presentation/d/187u9XixvtIJLeSjqdXI1j00_mUkJF9DP94rgsXihbk0/edit?usp=sharing">presentation I gave at our Scala guild meeting</a> a few months ago. It explains how we can use Rust code from Scala (on JVM) via <a href="https://docs.oracle.com/en/java/javase/17/docs/specs/jni/index.html">JNI</a> with the help of <a href="https://github.com/sbt/sbt-jni/">sbt-jni</a>. If you want to dive straight into example code, <a href="https://github.com/sideeffffect/scala-rust-interop">here’s the self contained example project</a>.<h2>Scala and Rust synergy</h2>In our team, we use both Scala and Rust. Both are very powerful languages in their own regard, each with unique strengths (and also weaknesses). Whereas Scala is very high level, with a powerful type system and sits on the battle-tested JVM, Rust allows you to control low level details, enabling great CPU and/or memory efficiency without sacrificing safety. Thus, these two languages complement each other very well. We already have applications written in Scala, and some written in Rust, depending on which technology was better suited for the job. But can we have one application written in both languages?<h2>Using native code from JVM</h2>On the Java Virtual Machine – since we run Scala on JVM – we have a couple of options of how to call native code.<ul><li>JNI (<a href="https://docs.oracle.com/en/java/javase/17/docs/specs/jni/index.html">Java Native Interface</a>) is the official way. It is fast, but a bit clumsy, it requires programmers to write a lot of boilerplate interoperability code.</li><li>JNA (<a href="https://github.com/java-native-access/jna">Java Native Access</a>) is relatively easier on programmers, but is slower, because it relies on dynamic behavior.</li><li>The <a href="https://openjdk.java.net/projects/panama/">OpenJDK project Panama</a> attempts to make it significantly easier to interoperate with native code, but is still in the experimental stage. (Check out SLinC to use it from Scala: <a href="https://github.com/markehammons/SLInC/tree/second_prototype">GitHub repository</a> and an <a href="https://dev.to/markehammons/slinc-update-5afi">introductory blogpost</a>.)</li><li>Of honorable mention is <a href="https://scala-native.readthedocs.io/">Scala Native</a>, which compiles Scala to native code (in the same manner as is typical for Go or Haskell or Rust for that matter), allowing for direct use of native libraries. But it is also in the experimental phase of its life and, more importantly, by its nature the code doesn’t run on JVM, so it’s not relevant for our use case.</li></ul>Because JNI is the official and most widely used option, with the benefits like most resources available online, easy googlability of issues, etc, I decided to explore it further. But the vast majority of articles or tutorials about JNI focus on calling C code from Java. There are also few resources and tools available for calling Rust from Java, or C from Scala. But it seemed like calling Rust from Scala hadn’t been done before. I wanted to change that – this very post is part of the effort.Scala, as such, has support for using JNI, using the <code>@native</code> annotation, but it requires a lot of boilerplate code. Thankfully, there is <a href="https://github.com/sbt/sbt-jni/">sbt-jni</a>, a suite of sbt plugins, libraries and macros, originally developed by <a href="https://github.com/jodersky">Jakob Odersky</a>, that make using JNI from Scala easier (helps with boilerplate on Scala side, boilerplate on the native side is still there). But it had only support for working with C and the CMake build tool. <a href="https://github.com/sbt/sbt-jni/pull/42">Adding support for Rust and Cargo</a> (the build tool for Rust) was surprisingly easy! Unfortunately, at that time sbt-jni wasn’t maintained, but after some time, the project was taken over by <a href="https://github.com/pomadchin">Grigory Pomadchin</a> and migrated under the <a href="https://github.com/sbt">github.com/sbt</a> organization and the PR was merged afterall.There is also a Rust library <a href="https://docs.rs/jni/latest/jni/">jni</a> which makes it easier to work with JNI on the Rust side. We will use that too.So let’s see how sbt-jni and jni help us to use Rust and Cargo with Scala.<h2>Calling Rust code from Scala with the help of sbt-jni and the crate jni</h2>To read all the details about how to use sbt-jni, have a look at its <a href="https://github.com/sbt/sbt-jni/#readme">README</a>. Here I’ll point out only the important bits relevant for Rust.<h3>The sbt part</h3>When working with sbt-jni, it is a good practice to have two sbt sub-projects. One is dedicated only to native code, Rust in our case. When packaged into a JAR, it will contain only the compiled <code>.so</code> library (on Linux). This is the sub-project that needs to have the plugin <code>JniNative</code> enabled. We should set the key <code>sourceDirectory</code> for task <code>nativeCompile</code> (this task comes from <code>JniNative</code>) to <code>baseDirectory</code> (in our case has value <code>"scala-rust-interop/native"</code>), because we want to have directory structure like this:

The sbt definition then should look like this: &nbsp;

The other sub-project is dedicated to the Scala wrapper around the native code. <code>sbtJniCoreScope := Compile</code> is an sbt-jni specific way to make sure that the <code>core</code> library will depend on the <code>sbt-jni-core</code> library which contains the class <code>NativeLoader</code>. More on <code>NativeLoader</code> later. When using sbt-jni, it is also a good practice to not depend on the <code>native</code> library, users of <code>core</code> should provide this dependency explicitly.

We don’t need to add any further Rust specific configuration to sbt, sbt-jni is clever enough to autodetect that we’re building a Rust project with Cargo if there is a <code>Cargo.toml</code> file in the directory <code>nativeCompile / sourceDirectory</code> points to. We’ll get to the content of the <code>native</code> Rust sub-project later, let’s first have a look at the Scala wrapper, the <code>core</code> sub-project.<h3>The Scala part</h3>We create a class that extends <code>NativeLoader</code>. This makes sure that the correct native library will be linked to the JVM process. The argument to <code>NativeLoader</code> should be the name of the library we want to wrap around, but without the <code>lib</code> prefix. Our class contains methods which don’t have definition, instead they are annotated with <code>@native</code>. When they’re called, the implementation in the respective native library will be executed. Note that the annotation doesn’t come from sbt-jni, but from Scala standard library.sbt-jni also provides an alternative to extending <code>NativeLoader</code>, the <code>@nativeLoader</code> macro, but that doesn’t work on Scala 3, so we ignore it in this post. The <code>NativeLoader</code> way works on both Scala 2 and Scala 3.

<h3>The Rust part</h3>Let’s now get to the <code>native</code> sub-project. The content of <code>Cargo.toml</code> is pretty simple, we add the <a href="https://github.com/jni-rs/jni-rs"><code>jni</code></a> library and the resulting artifact is to be a dynamically linkable library (<code>cdylib</code>).

The content of our Rust library is the most elaborate part. Of utmost importance is the signature of the function(s). <code>#[no_mangle] pub extern "system"</code> makes a function callable as if it were written in C. The name itself and the parameters and return type have to follow the package and name of the class and the parameters of the respective method in question.It may be tricky to write this manually from scratch, so we can use the help of the sbt-jni task <code>javah</code>. It generates C header files, which is a necessity when using JNI for C code. We write Rust, not C, but it’s easy to infer the expected Rust function signatures from the C prototypes.

That prototype we translate to Rust like this: &nbsp;

The first parameter <code>JNIEnv</code> is very important. Many JNI operations are done via it. Note that it is a wrapper from the Rust <code>jni</code> library, not directly a functionality from bare JNI itself. For example, accessing a field of a class is done like this: &nbsp;

<code>object</code> is the second parameter of our function, the object whose field we want to access. <code>"numerator"</code> is the name of the field and <code>"I"</code> is the type of the field (<code>"I"</code> stands for primitive <code>int</code> in Java bytecode). Note how both are strings and so are not statically verified by the compiler and so can fail in runtime – thus the <code>unwrap</code>s.We also shouldn’t forget that in Rust, <a href="https://doc.rust-lang.org/nomicon/ffi.html#ffi-and-panics">panics across FFI boundaries are an undefined behavior</a>. To avoid doing that, we should catch them using <code>catch_unwind</code>. We may then try to convert the error to a string. Regardless, if we catch a panic, we should indicate to the calling Scala code that the native code failed. A natural solution is to throw a JVM exception, using <code>throw_new</code>. Note that calling this function doesn’t immediately throw. You still need to finish the native function, for example by returning some value. It doesn’t matter which, because it will be discarded. Only then will the exception be thrown.

All together, the sample Rust code looks like this: &nbsp;

<h2>Conclusion</h2>In this post, we have seen a short tutorial on how to call to native Rust code from Scala on JVM via JNI. Scala and Rust can be used together for great synergy, yet there doesn’t seem to be any documentation on how to do that – I hope this post has helped to mitigate this blind spot. While using JNI can be a bit bothersome, there are tools and libraries that help, like the Rust crate <code>jni</code> and Scala’s suite sbt-jni. Initially, there wasn’t support for compiling Rust in sbt-jni, but support was eventually added and I’m happy to report that it already seems to be <a href="https://github.com/zio/zio-uring/tree/master/zio-uring-rs">used in the wild</a>.I would like to thank <a href="https://vorner.cz/">Michal Vaner</a> (our Avast ex-colleague) for his help on the Rust side, particularly for making me aware of the issue with panics across FFI and instructing me how to catch the errors and convert them to a sensible string. I’m also grateful to the <a href="https://github.com/jni-rs/jni-rs/graphs/contributors">authors of the Rust crate jni</a>, to <a href="https://github.com/sbt/sbt-jni/graphs/contributors">Jakob for creating sbt-jni and to Grigory for taking over the maintainership</a>.Head over to <a href="https://github.com/sideeffffect/scala-rust-interop">github.com/sideeffffect/scala-rust-interop</a> to get your hands on the code, it is executable (<code>sbt core/run</code>) and there are also sample tests (<code>sbt test</code>). I hope using Scala together with Rust becomes more popular in the future, the benefits are significant. Merry Scala and Rust hacking, everyone!

Unlocking Scala with Rust's Power

Scala and Rust interoperability via JNI

In a <a href="https://engineering.avast.io/automating-pr-merging-with-bulldozer-and-policybot/">previous blog post</a>, we’ve talked about Bulldozer and PolicyBot and how they automate merging Pull requests for you. In this post, we will talk about another piece of the automatization puzzle: <a href="https://github.com/spotify/missinglink">Missinglink</a> and it’s plugin for sbt, <a href="https://github.com/scalacenter/sbt-missinglink">sbt-missinglink</a>. This tool checks that all required classes and methods are present on the classpath.Together with Scala Steward Bulldozer/PolicyBot, it will allow us to achieve a fully automated luxury pipeline for keeping our projects up to date. Pipeline that will automate away as much work as possible while being safe to run without humans needing to supervise them. Since we use Scala, we will put an emphasis on sbt-missinglink, but Missinglink is a tool usable for anybody who runs on the JVM, because it works on the level of Java bytecode.<h2>The hazard of updating dependencies on the dynamic JVM</h2>Scala has a powerful type system, which is a great tool for ensuring that errors are caught during development or CI and not at runtime after deployment to production, where they can cause serious issues. However, in contrast to Scala and Java the language, the Java Virtual Machine is very dynamic. It links all classes together only at runtime and that can have unwanted consequences, especially surprising to developers used to thinking “if it type-checks then it will run correctly”. You can learn more about the dynamic nature of the <a href="https://worace.works/2021/04/04/jar-hell-part-1-compilation-classpath-libraries/">JVM at Welcome to JAR Hell series</a>.When Scala Steward opens a new PR updating a version of a dependency, we cannot be entirely sure it can’t break anything. Of course, in CI, we compile the project, so we can be sure that any class or method that our project directly uses is present with the expected signature (if not, the compilation will fail). But what if some other class or method in the library being updated, used only transitively by another library, changes? These issues are called <a href="https://docs.scala-lang.org/overviews/core/binary-compatibility-for-library-authors.html">“binary incompatibility” in Scala circles</a> (even more <a href="https://medium.com/zendesk-engineering/maintaining-binary-compatibility-in-scala-6e07157aac23">here</a>). It can lead to <code>NoClassDefFoundError</code> or <code>NoSuchMethodError</code> in runtime. Such issues can still be uncovered in CI by tests, but that depends on your test coverage, which is difficult to have 100%.Taken all together, this means that automatically merging every PR from Scala Steward (or any other updating tool, like Renovate or Dependabot) is too risky for our taste, let alone deploying directly to production after merge. We need a tool that will in CI make sure that a particular update of a dependency doesn’t introduce binary incompatibility.<h2>Missinglink to the rescue</h2>Missinglink is a tool developed by Spotify, designed to solve precisely the issues outlined above. It analyses the bytecode of all classes in all libraries (JARs) that our project uses. Because it sees all the code that can be called at runtime, it can verify that indeed all the called methods and used classes are available. By the virtue of working at the level of Java bytecode, it is agnostic of the language you work with — we use it for Scala, but it should work well for Java or Kotlin too.Since we compile our Scala programs with sbt, we use the <a href="https://github.com/scalacenter/sbt-missinglink">sbt-missinglink</a> plugin which wraps the Missinglink tool. You can add it to your <a href="https://github.com/avast/scala-server-toolkit/blob/2da9092c22db2909f8e1cb7bfac85c7c4da7852b/project/plugins.sbt#L9">plugins.sbt</a> like this:

Besides sbt, there is a plugin for Maven, but sadly none for Gradle. Yet! Will you be the one to take up the challenge?To run the tool in CI or locally on your computer, use the sbt task <code>missinglinkCheck</code>. If it finds conflicts, you will see a log output like this (Pureconfig, together with Circe, are common offenders with regards to binary incompatibility):

In this particular case, this output means that <code>rabbitmq-client-pureconfig</code> depends on a version of Pureconfig which is binary incompatible with the version of Pureconfig which is actually present in the project. The fix was to adjust the Pureconfig version used in the <a href="https://github.com/avast/rabbitmq-scala-client"><code>rabbitmq-client</code> project</a>. If we didn’t use Missinglink, our application would surely crash at runtime.In general, the output of sbt-missinglink can be quite long, especially overwhelming when setting it up for the first time. But in our experience, it is enough to focus on the lines <code>In artifact: some-library-1.2.3.jar</code> which tell you what the problematic libraries are and you can work from there. sbt-missinglink developers are <a href="https://github.com/scalacenter/sbt-missinglink/issues/21">aware of this problem</a>, so hopefully it will improve in the future.Another downside of Missinglink is that it can report false positives: methods/classes which are not present, but won’t be ever called in our application. Typically, those are Java libraries, often related to logging; we can speculate that those occur because these libraries use reflection heavily. Thankfully, you can filter such false positives out via <code>missinglinkExcludedDependencies</code>. The rule of thumb is that if you encounter a new conflict and it’s a Java library, you can safely exclude it, however, if it is a Scala dependency, chances are that it is a true positive and you should investigate further. For simplicity, at Avast, we reuse the same long list of offending JARs in every project:

Missinglink can consume a lot of memory for its analysis, but you can simply increase the heap size by passing <a href="https://github.com/avast/scala-server-toolkit/blob/2da9092c22db2909f8e1cb7bfac85c7c4da7852b/project/plugins.sbt#L1"><code>-J-Xmx</code></a> to sbt.We’ve made sure that (sbt-)missinglink is production ready for modern enterprise apps: it can now <a href="https://github.com/spotify/missinglink/pull/125">handle so called Multi-Release JARs</a> and can <a href="https://github.com/spotify/missinglink/pull/129">run concurrently</a> on <a href="https://github.com/scalacenter/sbt-missinglink/pull/34">all sbt modules at once</a>, thus reducing the wall time spent on the analysis (which could otherwise take a long time). You can still limit the degree of concurrency with <a href="https://github.com/scalacenter/sbt-missinglink#limiting-the-concurrency"><code>concurrentRestrictions</code></a>, fox example to 4 modules at a time:

This limit can help if you don’t have enough memory to give to sbt if it were to run the analysis on all modules at once.<h2>The quest for binary compatibility continues</h2>Preventing issues in CI, before they happen in production is great. Missinglink is a tool for preventing linking errors on the very dynamic Java Virtual Machine. It works great on its own, but in our opinion it’s a must if one wants to have automated merging of dependency update PRs. So now we have a pipeline set up that Scala Steward opens a PR updating a dependency, in CI we run compilation, tests and Missinglink and if all that passes, Bulldozer can safely merge such PR, no need for humans to check anything. It works even more reliably, because a human could only guess whether a dependency update could cause binary incompatibility.Preventing linking errors in CI helps us improve our automation game. Our team is considering automating the deployment of our applications from master on each push/merge and we can do that safely, because we know that our app won’t crash because of linking errors, caused by a dependency update or any other reason.There is also the other side of the binary compatibility coin. Missinglink ensures that the libraries you use won’t crash your app. But if you are a library developer, you need to make sure that you stay binary compatible, so that you won’t break your users’ apps. There are tools for that, like <a href="https://github.com/lightbend/mima">MiMa</a> and the <a href="https://github.com/scalacenter/sbt-version-policy">sbt-versioning-policy</a> which has recently released a 1.0 version. In our team, we’re now looking at these tools, but that will be for a future post.

Don’t suffer from NoClassDefFoundError or NoSuchMethodError and similar issues in production, use Missinglink to identify issues in CI!

Fully automated luxury pipeline for updating dependencies in Scala projects with Missinglink

As you might have read in our <a href="https://engineering.avast.io/running-scala-steward-on-premise/">previous post</a>, we’re happily using <a href="https://github.com/scala-steward-org/scala-steward">Scala Steward</a> for our public and internal Scala projects. It is a bot that keeps them up to date by opening Pull Requests with updated versions of dependencies. That alone is a great help, but it’s only half of the job, a developer has to come in, verify the PR that it makes sense and click the Merge button – it is still not fully automated. In this post, we will explore ways to increase automation regarding PRs and thus decrease the burden on developers.For our public projects, we use <a href="https://mergify.io/">Mergify</a>, a free service for automation of merging PRs. Check for example these projects: <a href="https://github.com/avast/scala-server-toolkit">SST</a>, <a href="https://github.com/avast/grpc-json-bridge">grpc-json-bridge</a>. For projects hosted on GitHub.com, it’s something we can greatly recommend, but what could we use for our internal GitHub Enterprise? Besides Scala Steward, we’re using <a href="https://github.com/renovatebot/renovate">Renovate</a> for non-Scala projects, which can merge its PRs, but we would prefer to have just one tool for automated merging. That’s where Bulldozer and PolicyBot come in.<h2>Bulldozer</h2>Developed by Palantir and released as Free software, <a href="https://github.com/palantir/bulldozer">Bulldozer</a> is a bot, a GitHub App to be more precise, for merging Pull Requests. It doesn’t have a publicly hosted instance, like Mergify for example, but it is published as a container image, which makes it easy to deploy internally at our organization.We deployed it to our internal Kubernetes. The app itself is configured via environment variables. This is an excerpt from the Kubernetes deployment configuration:

Note that <code>GITHUB_APP_INTEGRATION_ID</code> is what GitHub calls “App ID”. To get the id, Bulldozer first has to be registered as an App to your GitHub team. The link where to do that could look something like <code>https://git.company.com/organizations/your-team/settings/apps</code> and there you click the New GitHub App button in the top right corner. The specific details of what permissions are necessary and other installation related things can be found in the Bulldozer’s official <a href="https://github.com/palantir/bulldozer#deployment">README</a>.After you have Bulldozer App registered and the container up and running, you can install it for specific repositories or whole organizations in the Install App tab in the App’s administration panel (<code>https://git.company.com/organizations/your-team/settings/apps/bulldozer/installations</code>). We installed it for whole organizations, because it’s comfortable and safe, because the bot won’t touch repositories that don’t have the <code>.bulldozer.yml</code> configuration file in the root.Bulldozer’s single purpose is to merge PRs. It merges only those PRs for which the required status checks pass and that satisfy certain criteria. The important criteria for us were <code>labels</code> and <code>pr_body_substrings</code> and the build checks are the TeamCity CI and PolicyBot (more on that later). For a Scala project, the <code>.bulldozer.yml</code> configuration file could look like this:

Notice how we tailored the <code>pr_body_substrings</code> to the messages Scala Steward puts into its PRs, so that everything besides major upgrades of libraries will get automatically merged. We also have the <code>bulldozer-merge</code> label. That is a very useful feature (unrelated to dependency bots), if you open a PR and don’t want to wait for the CI to finish. Just add this label to the PR and right after all necessary checks are green, Bulldozer will merge the PR for you, you don’t have to go back to it. If you are using Restrict who can push to matching branches in Branch protection rules, make sure that Bulldozer is allowed push access there.As you can see, Bulldozer can be very powerful, but maybe even too powerful. It could wreak havoc on your repositories if something went really really wrong – by merging PRs, it’s changing the code in your <code>master</code> branch after all. It most likely won’t, it is developed by competent developers and it has even been vetted by our internal security team, but still, it’s better to be safe than sorry. Isn’t there a way to further curb the possibilities of what Bulldozer can and cannot do, what files it is allowed to change?<h2>PolicyBot</h2>This is Bulldozer’s sidekick. It is developed by the same team, deployed and configured in the same manner. The two bots can work independently, one without the other, but they play very well together. What <a href="https://github.com/palantir/policy-bot">PolicyBot</a> does is create a new kind of check in a GitHub PR that can be red or green based on a very granular criteria, like the full file names of the changed files.As Bulldozer, PolicyBot is easy to deploy to k8s:

Note that <code>GITHUB_OAUTH_CLIENT_ID</code> is what GitHub calls “Client ID”. Register the app first with GitHub and then install it for repositories or organizations. More details about the settings, deployment and other configuration is at PolicyBot’s <a href="https://github.com/palantir/policy-bot#deployment">README</a>. List of changed files is only a small fraction of what it can do, so check out the documentation, maybe you will be able to employ the bot for other purposes besides guarding version upgrade PRs.The configuration file is <code>.policy.yml</code> and only projects that contain it will get this new PR check. For Scala projects, we use configuration that looks something like this:

You can see that we limit the files in a Scala Steward’s PR that Bulldozer can merge to only those that contain versions of libraries, sbt plugins, sbt itself or of Scalafmt. If Scala Steward changed anything else, the PolicyBot’s check would be red and Bulldozer wouldn’t merge such PR. (By the way, that can happen for legitimate reasons, because Scala Steward can <a href="https://github.com/scala-steward-org/scala-steward/blob/master/docs/scalafix-migrations.md">apply Scalafix rules when updating certain libraries</a>, but those result in changes to the code so we want a developer to have a look at it before merging.) Something similar could be said about Renovate that is keeping our TeamCity configuration for the project up to date.PolicyBot comes with a nice web UI, where it explains how it came to the conclusion it made. You can get to it by clicking Details on the right side of the policy-bot: master row in the PR checks. Here’s an example of how it can look:

<h2>Conclusion</h2>Bulldozer and PolicyBot have saved us developers a ton of onerous work since we’ve deployed them. We hope that this post has inspired you to think about how you can automate your PR workflow. We owe a big thank you to the nice folks at Palantir who developed these two bots, we couldn’t have done this without them. They were also receptive to our improvements (<a href="https://github.com/palantir/bulldozer/pull/189">1</a>, <a href="https://github.com/palantir/go-baseapp/pull/32">2</a>, <a href="https://github.com/palantir/policy-bot/pull/211">3</a>), so thanks again, <a href="https://github.com/bluekeyes">@bluekeyes</a> in particular!To automate PRs to the maximum extent possible and to do that fearlessly, we should talk about preventing runtime failures caused by linking incompatible transitive dependencies and <code>ClassNotFoundException</code> or preventing PRs that could introduce such issues from being merged. But those are issues specific to Scala/JVM and this post is already long enough, so we will explore this in a future post, where we will talk about <a href="https://github.com/scalacenter/sbt-missinglink">sbt-missinglink</a>.

Ondřej Pelech

Posts by Ondřej Pelech

Scala and Rust interoperability via JNI

Fully automated luxury pipeline for updating dependencies in Scala projects with Missinglink

Automating PR merging with Bulldozer and PolicyBot