Understanding and Analyzing Appraisal Systems in the Underground Marketplaces

Understanding and Analyzing Appraisal Systems in the Underground Marketplaces

Speakers: Zhengyi Li

Conference: NDSS Symposium

YouTube: (no public video)

Overview

This talk, presented by Zhengyi Li at the NDSS Symposium, delves into the burgeoning phenomenon of appraisal systems within underground online marketplaces. These illicit platforms, much like their legitimate e-commerce counterparts such as Amazon or eBay, employ feedback mechanisms to build trust and assess product quality, a critical function in environments rife with anonymity and potential for fraud. The appraisal system represents a sophisticated evolution of these mechanisms, wherein vendors offer free samples, termed vouch copies, to qualified members, or appraisers, in exchange for detailed, in-depth technical reviews.

The core problem addressed by this research is the significant lack of systematic study into these appraisal systems. While similar concepts exist in legitimate contexts like Amazon Vine, their operation and implications within the underground remain largely unexamined. The challenge is compounded by the forum-like structure of many underground marketplaces, where various types of content—discussions, disputes, and reviews—are intermingled without clear categorization, making the identification of appraisal reviews particularly difficult.

This groundbreaking study provides the first comprehensive measurement and analysis of the appraisal system. It aims to demystify its ecosystem, elucidate the roles of its participants, and, most importantly, uncover its substantial potential as a novel source for extracting valuable Cyber Threat Intelligence (CTI). The detailed technical insights embedded within these appraisal reviews can significantly complement existing CTI gathering efforts, offering a unique window into the evolving threat landscape and the operational intricacies of cybercriminal activities.

Background

[▶ Watch: Introduction to underground market appraisal systems (0:00)]()

Underground online marketplaces serve as virtual hubs for illicit transactions, ranging from malware like ransomware and Remote Access Trojans (RATs) to various scam services. While anonymity is a foundational principle of these platforms, it inherently fosters an environment susceptible to deception. To counteract this and build a modicum of trust, many marketplaces have adopted feedback systems. The appraisal system stands out as a recent and highly evolved form of such feedback. In this system, vendors selectively distribute free trial samples, or vouch copies, to trusted members—the appraisers—who then post comprehensive, insightful reviews under the vendor's product listing. This mechanism is generally considered more reliable than traditional review systems due to vendors' stringent selection criteria for appraisers and the depth of evaluation provided, often covering price, functionality, unique features, and potential drawbacks.

Historically, the presence of appraisal systems in underground markets was nascent, with only a few platforms like Hack Forums and BlackHatWorld exhibiting limited activity before 2010. However, a significant shift occurred post-2014, marked by the emergence of new, prominent platforms such as Evolution and Nulled. This period saw the widespread adoption of appraisal systems and a steady increase in the number of active appraisers, indicating a maturing and institutionalization of this trust-building mechanism within the illicit ecosystem.

The broader context for this research is Cyber Threat Intelligence (CTI), defined as evidence-based knowledge providing context, mechanisms, indicators, implications, and actionable advice regarding existing or emerging threats. CTI is indispensable for comprehending the dynamic threat landscape and identifying adversary tactics. Traditional CTI sources include structured attack artifact datasets, technical blogs, and academic research. In recent years, underground marketplaces and forums have been recognized as valuable CTI sources, offering insights into malicious ecosystems. This study uniquely identifies appraisal reviews as a novel and exceptionally rich source of CTI, providing granular technical details about malicious products that are often absent from other sources. While prior research has examined general feedback systems in underground markets and CTI extraction from technical articles or hacker forums, this work distinguishes itself by specifically targeting the sophisticated and under-explored appraisal system.

Key Findings

[▶ Watch: Detailed explanation of underground appraisal system mechanics (2:00)]()

The comprehensive analysis of 18,701 unique appraisers and 56,229 appraisal reviews across eight marketplaces from 2008 to 2023 yielded several significant findings:

• Finding I: Prevalence and Growth of Active Appraisers: The study identified a substantial and growing number of appraisers, with Hack Forums hosting the most (43.1%, or 8,067), followed by BlackHatWorld (34.8%, or 6,505). The appraisal system first appeared in BlackHatWorld and Hack Forums around 2008 and saw widespread adoption after February 2014. The ratio of monthly active appraisers to active vendors remained stable, underscoring the system's integral role in marketplace operations. For instance, Hack Forums experienced a remarkable 5,040% growth in appraisers between 2009 and 2014. Official appraisers demonstrated higher activity, averaging six reviews compared to three from detected appraisers, with one top contributor on MPGH posting 277 reviews.

• Finding II: Stringent Merits for Appraiser Selection: Vendors employ strict criteria to ensure the trustworthiness of appraisers. Five common merits were identified: VIP Status (67.3% of listings), Minimum Number of Posts (29.2%, with 46.5% of listings requiring at least 500 posts; appraisers typically had over 1,000 posts), Minimum Reputation Score (8.9%, appraisers having significantly higher scores), Length of Membership (3.3%, appraisers averaging 816 days), and preference for Marketplace Staff (12%) due to their high prestige.

• Finding III: Identification of Less-Trusted Appraisers: Analysis of scam reporting sub-forums revealed instances of less-trusted appraisers, constituting 0.2% of the total (22 identified). Categories included appraisers posting fake positive reviews for friends (sometimes for monetary gain, e.g., $100), vendors using multiple accounts to post self-serving fake reviews, and the formation of fake appraiser groups dedicated to "bumping" sale threads with fabricated endorsements.

• Finding IV: Appraiser's Private Interactions with Vendors: An examination of 800,593 private messages from the Nulled dataset uncovered 26 appraisal-related conversations. These interactions sometimes showed vendor interference, where vendors provided templates for positive reviews or instructed appraisers to omit sensitive information (e.g., "Don't post any screenshots please, event names are secret"). Conversely, some appraisers posted negative reviews strategically to bargain for vouch copies, while vendors provided technical support to appraisers to ensure successful product testing and prevent negative feedback.

• Finding V: Comprehensive Appraisal of Diverse Products: Products undergoing appraisal were categorized into 11 types, including Account, Social Booster Services, Email, Video Game, Malware, RAT, Botnet, Website, Hosting, Making Money Guide, and Others. Website-related products were most frequently appraised (35.2%), followed by making money guides and accounts, partly due to BlackHatWorld's focus on SEO products. Appraisers often specialized, preferring to review products within the same category to establish expertise. Assessment merits were diverse, covering Delivery Speed (Turnaround Time, speed), Product Characteristics (e.g., control panel design, detectability, stability, functionality for malware; grammar, originality, ease of use for guides), Customer Service, and Value (price vs. quality).

• Finding VI: A Novel Taxonomy of Cyber Threat Intelligence (CTI): The study developed a new CTI taxonomy, specifically tailored for underground marketplaces and encompassing products beyond traditional malware. It identified 41 unique types of CTI across three primary categories:
• Website CTI: Related to black-hat SEO, including keyword search volume, competition score, CPC, article length, types and numbers of backlinks, search engine ranking scores, domain age, and geolocation of backlinks.
• Account CTI: Features for detecting fake accounts, such as account age, profile information, upvotes/downvotes, views, verification methods, and social booster service metrics (followers, likes, tweets, friends, geolocation of followers).
• Malware CTI: Traditional Indicators of Compromise (IOCs) like file hashes (MD5, SHA1, SHA256), filenames, operating system, version, size, programming language, dependencies, anti-virus detection results, FUD status (Fully Undetectable), and auto-update features.

• Finding VII: Appraisal Reviews as a Superior Source of CTI: A critical finding was that appraisal reviews provide significantly more, and often unique, CTI compared to product listings and non-appraisal reviews.
• Higher CTI Density: 50.2% of appraisal reviews contained CTIs, dramatically higher than 8.9% in listings and 2.7% in non-appraisal reviews. This was most pronounced in the website category (78.5% in appraisal reviews vs. 0.2% in listings).
• Unique CTIs: For malware, 492 out of 526 identified CTIs (a striking 93.5%) were unique to appraisal reviews, not found in listings or other reviews. For example, an appraiser detailed three filenames, sizes, MD5/SHA1 hashes, and antivirus scan results for a "Vox Office Builder," information absent elsewhere.
• Contradictory CTIs: Six instances were observed where appraisal reviews directly contradicted vendor claims. A vendor's claim of "Hyper Downloader" being FUD was disproven by an appraiser reporting immediate detection by their "Chinese version 360 AV."
• Comparison with Public CTI Sources: Submitting 493 malware hashes from appraisal reviews to VirusTotal and DigitalSide revealed that only 2 (0.4%) were labeled malicious, and 482 (97.8%) had no linked record, suggesting attackers avoid public submission. Furthermore, while most public white papers focused on malware, appraisal reviews covered a broader range of illicit products (crypters, miners, keyloggers, botnets) and provided granular assessments of malware's encryption algorithms, price, detectability, UAC bypass capabilities, and detection rates—details often missing from public reports.

Technical Deep Dive

[▶ Watch: Appraisal reviews identified as novel CTI source (3:10)]()

The study's robust technical approach centered on two main pillars: comprehensive data collection and sophisticated appraisal review identification.

Data Collection:

The measurement study leveraged a vast dataset spanning nearly two decades, comprising 17,340,789 communication traces from December 2006 to March 2023, collected from eight prominent underground marketplaces and forums focused on malware and cyber products/services. The data sources included:

1. CrimeBB Underground Marketplaces Dataset 89: This provided 12.7 million communication traces and 812,080 user accounts from seven marketplaces (BlackHatWorld, HackForums, MPGH, V3rmillion, OGUsers, Raid, and Nulled) from 2006 to 2020. An additional 753,933 traces from scam reporting sub-forums were also collected to investigate appraiser credibility.
2. Nulled Database 87: This dataset contributed 121,499 traces from February 2015 to May 2016 and 599,085 user accounts. It was combined with Nulled traces from CrimeBB (January 2018 - July 2019). Crucially, it also provided 800,593 private messages exchanged among 36,606 users, which were instrumental in examining potential collusion and vendor interference.
3. Dark Net Markets (DNM) Archive 49: From this, 9,385 traces were obtained from the Tor-based marketplace Evolution, covering February 2014 to November 2014. Marketplaces were filtered to include those with at least 50 listings or a primary focus on cyber products.
4. Self-Scraped Dataset: To bridge the data gap in CrimeBB from 2020 to 2023, the researchers developed Selenium-based scrapers 19 for five active marketplaces (BlackHatWorld, HackForums, MPGH, Nulled, V3rmillion). Data completeness was rigorously ensured by checking HTTP status codes, page sizes, session expiry, and handling CAPTCHAs.

The completeness of the self-scraped data was validated through over-time consistency checks and comparisons with statistics from other studies. Cumulative listing counts (Figure 2, page 4 of the paper) showed smooth upward trends, confirming good completeness, with observed plateaus on V3rmillion and HackForums in mid-2020 aligning with reports of decreased trading activity during the COVID-19 pandemic.

Appraisal Review Identification:

Identifying appraisal reviews amidst the vast, unlabeled forum data was a significant methodological challenge. The researchers developed a robust, hybrid approach combining groundtruth identification, machine learning, and keyword matching.

1. Groundtruth Appraisal Reviews: The process began by identifying official appraiser groups, such as the "Official Reviewers Group" and "Official Appraisers" in Hack Forums. These groups, often administered by marketplace staff, follow public application processes and specific review templates. Manually designed regular expressions (regex) were used to match these templates, yielding 1,927 groundtruth appraisal reviews from 379 appraisers in Hack Forums. For MPGH, the "Vouch Copy Profiles" sub-forum, where vendors recruit appraisers based on explicit requirements (e.g., 1,500 posts, 3-month membership) and appraisers post links to their profile threads, was utilized. This yielded 100 appraisers and 2,127 appraisal reviews. In total, 4,054 groundtruth appraisal reviews from 479 appraisers across Hack Forums and MPGH were gathered.

1. Appraisal Review Identification for Unofficial Appraisers (Scaling with ML): To scale this identification, a review classifier was trained.

• Data Annotation: 2,400 reviews and 2,400 non-reviews (300 per class per forum) were randomly sampled and manually annotated. Reviews were defined as containing an evaluation of a product's objective attributes, features, performance, quality, or value based on usage. Inter-coder reliability, measured by Cohen's kappa, was 0.79.
• Classifier Training: Three Deep Neural Network (DNN) models (TextCNN, LSTM, BiLSTM) and six statistical Machine Learning (ML) models (SVM, Naive Bayesian, Logistic Regression, K-Nearest Neighbors, Multi-Layer Perceptron, Random Forest) were compared. DNN models used a 256-dimension word embedding layer, while ML models employed word-count-based vectors.
• Model Selection: The LSTM model demonstrated superior performance, achieving a recall of 96.4% and a precision of 93.1% on the test set (Table III, page 6 of the paper). This LSTM classifier was then applied to the entire dataset, identifying 1,753,413 potential review traces.
• Keyword Matching for High Confidence: To ensure high precision specifically for appraisal reviews, a predefined list of keywords (Table IV, page 6), derived from groundtruth reviews (e.g., "vouch copy," "free sample," "honest review"), was applied to the classified reviews. This conservative, hybrid approach resulted in an impressive precision of 97.8%. In contrast, a classifier trained solely on groundtruth appraisal reviews without keyword matching achieved only 67.1% precision, underscoring the effectiveness of the hybrid methodology.
• Ultimately, this methodology identified 18,701 unique appraisers and 56,229 appraisal reviews across the eight marketplaces.

Workflow of the Appraisal System:

The study also elucidated the typical two-step workflow of an appraisal system in underground marketplaces, as depicted in Figure 3 (page 6) of the paper:

1. Appraiser Recruitment:

• Channels: Vendors recruit appraisers either directly through specific requirements in their product listings or by selecting from official appraiser groups established by marketplaces (e.g., "Official Reviewers Group").
• Promotion & Selection: Appraisers actively promote their services under vendor listings, sometimes even offering them as a separate business. Vendors or official groups select qualified appraisers based on predefined criteria (e.g., VIP status, post count).

1. Vouch Copy Appraisal:

• Distribution: Vendors send vouch copies (e.g., download links) to selected appraisers via private messages, email, or social messaging apps.
• Assessment & Review: Appraisers thoroughly test the product and conduct an in-depth evaluation.
• Posting & Feedback: Appraisers post their detailed reviews under the vendor's product listing, and vendors may then provide feedback.

This structured process ensures that appraisal reviews are not superficial but offer valuable, detailed insights into product quality.

Demo / Proof of Concept

[▶ Watch: Overview of comprehensive dataset collection (2006-2023) (4:00)]()

While the talk did not feature a live demonstration of a specific tool or a traditional proof-of-concept exploit, the authors meticulously detailed their implementation for the automated CTI extraction pipeline, which serves as a critical proof of concept for the feasibility and value of their methodology. This pipeline effectively transforms the raw, unstructured text of appraisal reviews into actionable threat intelligence.

The CTI extraction process employed a hybrid approach:

1. Regex for Fixed Patterns: For CTIs with well-defined, fixed patterns, such as MD5 hashes, SHA1 hashes, SHA256 hashes, and file sizes, specific regular expressions were designed. These regex patterns allowed for highly accurate extraction of these structured values from the review text. Examples of these patterns were provided (Table X, page 13, and the full list in Appendix A, Table XIV, page 18 of the paper).
2. NER-Based Methods for Complex CTIs: For CTIs that lacked fixed patterns and required contextual understanding, state-of-the-art Named Entity Recognition (NER) approaches were utilized. Specifically, the researchers adapted spaCy's NER engine 62, which is powered by deep convolutional neural networks.

• Model Training: To train the NER model, 100 unique reviews were randomly selected and meticulously annotated for each CTI type identified in their taxonomy (Table XI, page 13). The model underwent 1,500 iterations of training on this annotated data, with shuffling at each epoch and using a minibatch size of 4 to update neural network weights, while preserving other pipeline components.
• Model Evaluation: The adapted NER model was rigorously evaluated using an additional 50 randomly selected samples for each entity. The performance metrics, specifically F1-scores, for various CTIs ranged from 82.4% to 89.0% (Table XI), indicating a high degree of accuracy in identifying and classifying diverse CTI elements within the reviews.

CTI Extraction Results:

After applying this sophisticated CTI extraction model to 33,184 appraisal reviews within the website, account, and malware categories, the researchers successfully extracted a total of 23,978 CTI artifacts. These artifacts were associated with 16,668 (50.2%) of the analyzed appraisal reviews, demonstrating the high density of CTI within this source. The distribution of extracted CTIs by category highlighted the breadth of information:

• Website category: Yielded the highest number of CTI instances, with 15,508 artifacts (64.7% of the total).
• Account category: Contributed 7,099 artifacts (29.6%).
• Malware category: Accounted for 1,371 artifacts (5.7%).

This detailed implementation and the successful extraction of tens of thousands of CTI artifacts serve as a robust proof of concept, validating the hypothesis that appraisal reviews are indeed a rich and extractable source of valuable threat intelligence.

Defensive Implications

[▶ Watch: Methodology for identifying groundtruth appraisal reviews (6:00)]()

The findings of this study have profound implications for cybersecurity defenders, offering a novel and potent avenue for enhancing threat intelligence.

Firstly, the identification of appraisal systems as a source of unique and granular CTI means defenders can gain insights into emerging threats that are not yet visible through traditional intelligence channels. The fact that 93.5% of malware-related CTIs from appraisal reviews were unique, and that public sources like VirusTotal had no record for 97.8% of identified malware hashes, underscores the critical gap these reviews fill. Defenders can leverage this by actively monitoring underground appraisal systems (using automated tools similar to those developed in this study) to identify new malware variants, crypters, miners, keyloggers, and botnets, often before they become widespread or are submitted to public repositories.

Secondly, the revealed contradictory CTIs (e.g., an appraiser disproving a vendor's "FUD" claim) provide crucial validation and reality checks. This information can help prioritize defensive efforts, debunk false marketing by threat actors, and inform more accurate risk assessments. Defenders can use these details to refine their detection rules, enhance their endpoint security solutions, and improve their understanding of actual threat capabilities versus advertised ones.

Thirdly, the comprehensive CTI taxonomy developed in this research, covering website, account, and malware categories, offers a structured framework for threat intelligence analysts. This taxonomy can guide the collection, analysis, and categorization of intelligence from various sources, making the process more efficient and thorough. For instance, the detailed Website CTI (e.g., keyword search volume, backlink types) can inform proactive measures against black-hat SEO campaigns and phishing attempts, while Account CTI (e.g., fake account detection features) can aid in combating social engineering and account compromise.

Fourthly, understanding the assessment merits used by appraisers provides insight into the criteria threat actors prioritize, such as malware detectability, stability, UAC bypass capabilities, and programming language. Defenders can use this knowledge to anticipate adversary development trends, improve their own product testing methodologies, and focus on developing countermeasures that specifically target these weaknesses or strengths.

Finally, the analysis of less-trusted appraisers and vendor interference sheds light on the internal dynamics and trust manipulation tactics within underground markets. This contextual intelligence can help defenders understand the broader ecosystem, identify potential misinformation campaigns, and avoid being swayed by fabricated reviews when assessing threat actor capabilities or product quality. By systematically extracting and analyzing CTI from appraisal reviews, defenders can gain a proactive, in-depth, and often exclusive understanding of the cybercriminal landscape, enabling more effective resource allocation and strategic defense planning.

Key Takeaways

• Appraisal systems are a sophisticated and growing feature of underground marketplaces, serving as a critical trust mechanism for illicit transactions.
• Appraisal reviews are an exceptionally rich and under-explored source of unique Cyber Threat Intelligence (CTI), offering granular technical details often absent from traditional CTI sources or public repositories.
• Automated methods, combining machine learning (LSTM model with 96.4% recall, 93.1% precision) and keyword matching (97.8% precision), can effectively identify appraisal reviews and extract CTI artifacts (e.g., 23,978 artifacts extracted).
• CTI from appraisal reviews frequently contradicts vendor claims and provides crucial "real-world" insights into illicit product functionality, detectability, and capabilities, with 93.5% of malware-related CTIs being unique to these reviews.
• A new, comprehensive CTI taxonomy has been developed, tailored for underground marketplace products across website, account, and malware categories, offering a structured approach for threat intelligence.
• Defenders can leverage this novel CTI source for proactive threat intelligence, identifying emerging threats, validating adversary capabilities, and informing more effective defensive strategies against cybercriminal activities.

About the Speaker(s)

Zhengyi Li is a researcher who presented this detailed technical article at the NDSS Symposium. Based on the depth and scope of the work, which involved large-scale data collection, sophisticated machine learning methodologies, and the development of novel taxonomies, it is evident that Zhengyi Li is engaged in significant academic or industry research in the field of cybersecurity, with a focus on cybercrime ecosystems and threat intelligence. The presentation highlights a robust analytical approach to understanding complex socio-technical systems within underground markets.