AI Security Engineer

Menlo Security is the leader in Browser Security for human and agentic workforces. Our mission is to enable humans and agents to connect, communicate, and collaborate securely, without compromise. The Menlo Browser Security Platform protects organizations from cyberattacks by stopping threats across the web, documents, and email before they reach the user. With Menlo Agent Runtime Security (MARS), that protection now extends to the AI agents working alongside every employee. Menlo Security is trusted by major global businesses, including Fortune 500 companies and government agencies, to protect their most valuable asset, their data, and is backed by top-tier investors. We are looking for a Data & AI / ML Evaluation Engineer to own the data collection, mining, curation and continuous evaluation foundation for our various AI security products. High-quality models require high-quality datasets and ongoing enrichment and retraining. In this role, you will build automated data ingestion pipelines from surrogate web traffic and open sources, manage dataset curation, labeling workflows, model training and operate our benchmark and evaluation harnesses. This is a data and AI / ML software engineering role focused on building the data infrastructure that trains our proprietary Prompt Injection SLMs, Shadow-AI detection models, Malicious Macro classifiers, PII, Content Classification engines and more. Outcomes & KPIs Key Outcome(s) Owned: • Automated, continuous generation and curation of training, evaluation, and false-positive (FP) datasets across web, file, and macro threat surfaces. • Standardized, reproducible evaluation benchmarks that guide model improvements before deployment. • Automated data pipeline uptime and ingestion throughput from web/file sources. • Size, diversity, and labeling accuracy of threat dataset repositories (PI, Shadow-AI, PII, Macros). • Mean time to ingest live false-positive/false-negative feedback into training evaluation loops. • Accuracy and benchmark coverage across historical vs. newly trained SLMs. What You'll Do • Design and operate automated data collection, mining, and web-scraping pipelines to gather threat samples from surrogate traffic and web sources. • Build data processing, annotation, and dataset curation workflows, training and re-training various models, e.g. Prompt Injection (PI) SLMs, Shadow-AI detection models, Malicious Macro classifiers, and PII detection. • Build mechanisms to capture false-positive (FP) samples from live user/DDR flows and represent / integrate them into model retraining pipelines. • Develop automated benchmarking and evaluation harnesses to continuously test model detection quality, runtime performance, and accuracy against baseline models (e.g., Vertex AI). • Build automated pipelines for model training for various models. • Work closely with research and software engineering teams to turn raw detection telemetry into ongoing actionable model improvements. Required: • 2–4 years of experience in software engineering, data engineering, or automated QA/evaluation roles. • Strong programming proficiency in Python. Adv: C++, SQL. • Experience in model training and various ML, AI algorithms. • Experience building data pipelines, web scrapers, or automated data processing systems. • Familiarity with data structures, storage formats, and database configurations. • Methodical approach to dataset labeling, validation, bug logging, and metric tracking. Preferred / Nice to Have: • Hands-on experience with ML evaluation frameworks, dataset curation, or annotation tools. • Familiarity with text extraction, NLP preprocessing, or document parsing techniques. • Understanding of cybersecurity concepts (Prompt Injection, Malicious Macros, DLP/PII). • Basic expe