Senior ML Engineer - Malware Detection

Job description

**This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Senior ML Engineer - Malware Detection based remotely in United States.** The Senior ML Engineer will develop and deploy advanced machine learning solutions to detect web-based malware across large-scale Linux hosting environments. Working with an extensive dataset of real-world malware samples, analyst decisions, detection rules, and cleanup outcomes, you will transform production intelligence into more accurate and efficient security systems. Your work will span classical machine learning, transformer models, LLM-assisted triage, and automated signature generation. You will own the journey from raw data and experimentation to production deployment, with success measured by real detection improvements and controlled false-positive rates. Collaborating closely with malware analysts, infrastructure specialists, and fellow ML engineers, you will have access to dedicated GPU resources and modern AI-assisted development tools. This remote-first opportunity is ideal for an experienced ML practitioner who thrives on experimentation, rigorous evaluation, and delivering measurable cybersecurity impact. ### Accountabilities: You will lead the development, evaluation, and deployment of machine learning systems that improve malware detection at scale while balancing accuracy, operational costs, and resource constraints. Your key responsibilities will include: - **Machine Learning Model Development:** Design, train, fine-tune, and deploy ML models using production malware data to improve detection coverage and identify threats that traditional signatures and rules may miss. - **Malware Classification and Triage:** Develop models that distinguish malicious files from benign content, prioritize suspicious files for deeper inspection, and route samples to the most appropriate detection engines. - **Automated Signature Generation:** Build systems that generate detection signatures from emerging malware campaigns, helping security analysts respond faster to new and evolving threats. - **Context-Aware Threat Detection:** Develop methods to identify malicious content that requires contextual analysis, including phishing pages, webshells, script injections, and compromised website components. - **Detection and Cleanup Optimization:** Improve detection precision and cleanup effectiveness so that infected websites remain clean after remediation and recurring threats can be identified. - **Model Evaluation and Validation:** Establish rigorous evaluation frameworks using time-based and campaign-based dataset splits, historical false-positive and false-negative cases, and regression testing to prevent data leakage and verify improvements before deployment. - **Inference and Resource Optimization:** Optimize model throughput, latency, CPU usage, memory consumption, and cost per file to support reliable deployment across large hosting fleets. - **Production Deployment and Monitoring:** Move models from experimentation into production, using shadow deployments and controlled releases to gather evidence before enabling live detection. - **Continuous Learning and Retraining:** Incorporate fresh analyst feedback and newly identified false negatives into training datasets, establish regular retraining cycles, and monitor trends in detection rates and model performance. - **ML Infrastructure and Experimentation:** Develop reusable training pipelines, maintain dataset tooling, make effective use of GPU resources, and enable multiple parallel research and development efforts. - **Cross-Functional Collaboration:** Work closely with malware analysts, detection engineers, and infrastructure teams to investigate customer-reported detection failures, understand emerging threats, and translate operational findings into model improvements. - **Technical Documentation and Knowledge Sharing:** Document architectural decisions, evaluation methods, model limitations, hardware requirements, and production results. Participate in asynchronous collaboration and blameless post-mortems to continuously improve engineering practices. - **Delivery Against Milestones:** Train and evaluate an initial production-data model during the first month, bring multiple models into production or shadow testing by month three, and help establish ML as a major source of new detections within the first year. **Requirements:** The ideal candidate combines substantial production ML experience with hands-on transformer expertise, strong evaluation discipline, and the ability to engineer efficient models for adversarial security environments. You should be comfortable making evidence-based decisions, communicating asynchronously, and taking ownership of the full ML lifecycle. Key qualifications include: - **Production ML Experience:** At least five years of experience building and deploying machine learning systems in production, including at least two years in an adversarial domain such as cybersecurity, fraud, spam, or abuse prevention. - **Transformer Model Expertise:** Practical experience training and fine-tuning transformer models, including data preparation, code tokenization, LoRA/QLoRA or full fine-tuning, distillation, and quantization. Experience should extend beyond calling hosted model APIs. - **Classical Machine Learning:** Strong understanding of gradient boosting, linear models with sparse features, probability calibration, class imbalance, and the ability to determine when traditional ML methods outperform LLM-based approaches. - **Rigorous Model Evaluation:** Experience designing leakage-free splits by time and malware family, evaluating precision under fixed false-positive constraints, maintaining regression datasets, and understanding the limitations of offline evaluation when predicting production performance. - **Inference and Cost Optimization:** Demonstrated ability to measure throughput and latency and optimize models for CPU, memory, and operational cost constraints. Familiarity with technologies such as ONNX, llama.cpp, vLLM, and int8/int4 quantization is valuable. - **Technical Stack:** Strong Python, PyTorch, and Hugging Face skills, along with SQL expertise for analyzing large datasets using ClickHouse or comparable analytical databases. - **Web Technology Familiarity:** Experience with PHP or JavaScript, or a willingness to learn these technologies while collaborating with malware analysts. - **AI-Assisted Engineering:** Daily use of AI coding agents, with the ability to demonstrate how they support development, investigation, and engineering productivity. - **Written Communication:** English proficiency at B2 level or higher, with particularly strong written communication skills for technical documentation and asynchronous collaboration. - **Analytical and Problem-Solving Skills:** Ability to investigate complex detection problems, interpret model results, make sound technical trade-offs, and prioritize measurable improvements over experimental metrics alone. - **Ownership and Collaboration:** Comfort taking responsibility from experimentation through production deployment while working effectively with analysts, engineers, and other stakeholders. The following experience would be a strong advantage: - **Web Malware Security:** Experience with web malware, webshells, phishing kits, or WordPress security. - **Code-Based Machine Learning:** Experience developing models over source code using abstract syntax trees (ASTs), token features, code-focused language models, or graph models of program structure. - **Malware Analysis:** Knowledge of malware clustering, similarity analysis, and methods for identifying related samples or emerging campaigns. - **Training Infrastructure:** Experience with multi-GPU training on bare-metal infrastructure, MLflow or Weights & Biases, and dataset versioning. - **Systems Programming:** Familiarity with Go or Rust for efficient on-server inference. - **Security Research:** Published work, technical presentations, or conference talks focused on machine learning for cybersecurity. **Benefits:** - **Fully Remote Work:** Work remotely with flexibility to organize your schedule and work from different locations worldwide, subject to hiring arrangements. - **Flexible Working Hours:** Autonomy to structure your working day in a remote-first, asynchronous environment. - **Paid Vacation:** 24 days of paid vacation annually. - **Public Holidays:** 10 days of national holidays. - **Sick Leave:** Unlimited sick leave. - **Healthcare Support:** Financial compensation toward private medical insurance. - **Wellness Benefits:** Reimbursement for co-working spaces and gym or sports activities. - **Learning and Development:** An education budget and a strong focus on professional growth. - **Innovation Rewards:** The opportunity to receive a reward for developing an innovative idea that can be patented. - **Dedicated Technical Resources:** Access to dedicated GPU resources and potential additional computing investment supported by evidence-based proposals. - **Challenging Technical Work:** Opportunities to work with large-scale real-world malware datasets and develop production ML systems that address evolving cybersecurity threats. - **AI-Enabled Engineering Environment:** Access to AI coding agents and integrated development tools to support experimentation, implementation, and collaboration.