Sr. Principal Mission Engineer

Job description

Clarity Innovations is a trusted national security partner, dedicated to safeguarding our nation’s interests and delivering innovative solutions that empower the Intelligence Community (IC) and Department of Defense (DoD) to transform data into actionable intelligence, ensuring mission success in an evolving world. Our mission-first software and data engineering platform modernizes data operations, utilizing advanced workflows, CI/CD, and secure DevSecOps practices. We focus on challenges in Information Warfare, Cyber Operations, Operational Security, and Data Structuring, enabling end-to-end solutions that drive operational impact. We are committed to delivering cutting-edge tools and capabilities that address the most complex national security challenges, empowering our partners to stay ahead of emerging threats and ensuring the success of their critical missions. At Clarity, we are people-focused and set on being a destination employer for top talent, offering an environment where innovation thrives, careers grow, and individuals are valued. Join us as we continue to lead innovation and tackle the most pressing challenges in national security. # Senior Principal Mission Engineer (K8s & Proxmox) # Position Overview The position is for an on-site, cleared Senior Principal Mission Engineer as part of a team of software and platform engineers building a unified, multi-tenant control plane that abstracts away infrastructure differences across AWS, Azure, and on-premises environments. The platform allows application teams to provision secure, isolated Kubernetes clusters and workloads dynamically. The control plane runs Crossplane and Cluster API (CAPI). As the primary on-site engineer, you will be dedicated to deploying, managing, and maintaining our hardware and Proxmox-based system within a secure, classified space. In this role, you will be the primary on-site technical authority responsible for the physical and virtual health of our control plane footprint. You will bridge the gap between our remote engineering pipeline and the isolated, classified hardware infrastructure, handling everything from low-level virtualization troubleshooting to direct customer engagement and user training. The ideal candidate is highly independent, capable of driving projects from inception to completion with minimal supervision. # Key Responsibilities - **System Management & Virtualization Tuning**: Maintain, update, and optimize the on-site Proxmox VE hypervisor environments, Ceph distributed storage clusters, and associated underlying server hardware. - **Troubleshooting & Incident Resolution**: Diagnose and resolve complex hardware failures, kernel panics, network partitions, virtualization bottlenecks, Kubernetes pod crash loops, CNI networking failures, and node pressure conditions within the isolated classified space. - **Integration & Deployment**: Deploy automated system updates, patch air-gapped systems, and integrate new physical compute or storage nodes into the local cluster environment. - **Customer Engagement & Advisory**: Serve as the trusted, face-to-face technical representative for the program, managing day-to-day interactions with site stakeholders, mission owners, and local administrators. - **On-Site Training**: Conduct technical walkthroughs, operational training, and compliance briefings for customer personnel interacting with the Proxmox execution platform. - **Security & Configuration Control**: Partner with the program ISSO/ISSM to enforce local security benchmarks, apply security patches, and support configuration control board reviews for physical hardware modifications. # Qualifications - **Clearance**: Active Top Secret / SCI - **Experience**: 5+ years of hands-on experience managing enterprise Linux environments and bare-metal hardware infrastructure in secure environments. - **Hypervisor Expertise**: Strong, demonstrable experience configuring and troubleshooting Proxmox VE, KVM, QEMU, or equivalent enterprise virtualization technologies (e.g., VMware vSphere, OpenStack). - **Storage & Networking**: Practical experience with Ceph storage orchestration, software-defined networking, VLAN tagging, and low-level Linux network troubleshooting. - **Customer-Facing Skills**: Excellent interpersonal skills with a proven track record of managing technical relationships and conducting formal training for government or military personnel. # Preferred Qualifications - **Live-Production Incident Response**: Hands-on experience managing incident response lifecycles and resolving critical control-plane or etcd failures within live, high-scale production systems. - **Telemetry-Driven Debugging**: Advanced capability in utilizing telemetry, API server audit logs, distributed tracing, and metrics to perform root-cause analysis and resolve complex issues in live production environments. Salary Range: $100,000 - $330,000 We are an equal opportunity employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or veteran status.