Staff Backend Engineer - Trust & Control Safety Systems

Job description

**About HighLevel:**HighLevel is an AI-powered business operating system that gives agencies, entrepreneurs and SMBs the infrastructure to build, automate and scale. Today, HighLevel supports SMBs across 150+ countries, fueling community-driven growth rooted in real customer outcomes. To date, businesses operating on HighLevel have generated over $7 billion in ecosystem value, demonstrating the impact of shared infrastructure at scale. By centralizing conversations, automation and intelligence into one system, we help businesses move faster, reduce complexity and execute efficiently. Behind the platform, HighLevel powers more than 4 billion API hits and 2.5 billion message events daily. With 250 terabytes of distributed data, 250+ microservices and over 1 million domain names supported, our architecture is built for performance, resilience and long-term scalability. **Our people** With over 2,000 team members across 10+ countries, HighLevel operates as a global, remote-first organization built for speed and ownership. We value initiative, clarity and execution, creating space for ambitious people to build systems that support millions of businesses worldwide. Here, innovation thrives, ideas are celebrated and people come first, no matter where they call home. **Our impact** Every month, HighLevel enables more than 1.5 billion messages, 200 million leads and 20 million conversations for the more than 1 million businesses we support. Behind those numbers are real people building independence, expanding opportunity and creating measurable impact. We’re proud to be a part of that. Learn more about us on our [YouTube Channel](https://www.youtube.com/channel/UCXFiV4qDX5ipE-DQcsm1j4g) or [Blog Posts](https://blog.gohighlevel.com/general-atlantic-joins-highlevel/) **About the Team** Even a simple configuration error can lead to significant unintended usage. At HighLevel, we aim to build robust systems that protect our customers and the platform from accidental surges and intentional abuse alike. This team creates the intelligent safeguards that ensure our automation remains safe and reliable for everyone. We are looking for a Staff Engineer to architect the future of Trust, Controls & Safety. You will lead the technical direction of a real-time platform that secures critical workflows across HighLevel—including Signups, SaaS checkouts, User management, and global communication usage. This system processes 160M+ events and $4M+ in transactions daily, serving as the essential foundation for platform integrity. The challenge is to build a highly available, low-latency control plane that balances strict enforcement with a seamless user experience. By integrating synchronous metering with asynchronous signal correlation, you will help us stay ahead of evolving fraud vectors while ensuring that legitimate business activities, from new user signups to complex SaaS billing cycles, proceed without friction. As a Staff Engineer, you'll set the technical direction, own the hardest problems on the platform, mentor engineers across squads, and partner closely with EMs, PMs, Finance, Risk, Support and the channel teams. Getting this right means customers stop getting bills they didn't intend, and abuse stops being a cost the honest majority carries. ### Responsibilities - **Database architecture:** Redesign and optimize data models, queries, and indexing strategies across MongoDB, Firestore, and ElasticSearch - **Search at scale:** Own ElasticSearch reliability — ingestion, indexing, shard strategy, and query performance for billions of documents - **Reliability & performance:** Eliminate bottlenecks, improve replication health, and enforce predictable query and index latency - **System design:** Define data flow and integration boundaries between storage, cache, and APIs with clear contracts and fault isolation - **Observability:** Build visibility into every data path — metrics, traces, slow query logs, replication lag, and cluster health dashboards - **Testing & quality:** Make testing non-negotiable — enforce unit, integration, and load testing standards across all backend modules - **Engineering culture:** Drive RFCs, ADRs, and design reviews that push clarity and precision. Codify patterns that make good engineering repeatable - **Hands-on leadership:** Write code, design systems, and debug production issues. Lead by technical example, not by delegation - **Mentorship & influence:** Level up engineers around you — reviews that teach, feedback that sticks, and systems that outlive individuals ### Requirements - **Heavyweight Engineering Pedigree:** 8+ years of experience building high-scale, mission-critical backend systems where p99 regressions are unacceptable. - **Distributed Systems Mastery:** Deep expertise in concurrency, event-driven architectures, and consistency primitives (CAS, atomic dry-runs, idempotency). - **Financial Rigor:** Experience with systems where code correctness has direct financial impact—metering, ledgers, or payment gateways. - **Operational Instinct:** A track record of technical leadership through influence, driving architecture for multi-team platforms from design to production. - Experience with systems where **correctness has financial consequences** — metering, ledgers, billing, reservations, or comparable transactional domains - Strong experience with relational and NoSQL data models (especially with complex transactional data), plus working knowledge of **streaming and analytical stores** for time-windowed aggregation - Experience designing **policy- or rules-driven systems** that change through configuration rather than deploys — versioning, safe rollout, and rollback of behavioural change - Sound judgement on the **enforcement trade-off**: how to set thresholds and actions that stop real harm without punishing legitimate customers, and how to measure whether you got it right - Familiarity with **monitoring, SLOs, and root-cause analysis** in production environments - Familiarity with hallucination mitigation strategies (e.g., code constraints, test scaffolding, embeddings, context injection) - Track record of **technical leadership without formal authority** — driving cross-team architecture to a decision, writing designs others build from, and lifting the engineering standard around you - Excellent communication and cross-functional collaboration skills, including with Finance, Risk and Support stakeholders - A pragmatic, hands-on engineer who balances long-term vision with iterative execution ### Nice to Have - Experience with ElasticSearch cluster management, multi-tenant indexing, or lifecycle policies - Familiarity with ClickHouse or other OLAP systems for analytics - Background in event-driven systems, data pipelines, or message brokers - Contributions to open-source or public technical writing on database performance or systems design ### Preferred Experience - **Abuse & Fraud Expertise:** Experience building rules engines, velocity checks, or adversarial defense systems at massive scale. - **Modern Data Stack:** Hands-on with high-throughput tools like Go, Redis (Lua), ClickHouse, and Flink for real-time aggregation. - **Hyper-Growth Scaling:** Experience navigating the transition from early architecture to load-bearing platform at scale. - Exposure to **stream processing at scale** (Flink, Kafka Streams, Beam) and columnar analytical stores (ClickHouse, BigQuery) for real-time aggregation - Hands-on experience with **Node.js and Go** in production services, particularly on latency-sensitive or high-throughput paths - Experience working with AI Tools, Billing & Metered Systems - Contributions to open-source tools, internal platforms, or engineering blogs