Principal Infrastructure Engineer

Job description

**About Sezzle:** With a mission to financially empower the next generation, Sezzle is revolutionizing the shopping experience beyond payments, blending cutting-edge tech with seamless, interest-free installment plans that make shopping smarter and more accessible. We’re not just transforming payments; we’re redefining how people discover, interact with, and purchase the things they love while driving real impact on merchant sales through increased conversions and higher order values. As we continue to shape the future of fintech and retail, we’re building an innovative, dynamic team passionate about creating more than just a transaction but a truly unique shopping journey. If you’re excited about pushing boundaries in tech and delivering a game-changing experience for consumers and merchants alike, come join us at Sezzle and help create the future of shopping! **Compensation**: For this principal development role, with 12+ years of experience, the compensation range is **$12,500 - $20,800 USD based on location and experience level** per month and in gross amount. This range acknowledges the extensive expertise, leadership capabilities, and significant contributions expected at this level, offering a competitive salary to reflect the value of advanced skills and experience **About the Role:** We are seeking an exceptional **Principal Infrastructure Engineer** to design, build, operate, and scale the platform that powers Sezzle. Your focus will be the hardest infrastructure problems: increasing throughput, reducing latency, removing capacity bottlenecks, strengthening resilience, and making production operations more automated and predictable as the business grows. You will own complex technical initiatives from architecture and prototyping through implementation, production rollout, and ongoing operation. Your impact will come from the systems you build, the problems you solve, and measurable improvements in reliability, performance, and cost efficiency. Our stack runs on **AWS**, with workloads orchestrated on **Kubernetes** and data anchored in **Aurora RDS (MySQL and Postgres)**. You should know these technologies deeply and be comfortable moving between cloud architecture, networking, cluster internals, database performance, and application behavior to understand how the entire system scales. You will write code and infrastructure-as-code, debug production systems, and deliver changes that hold up under real traffic and failure conditions. Operational ownership is part of the job. You will **participate in the on-call rotation** and take a hands-on role in recovering from major incidents, including full outages. We need someone who can form and test hypotheses using logs, metrics, and traces, make sound mitigation decisions with incomplete information, and turn incident findings into lasting engineering fixes. You will also build and apply **AI-assisted infrastructure and SRE tooling** for incident investigation, capacity analysis, runbook automation, and toil reduction. You will evaluate these tools through practical results and apply appropriate access controls, validation, and auditability to their use in production. This role reports to engineering leadership and works closely with application engineers, Security, and Compliance. You will develop a deep understanding of how Sezzle's business operates and how customer journeys, transaction patterns, and product decisions shape infrastructure demand and behavior. You will use that understanding to identify and deliver improvements across teams and technical domains, connecting infrastructure decisions to better customer outcomes and business performance. **What you’ll do:** - **Own the technical architecture and evolution of core infrastructure**: identify system limits, prioritize technical improvements, and implement changes that support increasing traffic, data volume, and workload complexity. - **Connect business understanding to improvements across the system**: learn how key business workflows behave in production, trace their impact across applications, data, and infrastructure, and partner across teams to improve performance, reliability, and cost efficiency beyond any single service or team's scope. - **Engineer for scale and performance**: build capacity models, run load and stress tests, diagnose bottlenecks across compute, networking, Kubernetes, and databases, and validate improvements against throughput, latency, saturation, and cost per workload. - **Design and build AWS infrastructure**: implement resilient account, IAM, network, and service architectures; address service quotas, fault isolation, and multi-AZ or multi-region requirements as workloads grow. - **Build and operate the Kubernetes platform**: improve cluster architecture, lifecycle automation, workload isolation, resource allocation, autoscaling, safe upgrades, and deployment reliability. - **Scale and optimize Aurora RDS for MySQL and Postgres**: tune queries and indexes, address connection and replication bottlenecks, plan capacity, improve failover behavior, and implement safe schema changes and database migrations with application engineers. - **Improve reliability through engineering**: define and instrument service-level objectives and error budgets with service owners; implement failure isolation, backpressure, load shedding, and safe retry behavior where needed to prevent cascading failures. - **Participate in on-call and drive technical recovery during serious incidents**: use evidence-based triage, execute mitigations, communicate findings, and implement corrective actions from postmortems. - **Implement and test disaster recovery**: design backup, restore, and failover mechanisms against agreed recovery time and recovery point objectives; run recovery exercises and document measured results. - **Build infrastructure-as-code and operational automation**: make provisioning, configuration, deployments, upgrades, and recovery reproducible, reviewed, testable, and recoverable. Eliminate recurring manual work through code. - **Build observability that makes production diagnosable**: improve metrics, logs, traces, dashboards, and actionable alerts, with visibility into service health, scaling limits, and customer impact. - **Deliver safe infrastructure migrations**: design phased rollouts, compatibility checks, validation, and rollback paths for changes to shared production systems. - **Improve cloud cost efficiency through technical changes**: right-size resources, improve utilization, tune autoscaling and storage, and quantify savings while maintaining reliability and performance targets. - **Build and evaluate AI-assisted operational tooling**: apply AI to investigation, runbooks, anomaly analysis, and repetitive operations, with bounded permissions, reviewable actions, and measurable improvements in accuracy or toil. - **Make technical decisions clear and executable**: write architecture proposals, evaluate technology tradeoffs through prototypes and benchmarks, review changes affecting shared infrastructure, and document how systems operate and fail. **What We look for:** - Bachelor's degree in Computer Science or a similar technical field (required) - **12+ years** of experience across infrastructure, platform, site reliability, software development, or related engineering disciplines, with substantial hands-on depth designing and operating production infrastructure at scale. - **Deep expertise with AWS**: production experience across compute, IAM, multi-account architectures, and networking, including VPC design and private connectivity. - **Deep expertise with Kubernetes in production**: cluster lifecycle, scheduling, resource management, autoscaling, networking, and troubleshooting business-critical workloads. EKS experience is strongly preferred. - **Deep expertise with relational databases at scale**, specifically **RDS/Aurora (MySQL and/or Postgres)**: query performance, indexing, connection management, replication, high availability, failover, and verified backup and recovery. - **A track record of personally delivering infrastructure scaling improvements**: identifying constraints, measuring baseline behavior, implementing changes, and demonstrating gains in capacity, latency, reliability, or cost efficiency. - **Strong coding and automation skills**, using Golang, Python, or similar languages to build production tooling and eliminate operational toil, alongside infrastructure-as-code experience with Terraform or equivalent. - **Strong systems fundamentals**: Linux, networking, DNS, TLS, storage, concurrency, and distributed system failure modes, with the ability to debug problems across infrastructure and application boundaries. - Experience operating a **24/7, high-availability platform** where downtime has direct customer or revenue impact, including hands-on incident response and postmortem remediation. - **Willingness to participate in an on-call rotation** and demonstrated ability to recover production systems under pressure using evidence-based triage, safe mitigation, and clear technical communication. - Experience implementing and testing **disaster recovery** against defined recovery objectives, including restoring data and validating service recovery. - Practical experience with **observability, load testing, capacity planning, and safe CI/CD practices** for shared production infrastructure. - **Active use of AI tooling in engineering or operations**, with practical judgment about its limitations and how to verify generated code, recommendations, and operational actions. - Ability to carry ambiguous technical problems from investigation through production delivery and collaborate across engineering disciplines to resolve system-wide constraints. **Preferred Knowledge and Skills:** - Experience in **fintech, payments, or banking**, operating infrastructure with demanding reliability, security, and audit requirements. - Experience with **multi-region architectures, chaos engineering, and failure testing**, including the consistency and recovery tradeoffs of distributed data systems. - Proficiency with **Prometheus, Grafana, Loki, Tempo**, or comparable observability systems. - Experience building **internal platform capabilities and self-service tooling**, including deployment automation, progressive delivery, and reusable infrastructure components. - Experience building **AI-assisted incident investigation or operational automation** with restricted access, auditable execution, and clear human review points **About You:** - **You have relentlessly high standards** - many people may think your standards are unreasonably high. You are continually raising the bar and driving those around you to deliver great results. You make sure that defects do not get sent down the line and that problems are fixed so they stay fixed. - **You’re not bound by convention** - your success—and much of the fun—lies in developing new ways to do things - **You need action** - speed matters in business. Many decisions and actions are reversible and do not need extensive study. We value calculated risk-taking. - **You earn trust** - you listen attentively, speak candidly, and treat others respectfully. - **You have backbone; disagree, then commit** - you can respectfully challenge decisions when you disagree, even when doing so is uncomfortable or exhausting. You have conviction and are tenacious. You do not compromise for the sake of social cohesion. Once a decision is determined, you commit wholly. - **You deliver results** - you focus on the key inputs and deliver them with the right quality and in a timely fashion. Despite setbacks, you rise to the occasion and never settle. - **You go deep**: you investigate how systems behave under load and failure, follow the evidence, and fix underlying causes. - **You build and ship**: you turn architecture into working code, tested infrastructure, and safe production changes. - **You measure results**: you can explain the impact of your work in availability, latency, capacity, recovery time, cost, or hours of toil removed. - **You value simplicity**: you choose systems that are understandable, operable, and appropriate to the problem, and reduce unnecessary complexity. - **You earn trust**: you listen carefully, communicate clearly, challenge technical decisions respectfully, and follow through on commitments. - **You stay accountable for the outcome**: you verify that changes work in production and that fixes remain effective as the platform scales. **Sezzle’s Technology Stack:** - **Languages:** Golang, Python - **Database:** MySQL, Postgres - **DevOps and Cloud:** AWS, Kubernetes - **Version Control:** Git - **CI/CD:** Gitlab - **Open Source:** Sezzle is focused on using open source, and we build what we can before buying! **What Makes Working at Sezzle Awesome?** At Sezzle, we are more than just brilliant engineers, passionate data enthusiasts, out-of-the-box thinkers, and determined innovators; we are skilled musicians, yogis, cyclists, chefs, golfers, dog-lovers, and rock-climbers. We believe in surrounding ourselves with not only the best and the brightest individuals, but those that are unique and purpose-driven in all that they do. Our culture is not defined by a certain set of perks designed to give the illusion of the traditional startup culture, but rather, it is the visible example living in every employee that we hire. #Li-remote #full-time