Staff Software Engineer

Job description

## About Prophecy The leader in AI-native data preparation and analysis, Prophecy is revolutionizing how the world's top enterprises turn data chaos into reliable insights. We introduce the AI-native data lifecycle (generate, refine, deploy) where our industry leading AI agents and humans work hand-in-hand in visual and document interfaces to analyze, transform and prepare data, to ship trusted insights at enterprise scale. To learn more, visit us on LinkedIn. ## Position Summary This is an exceptional opportunity to be a technical leader on the team that powers the heart of Prophecy: the Runtime, our orchestration and execution service. Every data workflow , whether generated by our AI data agents or built by a human in the visual editor , is planned, scheduled, and run at enterprise scale by Runtime, directly against the customer's own platform (Databricks, Snowflake, BigQuery, and more). When a business user asks an agent a question and sees trusted results seconds later, Runtime is what made it happen. As a Staff Engineer, you will set the technical direction for this service: the distributed execution engine, the task coordinator, the multi-engine connectors, and the fast interactive previews that make the AI-native experience feel instant. You will own hard, ambiguous, high-leverage problems end-to-end , from architecture to production reliability , and raise the bar for the engineers around you. We're seeking A players to work with A players. We're a high-growth company on a once-in-a-lifetime journey to revolutionize the way data is prepared and analyzed. Be a part of it. ## The impact you will have **Own the engine behind the product.** Architect and evolve the runtime service that turns agent- and human-authored data workflows into high-performance jobs that run reliably in production and interactively during authoring. This is deep-tech at the center of the platform. **Make it fast and make it scale.** Design the distributed execution and task-coordination layer , scheduling, concurrency, resource management, streaming of sample data (interims) , so that pipelines run efficiently from a single interactive preview to large production workloads. **Run everywhere the customer's data lives.** Extend execution across multiple engines and warehouses , Databricks, Snowflake, BigQuery, Spark, dbt, Airflow , with first-class connectors, query pushdown, and consistent behavior across clouds. **Power the AI data agents.** Build the execution path that lets our agents generate, run, validate, and refine pipelines in a tight loop , the difference between a plausible answer and a trusted one. **Own reliability and observability.** Drive production robustness: fault tolerance, graceful degradation, recovery and restartability, monitoring, and the SLAs that enterprise customers depend on. **Lead technically.** Set architecture and standards, drive cross-team designs, and mentor engineers. Own features and initiatives end-to-end, as a founder would , from the requirement to delivery. We don't believe in hierarchy or years of experience for their own sake; we believe in ownership and impact. **What You’ll Need** - 5+ years of backend engineering experience building distributed systems or data-intensive services, with a track record of solving deep-tech problems - Deep expertise in distributed systems fundamentals: scheduling, task coordination, concurrency, resource management, and fault tolerance will help you succeed. - Strong proficiency in a systems language such as Go, Scala, or Java (our runtime is primarily Go and Python; functional-programming experience is a plus) - Hands-on experience running data at scale , with Spark and/or cloud data warehouses (Databricks, Snowflake, BigQuery) and query execution - A strong foundation in data structures, algorithms, and software design, with excellent analytical and debugging skills - Experience owning the reliability, performance, and observability of a production service - A demonstrated ability to lead technically , driving architecture, influencing across teams, and raising the quality bar for others - Comfort working in short, agile, iterative development cycles - Entrepreneurial spirit, priding yourself on getting things done with a high level of quality - Strong desire to work in a fast-moving startup environment with high ownership and attention to detail - You have been a top performer your entire career and hold yourself to very high standards - B Tech or above degree in CS or equivalent **Preferred Qualifications** - Experience integrating with multiple execution backends or building multi-cloud data connectors - You have built platforms or services that other engineers and products build on top of - Familiarity with AI/agent execution paths , running and validating LLM- or agent-generated code - Experience building an execution engine, orchestrator, or query engine (e.g. Airflow, Spark internals, query planners/optimizers such as Apache Calcite, or dbt) - You have shipped software over multiple releases, including systems where you drove the delivery of the first version - Ability to assess new technologies and make pragmatic choices that guide us toward a long-term vision **What You’ll Have At Prophecy** - Great company culture. - Competitive compensation. - Fair and Open Equity awards for everyone. - Amazing work. - Private medical insurance. - Manage your own growth and career. - Experience building a transformative product. - High learning opportunity. - End-to-end ownership of your project.