Senior Data Engineer

Why this role exists:

We’re looking for a Senior Data Engineer to design and operate a scalable data infrastructure powering a high-throughput, production-grade platform.

You will own data systems that support analytics, operational workloads, and ML-driven capabilities across a multi-tenant environment. This is a hands-on senior role with architectural responsibility and end-to-end ownership under strict reliability, performance, and SLA requirements.

You will influence core data architecture decisions and help evolve the platform to support growing scale and complexity.

What you’ll drive:

Core Responsibilities

• Design and operate batch and streaming data pipelines.
• Architect and evolve data storage layers (lake, analytical, operational).
• Develop scalable data models for analytics, ML, and operational systems.
• Ensure data quality, consistency, and observability across pipelines.

Platform & Reliability

• Implement real-time processing using event-driven architectures.
• Build ingestion and transformation workflows (ETL/ELT).
• Own backfills, reprocessing strategies, and data migrations.
• Implement CI/CD, monitoring, and SLA-driven operational practices.

Collaboration

• Partner with engineers, analysts, and data scientists to design reliable and scalable data solutions.
• Contribute to data platform standards and best practices.

Technical Environment

• Core: Python, SQL
• Streaming: Kafka
• Storage: PostgreSQL, Redis (ElastiCache), S3
• Analytics: AWS Athena, StarRocks
• Vector: PostgreSQL + pgvector
• Infrastructure: AWS, batch orchestration

What makes you a GR8 fit:

Must-have

• 5+ years of experience building and operating production-grade data systems under SLA constraints.
• Strong proficiency in SQL and Python.
• Hands-on experience with streaming systems (Kafka or similar).
• Experience with batch processing and workflow orchestration.
• Production experience with relational databases (PostgreSQL or similar), including performance tuning, indexing, and partitioning.
• Experience designing high-throughput and low-latency systems.
• Solid understanding of point-in-time correctness and reproducible data pipelines.
• Experience with caching systems (Redis or similar).
• Cloud experience (AWS preferred).
• Experience with analytical engines (Athena, StarRocks, ClickHouse, or similar).
• Experience with distributed processing frameworks (Spark, Flink, Beam, or similar).
• Experience implementing data quality frameworks and CI/CD for data pipelines.
• Familiarity with vector databases or columnar storage systems.

Nice-to-have

• Experience supporting ML workloads (e.g., feature pipelines or feature storage patterns).
• Familiarity with training-serving consistency concepts.
• Experience building shared data platform components.
• Experience building lightweight data-serving APIs (e.g., FastAPI).
• Experience working with containerized environments (e.g., Docker).