About the Role: The Hybrid Site Reliability Engineer/Data Engineer at Lumin Digital will play a critical role in ensuring the reliability, scalability, and performance of the Lumin Analytics data platform, while also designing, implementing, and managing robust data pipelines. Lumin Analytics provides financial institutions with critical insights about digital banking metrics and user engagement, enabling them to understand, predict, and influence customer behavior.

This role combines operational excellence with data engineering proficiency, requiring deep expertise in automation, infrastructure management, continuous deployment, data streaming, and data enrichment.

Responsibilities:

Design, develop, and manage scalable and reliable data pipelines and infrastructure for Lumin Analytics.

Develop and manage robust CI/CD pipelines to facilitate rapid, consistent deployment.

Monitor system health, troubleshoot infrastructure and data pipeline issues, and proactively meet Service Level Objectives (SLOs).

Engage in capacity planning, demand forecasting, and performance optimization to ensure scalability and reliability.

Implement monitoring and alerting systems emphasizing automation, self-healing mechanisms, and incident response effectiveness.

Collaborate closely with Development, Security, and Operations teams to ensure compliance with regulatory and security standards.

Conduct and participate in code and architecture reviews.

Participate in an on-call rotation to support system reliability and availability.

Advocate effectively for project resources with senior leadership.

AWS infrastructure with Terraform (Infrastructure-as-Code)

Docker and Kubernetes (Amazon EKS)

Data streaming with Amazon MSK (Kafka), Debezium CDC, ksqlDB

Data enrichment and querying with Amazon Redshift

Java (Kafka Streams, custom transformations)

Python (AWS Lambda, Glue, pandas, boto3, awswrangler)

Scala (AWS EMR, for ETL)

CI/CD pipelines

Apache Spark (Scala and Python)

Software engineering experience (Python, Scala, and Java preferred).

Strong understanding of data structures, data movement, and troubleshooting data issues.

Experience with streaming platforms and data transformation/enrichment (Kafka, Debezium, ksqlDB).

Solid experience with AWS (preferred), Azure, Google Cloud, Databricks, and/or Snowflake.

Proficiency in Linux/Unix environments, git version control, Docker, Kubernetes, Terraform, and CI/CD practices.

Advanced problem-solving skills, with an ability to anticipate operational and data-related issues.

Knowledge of networking protocols (HTTP, DNS, TCP/IP).

Strong written and verbal communication skills.

Commitment to continuous learning and self-improvement.

Familiarity with big data frameworks (Spark, Hadoop, Hive) and related programming languages (Python, Scala, Java).

Experience with relational and non-relational databases (MySQL, PostgreSQL, DynamoDB, Redis).

Knowledge of data quality assurance tools (Great Expectations, deequ, etc.).

Understanding of various data formats (YAML, JSON, Avro, Parquet, protobuf, Iceberg, Delta, Hudi).

Experience with Kafka broker and topic management (performance, sizing, ACL security).

Knowledge of distributed systems concepts (concurrency, race conditions, deadlocks).

Bachelor's degree in Computer Science, Computer Engineering, relevant technical field, or equivalent practical experience.

Minimal travel required (~12 days/year, ~2 team gatherings annually).

Participation in occasional incident response during off-hours.

$170,000 - $200,000 a year