Data Engineer

Palo Alto, CA
Information /
Full-time /
On-site
Pivotal is the leader in the emerging market of electric Vertical Takeoff and Landing (eVTOL) aircraft. We design, develop, and manufacture light eVTOL aircraft and are renowned for the BlackFly, the first light eVTOL to fly manned missions and enter the consumer market.

Efficient, compact, and simple, Pivotal vehicles are designed for a wide range of consumer, public service, and defense applications. Our distinctive tilt-aircraft architecture and scalable platform have been in development, preparing to enter the market for over 10 years. We recently announced our next-generation aircraft, the Helix, planned for general release and scalable production in January 2024.

Mobility is one of the most highly-valued areas of technology investment today. This is the right company, in the right space, the right strategy, at the right time. We invite you to join our amazing team and grow with us.

About the Job

As a Data Engineer, you will be responsible for building scalable, efficient, and cost-effective data solutions.  You will fortify and enhance the management, processing, and visibility of our data.  Your responsibilities will span a wide variety of activities, including collecting and understanding stakeholder requirements, developing and maintaining our data infrastructure, and optimizing and expanding our data processes and storage.

Responsibilities

    • Architect and develop data solutions across the SDLC, including stakeholder requirements, functional specs, design, implementation, integration, testing, and deployment
    • Build efficient, scalable, reliable, and easy-to-maintain ETL processes
    • Configure and leverage Airflow to orchestrate, schedule, and monitor data pipeline jobs
    • Categorize and store different data formats, e.g., numbers, text, images, videos, etc.
    • Sanitize collected data to ensure quality and optimal for use
    • Design and implement metadata systems to access processed data expediently
    • Develop APIs for data access and integration with external systems
    • Connect applications to allow data sharing with a single source of truth
    • Design, configure, and optimize robust and scalable data architectures in accordance with best practices
    • Develop, automate, and maintain cloud-based data infrastructure using IaC scripting
    • Integrate Elastic Search for enhanced data discovery and retrieval
    • Assist algorithm developers and data analysts in generating data-based solutions relevant to problems they are tackling
    • Configure and educate users on BI tools to query data repositories and generate reports
    • Generate ad hoc reports by crafting custom scripts and queries
    • Ensure data storage, retrieval, and processing systems meet performance requirements
    • Troubleshoot and test for security, performance, and availability of production systems
    • Implement data infrastructure monitoring tools to proactively identify and resolve issues
    • Follow best practices to develop HA and auto scaling solutions
    • Maintain code under version control in Git repositories
    • Create and maintain comprehensive documentation for data infrastructure, processes, and systems
    • Reduce costs by streamlining processes and optimizing cloud resources
    • Keep up-to-date with industry best practices and emerging technologies in data engineering

Qualifications

    • 3+ years of professional data engineering experience
    • Undergraduate or graduate degree in computer science, software engineering, or a related field, or equivalent technical experience
    • Strong understanding of computer science and software engineering fundamentals, especially data engineering and infrastructures, ETL processes, data integration, and database design
    • Strong foundation in writing and deploying scalable, secure object-oriented code into cloud environments
    • Programming fluency in Python, SQL
    • Experience with relational and NoSQL databases, especially PostGRES
    • Experience with sophisticated SQL commands and REST APIs
    • Experience with distributed computing environments such as PySpark or Hadoop
    • Experience with AWS Services (or equivalent): VPC, Lambda, Athena, Airflow, Glue, S3, IAM, RDS/Aurora/Redshift, and/or CloudWatch
    • Experience with Elasticstack and Kibana
    • Experience with IaC technologies such as Terraform or Cloud Formation
    • Experience with BI tools such as Tableau, Grafana, or similar
    • Experience with Git and building CI/CD pipelines
    • Experience in working closely with data analysts, data scientists, and engineers
    • Ability to understand different data sources and the advantages of different database types
    • Able to write in a clear and concise manner

Bonus Qualifications

    • Programming experience in protobuf, gRPC, Flask/Django, Javascript, and/or React
    • Experience with developing and monitoring real-time streaming pipelines
    • Experience with building and maintaining data lakes and different database formats
    • Experience developing applications using microservices and container technologies
    • Experience with Apache data management tools: Alluxio, Atlas, Flink, Hudi, Ignite, Kafka, Pulsar
    • Knowledge in advanced data cache/orchestration
    • Knowledge of GIS or map-related technologies
    • Demonstrate sophisticated troubleshooting and performance-tuning capabilities
    • Comfortable with Linux
    • Contribute to open source projects
    • Interest in RC planes, quad copters, or aviation

Personal Attributes

    • Enjoy thinking about the big picture and the small details
    • Thrive in a fast-moving, stimulating, and collaborative team environment
    • Resourceful, curious, and never satisfied with the status quo
    • Strong work ethic and extremely organized
    • Always thinking about how to increase efficiency and improve processes
    • Proactive about solving problems and suggesting solutions
    • Love learning and being challenged
Applicants must be eligible for employment in the United States and willing to work onsite at our HQ office in Palo Alto, CA.

Pivotal offers a comprehensive benefits package, including medical, dental, vision, and 401k plans.

Pivotal is an Equal Opportunity Employer. Pivotal does not discriminate on the basis of race, religion, color, sex, gender identity, sexual orientation, age, non-disqualifying physical or mental disability, national origin, veteran status or any other basis covered by appropriate law. All employment is decided on the basis of qualifications, merit, and business need.