Senior Software Engineer- Data Acquisition
Palo Alto, California
Engineering /
Full Time, Remote /
Remote
We are looking for people with experience gathering data from the web at scale. You’ll be part of Luma’s infrastructure team, working cross-functionally with Applied AI and Research on mission critical work-streams.
Responsibilities
- Own / lead engineering projects for data acquisition.
- Work with Research and Applied AI to ensure their data and technical needs are met.
- Develop and deploy highly scalable distributed systems capable of handling petabytes of data.
- Build and gather high quality datasets by collaborating closely with the research team.
- Build and maintain backend services for data storage.
- Architect and implement algorithms for data management including indexing & search.
- Conduct and analyze system performance experiments.
Experience
- Proficiency in Python.
- Experience of 5+ years in web-scraping at-scale.
- Experience building and working efficiently in very large distributed systems.
- Experience working with Kubernetes, proxies and other relevant tools.
- Experience working with research datasets.
- Please note this role is not meant for recent grads.
$180,000 - $250,000 a year
The pay range for this position in California is $180,000 - $250,000yr; however, base pay offered may vary depending on job-related knowledge, skills, candidate location, and experience. We also offer competitive equity packages in the form of stock options and a comprehensive benefits plan.
All your applications are reviewed by real people.
