Senior, Data Engineer (Crawler Service)

Asia /
Engineering – DevOps /
Full-time Onsite
Binance is the global blockchain company behind the world’s largest digital asset exchange by trading volume and users, serving a greater mission to accelerate cryptocurrency adoption and increase the freedom of money.

Are you looking to be a part of the most influential company in the blockchain industry and contribute to the crypto-currency revolution that is changing the world?


Responsibility:

    • Architecture design and research and development of distributed crawler and data acquisition system
    • Design and development of distributed crawler module service architecture and data storage architecture
    • Realization of daily network data capture requirements and quality monitoring of collected data
    • Reptile data extraction, cleaning, weight elimination, statistics, etc
    • Optimize the crawling strategy, make full use of bandwidth resources, avoid various restrictions, and improve the crawling effect

Requirements:

    • More than three years of Internet or enterprise-level web crawler development experience
    • Work conscientiously, meticulously and practically, have strong learning ability, take solving technical problems as fun, have ideas and dare to challenge
    • Familiar with Linux platform, solid basic skills in Java or Python, able to design and write crawler system independently is preferred
    • Familiar with the principles and techniques of web crawling, regular expressions, multithreading, HTTP protocol, and be able to obtain information from structured and unstructured data
    • Familiar with the concepts and processes of crawling, seed, parsing, downloading, deduplication, extraction, filtering, scheduling, asynchronous processing, etc
    • Familiar with one or more open source technologies in WebMagic/Scrapy/Heritrix/HtmlParser/Jsoup/HttpClient
    • Experience in verification code cracking, anti-crawling, distributed crawler architecture, data mining, and building data warehouses is preferred
    • A background in data mining, natural language processing, information retrieval, and machine learning is preferred
    • Have a good team spirit and cooperative spirit, full of enthusiasm for work and a sense of responsibility


Conditions
Do something meaningful; Be a part of the future of finance technology and the no.1 company in the industry
Fast moving, challenging and unique business problems
International work environment and flat organisation
Great career development opportunities in a growing company
Possibility for relocation and international transfers mid-career
Competitive salary
Flexible working hours, Casual work attire