Data Engineer
Shopee
- South Jakarta, Indonesia26, Pacific Century Place SCBD Lot 10, Jl. Jendral Sudirman Kav 52-53, Jakarta 12190, Indonesia, South Jakarta, DKI Jakarta, IndonesiaSouth JakartaDKI JakartaIndonesiaIndonesia
- Toàn thời gianFULL_TIME
Đã hết hạn 5 years ago
2019-10-31T17:00:00+00:00Miêu Tả Về Công Việc
As a Data Engineer, you will be partly a software engineer, a platform engineer, and a system engineer. You will design and grow the Shopee data platform to support a variety of big data applications using open-source technologies including Kafka, Hadoop, Presto, HBase, Spark, Hive, Druid, and our own creations. Some examples include a real-time data streaming platform, a unified query platform, a cluster management system, and a machine learning platform.
Job descriptions:
- Design, build, maintain, and scale our data infrastructure. In doing so, you will also have to monitor and debug the infrastructure that stream millions of data
- Explore and learn new technologies that can complement or improve our current stack
- Develop data ingestion and integration from data sources to data lake and/or data warehouse, based on data pipeline and framework, into query-able format
- Performance tuning and optimization of all data ingestion and data integration processes, including data platform and databases
- Work with regional team to allow centralised data warehousing
- Work with data analytics team to strive greater functionality in our data system
Trình độ tối thiểu
Requirements:
- Bachelor / Master / PhD in Computer Science or a related technical field
- At least 2 years working experience in software development, in at least one of these languages: Java, Scala, Go, Python, C/C++, under Linux / Unix
- Working knowledge of message queuing, stream processing, and highly scalable ‘big data’ data stores
- Uphold best practices and principle around clean code, testing, continuous integration
- Familiar with SQL and/or noSQL databases, eg MySQL, PostgreSQL, MongoDB etc
- Love to use and develop with open-source technologies
Skills below are optional but preferable:
- Experience with cloud platforms, eg Google Cloud Platform, Amazon Web Services
- Familiarity with writing ETL queries
- Strong scripting ability in Bash
- Familiarity with designing and operating of a robust distributed system.
- Experience in manipulating, processing and extracting value from large disconnected datasets
- Experience with big data tools: Hadoop, Spark, Kafka, etc
Tóm tắt công việc
- Trình Độ Công Việc
- Phó Giám Sát/Giám Sát
- Ngành Nghề
- IT and Software
- Yêu cầu học vấn
- Tốt nghiệp Cử nhân
- Nhà tuyển dụng trả lời hồ sơ
- Once in a while
- Địa chỉ văn phòng
- 26, Pacific Century Place SCBD Lot 10, Jl. Jendral Sudirman Kav 52-53, Jakarta 12190, Indonesia, South Jakarta, DKI Jakarta, Indonesia
Feel secure when applying: look for the verified icon and always do your research on a company. Avoid and report situations when employers require payment or work without compensation as part of their application process.