Data Ingestion Sre, Data Platform

Site Reliability Engineering (SRE) combines software and systems engineering to build and run large-scale, massively distributed services and infrastructures.
As a site reliability engineer in the data platform area, you will have the opportunity to manage the services and infrastructures in one of the largest dataplaforms in the world that directly supports the TikTok app.
You'll need to ensure the data, services and infrastructures are reliable, fault-tolerant, efficiently scalable and cost-effective.
In order to enhance collaboration and cross-functional partnerships, among other things, at this time, our organization follows a hybrid work schedule that requires employees to work in the office 3 days a week, or as directed by their manager/department.
We regularly review our hybrid work model, and the specific requirements may change at any time.
Responsibilities:
- End-to-End Service Lifecycle Management: Participate in and continuously improve the full lifecycle of services, from initial design and development to deployment, ongoing operation, and iterative optimization.
- Ensure Reliability and Scalability: Maintain highly reliable, fault-tolerant, and scalable systems that are both cost-effective and efficient, ensuring data, services, and infrastructure meet business needs.
- Performance Troubleshooting: Diagnose and resolve performance issues, including slow queries, resource contention, and bottlenecks across distributed storage engines and services.
- Cluster Scaling and Data Growth: Plan and implement strategies for scaling clusters effectively to accommodate increasing data volume while optimizing performance and cost-efficiency.
- Documentation and Incident Response: Develop and maintain clear runbooks, Standard Operating Procedures (SOPs), and lead sustainable, blameless incident response practices with post-incident analysis to drive continuous improvement.
- Big Data System Design: Architect and implement robust, scalable, and extensible big data systems that support the core business and products, ensuring seamless data flow and system integration.
- On-Call Rotation: Participate in on-call rotations for production incidents, ensuring critical issues are addressed swiftly, with availability to troubleshoot and resolve problems outside of regular business hours as needed.
- Incident Ownership and Analysis: Take ownership of incidents during on-call hours, coordinate escalations as necessary, and conduct thorough post-incident analyses to identify root causes and implement preventive measures.
Minimum Qualifications
- Bachelor's degree in Computer Science, a related technical field involving software or systems engineering, or equivalent practical experience.
- Experience writing code in Java, Scala, Go, Python, or a similar language.
Strong scripting skills (., Bash and Shell) for automation tasks.
- Experience with algorithms, data structures, complexity analysis, and software design: Solid understanding of how to build scalable and efficient systems.
- Basic SQL (MySQL, PostgreSQL, or similar): Strong understanding of traditional relational databases like MySQL or PostgreSQL.
Ability to write queries, perform joins, use aggregate functions, and optimize basic SQL queries.
- Systems and Infrastructure: Knowledge of Linux/Unix systems, as most infrastructure is based on Linux.
Familiarity with system internals, networking, and resource management (memory, CPU, storage).
- Hands-on experience with observability tools such as Prometheus, Grafana, & OpenTSDB: For monitoring, logging, and real-time performance tracking.
- CI/CD Tools: Familiarity with Continuous Integration/Continuous Deployment pipelines and tools (., Jenkins, GitLab CI).
Preferred Qualifications:
- Hands-on experience with distributed processing frameworks like Apache Spark and Apache Flink: Expertise in using these frameworks for large-scale data processing and stream processing.
- Familiarity with Apache Kafka and Apache Hive integrations: Experience with these tools in the context of data pipelines and stream processing.
- HiveSQL Knowledge: Solid understanding of HiveSQL (Hive Query Language - HQL), including the ability to write, optimize, and debug queries on large datasets stored in Hadoop-based systems (like Hive, HDFS, or similar).
- HiveSQL with Kafka Stream Integration: Knowledge and experience in integrating HiveSQL with Kafka Streams or Kafka Connect for real-time data processing
- Experience running production-grade services at scale: Understanding of cloud-native technologies, networking, and storage management to support high-availability and large-scale environments.
- Experience developing tools and APIs: To reduce manual intervention in systems administration and improve automation and operational efficiency.
- Expertise in designing, analyzing, and troubleshooting large-scale systems: Experience with Hadoop, Spark, Hive, Presto, Kafka, Flink, or comparable solutions is a strong plus.