Lead Data Engineer

1 month ago

Melbourne, Victoria, Australia Cognizant Full time

Job Description

Cognizant (Nasdaq-100: CTSH) is one of the world's leading professional services companies, transforming clients' business, operating and technology models for the digital era. Our unique industry-based, consultative approach helps clients envision, build and run more innovative and efficient businesses. Headquartered in the US, Cognizant is ranked 185 on the Fortune 500 and is consistently listed among the most admired companies in the world. Learn how Cognizant helps clients lead with digital at or follow us @Cognizant

Position Summary:

This position entails designing and building efficient data pipelines to attain a modern future state platform for collections operations. This involves migrating and streamlining the existing systems which service credit products including loans and cards that are used by collections operations teams. It requires building robust batch streaming pipelines to produce high-quality, consistent, and structured data and performing necessary encryption to restrict access to sensitive customer information. This requires strong knowledge in big data streaming technologies with ability to understand complex business requirements to implement data processing capabilities including ingestion, enrichment, transformation across cloud services.

Mandatory Skills:

· Solid experience on design patterns to implement strategic, tactical and operational products spanning across all disciplines of BI and Analytics including data ingestion, integration, modelling, performance acceleration, visualization etc.

· Solid functional understanding of the Open-Source Big Data Technologies

· Experience working on Spark Streaming

· Preferable work experience with Hortonworks distribution and Azure cloud platform.

· Proficient with HDFS, Hive, HBase, Spark and SQL.

· Experience in Azure cloud platform and services including ADLS2, HD Insight, Azure SQL, Azure Data Factory, Cosmos DB, Azure Blob Storage, and Azure Databricks is desirable.

· Expertise in various connectors and pipelines for batch and real-time data collection/delivery.

· At least 2+ years of experience in NoSQL databases such as HBase, MongoDB, and Phoenix.

· Must have strong experience with code management and build tools such as Git, Bitbucket, GitHub, GitLab, SVN, Maven, Gradle, and SBT.

· Must have hands on experience with CI/CD deployment tools such as Jenkins, Bamboo, and Azure DevOps.

· Proficient in relevant programming languages such as Java, Scala, Python, and SQL.

· Proficiency in analyzing business requirements for data architecting with both SQL/NoSQL databases.

· Performance tuning - table partitioning and indexing, process threading.

· Support multiple Agile Scrum teams with planning, scoping and creation of technical solutions for the new product capabilities, through to continuous delivery to production.

Duties and Responsibilities:

  • Managing data engineering initiatives, overseeing the development of pipelines that are dependable, optimized, verifiable, and sustainable. Hold significant responsibility in providing valuable inputs in design and data solutioning.
  • As a Lead Data Engineer, taking charge of data engineering initiatives to ensure the dependability, effectiveness, testability, and sustainability of pipelines.
  • Collaborating with business representatives, data analysts, and software developers to comprehend data requirements and devise strategies to fulfill those requirements.
  • Performing data encryption to ensure data masking of sensitive customer information using Spark and ingesting large sizes of data files into Hive, HBase and Phoenix tables.
  • Performing batch processing on Hive tables with Spark using Python to merge and transform large volumes of data.
  • Aid in creating data governance protocols and processes aimed at safeguarding customer personal information, ensuring business compliance with governmental regulations and security benchmarks.
  • Improving the overall Code quality and Code reliability by automated functional and unit testing, thus enabling early detection of bugs and defects.
  • Using CI/CD tools like Azure DevOps to build, test, and deploy the code and automated job schedules.
  • Developing python and shell scripts to perform load assurance tests on ingestion and transformations.
  • Providing guidance and support to new joiners on the best Data Engineering practices.
  • Performing code reviews and offering input regarding the design and execution of the code through peer reviews.
  • Performing System Testing across multiple sources to identify errors and supporting bug fixes in production releases.

Salary Range: >$100,000

Date of Posting :8/14/2024

Next steps:

If you feel this opportunity suits you, or Cognizant is the type of organization you would like to join, we want to have a conversation with you Please apply directly with us.

For a complete list of open opportunities with Cognizant, visit is committed to providing Equal Employment Opportunities. Successful candidates will be required to undergo a background check.

#LI-CTSAPAC

  • Senior Azure Data Engineer

    3 days ago

    Melbourne, Victoria, Australia NTT DATA Full time

    About the RoleWe are seeking a highly skilled Senior Azure Data Engineer to join our team at NTT DATA. As a key member of our analytics practice, you will be responsible for designing and developing cloud-based data platforms and associated technologies.Key ResponsibilitiesAnalyze current business practices, processes, and procedures to identify...

  • Senior Data Engineer

    3 days ago

    Melbourne, Victoria, Australia ECF Data, LLC Full time

    About the Role:We are seeking a highly skilled Data Engineer to join our team at ECF Data, LLC. As a key member of our data engineering team, you will be responsible for designing, building, deploying, testing, maintaining, monitoring, and owning scalable, resilient, and distributed data pipelines.Key Responsibilities:Design and implement data pipelines...

  • Reporting and Data Engineer

    3 days ago

    Melbourne, Victoria, Australia ECF Data, LLC Full time

    About the RoleWe are seeking a highly skilled Reporting and Data Engineer to join our team at ECF Data, LLC. As a key member of our data analytics team, you will play a critical role in designing, developing, and maintaining complex data solutions for our clients.Key ResponsibilitiesCollaborate with clients and internal stakeholders to understand business...

  • Data Engineering Lead

    4 days ago

    Melbourne, Victoria, Australia TeamLogic IT, SE Grand Rapids, MI Full time

    About the RoleWe are seeking a highly skilled Data Engineering Lead to join our team at TeamLogic IT in SE Grand Rapids, MI. As a key member of our organization, you will be responsible for leading our data analytics efforts and driving business growth through data-driven insights.Key ResponsibilitiesLead the development and implementation of data analytics...

  • Data Engineering Lead

    1 day ago

    Melbourne, Victoria, Australia Render Networks Full time

    About the RoleWe are seeking a highly skilled Data Engineering Lead to join our team at Render Networks. As a key member of our organization, you will be responsible for leading our data analytics and AI strategy, developing and implementing data solutions, and working closely with cross-functional teams to drive business growth.Key ResponsibilitiesLead data...

  • Data Engineering Lead

    4 days ago

    Melbourne, Victoria, Australia Cognizant Full time

    About the RoleCognizant is seeking a highly skilled Data Engineering Lead to join our team. As a key member of our organization, you will be responsible for designing and building efficient data pipelines to attain a modern future state platform for collections operations.Key ResponsibilitiesManage data engineering initiatives, overseeing the development of...

  • Data Architect

    4 days ago

    Melbourne, Victoria, Australia ECF Data, LLC Full time

    About the RoleWe are seeking a highly skilled Data Engineer to join our team at ECF Data, LLC. As a Data Engineer, you will play a critical role in designing, building, and maintaining our data infrastructure, ensuring the seamless flow of data across our organization.Key ResponsibilitiesDesign and implement efficient data pipelines using Microsoft Fabric,...

  • Senior Data Engineering Lead

    14 hours ago

    Melbourne, Victoria, Australia Bupa Full time

    About the RoleWe are seeking a highly skilled and experienced Senior Data Engineering Lead to join our team at Bupa. As a key member of our engineering team, you will play a critical role in providing thought leadership and technical expertise to drive the development of our data services.Key ResponsibilitiesTechnical Leadership: Provide technical guidance...

  • Senior Data Engineering Lead

    3 days ago

    Melbourne, Victoria, Australia Snaphunt Full time

    About the RoleWe are seeking a highly experienced Senior Data Engineering Lead to join our team at Snaphunt. As a key member of our data engineering team, you will be responsible for building and maintaining our data platform, primarily using PySpark.Key ResponsibilitiesDesign and implement scalable data pipelines using PySpark and Python.Drive best...

  • Data Engineering Lead

    4 days ago

    Melbourne, Victoria, Australia Jenny Barbour IT & Project Recruitment Full time

    Job SummaryJenny Barbour IT & Project Recruitment is seeking a highly skilled Data Engineering Lead to join our team. As a Data Engineering Lead, you will be responsible for leading, managing, and overseeing all aspects of our clients' Data & Analytics environment.Key ResponsibilitiesDevelop and drive the roadmap for delivery of the Data & Analytics...

  • Lead Data Engineer

    4 days ago

    Melbourne, Victoria, Australia Jenny Barbour IT & Project Recruitment Full time

    Position OverviewPermanent Role | Dynamic Team Environment | Reputable Australian BrandAs the Data Engineering Lead, you will take charge of directing, managing, and supervising all facets of our client's Data & Analytics framework. This includes the data warehouse and business intelligence tools, aimed at delivering insightful analyses that enhance business...

  • Lead Data Engineer

    4 days ago

    Melbourne, Victoria, Australia Jenny Barbour IT & Project Recruitment Full time

    Position OverviewPermanent Role | Dynamic Team Environment | Renowned Australian BrandAs the Data Engineering Lead, you will take charge of directing, managing, and supervising all facets of our client's Data & Analytics ecosystem, including data warehousing and business intelligence tools, to provide valuable insights and analyses aimed at enhancing...

  • Lead Data Engineer

    4 days ago

    Melbourne, Victoria, Australia Jenny Barbour IT & Project Recruitment Full time

    Position OverviewPermanent RoleDynamic Team EnvironmentRenowned Australian BrandAs the Data Engineering Lead, you will take charge of managing and directing all facets of our client's Data & Analytics framework, including data warehousing and business intelligence tools, to provide valuable insights and analyses aimed at enhancing business performance and...

  • Data Engineering Lead

    4 days ago

    Melbourne, Victoria, Australia Jenny Barbour IT & Project Recruitment Full time

    About the RoleWe are seeking a highly skilled Data Engineering Lead to join our team at Jenny Barbour IT & Project Recruitment. As a key member of our organization, you will be responsible for leading and managing all aspects of our clients' Data & Analytics environment.Key ResponsibilitiesDevelop and Drive Data StrategyDevelop and drive the roadmap for...

  • Data Engineering Lead

    4 days ago

    Melbourne, Victoria, Australia Jenny Barbour IT & Project Recruitment Full time

    Job DescriptionJob Title: Data Engineering LeadCompany: Jenny Barbour IT & Project RecruitmentJob Type: Permanent OpportunityLocation: Remote (Australian-based company)Job OverviewAs the Data Engineering Lead, you will be responsible for leading, managing, and overseeing all aspects of our clients' Data & Analytics environment - data warehouse and BI tools...

  • Data Analytics Engineering Lead

    1 day ago

    Melbourne, Victoria, Australia TeamLogic IT, SE Grand Rapids, MI Full time

    About the RoleWe are seeking a highly skilled Data Analytics Engineering Lead to join our team at TeamLogic IT in SE Grand Rapids, MI. As a key member of our organization, you will be responsible for leading our data analytics strategy and platform efforts.Key ResponsibilitiesLead data, analytics, and AI strategy and platform efforts at TeamLogic ITUse big...

  • Data Engineering Lead

    4 days ago

    Melbourne, Victoria, Australia Urbis Full time

    About this roleWho we are:At Urbis, we're a community of passionate problem solvers dedicated to creating meaningful, sustainable cities and communities. We're a people-focused organization that empowers our employees to be the best they can be.We're committed to gender equality and are recognized as an employer of choice by the Workplace Gender Equality...

  • Technical Lead

    4 days ago

    Melbourne, Victoria, Australia Tabcorp Full time

    About the RoleWe are seeking a highly skilled Technical Lead to join our Financial Crime & Risk team at Tabcorp. As a Technical Lead, you will be responsible for providing technical leadership and expertise to our engineering teams, driving the development of our data engineering capabilities, and ensuring the delivery of high-quality solutions.Key...

  • Data Engineer II

    4 days ago

    Melbourne, Victoria, Australia Frost Bank Full time

    About Frost BankFrost Bank is a leading financial institution committed to fostering an environment that reflects our values and encourages team members to thrive. With over 150 years of legacy, we provide unparalleled banking services and are dedicated to our customers' success.Job SummaryWe are seeking a highly skilled Data Engineer II to lead the design,...

  • Engineering Team Lead, Data Platform

    3 days ago

    Melbourne, Victoria, Australia Culture Amp Full time

    About Culture AmpCulture Amp is a leading employee experience platform that empowers companies to transform employee engagement, drive performance management, and develop high-performing teams. Our mission is to make a better world of work for millions of employees.About the RoleWe are seeking an experienced Engineering Team Lead with a passion for data and...