Senior Data Engineer - Shining Star

Job Description

Develop new tools, code, and services to execute data engineering activities involving data of varying types and conditions. Responsibilities include data movement, ingestion activities, developing code for data formatting and exploration, analyzing source data formats, and working with Data Scientists to determine optimal formats and transforms. The role involves developing ETL processes, maintaining data pipelines using tools like Spark, Apache Iceberg, Trino, OpenSearch, EMR cloud services, NiFi and Kubernetes containers, while ensuring data pedigree and provenance protection.

Mandatory Skills

• 10+ years' experience with:
  • Data lifecycle engineering
  • Development and maintenance of extract, transform and load (ETL) tools and services
  • Cloud and on-prem data storage and processing solutions
  • Python, SQL, Spark and other data engineering programming
  • COTS and open source data engineering tools such as ElasticSearch and NiFi
  • Processing data within the Agile Lifecycle

Desired Skills

• Experience with Apache Iceberg, Trino, OpenSearch
• EMR cloud services experience
• Kubernetes containers for data processing
• Advanced analytics and data preprocessing
• Enterprise data standards collaboration
• Data quality and standardization expertise