Software Engineer - ETL/Data Engineering

Role Overview

We are seeking a Software Engineer to support an enterprise IT program. The selected candidate will be joining an existing team supporting extract, transform, load (ETL) workflows for intelligence operations.

Position Summary

The Sponsor supports analysts through the provision of large datasets, methodologies, and data visualizations to address pressing intelligence questions. The Sponsor requires support to ingest, clean, store, and analyze data. The Sponsor needs experienced support in extract, transform, load (ETL) workflows. The work includes engaging regularly with data scientists, analysts, and managers.

Key Responsibilities

Load large datasets into the Sponsor's on-premises and Cloud environments
Develop and maintain ingestion algorithms and schemas for large datasets
Analyze new large-volume datasets to optimize the data ingest processes
Support the creation of Apache NiFi schemas for new data loads
Develop software tools that efficiently preprocess, modify, aggregate, load, index, and archive large data collections into clusters in near real-time
Ensure proper access controls are implemented
Generate metrics to track data ingest statistics to maintain data integrity and provenance
Document the data-flows according to standards set by Sponsor

Mandatory Skills & Requirements

Technical Requirements

Education/Experience - Demonstrated professional experience in Computer Science, Computer Engineering, Systems Engineering, or closely related discipline
AWS Cloud Services - Demonstrated professional experience with AWS cloud services, including long-term storage options, and cloud-based database services
Databricks - Demonstrated experience working with Databricks
SQL Database Management - Demonstrated experience understanding SQL database structures and mapping them between different SQL databases
Apache NiFi - Demonstrated professional experience working with Apache NiFi
Big Data Platforms - Demonstrated professional experience working with large data and high performance compute clusters such as Hadoop or similar
API Development - Demonstrated experience with API development techniques
ETL Processes - Demonstrated experience developing and deploying ETL processes for large data sets
Scripting - Demonstrated experience creating operating system level scripts to perform ETL operations on SQL databases
Version Control - Demonstrated professional experience with version control systems, preferably Git
Software Testing - Demonstrated experience testing the development of software solutions for the extraction, transformation, and loading of data using the most efficient languages for the task such as NiFi, Python, and SQL
Parallel Processing - Demonstrated experience implementing multiprocessing data-flows to parallelize ingest operations

Desired Skills & Qualifications

Sponsor Environment - Demonstrated experience with the Sponsor's data environment
Collaboration Skills - Demonstrated experience exhibiting strong coordination and collaboration skills
Full-Stack Integration - Demonstrated experience working with full-stack developers to deploy applications that leverage large data sets
Technical Communication - Demonstrated experience communicating technical concepts to non-technical audiences

Jobs at Janus Soft Inc

Apply to Software Engineer - SME