Data Engineer (ETL)

Data Engineer (ETL)

Irving, TX (remote to start)

Full time work week

Primary Responsibilities:

Candidates should possess strong knowledge and interest across big data technologies and have a background in data engineering.

• Build data pipeline frameworks to automate high-volume and real-time data delivery for our Spark and streaming data hub
• Transform complex analytical models in scalable, production-ready solutions
• Provide support and enhancements for an advanced anomaly detection machine learning platform
• Continuously integrate and ship code into our cloud production environments
• Develop cloud based applications from the ground up using a modern technology stack
• Work directly with Product Owners and customers to deliver data products in a collaborative and agile environment
   

Your Responsibilities Will Include
Developing sustainable data driven solutions with current new generation data technologies to drive our business and technology strategies

• Building data APIs and data delivery services to support critical operational and analytical applications
• Contributing to the design of robust systems with an eye on the long-term maintenance and support of the application
• Leveraging reusable code modules to solve problems across the team and organization
• Handling multiple functions and roles for the projects and Agile teams
• Defining, executing and continuously improving our internal software architecture processes
• Being a technology thought leader and strategist

Knowledge, Skills & Experience

Education

• BS degree in Computer Science, Data Engineering or similar
• Intermediate to senior level experience in an Apps Development role. Demonstrated strong execution capabilities

Required

• Experienced with Python and ETL
• Must have Search experience
• Search engine integration and data catalog/metadata store experience is preferred (Coveo)
• MongoDB experience preferred
• 5+ years of experience on designing and developing Data Pipelines for Data Ingestion or Transformation using Java or Scala or Python
• At least4 years of experience in the following Big Data frameworks: File Format (Parquet, AVRO, ORC), Resource Management, Distributed Processing and RDBMS
• 5+ years of developing applications with Monitoring, Build Tools, Version Control, Unit Test, TDD, Change Management to support DevOps
• At least 2 years of experience with SQL and Shell Scripting experience
• Experience of designing, building, and deploying production-level data pipelines using tools from Hadoop stack (HDFS, Hive, Spark, HBase, Kafka, NiFi, Oozie, Apache Beam, Apache Airflow etc).
• Experience with Spark programming (pyspark or scala or java).
• Experience troubleshooting JVM-related issues.
• Experience and strategies to deal with mutable data in Hadoop.
• Experience with Stream sets.
• Familiarity with machine learning implementation using PySpark.
• Experience in data visualization tools like Cognos, Arcadia, Tableau

Preferred

• Angular.JS 4 Development and React.JS Development expertise in a up to date Java Development Environment with Cloud Technologies
• 1+ years’ experience with Amazon Web Services (AWS), Google Compute or another public cloud service
• 2+ years of experience working with Streaming using Spark or Flink or Kafka or NoSQL
• 2+ years of experience working with Dimensional Data Model and pipelines in relation with the same
• Intermediate level experience/knowledge in at least one scripting language (Python, Perl, JavaScript)
•  Hands on design experience with data pipelines, joining data between structured and unstructured data
• Familiarity of SAS programming will be a plus
• Experience implementing open source frameworks & exposure to various open source & package software architectures (AngularJS, ReactJS, Node, Elastic Search, Spark, Scala, Splunk, Apigee, and Jenkins etc.).
• Experience with various noSQL databases (Hive, MongoDB, Couchbase, Cassandra, and Neo4j) will be a plus
• Experience in Ab Initio technologies including, but not limited to Ab Initio graph development, EME, Co-Op, BRE, Continuous flow

Other

• Successfully complete assessment tests offered in Pluralsight, Udemy, etc. or complete certifications to demonstrate technical expertise on more than one development platform.