||Senior Data Engineer
This position designs, develops, tests and maintains distributed data processing applications on a Big Data Platform.
- Scala (2+ Years)
- Java (5+ Years)
- Spark (1+ Years)
- Kafka (1+ Years)
- Hbase and/or Solr
- Real-Time and Batch Architecture
- Design, develop, document, test, and debug new and existing software systems and applications.
- Serve as technical expert on development projects.
- Drives full development life cycle including requirements analysis and design.
- Write technical specifications based on conceptual design and stated business requirements.
- Support, maintain, and document software functionality.
- Identify and evaluate new technologies for implementation.
- Analyze code to find causes of errors and revise programs as needed.
- Drive software design meetings and analyze user needs to determine technical requirements.
- Consult with end user to prototype, refine, test, and debug programs to meet needs.
- Complies with all security policies and procedures, to ensure that the highest level of system and data confidentiality, integrity and availability is maintained.
- Ability to use Unix/Linux command line programs and create/edit scripts.
- Able to analyze results of automated deployment scripts and makes modifications.
- Experience with Spark Map/Reduce distributed processing
- Able to prepare detail design documents from requirements.
- Demonstrable understanding of distributed, component based Real-Time and Batch architecture
- A solid understanding of event driven / asynchronous programming and multi-threading
- Demonstrable technical design skills
- Familiar with TDD/BDD.
- Common Developer Skills
- Competent (5+ years) Java or Scala
- Experience with Kafka messaging system
- Experience with HBase and/or SOLR
- Competent with SQL or PL/SQL and relational database concepts.
- Experience with source code control systems (GIT)
- Experience with secure programming techniques is highly desirable
- Able to perform code reviews to ensure the code meets the acceptance criteria.