Data Engineer

Part of the team in charge of building data pipelines for specific Business areas, you will work closely with other Data, DevOps and Visualization engineers, with the Scrum Master, Product Owner, Architects and Engineering Manager, to evolve the product, but also optimize its release cycle and improve user experience.

Technology stack: Python, PySpark, Hadoop, Hive, Databricks, Docker, Kubernetes.

Required technical skills

• Last 5 years in SW engineering, of which at least 3 years in data engineering (Python is a must)
• Experienced in structured & unstructured data, data lakes
• Great experience in Python Spark (have worked in the last 3 years)
• Good knowledge of SQL (Hive) and No-SQL (preferably Dynamo DB)
• Good knowledge of scheduling and orchestration (Oozie)
• Good knowledge of cloud platform tools (preferably Amazon Web Services - AWS)
• Ability to review complex big data architecture designs
• Average knowledge of DevOps (Kubernetes, Docker) and CI/CD release process (Jenkins)
• Good testing experience (unit test, component test, system test) with some of the most common tools in the market (PyTest, Nose, Behave, etc.)

Other requirements

• Freelancer registered in NL
• Presence on site 2 days a week on-site (Amsterdam)