PySpark Developer (Inside IR35)

Location: London, England Salary: £350 - £400 per day
Sector: Consultancy Type: Contract
Reference #: CR/080963_1623052733

PySpark Developer (Inside IR35) / London or Dublin / 6 months / Start ASAP

Key responsibilities:

* Fundamentals of Spark using the Dataframe API
* Understanding partitioning of data
* Analysing and performance tuning Spark queries e.g. looking at the DAG
* Knowledge of Hadoop and its ecosystem of technologies especially Hive Python, OOP concepts using Python
* Knowledge of Conditional Statements & Loops: If-else Control Structures, For/While Loops
* Demonstrate a comprehensive understanding of Complex Data Types: Shallow & Deep Copies, Working with Lists & Tuples, Dictionaries & Sets
* Understand Fundamental Data Structures & their Implementation
* Good knowledge of Exceptions & Command Line Arguments
* Contributes to quality assurance by writing unit and functional tests.
* Ensures development happens for all Software Components in accordance with Detailed Software Requirements specification, the functional design and the technical design document.
* Basic knowledge of UNIX
* Demonstrate source control knowledge (preferably GIT)
* Ability to analyse databases directly using query language tools such as SQL
* Experience on ETL process on Big Data
* Have an understanding of data relationships, normalisation