GovWire

This describes the role of a data engineer and the skills required, including:

• an introduction to the role, telling you what you would do in this role and the full list of skills
• a description of the levels in this role, from data engineer to head of data engineering, specifying the skills you need and the corresponding skill levels (awareness, working, practitioner, expert)

This role is part of the Digital, Data and Technology Profession in the Civil Service.

Introduction to the role of data engineer

A data engineer develops and constructs data products and services, and integrates them into systems and business processes.

Skills needed to be a data engineer

You will need the following skills for this role, although the level of expertise for each will vary, depending on the role level.

• Communicating between the technical and non-technical. You can communicate effectively across organisational, technical and political boundaries, understanding the context. You can make complex and technical information and language simple and accessible for non-technical audiences. You can advocate on behalf of a team and communicate what it does, to create trust and authenticity. You can successfully respond to challenges.
• Data analysis and synthesis. You can translate data into valuable insights that inform decisions. You can effectively involve teams in analytics and synthesis to increase consensus and challenge assumptions. You can identify and use the most appropriate analytical techniques, and you have an understanding of analytical tools. You can demonstrate numeracy. You can show an awareness of advances in digital analytics tools and data manipulation products, and can keep up to date with them. You can collect, collate, cleanse, synthesise and interpret data to derive meaningful and actionable insights.
• Data development process. You can integrate and separate data feeds to map, produce, transform and test new data products.
• Data innovation. You can recognise and exploit business opportunities to ensure more efficient and effective performance of organisations. You can explore new ways of conducting business and organisational processes.
• Data integration design. You can develop data services that are fit for purpose, resilient, scalable and future-proof, to meet user needs. You can demonstrate an understanding of how to expose data from systems (for example, through APIs), link data from multiple systems and deliver streaming services.
• Data modelling. You can produce data models and understand where to use different types. You can understand different tools and compare different data models. You can reverse engineer a data model from a live system. You can understand industry-recognised data modelling patterns and standards.
• Metadata management. You can understand a variety of metadata management tools. You can design and maintain the appropriate metadata repositories to enable the organisation to understand its data assets.
• Problem resolution (data). You can log, analyse and manage problems to identify and implement the appropriate solution. You can ensure that the problem is fixed.
• Programming and build (data engineering). You can design, write and iterate code from prototype to production-ready. You can understand security, accessibility and version control. You can use a range of coding tools and languages.
• Technical understanding. You can demonstrate knowledge of the specific technologies necessary to fulfil the responsibilities and tasks of the role. You can apply the required breadth and depth of technical knowledge. You can actively keep informed of industry developments to make cost-effective use of new and emerging tools and technologies.
• Testing. You can plan, design, manage, execute and report tests, using appropriate tools and techniques. You can work within regulations. You can ensure that risks associated with deployment are adequately understood and documented.

Data engineer

A data engineer delivers the designs set by more senior members of the data engineering community.

At this role level, you will:

• implement data flows to connect operational systems, data for analytics and business intelligence (BI) systems
• document source-to-target mappings
• re-engineer manual data flows to enable scaling and repeatable use
• support the build of data streaming systems
• write ETL (extract, transform, load) scripts and code to ensure the ETL process performs optimally
• develop business intelligence reports that can be reused
• build accessible data for analysis

Skills needed for this role level

• Communicating between the technical and non-technical. You can show an awareness of the need to translate technical concepts into non-technical language. You can understand what communication is required with internal and external stakeholders. (Skill level: awareness)
• Data analysis and synthesis. You can undertake data profiling and source system analysis. You can present clear insights to colleagues to support the end use of the data. (Skill level: working)
• Data development process. You can design, build and test data products based on feeds from multiple systems, using a range of different storage technologies, access methods or both. You can create repeatable and reusable products. (Skill level: working)
• Data innovation. You can show an awareness of opportunities for innovation with new tools and uses of data. (Skill level: awareness)
• Data integration design. You can deliver data solutions in accordance with agreed organisational standards that ensure services are resilient, scalable and future-proof. (Skill level: working)
• Data modelling. You can explain the concepts and principles of data modelling. You can produce, maintain and update relevant data models for specific business needs. You can reverse-engineer data models from a live system. (Skill level: working)
• Metadata management. You can work with metadata repositories to complete complex tasks such as data and systems integration impact analysis. You can maintain a repository to ensure information remains accurate and up to date. (Skill level: working)
• Problem resolution (data). You can explain the types of problems in databases, data processes, data products and services. (Skill level: awareness)
• Programming and build (data engineering). You can design, code, test, correct and document simple programs or scripts under the direction of others. (Skill level: working)
• Technical understanding. You can understand the core technical concepts related to the role, and apply them with guidance. (Skill level: working)
• Testing. You can correctly execute test scripts under supervision. You can understand the role of testing and how it works. (Skill level: awareness)

Senior data engineer

A senior data engineer designs and leads the implementation of data flows to connect operational systems, data for analytics and business intelligence (BI) systems.

At this role level, you will:

• recognise opportunities to reuse existing data flows
• lead the build of data streaming systems
• optimise the code to ensure processes perform optimally
• lead work on database management

Skills needed for this role level

• Communicating between the technical and non-technical. You can communicate effectively with technical and non-technical stakeholders. You can support and host discussions within a multidisciplinary team, with potentially difficult dynamics. You can be an advocate for the team externally, and can manage differing perspectives. (Skill level: working)
• Data analysis and synthesis. You can undertake data profiling and source system analysis. You can present clear insights to colleagues to support the end use of the data. (Skill level: working)
• Data development process. You can design, build and test data products that are complex or large scale. You can build teams to complete data integration services. (Skill level: practitioner)
• Data innovation. You can understand the impact on the organisation of emerging trends in data tools, analysis techniques and data usage. (Skill level: working)
• Data integration design. You can select and implement the appropriate technologies to deliver resilient, scalable and future-proofed data solutions. (Skill level: practitioner)
• Data modelling. You can understand the concepts and principles of data modelling. You can produce relevant data models across multiple
  

  View the original news story