Cloud Data Engineering for Dummies

Introduction

If data is the heartbeat of the enterprise, then data engineering is the activity that ensures current, accurate, and high-quality data is flowing to the solutions that depend on it. As analytics become progressively more important, data engineering has become a competitive edge and central to the technology initiatives that position companies for success.

Data engineers oversee the ingestion, transformation, delivery, and movement of data throughout every part of an organization. They extract it from many sources. They load it into data warehouses and data lakes. They transform it for data science initiatives, massage it for predictive analytic systems, and deliver it for business intelligence dashboards.

Today, many data engineering activities are moving from highly technical data professionals into the hands of business users. As IT cedes control of some types of data, user-friendly data engineering tools give business users more independence.

For example, data scientists need massive amounts of data to develop and train machine learning models, which constantly churn through new data to automate processes and make predictions. Marketing professionals amass data from weblogs, clickstreams, mobile applications, and social media networks to understand customer behavior. Retailers combine point-of-sale data with weather data to forecast consumer demand and automate supply chains.

In this book, you’ll discover how your business and IT teams can effectively collaborate as the entire organization adopts modern data engineering tools and procedures. You’ll see how IT teams lay the data engineering foundation to build reliable, high-performance data pipelines that can benefit the entire organization. You’ll also learn how self-service data preparation and integration activities can be extended to analysts, data scientists, and line-of-business users. Lastly, you’ll discover how a cloud data platform enables your organization to amass all of its data in one central location, making it easy to consolidate, cleanse, transform, and deliver that data to a wide range of analytic tools and securely share it with internal and external data consumers.