What is Data Engineering?
Data is everywhere, during the morning walk, while exercising in the gym, while having a cup of coffee, during lunchtime, and during the work hours, we encounter with data.
Data is very useful. Why? Because we can use this data to make decisions. But can we use all of the available data? or Do we need to put certain constraints over that data to process it?
Yes! you are right. We need to makeover our data to be used for performing analysis and getting insights.
And here comes the engineering over data that is; Data Engineering. Data Engineering is the process of extracting, cleaning, and optimizing the data for further processing. Data Engineering is the foremost step in almost every data science project and product. The person who performs data engineering is called Data Engineers.
Data Engineers are those smart people who perform ETL (extract, transform, load), create data pipelines, provide data models, ensure reliable data flow, and hence prepare the data to be used by Data Analysts and Data Scientists. Then Data Analysts/Scientists use this data to perform predictive, descriptive, and/or associative modeling using Machine Learning and Statistical techniques.
Now we have got a brief overview of Data Engineering, Let's talk about the next plan:
The plan is to study Data Engineering from the basics, hence we will walk through the process from the fundamental concepts. Udacity provides a really amazing Nano Degree program in Data Engineering, I didn't take that course however, I have downloaded its syllabus for topic selection. Hence, I will try to add those topics in the upcoming Data Engineering Articles in this blog.
To be honest, Data Engineering(DE) is an interesting field. I hope we will enjoy learning and exploring DE together.
Next Article Topic:
- Data Modeling
Happy Learning!
Comments
Post a Comment