Skip to main content

How Machine Learning Works ? - Data Rules the Kingdom

Welcome Back Readers !, For those who have not read the previous post on How Machine Learning works ? Please do read previous one before continuing this so that it would be more easy to understand. You can find the previous post here How Machine Learning Works ? - Mathematics Every Where.

For other who are ready to join with me in this journey to understand the Machine Learning, Thank you for being with this post. Till now we are discussing on the the title "How Machine Learning works? " and to understand that we have seen in the first post that there exist the patterns or trends in the data and we try to find that in terms of a mathematical equation called a model and try to fit to the new data that we have not came across for predicting the values. Wow may be we got the entire process complete with this. But there are few thinks that needs to be cared while training the model or creating the model or while processing the data for the model creation.

We will be talking here about two factors that we need to take care in the model creation,

  1. Data Cleaning
  2. Data transformation

Data Rules the Kingdom

Human right from the advent of computer and the internet, started producing data that now in the current period accounts for more than Tera and Peta and Zeta bytes of data. Fine we store all the data in the severs and data centers, in the form of various database, data warehouse or using the Hadoop - Big data systems. But why don't we let them go ? Why do we need the need to be retained ? What does a data provide us ? 

Many big companies like Google, Facebook, Microsoft etc.. collect the data from the user, what do they do with that huge data ? We live inside home, we go for shopping every day, we may do exercise etc. but how do the Google or any company comes to know about that ? Each user in the world is tracked once you visit the particular website through cookies, or through the GPS in the android phones and other mobiles. Ahhhh !! So are we spy'ed ? The answer is NO, they don't collect the data without knowing us, we are the people who agree the terms and conditions to collect the data without reading the agreement :-P.

Fine lets come to the point, why do we need data ? why do they need data ? 

"Data is precious " - Data helps the business people to improve the business, they also help the common man to improve himself and many boons and the same time data can also be used for destruction.

Imagine a business man collect the data about the daily activities of a person, the business man can now target only the people who do jogging every day and sell the shoes with offer rather than to all with that offer, Mutually both are benefited. This is what we actually do with data, we try to understand the information from data and apply to every problem that exist in the world. We do this through the Data Science, Data Analytics , Machine Learning fields ... etc

"One who have the data - He is the master of the future" - this proves that data rules the kingdom.

Data Cleaning

Data that we collect from the environment does not exist in the pure and structure format. It will have unwanted data mixed in that and we call those data as NOISE. Is the noise the only problem in the data ? No. data that we collect might have missing data that we are interested in. The data collecting devices or the methodology might be error prone. Like this we have lot of problems. Making a data clean from all these is the process that we call as Data Cleaning.

We need to remove the data that are of not of interest from the given set and also try to find the missing data through various methods like filing the blanks with mean value or mode or Variance.

Some theory here :-(

Mean , mode and variance is called the central tendency of the given data. That means for a given set of data the above component gives the representational value which lies in the center of the distributed values. Example Mean gives the value that is the average (Add all the values and divide by the total number of values). Mode gives the highest occurring value in the given set of data. Variance is value that represents how much each data point in the given set of data vary from the mean.

Well the boring theory is done ! The above mentioned steps are few commonly followed methods to remove the noise and the missing data problems. There are still lot more algorithms for doing the same like Bin cleaning etc. Almost 70 to 80 percent of the work of a Data scientist or a Machine learning professional goes into data cleaning and making it ready for next step. Also being in this profession its mandatory that we need to understand the data that we are dealing with. Imagine we are dealing with the car manufacturing unit data, we need to know the various business terms that they use in the domain. The Domain knowledge of the business is very important to understand the data and what make the importance in solving my problem. Not all the data that we collect makes the unit of interest for solving the given problem.

Data Transformation

This is going to be the next step after Data cleaning. OMG still is it not over ? Yes though we spent 70 to 80 percent of time in cleaning the data. Now we need to transform the data to other forms if needed. 

I can understand that there are lot of questions about why we need transformations !! Will come to know about that shortly.

Numbers - Primary weapon

We know by now that models are represented in the mathematical equation to guide the trends or patterns in the data. So to train the model means to find the parameters in the equation for example if you take y = mx + c , we need to find the m and c from the data that we have. They are a guiding values that exist in the form of numbers (decimal or natural numbers or integers etc), we need our attributes that of interest that exist in the other data types needs to be converted to numbers but retaining the semantics of the data.

For example,  Take any attribute "Gender" in the Employee data. It is a categorical attribute (Means it is not a continuous or discrete value but represents a category), Imagine we are gonna try applying the y = mx + c equation means it is the Linear Regression model equation. m and c are the values in number type so how to pass the "Male" or "Female" values of the data ?

In these situations we convert or represent the Male with 0 and Female with 1 and apply the Linear equation. This is not the only case that needs the transformation, there are lot of cases we encounter while solving the problem.

If you take a complete text attribute like comments in the review by the people you need to find the emotions from the text. We can see the process that are involved in both.

Data Cleaning : 
  1. We need to remove the words like is, the, or, a etc which are not of interest. Theses words are called stop words. NLTK library in python helps in doing this
  2. Next, Machine doesn't need to know the words like Loved, Loving but need to know the word love. So this is called rooting process. We transform all the words to its stem word. NLTK again helps in this.
All cleaning is done.

Data Transformation:
  1. We have the bag full of words that we obtained from each comments. This bag of words is called the corpus.
  2. The words needs to be converted to the numbers, so we represent each word is 0 and 1 means is the word available in that comment or not. The big matrix of each row and column with the words having 1 and 0 will be formed. This is called the sparse matrix.

Now this sparse matrix can now be applied to any classification models that exist to identify sad, happy reactions.

Hope we have might have by now understood the importance of data and the transformation steps needed to make it applicable for the next steps.

Machine learning finds various applications in the almost all industry not limited to health, logistics, trading, stock market etc.

By now we might have got the high level idea about How the machine learning works ? 

Once again Thanks for being in my journey to explore more on Machine learning. I am planning to write on various types of algorithms that exist in the machine learning in my next blog. 

Stay Tuned with the thirst for machine learning !


Post a Comment

Popular posts from this blog

How to access the each view of item by position in Recycler View in android ?

How to access the each view of item by position in Recycler View in android ? There are some methods to access the view of each item view to change or update the view during the run time. To access the view of the item view , consider the Recycler view as below, RecyclerView mainRecyclerView = (RecyclerView)view.findViewById(; RecyclerAdapter adapter = new RecyclerAdapter(mContext); mainRecyclerView.setAdapter(adapter); main.setLayoutManager(new LinearLayoutManager(mContext));  To access the itemView of each position the following functions can be used,  1. View itemView = mainRecyclerView.getChildAt(positionOfItem);  2. View itemView = mainRecyclerView.findViewHolderForAdapterPosition(Position).itemView;  3. View itemView = mainRecyclerView.findViewHolderForPosition(Position).itemView;  4. Long itemId = mainRecyclerView.getAdapter().getItemId(position);       View itemView = mainRecyclerView.findViewHolderForItemId(itemId);  5. View

A.P.I call or async await not working in Array forEach ?

H ello Readers, Welcome back. You would have wondered why does forEach function on the array does not wait for any asynchronous function even if we provide the async await in the node js. If you are the person, wondered about this ever, this post is right for you. Non working example : Lets consider the below snippet, this will not wait for the asynchronous process to wait. I am making a setTimeout to mock the API call async process. This will result the count as, Output : count = 0 OMG !! Why it doesn't work ? Answer probably might be lying inside the Array prototype from JavaScript. Lets take a look inside the "Array.prototype.forEach" function. From the snippet its clear that, the for loop in which they call the callback function does not wait for the asynchronous process to run in the callback. So this forEach is not build for asynchronous process itself. So can't I use forEach any more for running a asynchronous function ? Answer g

Machine Learning - Types of Learning - Classification Explained

W elcome back to yet another post on the Machine Learning - Types of learning. In the last post we read about Regression type of learning and its various methodologies. If you did not come across that post yet please click on this link to read that. After reading that, this interesting post will seem more interesting 😎 For the people who did not read the previous post short recap: we are seeing each and every type of learning methodology in machine learning that makes the machines to learn. Following types of learning procedures can be identified, Regression - Visited last week - click here to read this Classification  - This is our topic today  Clustering Rule association  Ensemble  Reinforcement Deep Learning Well let's dive deep into the classification. Classification: Before we see how the classification works, Let me tell you in english, in general what does this classification means !. Wikipedia says that classification is the process