NMLS - 2. Anomaly Detection - Unsupervised Learning - Introductory Machine Learning

 

In 2022 I started the new  Machine Learning Specialization by Andrew NG.

I completed it's first two courses in 2022. I'll share the jupyter notebooks from this specialization soon.

Now, in 2023 when I am finally free from my M1, I am going to complete the specialization. Here is the third Assignment of course 3 (last course) on Recommender Systems.

 

Problem Statement:

307 measurements of two features

  • throughput (mb/s)
  • latency (ms)

 of several servers, were given, to us, we were tasked to find anomalous behavior of a server.

 

Procedure:

Since in addition to unlabelled data, we, also had some labelled data, so, we made a training dataser from unlabelled data and a cross validation set from labelled data.

First we used numpy to find the mean and variance of the features.

Then we using mean and variance we created probability distribution functions for each feature.

Assuming that all features are statistically independent, the total probability for a sample were calculated by simply multiplying the probabilities corresponding to each of it's features.

Probabilities for all of the samples were computed.

The difference of the highest value of probability and the lowest one was divided by 1000 to get a step size.

A loop was run from the lowest value to the highest value in the steps of the step-size calculated in last step.

During each iteration the threshold (epsilon - used to declare samples as anomalous vs non-anomalous) was set equal to the current value of 'iter'.

F1 score was computed for each value threshold/iter. The threshold that generated the highest F1 score was selected.


Relevant  github link (my github repo)

Relevant  video explanation (by me in Urdu-langauge)

Comments

Post a Comment

Popular posts from this blog

Ai SDK ui - 1 - Learning AI SDK UI with Next.js

I recently started learning AI SDK UI . I’ve been working with Next.js already, but this library adds something special. It's  framework agnostic !  It works with Angular, Vue, Svelte, and React. Of course, I’m doing it with Next.js (React) because that’s what I know best. One of the first things I tried was the useChat hook. This makes building a chatbot UI feel way simpler. Normally, handling messages, updating state, and keeping track of what the AI is saying can be messy. With useChat , a lot of that heavy lifting is done for you. Here are some cool things I noticed about it: Message streaming out-of-the-box. You don't need to implement manual streaming! The hook manages states like input, messages, status, and errors. You don’t need to write all that boilerplate yourself. I also liked that it handles more than just happy paths. For example: You can stop a response midway if you want. It provides it out of the box! There are even helpers for modifying message...
Read more

NMLS - 1. Clustering - Kmeans - Unsupervised Learning - Introductory Machine Learning

In 2022 I started the new  Machine Learning Specialization by Andrew NG. I completed it's first two courses in 2022. I'll sgare the jupyter notebooks from this specialization soon. Now, in 2023 when I am finally free from my M1, I am going to complete the specialization. Here is the first Assignment of course 3 (last course) on Clustering. Procedure : Distances between each data-point and all of the cluster centroids were computed The cluster centroid closest to the each data-point was assigned to that data-point The average positions of all of the data-points in a cluster were computed Corresponding centroids were moved to these new average positions The procedure was repeated for a specific number of iterations. Relevant  github link Relevant  video explanation (in Urdu)
Read more

NMLS - 3. Recommender Systems - Collaborative Filtering - Introductory Machine Learning

  In 2022 I started the new  Machine Learning Specialization by Andrew NG. I completed it's first two courses in 2022. I'll share the jupyter notebooks from this specialization soon. Now, in 2023 when I am finally free from my M1, I am going to complete the specialization. Here is the third Assignment of course 3 (last course) on Recommender Systems.   We were given a dataset which contained 'nu' users and 'nm' movies. The ratings that each user gave to various movies were present in the dataset. The task was: We have a new user who has not yet rated a lot of movies. We have have to recommended him movies based on his previous ratings.   Procedure : The cost function was implemented using For Loop & then without any loop (Vectorized Implementation) A list of movie ratings for a new user was created and was added to the training data The rows(movies) were normalized by taking average of each row for non-zero values and then subtracting each value from the...
Read more