Add decision boundary

logistic_regression/binary.org

@@ -77,3 +77,21 @@ For more informations on binary logistic regression, here are usefull links:
 - [[https://ml-cheatsheet.readthedocs.io/en/latest/logistic_regression.html][Logistic Regression -- ML Glossary  documentation]]
 - [[https://math.stackexchange.com/questions/2503428/derivative-of-binary-cross-entropy-why-are-my-signs-not-right][Derivative of the Binary Cross Entropy]]
 
+* Desision Boundary
+
+The method used here is similar to the one used [[https://scipython.com/blog/plotting-the-decision-boundary-of-a-logistic-regression-model/][here]]. In binary logistic regression, decision
+boundary is located where:\\ \[g_w(x_1,x_2)=0.5 \implies h_w(x_1,x_2)=0\]
+In addition we now that our decision boundary has the following form  \[x_2=ax_1+b\]
+Thus, we can easily deduce b since if $x_1=0$ we have $x_2=a\times 0 + b \implies x_2=b$. Thus:
+\begin{equation}
+h_w(0,x_2)=w_1 + w_3x_2=0\\
+\implies x_2=\frac{-w_1}{w_3}
+\end{equation}
+To deduce the a coefficient, it is slighly more complicated. If we know two points $(x_1^a,x_2^a)$ and $(x_1^b,x_2^b)$ 
+on the decision boundary line, we know that $a=\frac{x_2^b-x_2^a}{x_1^b-x_1^a}$. thus if we compute:
+\begin{align*}
+h_w(X_1^b,x_2^b)-h_w(X_1^a,x_2^a)&=\cancel{w_1}+w_2x_1^b+w_3x_2^b\cancel{-w_1}-w_2x_1^a-w_3x_2^a = 0 \\
+&\implies  w_2(x_1^b-x_1^a)+w_3(x_2^b-x_2^a) = 0 \implies  \frac{w_2}{-w_3}=\frac{(x_1^b-x_1^a)}{(x_2^b-x_2^a)}=a
+\end{align*}
+Thus we have the decision boundary defined as follow:
+\[ d(x) = \frac{w_2}{-w_3} x - \frac{w_1}{w_3} \]

logistic_regression/binary.py

@@ -97,8 +97,12 @@ scatter=plt.scatter(x_1,x_2,c=np.round(h(x_1,x_2)),marker="o")
 handles, labels = scatter.legend_elements(prop="colors", alpha=0.6)
 legend = ax.legend(handles, ["Class A","Class B"], loc="upper right", title="Legend")
 
-x=np.arange(0,10,0.2)
-plt.plot([1,2],[2,2])
+# Plot decision boundaries
+x=np.arange(0,10,0.01)
+y=-w1/w3 +(w2/-w3)*x
+plt.fill_between(x,y,np.min(y),alpha=0.2)
+plt.fill_between(x,y,np.max(y),alpha=0.2)
+plt.plot(x,y,"--")
 
 # Save
 plt.tight_layout()