1. How it start
this is not the full start, I joined & help at jan 19, 2022. (Fun fact thats the same date as birthday lol)
So this is how it went. on January jays webserver went down for almost a week. the reason? e-z.host.
yada yada yada, Jay’s vps dying moment.
2. Investigate Problem
So first thing I do was check Cloudflare logs, see what traffic is going trough and see if Cloudflare was blocking it. Turns out 99% of the traffic is going pass Cloudflare and straight into the Server, no shit it was dying.
3. Patch the issue & tesing
The first thing I did was disable ICMP Ping on the server, this is to prevent the server from getting ping of death. and it works! for the server… traffic still going trough Cloudflare, and the server is still accepting it.
Next What I did was enable firewall on the server, I dont know why tf is it disabled in the first place. I enabled it and configured it to only allow incoming traffic from Cloudflare, My IP, Troplo IP(the other admin).
After that I check the traffic on Cloudflare again, and sure it was, it’s still going trough the server. So I created a firewall rule @ Cloudflare to ban IP address that is requesting alot of stuff in a short time. that works. but it backfired. Regular user are getting banned as well.
So I reverted that change, and quickly unbanned all IP that the firewall rules banned. I got another stupid idea again, to add NGINX global ratelimit. well that kinda works but still… normal users get ratelimited just by visiting the site.
After that I contacted troplo to setup heavy ratelimit on the express server instead. it works but the traffic still going trough Cloudflare, and made the express server died. thus the ratelimiting is useless af.
So I check Cloudflare once more, and see what endpoint is being attacked.
Turns out it was the /api/user end point and /api/domain endpoint. it was being spammed with request.
So I quickly created a WAF rule to block sus traffic going trough every endpoint, and it worked.
After that I started monitoring the traffic again, the Server is not dying anymore and Cloudflare finnaly stopped sending the attacker traffic to the server.
4. It works!
- Discord: discord.darrennathanael.com
- Connect: connect.darrennathanael.com