rot13 Sami fnzv

Loadstressing from the cloud with DTBOT

Topics: ddos

Today i finally decided to opensource some of my code created to reach my maximum level of lazyness, Automatically loadstressing web infrastructures via Telegram.
The other challenge was to see/prove if Golang can be a replacement/alternative for Python scripting.

Repo: https://github.com/fnzv/DTBOT

Here is the diagram to better explain what i wanted to do:


Disclaimer before i even start
I'm not responsible for anything you do with this tool, this was made only for legit web loadstressing/benchmarking YOUR OWN infra.
I know that most of the code can be written more efficently/well, don't hate on my exec_shell() ahah
end of disclamer




The main "ingredients" are:

  • Ansible
  • Golang
  • Telegram
  • At least one cloud provider with some resources


It all starts from the Telegram Bot that keeps listening commands from the allowed "chat_id" configured and whenever a predefined command is sent the bot (Written in Golang) runs the Ansible playbook with extra args and gives feedback to the user via Telegram.

This is a classic example for load stressing from Openstack using DTBOT:

  1. User writes to Loadstresser bot chat "/create 5" which triggers the bot to execute the underlying Ansible playbook to deploy 5 VMs on the Openstack Configured Credendials.
    If you check the logs (/var/log/dtbot.log) with a small Ansible background you can understand what's really happening:
    2018/05/19 14:35:46 Command: source /etc/dtbot/os_creds && ansible_python_interpreter=/usr/bin/python3 ANSIBLE_HOST_KEY_CHECKING=False ansible-playbook -vv /etc/dtbot/playbooks/create-infra.yaml --extra-vars="total_nodes=5 telegramtoken=botTOKEN telegramchatid=CHATID"
  2. After a few minutes User recieves feedback that VMs are ready and can start loadstressing with: /load http://example.org <Num clients> <Num VMs involved> <Time in seconds>

The defined command /load was created for simplicity and uses WRK (https://github.com/wg/wrk) as a stresser which works great out of the box without complex configuration files.

Sublime's custom image

After some time passed loadstressing i decided to add a bit of complexity with Jmeter configurations and custom bash scripts so any User can configure or use it's own loadstressing tool (jmeter, vegeta, nghttp2, locust.io ..).

The defined commands for custom Jmeter scripts are /loadj (Openstack) and /loadj_aws (AWS) which follows the exact previous work flow (Telegram -> Golang -> Ansible) but loads a remote configuration file (.jmx in case of Jmeter) and executes the tool with the custom configuration file.

Note: The remote configuration file must be RAW (gist/any pastebin can be used for this).

Example: /loadj or /loadj_aws (to run jmx conf on all AWS nodes)

You can find a simple .jmx example inside the repo under examples/

If you reached that point and you still asking what DT stands for.. well it's just "DownTime" :)



Brief How To/Usage (more info on github repo):

  1. Create a bot and save the bot Token, you can do it by writing "/newbot" to BotFather (https://telegram.me/botfather)
  2. Use the Quick-Install of dtbot on a Ubuntu 16.04 machine and configure it. Required configuration files are located under /etc/dtbot/ :

    • dtbot.conf ( Chat ID and Telegram Token, to find what chat id you have just write some messages to your bot and then open from the browser this url: https://api.telegram.org/bot/getUpdates )
    • os_creds (if you want to create VMs on the Openstack Provider) - Openstack credential source file
    • aws_creds` (if you want to create VMs on AWS) - AWS ACCESS and SECRET key source file (you just need the exports for those enviroment variables)
  3. (re)start dtbot via systemd: service dtbot restart
    If everything is fine you should see "Authorized on account BOT_NAME" on /var/log/dtbot

3.5) Take some time to adjust the Ansible Playbooks based on your cloud enviroment (AWS or Openstack):

  • /etc/dtbot/playbooks/aws-create-infra.yaml - You can keep it as-is but you need to change the "key_name:" with one present in your account, this VM should be able to SSH into newly created AWS instances with this key so generate a new key on the machine and add it to AWS)
  • /etc/dtbot/playbooks/create-infra.yaml - The only part that needs to be changes is the "flavor:" and "image:" name wich changes based on the Openstack provider
  • Other changes that might be done are always the same but on also the other playbooks: info.yaml,ddos.yaml (Openstack flavor,image)
  • Try to send some commands to your Telegram Bot:

    /help - shows the command list
    /create N - Deploys N VMs on Openstack, multiple runs won't deploy more VMs but just checks is N VM is present
    /create_aws N - Deploys N VM on AWS, multiple runs will deploy more VMs
    /stop N - Stops loadstressing tasks on N VMs (Openstack)
    /stop_aws - Stops all loadstressing tasks on ALL AWS VMs 
    /destroy N - Deletes N VMs created on Openstack (0 to N)
    /destroy_aws - Deletes ALL loadstressing VMs created on AWS (Will just shutoff all VMs accessible by the dtbot key and therefore will be deleted because of 'delete on shutoff')
    /load <URL> <Num clients> <Num VMs involved> <Time in seconds> - Start load stressing on Openstack N VMs
    /load_aws <URL> <Num clients> <Time in seconds> - Start load stressing on ALL AWS create VMs
    /loadj <URL> <Num VMs involved> - Executes given JMX Jmeter script on N VMs (Openstack), URL must be raw and displaying directly the text
    /loadj_aws <URL> - Executes given JMX Jmeter script on all AWS VMs, URL must be raw and displaying directly the text
    /load_custom <URL> <Total nodes> - Executes custom bash script provided on Openstack VMs, URL must be raw and displaying directly the text
    /load_custom_aws <URL> - Executes custom bash script provided on all AWS VMs, URL must be raw and displaying directly the text
    /info N - Gathers info on N VMs on Openstack (Established connections and ifconfig stats), useful to check current stresstest status. (Example: start /load then after all the VMs started check /info N to see stats/data)
  • Start loadstressing and tune your infra cache,db & webserver... repeat :)

Pro-Tips:

  • The bot can be added to Telegram Groups and accept commands from all members of the group, just find out the chat_id of the group and add it into the dtbot.conf
  • Check the logs on /var/log/dtbot.log to see what's happening and in case change parameters/values on ansible playbooks.
  • Execute manually Ansible on the dtbot VM to see if something is wrong (ansible-playbook -vv, you can copy/paste the command from the logs)
  • On AWS use a different region from your production env
Previous PostMonitoring trains the sysadmin way
Next Post

© 2020 Sami Yessou

This site is built with GatsbyJS. You can find the source code on GitHub.