Monitoring trains the sysadmin way

After discovering that the site viaggiatreno.it and lefreccie.it kindly offers some API to their train data i decided to implement my own monitoring system to get a complete overview of what is happening in the public train system and never miss a train.




Master Plan:

1) Scrape all data available (Train departure/arrival,delays,stations….)
2) Standardize the format so i can implement pluggable systems (Grafana, Telegram Bot, Website, Twitter..)
3) At least have fun when i hear “We are sorry for the inconvenience” while i check my systems


Scraping all the relevant datasets

All the data is collected with a script every 30 minutes using as input the site APIs and station lists, the ouput will be saved into InfluxDB (Legit, delay time tracking with timeseries DBs) and a local folder for historical data that i will use later with git.


Standardize format

To allow multiple systems comunicate together you always need to take raw data (train datasets) and standardize it into a more pluggable format:

Developing “Pluggable” systems:

That’s it, i could have gone far into dashboarding and alerting but this setup seems to work fine for me.
I tried integrating Elasticsearch + Kibana for more fun stuff but Influx + Grafana did the job very well (it just works.. and no json decoding fights).

Sublime's custom image