BravoSystems uses cloud and bare-metal servers across two data centers. For the cloud deployments, we use Amazon Web Services and it’s mostly used for new and experimental projects. Our legacy and stable projects are deployed to our datacenters. Infrastructure in our datacenters is provisioned through a mix of Ansible<\/a> scripts and custom tools. We also have a Kubernetes <\/a>cluster which is used in some of our product teams. For infrastructure monitoring, we use Nagios<\/a> and Zabbix<\/a>. <\/p>\n\n\n\n Figure 1. Part of our tech stack<\/em><\/p>\n\n\n\n We also built a Platform for abstracting away infrastructure from product teams. This was an upgrade from the simple Kubernetes cluster which we already had in production. We are planning to migrate legacy products alongside all new ones that are already there.<\/p>\n\n\n\n The Platform is managed through Terraform<\/a>, Ansible, and Helmfile<\/a>, and the heart of the platform is Kubernetes. Deployed services are packaged using Docker<\/a> and Helm<\/a>. On the ingress side, we use envoy<\/a> as a front proxy for Kubernetes. For the storage provider, we use Portworx<\/a> – cloud-native storage and data management platform. The Platform also provides monitoring and centralized logging based on Prometheus<\/a>, Loki<\/a>, and Grafana<\/a>.<\/p>\n\n\n\n The important part is also CI\/CD solution which relies on all the above. With our solution, we minimize efforts to have a fully functional CI\/CD solution for any team.<\/p>\n\n\n\n <\/p>\n\n\n\n Since we are ingesting terabytes of data daily, we needed to set up proper data infrastructure to be able to efficiently use that data. For distributed storage, we use Hadoop<\/a> cluster. In addition, we backup everything on AWS S3<\/a> and Glacier<\/a>. Analytical data is currently stored in Amazon RedShift<\/a> or Clickhouse<\/a> and we are currently migrating to Druid<\/a>. For metadata management and governance we use Apache Atlas<\/a>. <\/p>\n\n\n\n For ETL orchestration and management we use Apache Airflow<\/a> with a combination of our internal tools. For data streaming, we use Kafka<\/a>. ETLs are mostly written in Python or Java. <\/p>\n\n\n\n Figure 2. Apache Atlas showing lineage of one of our Spark ETL jobs.<\/em><\/p>\n\n\n\n Our primary backend language is Java because of its performance and extensive support for external ecosystems like Apache, which we use a lot (Hadoop, Spark, Kafka..).<\/p>\n\n\n\n Other languages we use are PHP and NodeJs. We use PHP in our legacy products. NodeJs is mostly used for new and experimental projects because of a faster learning curve for junior developers, so we can take those projects to market as soon as possible. Our teams use MySQL and MongoDB as the production database. Where caching is needed, we use Redis<\/a>.<\/p>\n\n\n\n For the majority of tracking and logging, we use Kafka, which stores data into Hadoop using various streaming applications. <\/p>\n\n\n\n Several teams are using ELK<\/a> (Elasticsearch, Logstash, and Kibana) for application monitoring. <\/p>\n\n\n\n Usually, alerting mechanisms are set up per team i.e. the team that builds a service is responsible for monitoring that service. <\/p>\n\n\n\n Our primary frontend language is Javascript, and we build our projects with ExtJs, VueJs, and ReactJs. For testing, we mostly use Jest<\/a> for the unit and Testcafe<\/a> for functional testing. We have also built our internal test tool based on Selenium<\/a> and Sikuli<\/a> called Hodor for end-to-end tests which enables us to test browser extensions. We have used Cucumber<\/a> to write acceptance tests in one of our products.<\/p>\n\n\n\n As for web servers, we use Apache HTTP<\/a> and nginx<\/a>. <\/p>\n\n\n\n Figure 3. Cucumber reports showing test executions<\/em><\/p>\n\n\n\n Some of our legacy products are still deployed on bare-metal servers, which are managed by a centralized admin team. Also, in some teams where we use the on-prem Kubernetes cluster or AWS, that team manages deployment configuration themselves. Our in-house Platform made deployment very simple.<\/p>\n\n\n\n For Continuous Integration and Deployment, we use Jenkins<\/a> to run our pipelines. For artifacts management, we use Artifactory<\/a>, where we store built release candidates, libraries, and various other artifacts. Artifactory also serves as a Docker registry and a maven repository. Code collaboration is done using Bitbucket<\/a>, and we use Jira as a bug\/issue tracking tool. We use Vagrant<\/a> for managing some development environments and Docker for packaging most of our deployments.<\/p>\n\n\n\n Teams are mostly using Feature Branching as git workflow although we also use trunk-based development and feature toggling to enable experimentation and combination of work-in-progress and completed features.<\/p>\n\n\n\n Our products have various KPIs which we regularly monitor. KPI reports are mostly built on Tableau<\/a> and Zeppelin<\/a>. We also use Metabase<\/a> for various reporting. The analytics team built an anomaly detection tool called Sherlock which uses prediction models to check our most important KPIs.<\/p>\n\n\n\n Figure 4. Anomaly detection tool Sherlock finds anomaly during Thanksgiving day.<\/em><\/p>\n\n\n\n Alerts trigger email and Slack notifications that are sent to all relevant people. DataScience teams use Jupyter<\/a> which is connected to our data platform. On top of our Hadoop cluster, we use Apache Spark<\/a> and Hive<\/a>.![\"\"](\"https:\/\/bravosystems.com\/blog\/wp-content\/uploads\/2020\/08\/pasted-image-0-1.png\")
<\/figure><\/div>\n\n\n\n![\"\"\/](\"https:\/\/lh6.googleusercontent.com\/HeofPW9q89gTyL0hetTDgyBMFsSPMH2vGKd-eiGGdMsrgz9u-Mx8-nFC6FzNYlN7C_sNi86bppx15TY2UD24douS7mdKJ70V45E4UlwbMdzVKHIJtk37s5zLHHX6DoPIrZhhf8PB\")
Backend\u00a0<\/strong><\/h2>\n\n\n\n
Frontend<\/strong><\/h2>\n\n\n\n
![\"\"](\"https:\/\/bravosystems.com\/blog\/wp-content\/uploads\/2020\/08\/pasted-image-0-2-1024x723.png\")
<\/figure><\/div>\n\n\n\nDevelopment and Deployment<\/strong><\/h2>\n\n\n\n
DataScience: Machine Learning and Analytics<\/strong><\/h2>\n\n\n\n
![\"\"](\"https:\/\/bravosystems.com\/blog\/wp-content\/uploads\/2020\/08\/pasted-image-0-1-1-1024x614.png\")
<\/figure><\/div>\n\n\n\n
We use various machine learning algorithms in our products, like LightGBM<\/a> and XGBoost<\/a>, and various unsupervised ML algorithms. Besides that, we also work on building our NLP (Natural Language Processing) solutions.<\/p>\n\n\n\nNext steps<\/strong><\/h2>\n\n\n\n