![Build Status](https://npontu.com/images/logos/logo-inner.png) --- # WELCOME TO [NPONTU TECHNOLOGIES](https://npontu.com/). --- **Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)*
STARTER PACK # STARTER PACK All Engineering personnel will be provided access to the following: - A company email which will look as follows first letter of first name + surname + @npontu.com (example; an employee named Kofi Mensah email will be kmensah@npontu.com). Kindly note the email convention may change if the email address is taken. Employees can login to their mail at mx.npontu.com - A company email signature will also be provider **Version Control** - Personnel will be given access to our Company gitlab accounts. That is https://gitlab.npontutechnologies.com and http://gitlab.deywuro.com/ when working on older projects **Main Languages** - PHP - Python - Javascript and Jquery **Frameworks** - Laravel - Django - Flask **Databases** - Mysql - Postgres - Elasticsearch - Mongodb **Event store, streaming, queuing and caching tool** - Kafka - RabbitMQ - Redis **Tools** - Kedebah; Personnels will be given access to Kedebah.com to monitor and manage project tasks and activities - Test Environment; Personnels will be provided access to sandbox environments for safe testing of applications
APPLICATIONS INFRASTRUCTURE # APPLICATIONS ---- ![APPLICATIONS](Infrastructure_Npontu.jpg) ### Service: **Application:** are built with php, python and javascript at Npontu. USSD application are normally built with pure php and they are hosted on server 1 (for vodafone) and server 3 (for mtn and AirtelTigo). Laravel, Django, Flask are a few of the frameworks used for building. PHP versions 7.2, 7.4, 8.0, 8.1 and python 3 is currently used in production. Nginx is currently the only webserver used. All projects are own and controlled by npontucontrl. All applications can be found at `/home/sbj` or `/var/www/html/` **PHP**: When installing **php Centos 7**, follow the code below; ``` sudo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm sudo yum -y install https://rpms.remirepo.net/enterprise/remi-release-7.rpm sudo yum -y install yum-utils sudo yum-config-manager --disable 'remi-php*' sudo yum-config-manager --enable remi-safe sudo yum -y install php80 ``` Use the command below to install additional PHP extensions: `sudo yum install php80-php-xxx` Example: `sudo yum install php80-php-{cli,fpm,mysqlnd,zip,devel,gd,mbstring,curl,xml,pear,bcmath,json}` When installing **php Ubuntu**, follow the code below; A user guide for installation can be found at https://blog.containerize.com/2021/05/21/how-to-install-multiple-php-versions-with-nginx-on-ubuntu/ ``` sudo apt-get install software-properties-common -y sudo add-apt-repository ppa:ondrej/php sudo apt-get update -y apt-get install php7.0 php7.0-fpm php7.2 php7.2-fpm -y ``` After PHP is installed on a server the following tasks are carried out A user guide can be found at https://www.getpagespeed.com/server-setup/nginx-and-php-fpm-what-my-permissions-should-be The .conf file in php-fpm.d/www.conf is updated with the username npontucontrl. The npontucontrl user should be created if it doesn't exist. The listen's port should be updated to an available port. ``` listen = 127.0.0.1:9080 listen.owner = npontucontrl listen.group = npontucontrl listen.mode = 0660 user = npontucontrl group = npontucontrl ``` PHP installed on centos operating systems are located at `/etc/opt/remi/` PHP installed on ubuntu operating systems are located at`/etc/php` **PHP-FPM** Restart php-fpm ``` systemctl status phpXX-php-fpm systemctl restart phpXX-php-fpm ``` **Composer** running composer ``` phpXX /usr/local/bin/composer update phpXX /usr/local/bin/composer install ``` **Artisan** Running artisan ``` phpXX artisan config:clear phpXX artisan config:cache ``` ### TO DO: ROUTINE TASKS 1. Monitoring and managing server disk space, CPU utilization and performance IMPROVEMENT TASKS 1. Make applications stateless 2. Containerize applications to able easy deployment of multiple instances of applications to help achieve 99.999% SSL uptime 3. Add failover server to gitlab CI/CD pipeline. This will keep the failover server upto date. 4. Implement clickstream tracking to help improve monitoring and customer retention. ### THINGS TO KNOW: 1. Npontu manages the ndc server 2. npontucontrl user is used to manage all projects ---- ---- # DEYWURO ---- ![DEYWURO](Infrastructure/Deywuro_Infrastructure.png) ### Service: **Kannel:** Kannel is the SMS gateway Npontu uses to process SMS. Its installed on 3 server, with server 1 (5.9.61.79) and server 8 (95.216.10.31) being production environments and server 7 (95.216.10.32) being a sandbox environment. We have access to the following vendors: MTN, Airtel, Vodafone, RMS and infobip. For further checks visit /opt/kannel/etc/smsc.d on any server with kannel installed. Method of procedures for kannel tasks can be found at https://npontuiss.atlassian.net/wiki/spaces/ISS/pages/36438017/RESTARTING+KANNEL+BOXES - **Server 1 Kannel :** is mostly used to process api, smpp messages and foreign SMS traffic. The current active routes on server 1 are vodafone, rsms, mtnnpontu2 and mtnnpontu3. - **Server 8 Kannel :** is mostly used to process mtn messenger SMSes and bulk SMS. The current active routes on server 8 are mtnnpontu and mtnmsg. **Application:** The Deywuro application is written php with the laravel framework. Nginx is used as the web server and php-fpm version 7.2 is used. The application is broken into the frontend, sms apis, reporting apis and bulk processing scripts. The bulk processing scripts, dlr api and reporting apis are written in python while the sms api and frontend are written with php. Kindly note the index.php file in /var/www/html/bulksms is used to process all api messages and etz_index.php is used to process all Etranzact messages. Currently api messages are processed on 88.198.64.11. The dlr apis are hosted on 88.198.196.190 and 94.130.18.107. The Api is built with python and RabbitMQ is used to manage performance. - py-dlr.deywuro.com and py-dlr.mtnmessenger.com for bulk sms dlrs - dlr.deywuro.com and api-py-dlr.mtnmessenger.com for api sms dlrs **Database:** Deywuro main database is hosted at 144.76.195.8 with its backup hosted at 176.9.1.26. Deywuro has 3 backup policies. - Daily dump to file of all records without the logs tables - Realtime replication of key tables to secondary database and elasticsearch. The script can be located on 148.251.12.91 at /var/www/html/deywuro_backup_scripts - Realtime replication of changes to kafka and mysqlbin logs. ### TO DO: ROUTINE TASKS 1. Yearly SSL certificate renewal with Stanbic bank for deywuro.com 2. Monthly rotation of Kannel's MT table on server 2. (Performanced after the monthly report is generated on the 1st of the month) 3. Monitoring and managing server disk space, CPU utilization and performance 4. Monitoring and managing sms route performance 5. Monthly archiving of deywuro logs data to archive database __ IMPROVEMENT TASKS 1. Automate archiving script 2. Make the deywuro application stateless 3. Deploy multiple instance deywuro architecture to help achieve 99.999% SSL uptime 4. Improve Error reporting and monitoring 5. Improve script the reports low traffic 6. Transfer data dump to AWS S3 bucket ### THINGS TO KNOW: 1. Deywuro SSL is paid for on godaddy and its renewed yearly. The SSL cert expires in October. 2. The Redis for kannel is password protected. The password can be found in the kannel configs 3. Etranzact has its own custom api http://etz.deywuro.com/ # KEDEBAH ---- ![KEDEBAH](Infrastructure/Kedebah_Infrastructure_Npontu.drawio.png) ### Service: **Application:** kedebah is a mutiple tenant application built with the laravel framework. **Database:** Kedebah's main database is hosted at 95.217.63.56 with its backup hosted at 138.201.127.133. Kedebah has 2 main backup policies. - Daily dumps to files. - Realtime replication of changes to mysqlbin logs. Kindly note a realtime replication setup for kedebah has not been setup. As it's in development and the database schema is constantly changing the setup will break frequently. ### TO DO: ROUTINE TASKS 1. Monitoring and managing server disk space, CPU utilization and performance IMPROVEMENT TASKS 1. Make kedebah application stateless 2. Containerize kedebah application to able easy deployment of multiple instances of kedebah to help achieve 99.999% SSL uptime 3. Add failover server to gitlab CI/CD pipeline. This will keep the failover server upto date. 4. Implement clickstream tracking to help improve monitoring and customer retention. ---- ----
MONITORING & ALERTS # MONITORING AND ALERTING ---- ### Service: **Grafana**: is a monitoring application hosted at https://monitor.snwolley.com:3000/. SMS, application, database and server metrics are monitored using grafana. **Prometheus** is an alerting tool hosted at https://prometheus.snwolley.com/. Prometheus alerts on possible issues on our systems through Telegram. **Kafka** is used for event processing. Its hosted on 3 servers 5.9.8.66, 136.243.14.79 and 95.217.45.171. Most producer and consumer scripts are located at /var/www/html on the Grafana server 148.251.12.91 **Elasticsearch** is a document database focused on search. Its hosted on 3 servers 65.21.134.93, 95.217.61.218 and 95.217.56.233. Kindly note the elasticsearch servers have a deny all policy implemented, therefore access must be granted to a server before elasticsearch will be accessible from that server. **Filebeat** is an agent used to gather and transfer log data to kafka to more processing. This agent is installed on a target machine and configured to transfer data to kafka for processing. ### TO DO: ROUTINE TASKS 1. Monitoring and managing server disk space, CPU utilization and performance IMPROVEMENT TASKS 1. Automate kafka SSL renewal 2. Improve current alert system by raising every alert as a ticket that can be closed ### THINGS TO KNOW: 1. Kafka's SSLs are renewed ones every year in September, failure to renew the SSL cert will lead to the total failure of the alerting and monitoring tools. ---- ----
SERVERS # SERVERS Npontu has 2 vendors for server provisioning [AWS] (https://aws.amazon.com/) and [Hetzner] (https://accounts.hetzner.com/login). AWS is currently used as a testing and data archiving environment. Hetzner currently host all production environments. **Hetzner** There are 29 dedicated servers and 4 cloud servers on hetzner. A breakdown of the servers by use are as follows; **Kannel servers:** There are 3 servers (5.9.61.79, 95.216.10.31 and 95.216.10.32) licensed to use kannel with the custom smppbox. **Database servers:** There are 10 database servers. that is; - Deywuro's main database is hosted at 144.76.195.8 with its backup hosted at 176.9.1.26 - Kedebah's main database is hosted at 95.217.63.56 with its backup hosted at 138.201.127.133 - Ussd and mtnmessenger database is hosted at 95.216.10.33 - G-money's main database is hosted at 95.216.10.32 with its backup hosted at 65.21.77.41 - Kannel's database is hosted at 88.198.26.111 - All other live applications database are hosted at 65.21.134.82 with its backup hosted at 95.217.34.208 **Application servers** - Server 1 - 5.9.61.79 - hosts G-money portal, GCB's sms apis, gateway for vodafone payment and ussd - Server 4 - 144.76.58.179 - hosts deywuro and rotary website - Server 3 - 5.9.86.210 - hosts gateway for mtn and airteltigo payment and ussd - AI server - 94.130.18.107 - hosts ai projects, 4am report scripts, bulk dlr api - Api dlr server - 88.198.196.190 - hosts api dlr for deywuro - Deywuro app server - 88.198.64.11 - hosts deywuro's api and a backup deywuro application - Sandbox server - 148.251.89.119 - hosts all applications in development - Grafana & scripts server - 148.251.12.91 - hosts [Grafana](https://monitor.snwolley.com:3000/), kafka consumers and producer scripts, 4 am processing scripts **Elasticsearch servers;** There are 3 elasticsearch servers hosted at 65.21.134.93, 95.217.61.218 and 95.217.56.233 **Kafka servers;** There are 3 kafka servers hosted at 5.9.8.66, 136.243.14.79 and 95.217.45.171 **Gitlab servers;** There are 2 mail server. 95.216.150.21 hosts the [old gitlab](http://gitlab.deywuro.com/) and 95.216.244.146 hosts the [current gitlab](https://gitlab.npontutechnologies.com/) . **Mail servers;** There are 2 mail server. 95.216.150.20 hosts the main company email and 65.108.250.148 hosts the kedebah email.