DevOps for LTE service provider
End-to-end automation of assembly, testing and delivery resulted in a productivity increase of 800% with the whole cycle taking up 30-60 minutes.
Problem
Customer
The customer is one of the largest LTE service providers in Europe with a user base of about 28 million people. The mobile communication operator is constantly developing and implementing various solutions. And since the subscriber base is quite large, with services available 24/7, it was very important to create an automated deployment system, providing work without crashes and downtime. They addressed Itransition for DevOps services.
Problems
The main pain point of the project was to come up with a mechanism within XL Deploy to enable the customer’s team to deploy a whole release, rather than individual components. After trying several options, it was decided to customize the dependency orchestrator and create UI.
The project was characterized by rapid development with simultaneous focus on many directions of varying complexity.
The most challenging aspects of the project were providing support in absence of the customer’s team (with team members on vacation, sick leave, etc.), receiving approvals, and communicating with a large dispersed team.
Another challenge was working with a large system. As a consequence, great emphasis was put on integration, as well as providing ample flexibility to deal with constantly changing requirements, plans, and deadlines.
These challenges were resolved with the help of a thorough business approach and organization of work: the customer provided Itransition time for training, organized and conducted knowledge transfer between teams which greatly facilitated the process of initiating specialists into the project and subprojects. The work of the project’s architect was impressive, and the customer’s choice of tools for execution and automation of assembly and delivery, XL deploy, proved to be optimal for meeting project goals.
Project Goals
The main goal of the project was deployment automation: delivery automation for the modem environment conducted with the help of XL Deploy, development of the existing website, infrastructure organization and support (including the billing system of the mobile operator).
Other project goals included:
- Developing plug-ins for XL Deploy for delivery to JBoss, GlassFish, Tibco
- Support of the customer’s Atlassian products (FishEye, Jira)
- L1 / L2 in times when the customer’s key team members were absent (supporting test environments for testers and developers; eliminating component deployment issues (error analysis and troubleshooting); resolving issues when services were down)
- Works on CI / CD (Jenkins, XL Deploy).
Itransition worked with 2 development environments (voice and modem telecommunications), and with about 20 QA environments for individual component modules.
Solution
The result is a customized product that fits the customer’s goals. The solution automates such business processes as assembly, delivery and testing of voice and modem system components (36 components in total, making up 25% of the customer’s infrastructure), which was the focus of the customer support team.
Before Itransition’s team involvement on the project, about 36 components were collected and delivered manually. Another 100 of the remaining components were planned to be automated.
Prior to configuring delivery automation, developers spent about 8-20 man-hours on each delivery. Itransition helped the customer establish a proper delivery pipeline facilitating continuous deployment of software to QA and then to production in an efficient and automated manner, with every change to code deployable at any time, allowing the systems to absorb rapid change.
Since automating end-to-end delivery, the whole cycle (assembly-> testing-> delivery) takes about 30-60 minutes, with a productivity increase of 800%. Besides faster feature delivery, other business and technical benefits of automating deployment cycles are continuous software delivery, decreased problem complexity and stable operating environments.
Process
Itransition’s DevOps engineer analyzed the structure of the environments and the bash scripts, made a list of services requiring automation, drew a roadmap and recommended processes to be leveraged.
The main focus was automating deployment cycles across the delivery infrastructure to mitigate risks and increase productivity. At the first stage, Itransition’s team perfected the customer’s initial work with bash scripts for the environment deployment and maintenance. Itransition’s DevOps engineer came up with a full concept of automated delivery, created version variations and packs containing different versions, as well as developed necessary tools for realizing the plan.
It turned out that XL Deploy was not able to work with Oracle GlassFish Server (the main service of the modem environment) and WildFly in its entirety, so the DevOps engineer wrote plugins extending the functionality of the existing program and created the necessary add-ons in the form of web GUI and additional databases with presets for data transfers. Work on processes delivery and automation using XL Deploy and Maven included plugins for WildFly, CDI, GlassFish, SSO and WebUI for one-click deployment (partially realized), and creating about 20 components of the modem environment for XL Deploy. All the components were assembled using Jenkins and imported into XLD after a successful build in the testing environment.
The solution is integrated with Citrix NetScaler Unified Gateway for accessing the customer’s environment.
30% of the DevOps engineer’s time was spent on supporting Atlassian products. Work on Jira, for example, included not only creating, modifying, and setting up project tasks but also setting up fields and layouts, organizing workflow, and managing access rights.
The solution is used by the customer’s employees and developers both from Itransition and other companies.
Team
The project team was fairly distributed and dynamically developing. In addition to customer’s own development and QA teams, other contractors were actively involved such as developers, testers and telecommunication equipment suppliers. Itransition’s team consisted of 1 DevOps engineer, 1 customer support expert, 10 developers and 5 QA specialists. The customer support team provided Atlassian-based full-cycle DevOps enablement L2 support (maintenance, backup, 24/7). In absence of key engineers, L1 also provided support for internal services.
The customer provided a challenging release date plan, so the team often had to work with tight deadlines. Effective communication and collaboration of such a large team was carried out with the help of email, Jira and Skype, with weekly meetings to monitor the progress. Itransition’s team managed to work effectively in a distributed team on a project in which communication between specialists (testers, developers, architects, analysts, IT experts) from different countries played a very important role.
Testing
The main testing goal was optimizing the system and improving business processes. The QA team consisted of 5 testers. The DevOps engineer worked closely with Itransition’s developers and testers, the customer’s QA team and other contractors. Developers created new features and added to existing functionality, while the DevOps engineer oversaw deployment, maintenance of testing and development environments, with continuous testing of Itransition’s DevOps solution (including the launch of XL Deploy) throughout the whole project.
