AWS Transform for mainframe refactor

Project Manager guide

Setup the AWS Blu age DevOps Pipelines after the CDK scripts

AWS Blu Age DevOps Pipeline - Setup Guide

Check Modernized application

If there is an issue with your Jenkins jobs, it is mostly because the application didn’t run properly. So after each deployment, you have to check if the application runs well and without error in the logs.

You can either check the logs in CloudWatch by viewing at the following log group: /ecs/app-<ENV> or you can connect to the EC2 and run some docker commands (from my point of view it’s easier and you have more visibility):

  1. Go to the EC2 console
  2. Connect to ec2-<AWS_REGION>-app-<ENV>

Watch the logs:

sudo su  
docker ps  
docker logs -f <CONTAINER_ID>

Get inside the container:
You can also get inside the container to check if all your files have been well imported and well configured:

docker exec -it <CONTAINER_ID> bash

Modify Auto Scaling Groups (ASG)

All the EC2s created with ECS use an Auto Scaling Group. It means that, you cannot modify the properties of an EC2 as easily as usual. You will have to modify the launch template of the ASG concerned. If you want to modify it:

  1. Go to the EC2 console
  2. In the left panel, at the very bottom, click on Auto Scaling Groups
  3. Click on the ASG you want to modify
    1. You can edit, the Group details section if you want to modify the capacity of your instances.

    2. You can click on the ID of your launch template in the Launch template section and modify the security group, IAM role or whatever you want regarding the properties of the EC2s.

      Modify Parameter Store

      Parameter Stores are often use to store variables such as path, DNS, Docker image versions and so on.
      Here is a list of the parameter stores used and what they store:

  4. /app/VARIABLES: Stores variables about the modernized application
  5. /jenkins/VARIABLES: Stores variables about Jenkins and tests running on it
  6. /lambda/slack-notifications/VARIABLES: Stores variables useful for this lambda
  7. /selenium/VARIABLES: Stores variables about Selenium
  8. /sonarqube/VARIABLES: Stores variables about Sonarqube
  9. /dop/VARIABLES: Stores configuration variables for customizing the CI

Note: Other than the /dop/VARIABLES, I recommend not modifying these variables but sometimes you may have to do it. So you need to know that most of the variables are dynamics, except docker image name and version variables. If you change them you will have to run the codepipeline-docker-image-builder pipeline to apply the change.

Update Docker image version

  1. Go to the Parameter Store
  2. Click on the Parameter you want to modify
  3. Modify the DOCKER_IMAGE_NAME or VERSION variables
  4. Run the codepipeline-docker-image-builder to fetch the new image and push it on the ECR
  5. Then you will have to restart the ECS service for Sonarqube, Selenium or Chrome Node.
    1. For Jenkins you need to connect to the Jenkins EC2 and make the change manually
    2. For Tomcat, you need to rebuild the image with the codepipeline-app-<ENV> pipelines

Add a new CI/CD environment

  1. Go to the Parameter Store
  2. Click on /dop/VARIABLES
  3. Add a new environment in the ENVIRONMENTS variable

Note: environment value must match an existing branch name from both the modern-application and server git repositories.

Doing so will trigger the codepipeline-building-pipeline-factory codepipeline, which will create necessary AWS resources for the new environment.

Restore Gitlab

If your Gitlab repositories are corrupted or your lost your repositories, you may want to restore your repositories. The Lifecyle Manager backups the Gitlab EC2 twice a day except the week end. Here is how to restore your previous version of Gitlab.

The Gitlab EC2 doesn’t need to be terminated

  1. Go to the EC2 console
  2. At the bottom of the left panel, click on Auto Scaling Groups
  3. Click on the gitlab ASG
  4. In the Advanced configurations section, click on Edit
  5. In the Suspended processes field, tick check the boxes Launch and Terminate
  6. Click on Update
  7. Go back to the EC2 instances console
  8. Select the ec2-gitlab EC2
  9. Click on Actions > Monitor and troubleshoot > Replace root volume
  10. In Restore, select Snapshot
  11. Select the Gitlab snapshot you want to restore. You can get more information about the snapshots available on your account here
  12. Click on Create replacement task
  13. Wait for Gitlab to be up again. It can take a few minutes
  14. Optional: If Gitlab does not come back maybe the cluster did not restart the task.
    1. Go to the ECS console
    2. Click on the gitlab cluster
    3. Select the service and click on Update
    4. Tick Force new deployment
    5. Click on Update at the very bottom
    6. Wait for gitlab to be up again. It can take a few minutes
  15. Once Gitlab is up, you can uncheck the Launch and Terminate boxes of the gitlab ASG done at step 5 and click on Update

IMPORTANT: Do not forget to untick the boxes Launch and Terminate of the gitlab ASG

The EC2 needs to be terminated

  1. Terminate the Gitlab EC2 by changing the termination protection to false beforehand
  2. Let the Gitlab ASG recreates a new EC2

Note: If you followed the method with no EC2 termination before this one. Do not forget to untick the Launch and Terminate boxes of the ASG if checked before

  1. Once the new EC2 created, go to the EC2 console
  2. In the left panel, click on Snapshots
  3. Select the snapshot you want to restore
  4. Click on Actions
  5. Click on Create a volume from snapshot
  6. (Optional) Name your volume by giving it a tag. (eg. Key: “Name“, Value: ”gitlab“)
  7. Click on Create volume
  8. At the top of your screen a green banner says that your volume is created. Click on its ID
  9. Click on Actions > Attach volume
  10. Select the Gitlab instance
  11. Select the /dev/sdg device name
  12. Click on Attach volume
  13. Connect to the Gitlab EC2 and type the following commands:
sudo su  
lsblk -o NAME,UUID,MOUNTPOINT # This command allows you to know the parition name of the /dev/sdg device name  
xfs_repair -L /dev/<PARTITION_NAME> # Likely PARTITION_NAME=nvme1n1p1 according to the previous command  
xfs_admin -U generate /dev/<PARTITION_NAME>  
mount /dev/<PARTITION_NAME> /data  
docker ps # Copy the CONTAINER ID

docker stop <CONTAINER_ID>; rm -rf /etc/gitlab/* /var/log/gitlab/* /var/opt/gitlab/*; cp -r /data/etc/gitlab/* /etc/gitlab/; cp -r /data/var/log/gitlab/* /var/log/gitlab/; cp -r /data/var/opt/gitlab/* /var/opt/gitlab/; chown -R ec2-user: /etc/gitlab; chown -R ec2-user: /var/log/gitlab/; chown -R ec2-user: /var/opt/gitlab/
  1. Wait for the ECS cluster to recreate the Gitlab task
  2. (Optional) If Gitlab does not come back, follow these steps: Optional: If Gitlab does not come back maybe the cluster did not restart the task.