Setting up a CI/CD pipeline for API development with WSO2 API Manager, Github, and Jenkins

9 min readSep 6, 2020

Today APIs have taken a key role in generating revenue and getting more business into the enterprises. With this, the quality and reliability of the APIs have become very critical.

WSO2 API Manager provides the main access point from which to manage, supervise, and have secure access to the APIs.

Many enterprises are using multiple environments like development, testing, and production to ensure the APIs are developed and tested well before it is moved to production. Manually promoting APIs from lower environments to upper environments is a repetitive, time-consuming, and error-prone task. Hence it is important to have an automated process for this.

For this, you might need to have a fully automated CI/CD pipeline. The CI/CD process should cover the CI/CD aspects of both the actual backend APIs and the API Proxies created in the API Manager that fronts them. In this article, we are focusing on how to build a CI/CD pipeline for those API Proxies.

WSO2 API Manager comes with a CLI tool — API Controller which can be used to build a CI/CD pipeline for API development very easily.

In this article we will be discussing two main development flows for APIs and how they fit into the CI/CD pipeline.

API Developer creates an API in a locally running API Manager. — The API will be then exported using API Controller into an API Project. The project is committed to a Git repository which will trigger a CI/CD pipeline that pushes it to the upper environment.
API Developer creates an API Project using API Controller itself — Here the API Developer has an OpenAPI definition of the API and creates an API Project using it from API Controller. The project is committed to a Git repository which will trigger a CI/CD pipeline that pushes it to the upper environment in the same way as above.

The below diagram illustrates the two different flows and we will discuss each of them.

Pre-requisites

WSO2 API Manager 3.2.0
API Controller 3.2.0
Jenkins
Git

Setting up and start WSO2 API Manager

This tutorial assumes that the API Manager is running on live.apis.com hostname. If you are using a different hostname make sure to use it in the occurrences of the above host in this tutorial.

To change the hostname of the API Manager follow the official documentation.

You can start the API Manager instance by running:

Linux/MacOS: <APIM-HOME>/bin$ ./wso2server.sh
Windows: <APIM-HOME>\bin> wso2server.bat

Setting up API Controller

You can follow Getting Started with WSO2 API Controller section in the official documentation to setup API Controller with easy steps.

Setting up Jenkins

There are several methods included in the official Jenkins documentation. If you have already installed Java, you can easily use the war file approach.
You can also follow one of my previous articles as well.

Setting up the Git repository

A Git repository will be used to store the API projects and it will be pulled by Jenkins when deploying APIs.

Create a new Git repository and clone it.
Add the below content as the Jenkinsfile in the repository root folder.

Commit the Jenkinsfile to the Git repository and push.

Note:

In the above Jenkinsfile, there are three API Controller commands are used:

apictl add-env -e live --apim https://live.apis.com:9443

Registers the running API Manager instance at https://live.apis.com:9443 as the live environment with API Controller.

apictl login live -u admin -p admin

Logs in to the live environment using the admin credentials. For simplicity, the username and passwords are mentioned in plain text. But when running this in production, we should use a proper credentials storing mechanism such as Jenkins Credentials.

apictl vcs deploy -e live

Deploys the APIs (and of course other artifacts like Applications and API Products) to the given API Manager environment with -e flag.

Initialize the repository with API Controller

When using API Controller’s native git integration support, it is mandatory to initialize the git repository with API Controller. It is done with the simple and one-time command apictl vcs init. This will create a file vcs.yaml in the repository root folder which is used to identify the repository uniquely by API Controller.

Run apictl vcs init from the repository root.

apictl-git-cicd$ apictl vcs init Successfully initialized GIT repository

Now commitvcs.yaml to the Git repository and push.

Now, our initial repository content is ready.

Setting up the Jenkins Pipeline

From this step, we’ll move on to creating a Jenkins Pipeline that executes on the git repository content and push API artifacts to the live API Manager instance.

Select New Item from the main menu.
Give a name (eg: APIM-CICD-Pipeline) to the item and select the Pipeline option.

Under the pipeline settings, select configurations as below. Give the URL of the repository you created before under Repositoy URL
For the Script Path give JenkinsFile
Click Save

In the next screen, click on Build Now to schedule a build.

In the Console Output of the first build #1 should give a SUCCESS result if everything is in order.

Adding the first API —Creating the API Project using API Controller itself

API Developers can create WSO2 API Manager API Projects using API Controller itself. For this, the API Developer doesn’t need to run an API Manager locally to create an API Project. Instead, the API Controller can create a project on behalf of the API Developer. If the API Developer has an OpenAPI specification of the backend API, then it can be provided to the API Controller to create the API Project.

Run apictl init as below by providing the OpenAPI specification URL with the --oas flag. You can use the OpenAPI Spec URL: https://raw.githubusercontent.com/OAI/OpenAPI-Specification/master/examples/v3.0/petstore.yaml

apictl-git-cicd$ apictl init petstore --oas https://raw.githubusercontent.com/OAI/OpenAPI-Specification/master/examples/v3.0/petstore.yaml -f --initial-state=PUBLISHEDInitializing a new WSO2 API Manager project in /home/malintha/wso2apim/cur/medium/git-cicd/apictl-git-cicd/petstore
Project initialized
Open README file to learn more

The above command will create a project petstore with PUBLISHED as the initial status. This means, once the API is deployed to API Manager, this will directly become Published.

This is the structure of the created project.

apictl-git-cicd$ tree petstore
petstore/
├── api_params.yaml
├── Docs
│   └── FileContents
├── Image
├── Interceptors
├── libs
├── Meta-information
│   ├── api.yaml
│   └── swagger.yaml
├── README.md
└── Sequences
    ├── fault-sequence
    ├── in-sequence
    └── out-sequence10 directories, 5 files

Copy a thumbnail image into the Image folder. Make sure to have the file named with icon and you can have any supported image formats such as jpg, png. Eg: icon.jpg, icon.png
You need to set the backend endpoint of the API by editing api_params.yaml .

environments:
  - name: live
    endpoints:
      production:
        url: https://live.petstore.com
      sandbox:
        url: https://sandbox.live.petstore.comdeploy:
  import:
    update: true
    preserveProvider: true

Commit the full project to the Git repository.
Now, run a Build Now in the Pipeline.

If the deployment is successful, the Jenkins console logs should indicate it.

If we login to the Publisher in the Live environment, we should be able to see the new API you just deployed through the CI/CD pipeline without even touching the Live API Manager Publisher UI !! (https://live.apis.com:9443/publisher)

Adding Another API — by exporting an API from another API Manager instance

API Developer also has the option to create the API in a local API Manager instance then export it as a project. The exported project can be used for the same CI/CD flow as earlier.

Add the API to a locally running API Manager

Start another API Manager instance in your local machine. (If you are running the earlier API Manager instance (live) also in the local machine, make sure to port-offset the second instance by 1.
Login to Publisher using https://localhost:9443/publisher (or use the port offset)
Create a new API with below details

> Name: WeatherAPI

> Version: 1.0.0

> Context: /weather

> Endpoint: https://local.weatherapi.com (this is a dummy endpoint for demo purpose)

> Business Plan(s): Unlimited

Add a thumbnail image (or do any other changes as necessary)
Publish the API (Lifecycle Menu -> Publish)

Export the API from API Controller

Now we need to export the created API using the API Controller.

First, add an environment for the local API Manager if you have not added already. Specifylocal as the name of the environment.

$ apictl add-env -e local --apim https://localhost:9443Default token endpoint ‘https://localhost:9443/oauth2/token' is added as the token endpoint 
Successfully added environment ‘local’

Login to the local environment. Here I have used admin user and the password for simplicity. Instead you can also create a custom user to perform API controller operations.

$ apictl login localUsername:admin
Password:*****
Logged into local environment
WARNING: credentials are stored as a plain text in /home/malintha/.wso2apictl.local/keys.json

List the APIs in the local environment where you can see the API you just created.

$ apictl list apis -e localID                                     NAME                VERSION             CONTEXT             STATUS              PROVIDER
4aa8ba7a-88b4-432a-b8e3-338dfb0b3f70   WeatherAPI          1.0.0               /weather            PUBLISHED           admin

Now, export the API using the export-api command.

$ apictl export-api -n WeatherAPI -v 1.0.0 -e localSuccessfully exported API!
Find the exported API at /home/malintha/.wso2apictl/exported/apis/local/WeatherAPI_1.0.0.zip

Extract the exported API Project to the git repository.
The backend endpoint of the API may require to change in the live environment. Change it by editing api_params.yaml .

environments:
  - name: live
    endpoints:
      production:
        url: https://live.weatherapi.com
      sandbox:
        url: https://sandbox.live.weatherapi.comdeploy:
  import:
    update: true
    preserveProvider: true

Now we are ready to push the API project to the live environment.

Push the project and sync to the Live environment

Commit the project to the repository and push.

$ git add WeatherAPI-1.0.0$ git commit -m "Adding WeatherAPI-1.0.0"
[master 2b74148] Adding WeatherAPI-1.0.0
 4 files changed, 213 insertions(+)
 create mode 100644 WeatherAPI-1.0.0/Image/icon.png
 create mode 100644 WeatherAPI-1.0.0/Meta-information/api.yaml
 create mode 100644 WeatherAPI-1.0.0/Meta-information/swagger.yaml
 create mode 100644 WeatherAPI-1.0.0/api_params.yaml$ git push origin master
Counting objects: 9, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (8/8), done.
Writing objects: 100% (9/9), 295.32 KiB | 956.00 KiB/s, done.
Total 9 (delta 0), reused 0 (delta 0)
To git+ssh://github.com/malinthaprasan/apictl-git-cicd
   f258492..2b74148  master -> master

Now, run a Build Now again in the Jenkins Pipeline.
In the Console Output , can see the below output. API Controller detects the new API which was added in the latest commit and it will be deployed to the live environment.

In the Publisher portal, see the new API just added.

Backend endpoints are changed as specified in api_params.yaml

Congratulations!!!

You have just completed building a CI/CD pipeline to fully automate pushing APIs to another environment. Note that, you haven’t done any changes in the second environment Publisher portal to push the two APIs. And you just covered two main flows for pushing APIs to another environment. 1st API was created using the API Controller itself. 2nd API was created in a local API Manager and exported from API Controller.

The full repository with contents we pushed can be accessed below.

malinthaprasan/apictl-git-cicd

Contribute to malinthaprasan/apictl-git-cicd development by creating an account on GitHub.

github.com

From this article, I was mostly focusing on the main skeleton of a CI/CD pipeline that fully automates the API deployment. There are several aspects that are not covered here. I hope to cover them from a few articles coming up.

Automated testing
Integrating multiple environments (dev, testing, prod)
How to bring other assets like applications, subscriptions, API Products into the CI/CD flow.
Security best practices.

See you soon!.