sczyh30's Metaspace

GSoC 2016 | Vert.x Blueprint Application

2016-08-18T16:00:00.000Z

This blog post is intended to be an illustration of my work for GSoC Final Evaluation.

Overview

Vert.x is a toolkit for the Java Virtual Machine enabling the implementation of reactive, highly concurrent, polyglot applications. The idea of the project is to implement a set of applications using the Vert.x toolkit to illustrate how different classes of applications can be done. This project aims to provide guidelines to Vert.x users to implement various applications.

The blueprint project contains three parts: Todo Backend, Vert.x Kue and Online Shopping Microservice. Both runnable code and very detailed documents and tutorials (both in English and Chinese) are provided.

Vert.x Blueprint - Todo Backend

The project repository: sczyh30/vertx-blueprint-todo-backend.

This blueprint is a todo-backend implementation using Vert.x and various persistence (e.g. Redis or MySQL). It is intended to be an introduction to basic Vert.x web RESTful service development. From this blueprint, developers could learn:

What is Vert.x and its basic design
What is and how to use Verticle
How to develop a REST API using Vert.x Web
How to make use of asynchronous development model
Future-based reactive asynchronous pattern
How to use persistence such as Redis and MySQL with the help of Vert.x data access components

The tutorial documents:

Vert.x Blueprint - Vert.x Kue

The project repository: sczyh30/vertx-blueprint-job-queue.

This blueprint is a priority job queue developed with Vert.x and backed by Redis. It’s a Vert.x implementation version of Automattic/kue that can be used in production.

Feature document of Vert.x Kue is available here: Vert.x Kue Features.

This blueprint is intended to be an introduction to message-based application development using Vert.x. From this blueprint, developers could learn:

How to make use of Vert.x Event Bus (distributed)
How to develop message based application with Vert.x
Event pattern of event bus (Pub/sub, point to point)
How to design clustered Vert.x applications
How to design and implement a job queue
How to use Vert.x Service Proxy
More complicated practice about Vert.x Redis

The tutorial documents:

Vert.x Blueprint - Online Shopping Microservice

The project repository: sczyh30/vertx-blueprint-microservice.

This blueprint is an online shopping microservice application developed with Vert.x. It is intended to be an illustration on how to develop microservice applications using Vert.x toolkit. From this blueprint, developers could learn:

Microservice development with Vert.x
Asynchronous development model
Reactive and functional pattern
Event sourcing pattern
Asynchronous RPC on the clustered event bus
Various type of services (e.g. HTTP endpoint, message source, event bus service)
Vert.x Service Discovery
Vert.x Circuit Breaker
Microservice with polyglot persistence
How to implement an API gateway
Global authentication (OAuth 2 + Keycloak)

And many more things…

The tutorial documents:

Final Summary && Future

I really enjoy the work! During the work period I’ve learned a lot about Vert.x, various asynchronous development model, distributed systems, microservice architecture and many more things… This motivates me to investigate new technologies, and the infrastructure of Vert.x. This would be an unforgettable experience.

I’ll keep track of the blueprint project and planning to make some more improvements on the microservice blueprint to illustrate more usage about Vert.x and microservices (e.g. recommendation, Kafka, full-index search). And soon I’d like to contribute to Vert.x Microservice Toolbox to add more features and integration with other technologies. As Vert.x Scala is nearly technical preview, I’m also willing to contribute to Vert.x Scala version.

Thanks

I would like to give my sincere thanks to:

My mentors, Paulo Lopes and Clement Escoffier. They provided me guidance and gave me a lot of help during the work :-)
The Vert.x Community.
The Google Summer of Code team, for this wonderful experience.

Vert.x Microservice Blueprint | Some changes && refactoring

2016-08-10T16:00:00.000Z

Simplified architecture

Some of the unecessary components has been removed:

Payment Microservice: The logic should be refactored in order to fit for real production logic.

New diagram:

API Gateway

As suggested by Clement and Paulo, I use HTTP-HTTP pattern for API Gateway eventually. The API gateway itself is using HTTPS, but it communicates REST endpoints via HTTP.

So the API Gateway is responsible for:

Dispatching requests to corresponding endpoint.

How to implement this?
My solution: When publishing endpoints to the service discovery infrasture, every endpoints must provide an api-name to identify the api, then when a request arrives from the gateway, the gateway will get the prefix pattern /api/:name/* and find any corresponding endpoints via discovery, then consume the HTTP endpoint.

Simple load balancing
Health check
Failure handling (using Vert.x Circuit Breaker)
Global auth state storage(in route context and session scope)

Auth

It’s convenient to use Keycloak via Vert.x OAuth2.

GSoC 2016 | Work Period | Week 11

2016-08-06T16:00:00.000Z

Objectives

In this week, I’m planning to do the following stuff:

Architecture enhancement
A simple SPA frontend for the microservice blueprint (use Vue.js)
Monitor dashboard
A simple recommendation component
- Maybe I should write a individual component providing Neo4j integration with Vert.x
Run the application with Docker Compose
Update tests
Update documents and tutorials

Issues

Solved: Global auth

How to manage global auth in a microservice application. For example, we login in from the login page and our login data is saved in the session. When we visit resources that are protected behind the auth, how we validate the user info (e.g. via session data or use SSO)?

Solved: Use OAuth 2 + Keycloak is a good point. The API gateway is responsible for storing authentication principal and validation.

Thought: How to integrate well with SPA frontend?

Solved: Need a better mechanism for API gateway

Solved: use HTTP-HTTP pattern. The API Gateway is a individual verticle and is responsible for dispatching requests, managing global authentication, handling failure, health checking and simple load balancing. The API Gateway itself uses HTTPS, but it interacts with the downstream REST endpoints via HTTP.

Achievement

Refactored the architecture of the online shopping microservice, simplified some components (more features will be added into next version)
Enhanced the logic of the entire online shopping process
Code refinement and bug fixed
Added some tests (some tests are in local and weren’t committed as it requires some local envs)

Vert.x Microservice Blueprint | Basic Design

2016-07-31T16:00:00.000Z

The blueprint is an online shopping microservice application. You can treat it as a very simple C2C market like eBay.

Here is a diagram illustrating the microservice architecture:

Currently I designed these components:

account-microservice: Component for user account
product-microservice: Component for products we sell and buy
store-microservice: Component for individual stores in the market
checkout-microservice: Compoment for order checkout
payment-microservice: Compoment for payment transaction processing
order-microservice: Compoment for order dispatching, processing, analyzing and storage
- NOTE: this component should be divided into several sub-components
recommendation-microservice: Compoment for simple recommendation for friends and favorite products with Neo4j
cache-infrastructure: Cache component for services
api-gateway: API gateway (with load balance)
shopping-ui: A SPA front-end for the microservice
monitor-dashboard: Dashboard for performance monitoring and backend metrics data demonstration

Here is the user workflow: User login and add products to chart -> Checkout -> Go to payment page and pay for the products -> Payment transaction success, order submitted -> (Wait for delivery...)

All events are sent on the event bus.

GSoC 2016 | Work Period | Week 9 - Week 10

2016-07-31T16:00:00.000Z

Objectives

In this week, I’m planning to do the following stuff:

Initial work for Online Shopping Microservice Blueprint
Prepare for next blueprint application (high performance)

Good luck~ :-)

Issues

API gateway

How to implement an API gateway with load balance.

Checkout model

TODO: The checkout-payment-order model should be changed.

Deployment

In our design, the online shopping microservice blueprint contains many components, so we need to pay attention to how to deploy such a microservice application (e.g. use Docker Compose, Kubernetes; Deploy on Openshift)

Order dispatching design

In our current design, orders are sent via clustered event bus. But a real production environment, orders are supposed to be dispatched to message queues with recovery and persistence ability(e.g. Kafka). And the order processing component and order persistence component should pull orders from the MQ.

Integration with other technologies

e.g. Kafka

Achievement

Initial work for Online Shopping Microservice Blueprint

GSoC 2016 | Work Period | Week 7 - Week 8

2016-07-15T16:00:00.000Z

Objectives

In this week, I’m planning to do the following stuff:

Finish Chinese documentation of Vert.x Kue
Work on microservice blueprint
Learn more about Vert.x Microservice

Notice: Because my semester final exams is pending this week and next week, I’ll have to be busy with them. So I might spend less time on it (have time in the evening). After the final exams I’ll speed up and make a wonderful blueprint!

Good luck~ :-)

The final exams finally finished!!!

GSoC 2016 | Work Period | Week 5 - Week 6

2016-07-02T16:00:00.000Z

Objectives

In this week, I’m planning to do the following stuff:

Some extra things to implement or fix:
- Job active TTL check
- Job search functionality
- Render issue with _filter.jade
Finish documents and tutorials of Vert.x Kue
Finish tests
Add Docker Compose file
Prepare for the microservice blueprint

Good luck~ :-)

Issues

TODO: Job search functionality

This need a full-index engine.

TODO: Run Vert.x Kue in Docker Cluster

Maybe we need some configurations for Hazelcast to support it on Docker.

Achievement

Finish English documents and tutorials of Vert.x Kue
Finish part of Chinese documents and tutorials of Vert.x Kue
Initial design for bluepint microservice

These days my progress is slow…Because I have to prepare for the final exam(7.1 - 7.15) these days…

Vert.x Kue | A brief introduction to how it works

2016-06-20T02:00:00.000Z

In this blog post, I’ll give a brief introduction to illustrate how Vert.x Kue works.

In the workflow, first we deploy the KueVerticle in clustered mode and register the JobService on the event bus. The JobService consists of various logic of Job. After that, we could then deploy KueHttpVerticle to provide REST API. Most of the apis calls Future-based methods in Kue. Finally we could deploy our job processing verticle. The verticle usually consists of three parts:

create Kue instance

When we create Kue instance, the inner JobService proxy and redis client will be created. Also, the static Vertx field in Job class will also be initilized.

create and save Job

Here we could set job properties and subscribe events from the job address(via event bus).

process the Job

When we call the process or processBlocking method, Vert.x Kue will create and then deploy some KueWorkers, which is a kind of verticle(or worker verticle, if calling processBlocking).

In KueWorker, we first get Job from Redis backend (zpop operation):

private void prepareAndStart() {
  this.getJobFromBackend().setHandler(jr -> {
    if (jr.succeeded()) {
      if (jr.result().isPresent()) {
        this.job = jr.result().get();
        process();
      } else {
        throw new IllegalStateException("job not exist");
      }
    } else {
      jr.cause().printStackTrace();
    }
  });
}

If the KueWorker successfully gets the job, it will do process procedure. First we set started_at flag on the job and then save it to the backend. Then we emit the job start event on the event bus(both job address and worker address). Next, we will process the job using the given handler. Simultaneously, we consume done and done_fail events on the job-specific address. If done event received, the job should be completed and we emit complete event. If done_fail event received, the job should be failed so we first do a failed attempt. If attempt failed, then send error event on worker address. Else send failed or failed_attempt event depending on the attempt status.

There are three kinds of address:

/**
   * Handler address with certain job on event bus
   * Format: vertx.kue.handler.job.{handlerType}.{addressId}.{jobType}
   *
   * @return corresponding address
   */
  public static String getCertainJobAddress(String handlerType, Job job) {
    return "vertx.kue.handler.job." + handlerType + "." + job.getAddress_id() + "." + job.getType();
  }

/**
 * Global worker address with specific event type
 */
  public static String workerAddress(String eventType) {
    return "vertx.kue.handler.workers." + eventType;
  }

  /**
   * Worker address with specific event type and job-uuid on event bus
   * Format: vertx.kue.handler.workers.{eventType}
   *
   * @return corresponding address
   */
  public static String workerAddress(String eventType, Job job) {
    return "vertx.kue.handler.workers." + eventType + "." + job.getAddress_id();
  }

Diagram

GSoC 2016 | Work Period | Week 4

2016-06-19T16:00:00.000Z

The GSoC Midterm Evaluation is coming~ Boost~O(∩_∩)O

Objectives

In this week, I’m planning to do the following stuff:

Enhance Event system and KueWorker in Kue
- Implement all basic events in original Kue
- If possible, do some performance optimization
Add test cases and fix bugs
Add more examples
Finish all Vert.x Kue REST apis
Adapt Vert.x Kue REST API with original Kue’s UI
Code refinement
Initial document and design diagram update
Learn more about new features in Vert.x 3.3.0

Achievement

initial-work implementation of Vert.x Kue (including Vert.x Kue REST API and UI)
- See: sczyh30/vertx-blueprint-job-queue
Bugs fixed
Add more examples
Some refactor and code refinement
Feature document draft

GSoC 2016 | Work Period | Week 3

2016-06-12T15:00:00.000Z

Objectives

In this week, I’m planning to do the following stuff:

Core component of Vert.x Kue implementation complete
- Event system in Kue
- KueWorker
- Job
Basic implementation of Vert.x Kue HTTP component
- REST API implementation
- A simple UI (Vert.x Kue UI)
Refine the code of Vert.x Kue
- The Job class is messy and need refinement
- More concise
Learn more about microservice (as well as Vert.x Microservice Toolbox)
Write a brief introduction to new features of Vert.x 3.3.0 (in Chinese, will publish in Vert.x China Group)

PS: The Dragon Boat Festival holiday will be coming from Thursday to Saturday this week so I may not spend very much time on work during the holiday~ Have a great week :-)

Issues

Solved: Regex match in route path

In original Kue’s API, there is a route with path /jobs/:from..:to/:order?. I tried this in Vert.x Web(/jobs/:from..:to) but Vert.x Web cannot match it. So I used regex match \/jobs\/([0-9]*)\.\.([0-9]*)(\/[^\/]+)?:

public static final String KUE_API_JOB_RANGE = "\\/jobs\\/([0-9]*)\\.\\.([0-9]*)(\\/[^\\/]+)?";

// ...

router.getWithRegex(KUE_API_JOB_RANGE).handler(this::apiJobRange);

But nothing matched, which made me confused. And path like /jobs/:from/to/:to can be matched.

Solution: First use a supported pattern to catch string like 1..10 then use regex parse.

Achievement

Implementation of Vert.x Kue core

Job processing interface KueService now provides two methods: process method do asynchronous procedure(use common verticles), while processBlocking method could do blocking procedure(use worker verticles)
Add attempt support (could attempt when fails)
- TODO: need enhancement and test
Add worker and job cardinality metrics support
Add support for getting jobs with range(rank)
Basic refinement of Job class
Refactored most of async methods to Future based style (in order to support monadic transform and say goodbye to callback hell)
Fix issue: Sometimes throw java.lang.ClassCastException caused by wrong time flow when processing a job
Fix issue: Generated json converter sometimes could not distinguish some properties(e.g. id and jobMetrics) correctly

See here: sczyh30/vertx-blueprint-job-queue

Basic implementation of Vert.x Kue HTTP component

Basic implementation of Vert.x Kue REST API(get job, create job, delete job, get job by range(or state), etc)
Basic adaption between Vert.x Web and original Kue UI (rendered with vertx-web-templ-jade; still have some problems to solve)

Wrote a brief introduction to new features of Vert.x 3.3.0 in Chinese

GSoC 2016 | Work Period | Week 2

2016-06-05T16:00:00.000Z

Objectives

In this week, I’m planning to do the following stuff:

Work on the Vert.x Blueprint - Kue (Initial implementation)
Learn more about event-driven and message system

Knowledge got

This week I’ve learned a lot!

Clustered Verticles and how to interact with each other
Vert.x Event Bus (send/recv and pub/sub)
Vert.x Service Proxy
More understanding about Redis(sorted set, list, pub/sub, transaction, etc)
More understanding about asynchronous and reactive pattern(Handler / Future)

Issues

How to implement priority?

I imitate Kue in Node.js: use sorted sets in Redis and mark the priority as the score(weight). Use zadd to add a certain id of job and when necessary, use zpop(implemented with transaction) to get a job with higher priority.

How to emit and receive events?

In Node.js, we could use EventEmitter to emit and listen to events. And in Vert.x, we could make full use of EventBus, which supports three kinds of pattern. For example, in Node.js we may write this:

1
2
3

self.on('complete', fn);
// ......
self.emit('complete', res);

And in Vert.x we could write:

// listen to an address
eventBus.consumer(Kue.getHandlerAddress("complete", this.type), message -> {
      completeHandler.handle(new Job((JsonObject) message.body()));
});

// send message to an address
eventBus.send(Kue.getHandlerAddress("complete", type), job.toJson());

Achievement

Refined the design of Vert.x Kue

Kue: like the Queue in Kue. We could createJob and process the job
KueVerticle: a verticle where the KueService is registered
KueWorker: a worker verticle that processes the job
KueService: create KueWorker and deploy it
Job: the job entity, contains numerous logic about job

The backend of Vert.x Kue is Redis.

Basic usage (design) of Vert.x Kue:

// write this and then run with Vert.x Launcher in clustered mode
public class ExampleProcessVerticle extends AbstractVerticle {

  @Override
  public void start() throws Exception {
    // must first create kue
    Kue kue = Kue.createQueue(vertx, config());

    Job job0 = kue.createJob("video", new JsonObject().put("id", 3001))
      .priority(Priority.HIGH)
      .onComplete(e -> {
        System.out.println("Video result: " + e.getResult());
        System.out.println("Haha");
      });

    job0.save();

    kue.process("video", 1, r -> {
      if (r.succeeded()) {
        Job job = r.result();
        // consume 2 seconds
        vertx.setTimer(2000, l -> {
          job.progress(100, 100);
          System.out.println("Video id: " + job.getId());
        });
      } else {
        r.cause().printStackTrace();
      }
    });
  }
}

Initial concept implementation of Vert.x Kue

See here: sczyh30/vertx-blueprint-job-queue

GSoC 2016 | Work Period | Week 1

2016-05-28T16:00:00.000Z

This is the summary of Google Summer Of Code 2016, Week One.

Objectives

In this week, I’m planning to do the following stuff:

Refine the code and document of Vert.x Blueprint - Todo Backend
Work on the Vert.x Blueprint - Kue (Basic Design)

Issues

How to use Vert.x Codegen with Gradle

We need to understand the essence of Vert.x Codegen - an APT(annotation processing tool), so we could write a task to do processing:

task annotationProcessing(type: JavaCompile, group: 'build') {
  source = sourceSets.main.java
  classpath = configurations.compile
  destinationDir = project.file('src/main/generated')
  options.compilerArgs = [
    "-proc:only",
    "-processor", "io.vertx.codegen.CodeGenProcessor",
    "-AoutputDirectory=${project.projectDir}/src/main"
  ]
}

And in compileJava we reference the annotationProcessing task:

compileJava {
  targetCompatibility = 1.8
  sourceCompatibility = 1.8

  dependsOn annotationProcessing
}

sourceSets {
  main {
    java {
      srcDirs += 'src/main/generated'
    }
  }
}

Then we could process the annotations with Gradle~

The Chinese version is here: 在 Gradle 中使用Annotation Processing Tool | Vert.x Codegen 示例

Achievement

Because there were many temporary experiments last week, I didn’t work during Friday to Sunday. Here are achievements of this week:

Finished the Vert.x Blueprint - Todo Backend project

Basic design of Vert.x Kue

In Kue(Node.js), we use kue.createQueue() to create a Kue instance and then use create(type, data) to create a job and then save the job to Redis. After that, we invoke job.process(type, callback) to process the job.

And in my design of Vert.x Kue, I designed a KueService that provide basic functions and wrapped a class Kue like Queue in Node.js version. I use Vert.x Service Proxy and register it to a KueVerticle. Our user could create a custom verticle(e.g. ExampleVerticle) and create and process jon via Kue class.

As for processing jobs, I’m planning to use Worker Verticles or just use executeBlocking method.

Learned something about Microservice

Microservice patterns(Quorum, Circuit Breaker Pattern, Compensating Transaction Pattern, etc.)
Usage and reactive thoughts of Netflix Hystrix

GSoC 2016 | Vert.x | Community Bonding Period

2016-05-21T16:00:00.000Z

In this blog I’ll summarize the four weeks during the Community Bonding Period of GSoC 2016 :)

Week 1

The stuff of the first week:

Make contact with my mentors
Set up development environment
Do more research on my project
Read more documents and articles about Vert.x

Week 2

Discuss with mentors to refine my summer timeline and prepare a good project plan
Translate the document of Vert.x - From zero to (micro)-hero Microservice Lab into Chinese(a wonderful lab!)
Learn more about various components of Vert.x
Work on Vert.x Blueprint - Todo Backend, use multiple persistence

Week 3

Objectives

Refine project plan
Work on the English document of Vert.x Blueprint - Todo Backend
Refine the code and document of Vert.x Blueprint - Todo Backend
Learn more about EventBus and Vert.x Service Proxy
Design the structure of Vert.x Blueprint - Kue
Initial structure of Vert.x Blueprint - Kue

Issues

I viewed the code of Kue (Node.js version) and find it weird. Maybe I could write a Kue class referring to the encapsulation of KueVerticle. Every components of the Kue could use EventBus to interact with each other.

Week 4

Objectives

In this week, I’m planning to do the following stuff:

Finish Chinese version document of Todo Backend
Work on the Vert.x Blueprint - Kue (Initial implementation)
Learn from Kue’s design

Issues

Solved: How to use Vert.x Codegen in Gradle (for the lack of document)

Achievement

Learned something about Kue’s design
Refactored some of Vert.x Blueprint - Todo Backend code
- Use @DataObject annotation to generate converter
- Use Docker Compose to link two containers
- Use Vert.x Launcher and read configure from json file
- Resolve the id issue using incremental id (like auto_incr in db)
- Remove redundant code
- Update some part of documents

The Community Bonding Period is ending and the code period will be coming~ Exciting! Wish an amazing code summer with Vert.x!

Working on GSoC 2016 with Vert.x Community

2016-04-28T16:00:00.000Z

Exciting! I have been accepted for Google Summer of Code 2016 with Vert.x Community. My project is Vert.x Blueprint. Looking forward to an amazing summer with Vert.x~~

About my project

Vert.x is an excellent full-stack toolkit which can help us build FRP-sytle, high concurrent, scalable applications. And I hope more developers could make full use of Vert.x to build their applications rapidly and concisely. So it is necessary to implement a series of applications using various part of the Vert.x toolkit to demonstrate the ability to build many kinds of applications.

My goal is to implement some different applications using Vert.x and write detailed documents (both in English and Simplified Chinese) as guideline for the newcomer to Vert.x in order to help them understand the foundations of a vert.x application rapidly and then build complex applications. If time is permitted, I am willing to contribute to Vert.x Microservice Tools or Vert.x Scala Version~

My new blog in English

2016-04-25T16:00:00.000Z

A blog in English independently is very necessary!

This time I chose a concise and beautiful theme, Apollo. I love this theme~~

Tip Test

Note: This tip is only for font test!
仅供测试字体使用。