CKAD Certified

February 23, 2020Tech Notes 2 minsComments Off on CKAD Certified

Early this week, I took the 2nd try of the CKAD exam and passed it. I scored 89% this time despite the situation where I ran out of time and couldn’t finish the last question completely. Now I am CKAD certified!

I had my first try of the exam in Aug last year without many preparations. As CNCF gives you a free 2nd chance if the 1st one is not successful, I wanted to take the 1st exam as an opportunity to test my Kubernetes knowledge, to get familiar with the test environment, and to sense how difficult it is. I scored 64% of the 1st exam which is 2% short of the passing score.

Late last year I was busy on my work and until recently I got some time to properly prepare for and take the exam again. I spent about 2 weeks to polish my skills with kubectl and other tools, and used the resources in the following two github repos heavily.

Here are several my tips regarding the exam which I hope could help those who are preparing for the certificate.

  • The only tools that you can use in the exam environment are kubectl, vim and tmux. So be very familiar with them.
  • In the exam, you are allowed to open another browser tab to connect to https://kubernetes.io/docs, but you may not have enough time to read the docs in detail. I relied on kubectl explain more than checking the docs.
  • As the exam environment runs in Chrome, a big screen definitely helps.
  • Most importantly, a lot of practices. The history showed that I tapped k/kubectl for 1372 times and vim for 429 times in 2 weeks before the exam.

美式意餐

January 14, 2020狮城攻略 3 minsComments Off on 美式意餐

虽然是星期一，晚饭时间林肯中心二楼的鼎泰丰还是有许多人排队，位子难等。这家鼎泰丰开了有10年了吧，印象中我就只在非用餐高峰时间，比如下午三四点钟，在这里吃过东西。从没在中午或晚上的用餐时段，成功等到过位子。今天也不例外。于是我转头去了他家楼下的Maggiano’s Little Italy。

林肯中心的Maggiano’s Little Italy

这是一家美式意大利餐厅，算是Bellevue这里的老字号了。我第一次在这家餐厅吃饭，还是在2011年，也是因为没排到鼎泰丰的位子。那时我在Redmond培训，也是冬天，培训内容繁多还有考试挺累。那时Bellevue的鼎泰丰也才刚开业。好不容易到了周末，想说去鼎泰丰吃顿像样的中餐，可是谁知他家的队伍排满了走廊。我实在没时间排队，于是就发现了Maggiano’s Little Italy。

所谓美式意大利餐，一大特点就是，分量很大。就比如Maggiago’s装pasta的盘子很大，一盘pasta的分量，在意大利怕是要装两盘。这导致的后果就是，一个人去吃的话，没办法多点几样尝尝，一份主菜就吃饱了。

我一直以为Maggiano’s是Bellevue本地的一家餐馆，刚上他们的网站看了一下才知道，原来是它是一间从芝加哥开始的连锁餐厅。只是在华州似乎只有Bellevue这一间店而已。看他的网站介绍，故事平平无奇，连创始人的名字都没有，不免有一些小失望。

Configuring VNET integration of Azure Redis and Web App

March 9, 2019Tech Notes 2 minsComments Off on Configuring VNET integration of Azure Redis and Web App

To configure Azure Redis with the VNET support, we can follow the steps described in this document. And to integrate Azure web app with a VNET, there is a detailed document for it as well. In this post, I listed some of the common issues that one might hit during the configuration.

  1. The VNET integration for Azure Redis requires an empty subnet for the VNET that is created with Resource Manager. This subnet needs to be created before you create Azure Redis. Otherwise, the configuration would fail.
  2. The subnet for Azure Redis can be protected with a network security group (NSG). Usually the default NSG rules are good enough for protecting the connections. If you need further hardening, you will have to create rules based on the ports list in the Azure Redis document.
  3. To troubleshoot the connection between Azure web app and Azure Redis, you can use the Kudu of web app. There are two tools built in with the web app for network troubleshooting:
    nameresolver.exe can be used to test the DNS functionalities, and
    tcpping.exe     can be used to test if the host and port can be pinged. But you cannot test the Redis function directly from the Kudu.
  4. Once the VNET integration is configured, the Redis console in Azure Portal will not work anymore. To test the Redis functions with tools such as redis-cli, you will have to build a VM in the VNET and connect to Azure Redis from it.
  5. If somehow your web app cannot access the Azure Redis, although the network configurations are correct, you can try to sync the network for App Service Plan. See this issue for details. Make sure you don’t hit any error when syncing the network.

Move WordPress sites to a new domain

February 20, 2019Tech Notes 2 minsComments Off on Move WordPress sites to a new domain

Today I finally decided to move this site from its old domain to this new one, chunliu.me, which I got last year. For a personal hosted WordPress site, changing domain name is not a straight forward task, especially when you don’t want to break the existing external links that point to the site. Here is how I did it, with a lot of search on internet of course.

Copy the database and WordPress folder

To minimize the impact to the existing site, I duplicated the MySQL database of this site and its WordPress folder. Copy WordPress folder is easy, simply use cp command to copy the whole folder. Copy database is a bit tricky because I forgot the password of the MySQL root user. I used the following way to reset the password of the root user.

  1. Run mysql with --skip-grant-tables.
    $ sudo service mysql stop
$ sudo mkdir -p /var/run/mysqld
$ sudo chown mysql:mysql /var/run/mysqld
$ sudo mysqld --skip-grant-tables --skip-networking &
$ jobs
  2. Update the password of root user with mysql client.
    $ mysql -u root
mysql> FLUSH PRIVILEGES;
mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'MyNewPass';
mysql> QUIT;
  3. Restart mysql service.
    $ sudo pkill mysqld
$ jobs
$ sudo service mysql start

With the above steps, the database can be copied with the following command.

mysqldump -u root --password=<pwd> <original db> | mysql -u root -p <new db>

Update the URL of the new site

With the copied WordPress and database, the URL needs to be updated in the following several places.

  • wp_config.php: update DB_NAME and DOMAIN_CURRENT_SITE;
  • wp_options table: update site_url and home;
  • wp_site and wp_blogs table: update domain;

Create a URL rewrite rule in the old site

The last thing is to create a url rewrite rule in the old site so that all requests to the old site can be redirected to the new site. This will help to keep the external links without broken.

Edit the .htaccess file of the old site with the following code.

RewriteEngine On
RewriteCond %{HTTP_HOST} ^olddomain\.com$ [OR]
RewriteCond %{HTTP_HOST} ^blog\.olddomain\.com$
RewriteRule (.*)$ https://newdomain/$1 [R=301,L]

With that, the site is moved to the new domain successfully.

Build Hexo site with Azure Pipelines

January 17, 2019Tech Notes3 minsComments Off on Build Hexo site with Azure Pipelines

Azure Pipelines has the very good integration with GitHub. It provides 10 free parallel jobs and unlimited minutes for open source projects. It is a very good choice for open source projects to have the CI/CD capability.

As I have configured the site to use hexo’s git deployment, an Azure build pipeline is enough to carry out the deployment. And in the build pipeline, I need the following three tasks.

  • Install hexo-cli.
  • Install all npm packages used by the site with npm install.
  • Configure git credential and run hexo deploy.

The first 2 steps are quite straight forward. The tricky part is how to use script to configure git credentials. git credential store only accept inputs from stdin. I did some search and tests, and ended up with a bash pipe to feed in the credentials. The following is a sample for credential store. 

git config --global user.email "<git email address>"
git config --global user.name "<git user name>"   
git config --global credential.helper 'store --file ~/.my-credentials'
printf "protocol=https\nhost=github.com\nusername=<username>\npassword=%s\n\n" "$GHP" | git credential-store --file ~/.my-credentials store
./node_modules/.bin/hexo clean && ./node_modules/.bin/hexo deploy

As credential store will save the credential to a file on disk, it is not a good practice for a shared build agent. I ended up to replace the credential store with credential cache, and exit the cache when deployment is done.

The following is the yaml file of the build definition. $GHP is an environment variable storing the personal token of GitHub. The actual value is saved in Azure DevOps encrypted. 

resources:
- repo: self
queue:
  name: Hosted Ubuntu 1604
  demands: npm

steps:
- script: |
   echo "install hexo-cli"   
   npm install hexo-cli

  displayName: 'Install hexo-cli'

- task: Npm@1
  displayName: 'npm install'
  inputs:
    workingDir: /

    verbose: false

- script: |
   git config --global user.email "<git email address>"   
   git config --global user.name "<git user name>"   
   git config --global credential.helper cache   
   printf "protocol=https\nhost=github.com\nusername=<username>\npassword=%s\n\n" "$GHP" | git credential-cache store   
   ./node_modules/.bin/hexo clean && ./node_modules/.bin/hexo deploy   
   git credential-cache exit
  displayName: 'hexo deploy'
  env:
    GHP: $(password)

Now when I update the site, I just need to commit the changes and push it to GitHub. Azure Pipeline will take care of the build and deployment automatically.

Spring Boot, Azure Database for MySQL, and Azure App Service – Part 1

October 30, 2018Tech Notes , , 3 minsComments Off on Spring Boot, Azure Database for MySQL, and Azure App Service – Part 1

I recently played with Java and Azure App Service. What I was trying to find out is how the development experience would look like for Java developers if they want to build their applications with Azure App Service and Azure Database for MySQL.

There are some documents on Microsoft doc site, such as this one. It might be good enough for an experienced Java developer, but for someone like me who has limit Java experience, it is not easy to follow, and the sample is also too simple to make any sense for a real development. So I decided to try it myself and documented my experience here for others to reference. There would be a series of posts, and this is the first one. 

Prepare the dev environment

So instead of installing IntelliJ or Eclipse, I choose to use VSCode as my Java IDE. On my computer I’ve already had the VSCode installed. According to this tutorial, I just need to install JDK and Maven. I am a bit lost with the Java terms like Java SE, JDK, JRE and their versions, but I don’t want to be bothered. I choose to install OpenJDK because Oracle JDK requires a license. So here are steps to install OpenJDK. 

  1. Download OpenJDK from here. Windows version of OpenJDK is a zip file. Unzip it to C:\Program Files\Java so the root fold of the JDK would be something like C:\Program Files\Java\jdk-11.0.1
  2. Add an environment variable JAVA_HOME, set its value to the root of the JDK, for example, C:\Program Files\Java\jdk-11.0.1
  3. Add C:\Program Files\Java\jdk-11.0.1\bin to the system path. 
  4. With the above steps, OpenJDK is installed completely. To test if it works, open a command window and run java -version. It should print out the OpenJDK version and runtime information. 

When OpenJDK is installed, you can follow the vscode tutorial to download and install maven, and the Java Extension Pack for vscode. 

Create a MySQL database

Instead of installing MySQL on my local computer, I choose to create an Azure Database for MySQL instance as the dev database environment. It is easy to provision an Azure Database for MySQL instance. Azure has quick start for it. I also run the following SQL query to configure the database in Azure Cloud Shell. 

CREATE DATABASE tododb; -- Create a database
CREATE USER 'springuser'@'%' IDENTIFIED BY 'Spring1234'; -- Create a database user
GRANT ALL PRIVILEGES ON tododb.* TO 'springuser'@'%'; -- Grant user permissions to the database
FLUSH PRIVILEGES;

With the above preparation, we have a Java development environment and a MySQL database ready for the development. In the next post, I will start to create a Spring Boot REST API app with VSCode. Stay tuned. 

Upgrade Ubuntu Server From 16.04 to 18.04.1

October 10, 2018Tech Notes 5 minsComments Off on Upgrade Ubuntu Server From 16.04 to 18.04.1

I have received several notifications from my Ubuntu server running in Azure for asking me to upgrade the server to Ubuntu 18.04.1. When Ubuntu 18.04 was first released, I didn’t upgrade the server. I was afraid there could be compatibility issues and I don’t want to break the server. With the release of 18.04.1, it seems the version is stable enough for an upgrade. So I decided to upgrade the server. 

Here is what I did.

First of all, I updated the server with apt update && apt upgrade, and then I backed up my server with Azure VM backup. In case upgrade failed, I can restore the VM back.

Then I ran do-release-upgrade to upgrade the server. The os kernel seemed to upgrade successfully, but the software package upgrade failed with the following output. 

authenticate 'bionic.tar.gz' against 'bionic.tar.gz.gpg' 
extracting 'bionic.tar.gz'

 libpython3.6-stdlib:amd64
 python3.6
 python3-apt
 python3
 python3-cffi-backend
 apt-xapian-index
 python3-xapian
 python3-gi
 mailutils
 python3-markupsafe
 python3-systemd
 python3-gdbm:amd64
 python3-lib2to3
 python-apt
 dh-python
 python3-distutils
 libpython3-stdlib:amd64
 python3-yaml
 python3-pycurl
 python3-dbus

Upgrade complete

The upgrade has completed but there were errors during the upgrade
process.

To continue please press [ENTER]

I did some search on the internet. It seems a common issue. To solve this issue, I ran the command sudo mv /usr/share/dbus-1/system-services/org.freedesktop.systemd1.service /usr/share/dbus-1/system-services/org.freedesktop.systemd1.service.bak as it is mentioned here. 

After the issue was fixed, I just ran sudo apt-get dist-upgrade to upgrade all packages, and I chose to keep all local copies of configurations. After that, the upgrade completed successfully with all software and services running normally.  

CQRS和Event Sourcing模式

July 20, 2018Tech Notes , , 7 minsComments Off on CQRS和Event Sourcing模式

这两天重读了微软Patterns & Practices团队几年前写的CQRS Journey。这本书我几年前就看到过，只是当时我工作的重点不在应用架构上，当时没读下去。这两天重读，对CQRS和Event Sourcing (ES)模式有了新的认识。在云计算和微服务大行其道的当下，这两个模式在应用架构方面仍然是非常有参考价值的。这篇文章算是读书小结吧。

什么是CQRS模式？

CQRS是Command and Query Responsibility Segregation的缩写，中文直译过来，就是命令与查询责任分离的意思。这里涉及到两个定义，何为命令？何为查询？

  • 命令会改变对象的状态，但不返回任何数据。
  • 查询则相反，会返回数据，但并不改变对象的状态。

如果将查询和命令简化理解成对数据的读写操作，CQRS模式的含义就是，应用架构中负责模型读写的模块应当分离。这里的分离，不单是程序代码或逻辑上的分离，也包括数据模型，甚至是数据存储的分离。CQRS Journey的这张图描述了CQRS的一种典型应用。

虽然不是必须的，但CQRS模式通常与Domain Driven Design (DDD)同时使用。CQRS不是一种全局性的架构模式，它只适用于特定的bounded context。对于领域模型十分复杂的场景，CQRS模式可以增强架构的扩展性和灵活性，同时降低模块的复杂度。由于读和写的模型分离，可以分别针对读写操作优化，同时避免的数据锁定，对于性能提升也有帮助。这是CQRS模式带来的好处。

于此同时，CQRS模式也有其局限性。首先，它并不是一种易于实现的模式。因为读写责任的分离，它不如CRUD来的直观。而且读写数据同步和确保数据一致性会是一个问题。比较常见的作法，是通过事件(Event)来将写操作的结果同步到读操作的数据库中。为了确保数据的一致性，常常借用Event Sourcing模式来实现事件的存储和分发。同时由于通过事件实现的数据同步，其实是异步完成的，在分布式系统中，需要考虑数据的最终一致性(Eventual Consistency)。在CQRS模式中，通常写数据具有完全一致性(Full Consistency)，而读数据则具有最终一致性。

这些局限决定了，CQRS模式并非适用于所有场景，或所有的bounded context。它通常只适用于复杂多变，涉及多方操作的场景。而对于业务简单，操作方单一，以及非核心的bounded context，使用CQRS模式可能会增加开销，但并不能带来明显的好处。

什么是Event Sourcing模式？

Event Sourcing (ES)模式是一个关于如何存储domain model状态的模式。这个模式不直接存储模型的状态，而是存储模型状态变化的历史。应用想要获取模型的当前状态时，需要重演整个历史来得到当前状态。一个常用的解释ES模式的场景，是银行账户。

账户余额的直观存储方式，是存储余额本身。当用户存入100元时，余额假定是100，其后用户取出10元，余额变为90，其后用户又存入50元，余额变为140。当用户查询余额时，系统直接获取当前余额。

同样的场景，使用ES模式时，系统不存储余额本身，而是存储用户的行为，即Event。当用户存入100元时，存储“存入100”这个Event，当用户取出10元时，存储“取出10”。当用户查询余额时，系统获得所有的Event，然后进行计算，得到余额。这实际上是金融机构存储交易信息的方式。Bitcoin也是使用同样的方式存储账户的send和receive操作，并且将所有的Event使用blockchain链接起来。

ES模式带来的好处显而易见，比如它能简化写操作，所有Event一旦发生，就变为immutable，写操作就变为简单的添加纪录，避免了复杂的锁定和冲突。同时ES模式保留了所有状态的历史，容易做audit，或纠错。而ES模式的一个问题是，随着历史数据的增加，查询操作的性能可能会降低。

虽然不是必须的，但通常CQRS模式和ES模式会被同时使用。因为通常CQRS模式使用Event来同步读写两端的数据，使用ES模式存储Event有助于这种数据同步，在数据出现不一致的情况时（根据CAP理论，这在分布式系统中是不可避免的），读操作端可以通过replay event的历史，来确保数据的最终一致性。

被删答案2

June 11, 2018杂谈 1 minComments Off on 被删答案2

又一个很久之前发的答案，被举报政治敏感而遭删除。这个答案应该是在2012年左右写的，比英国脱欧早很久。可惜知乎发送的答案备份，居然没有原答案发布的时间，确切的时间不得而知了。

在可见的未来，不可能。
欧洲各国政治制度基本相同，但是现在欧盟也快要维持不下去了。中日韩历史上恩怨情仇就理不清，现在的政治制度有差异极大，还要照顾朝鲜，俄罗斯和美国的感情，根本不存在结盟的基础。对于天朝而言，与日本结盟，岂不是少了一条转移国内愤青爱国热情的对象？

Deploying a Service Fabric cluster to run Windows containers

May 21, 2018Tech Notes 2 minsComments Off on Deploying a Service Fabric cluster to run Windows containers

From container perspective, Service Fabric is a container orchestrator which supports both Windows and Linux containers. In legacy application lift and shift scenarios, we usually containerize the legacy application with minimal code change. And Service Fabric is a good platform to run these containers.

To deploy a Service Fabric cluster on Azure which is suitable for running containers, we can use ARM template. I created a template with the following special settings:

1 – An additional data disk is attached to the VMs in the cluster to host the downloaded container images. We need this disk is because by default all container images would be downloaded to C drive of the VMs. The C drive may run out of space if there are several large images downloaded.

[code lang=”HTML”] "dataDisks": [ { "lun": 0, "createOption": "Empty", "caching": "None", "managedDisk": { "storageAccountType": "Standard_LRS" }, "diskSizeGB": 100 } ] [/code]

2 – A custom script extension is used to run a custom script to format the data disk and change the configuration of dockerd service.

[code lang=”HTML”] { "properties": { "publisher": "Microsoft.Compute", "type": "CustomScriptExtension", "typeHandlerVersion": "1.9", "autoUpgradeMinorVersion": true, "settings": { "fileUris": [ "https://gist.githubusercontent.com/chunliu/8b3c495f7ff0289c19d7d359d9e14f0d/raw/2fdcd207f795756dd94ad7aef4cdb3a97e03d9f8/config-docker.ps1" ], "commandToExecute": "powershell -ExecutionPolicy Unrestricted -File config-docker.ps1" } }, "name": "VMCustomScriptVmExt_vmNodeType0Name" } [/code]

The customer script is as follows: