You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Super fast and lightweight DB migration & evolution tool written in go.
4
4
@@ -8,48 +8,83 @@ migrator can run as a HTTP REST service. Further, there is a ready-to-go migrato
8
8
9
9
# Usage
10
10
11
-
Important: Migrator since its inception supported both CLI and REST API. However, CLI is deprecated as of v2.2 and will be removed in migrator v3.0. Starting v3.0 only REST API will be supported.
11
+
migrator exposes a simple REST API which you can use to invoke different actions:
12
12
13
-
Short and sweet.
13
+
* GET /config - returns migrator config, response is `Content-Type: application/x-yaml`
14
+
* GET /diskMigrations - returns disk migrations, response is `Content-Type: application/json`
15
+
* GET /tenants - returns tenants, response is `Content-Type: application/json`
16
+
* POST /tenants - adds new tenant, name parameter is passed as JSON, returns applied migrations, response is `Content-Type: application/json`
17
+
* GET /migrations - returns all applied migrations
18
+
* POST /migrations - applies migrations, no parameters required, returns applied migrations, response is `Content-Type: application/json`
19
+
20
+
Some curl examples to get you started:
14
21
15
22
```
16
-
$ Usage of ./migrator:
17
-
-action string
18
-
when run in tool mode, action to execute, valid actions are: ["apply" "addTenant" "config" "diskMigrations" "dbTenants" "dbMigrations"] (default "apply")
19
-
-configFile string
20
-
path to migrator configuration yaml file (default "migrator.yaml")
21
-
-mode string
22
-
migrator mode to run: ["tool" "server"] (default "tool")
23
-
-tenant string
24
-
when run in tool mode and action set to "addTenant", specifies new tenant name
Port is configurable in `migrator.yaml` and defaults to 8080. Should you need HTTPS capabilities I encourage you to use nginx/apache/haproxy for TLS offloading.
32
+
33
+
There is an official docker image available on docker hub. migrator docker image is ultra lightweight and has a size of 15MB. Ideal for micro-services deployments!
34
+
35
+
To find out more about migrator docker container see [DOCKER.md](DOCKER.md) for more details.
36
+
37
+
# Configuration
38
+
39
+
migrator requires a simple `migrator.yaml` file:
28
40
29
41
```yaml
42
+
# required, base directory where all migrations are stored, see singleSchemas and tenantSchemas below
30
43
baseDir: test/migrations
44
+
# required, SQL go driver implementation used, see section "Supported databases"
31
45
driver: postgres
32
-
# dataSource format is specific to DB go driver implementation - see below 'Supported databases'
46
+
#required, dataSource format is specific to SQL go driver implementation used, see section "Supported databases"
# should you need more control over HTTP headers use below
71
+
webHookHeaders:
72
+
- "Authorization: Basic QWxhZGRpbjpPcGVuU2VzYW1l"
73
+
- "Content-Type: application/json"
74
+
- "X-CustomHeader: value1,value2"
75
+
```
76
+
77
+
migrator supports env variables substitution in config file. All patterns matching `${NAME}` will look for env variable `NAME`. Below are some common use cases:
Migrator will scan all directories under `baseDir` directory. Migrations listed under `singleSchemas` directories will be applied once. Migrations listed under `tenantSchemas` directories will be applied for all tenants fetched using `tenantSelectSQL`.
85
+
# migrator under the hood
86
+
87
+
migrator scans all directories under `baseDir` directory. Migrations listed under `singleSchemas` directories will be applied once. Migrations listed under `tenantSchemas` directories will be applied for all tenants fetched using `tenantSelectSQL`.
53
88
54
89
SQL migrations in both `singleSchemas` and `tenantsSchemas` can use `{schema}` placeholder which will be automatically replaced by migrator with a current schema. For example:
55
90
@@ -59,34 +94,6 @@ create table if not exists {schema}.modules ( k int, v text );
59
94
insert into {schema}.modules values ( 123, '123' );
60
95
```
61
96
62
-
# Server mode
63
-
64
-
When migrator is run with `-mode server` it starts a HTTP service and exposes simple REST API which you can use to invoke migrator actions remotely.
65
-
66
-
All actions which you can invoke from command line can be invoked via REST API:
67
-
68
-
* GET / - returns migrator config, response is `Content-Type: application/x-yaml`
69
-
* GET /diskMigrations - returns disk migrations, response is `Content-Type: application/json`
70
-
* GET /tenants - returns tenants, response is `Content-Type: application/json`
71
-
* POST /tenants - adds new tenant, name parameter is passed as JSON, returns applied migrations, response is `Content-Type: application/json`
72
-
* GET /migrations - returns all applied migrations
73
-
* POST /migrations - applies migrations, no parameters required, returns applied migrations, response is `Content-Type: application/json`
74
-
75
-
Some curl examples to get you started:
76
-
77
-
```
78
-
curl http://localhost:8080/
79
-
curl http://localhost:8080/diskMigrations
80
-
curl http://localhost:8080/tenants
81
-
curl http://localhost:8080/migrations
82
-
curl -X POST http://localhost:8080/migrations
83
-
curl -X POST -H "Content-Type: application/json" -d '{"name": "new_tenant"}' http://localhost:8080/tenants
84
-
```
85
-
86
-
Port is configurable in `migrator.yaml` and defaults to 8080. Should you need HTTPS capabilities I encourage you to use nginx/apache/haproxy for TLS offloading.
@@ -112,13 +119,84 @@ Currently migrator supports the following databases and their flavours:
112
119
* Microsoft SQL Server 2017 - a relational database management system developed by Microsoft, driver used: https://github.com/denisenkom/go-mssqldb
113
120
* Microsoft SQL Server - original Microsoft SQL Server
114
121
115
-
#Do you speak docker?
122
+
#Quick Start Guide
116
123
117
-
Yes, there is an official docker image available on docker hub.
124
+
You can apply your first migrations with migrator in literally a couple of minutes. There are some test migrations which are placed in `test/migrations` directory as well as some docker scripts for setting up test databases.
118
125
119
-
migrator docker image is ultra lightweight and has a size of approx. 15MB. Ideal for micro-services deployments!
126
+
Let's start.
120
127
121
-
To find out more about migrator docker container see [DOCKER.md](DOCKER.md) for more details.
128
+
## 1. Get the migrator project
129
+
130
+
For building migrator from source code `go get` is required:
131
+
132
+
```
133
+
go get -d -v github.com/lukaszbudnik/migrator
134
+
cd $GOPATH/src/github.com/lukaszbudnik/migrator
135
+
```
136
+
137
+
migrator supports the following Go versions: 1.8, 1.9, 1.10, and 1.11 (all built on Travis).
138
+
139
+
For running migrator on docker Go is not required and `git clone` is enough:
Script will start container called `migrator-postgres`.
155
+
156
+
Further, apart of starting test DB container, the script also generates a ready-to-use test config file. We will use it too.
157
+
158
+
## 3. Build and run migrator
159
+
160
+
When building & running migrator from source code execute:
161
+
162
+
```
163
+
./setup.sh
164
+
go build
165
+
./migrator -configFile test/migrator.yaml
166
+
```
167
+
168
+
> Note: There are 2 git variables injected into the production build (branch/tag and commit sha). When migrator is built like above it prints empty branch/tag and commit sha. This is OK for local development. If you want to inject proper values take a look at `Dockerfile` for details.
169
+
170
+
When running migrator from docker we need to update `migrator.yaml` (generated in step 2) as well as provide a link to `migrator-postgres` container:
171
+
172
+
```
173
+
sed -i "s/host=[^ ]* port=[^ ]*/host=migrator-postgres port=5432/g" test/migrator.yaml
174
+
sed -i "s/baseDir: .*/baseDir: \/data\/migrations/g" test/migrator.yaml
The Ruby framework has an undesired functionality of making a DB call each time to check if given migration was already applied. Migrator fetches all applied migrations at once and compares them in memory. This is the primary reason why migrator is so much better in the second test.
234
+
The Ruby framework has an undesired functionality of making a DB call each time to check if given migration was already applied. migrator fetches all applied migrations at once and compares them in memory. This is the primary reason why migrator is so much better in the second test.
157
235
158
236
flyway results are... very surprising. I was so shocked that I had to re-run flyway as well as all other tests. Yes, flyway is 15 times slower than migrator in the first test. In the second test flyway was faster than Ruby. Still a couple orders of magnitude slower than migrator.
159
237
160
238
The other thing to consider is the fact that migrator is written in go which is known to be much faster than Ruby and Java.
161
239
162
-
# Installation and supported Go versions
163
-
164
-
To install migrator use:
165
-
166
-
```
167
-
go get github.com/lukaszbudnik/migrator
168
-
cd migrator
169
-
./setup.sh
170
-
```
171
-
172
-
Migrator supports the following Go versions: 1.8, 1.9, 1.10, and 1.11 (all built on Travis).
173
-
174
240
# Contributing, code style, running unit & integration tests
0 commit comments