Sometimes we will need to use a service like Cloud Bigtable to store a bunch of data for our application. It sounds like an awesome option, with many built-in capabilities and incredible scalability. All sounds great, just one tiny question, how do I test my application?
In this tutorial, we will learn about some tools very useful to test the parts of our application that connect to Bigtable.
We will use a Bigtable emulator provided by Cloud SDK. Our application will connect to this emulator during testing, instead of the real Cloud Bigtable, this way we will not populate the real tables with test data.
It is important to note that the emulator stores data only in memory, which means that the data saved to the emulator will not persist across runs.
You can use the emulator with all Cloud Bigtable client libraries except for the PHP client library.
The emulator does not provide administrative APIs to create or manage instances and clusters. After the emulator starts up, you can connect to it using any project and instance name to create tables and read or write data.
Before we start, you need to:
- Install Cloud SDK
- Configure gcloud credentials
- After Cloud SDK is installed, install the cbt tool as a Cloud SDK component:
gcloud components update beta gcloud components install beta cbt
Running the emulator locally
Now, the next thing we will do is run the emulator locally, to get a sense of how it works before incorporating it into our project.
To run the emulator with default host (localhost) and port (8086), run:
gcloud beta emulators bigtable start
If you do not have the bigtable emulator installed, it will prompt you to install it, just run the command again when you are done.
And to set specific host and port (which we will, once we incorporate it to our application), run:
gcloud beta emulators bigtable start --host-port=[HOST]:[PORT]
Connect to the emulator locally
Next, we will want to connect to the emulator locally. If the emulator is not running, we can use the
cbt tool to see what the real bigtable has in it, but how do we get
cbt to connect to the emulator instead?
The answer is very simple, just set the correct environment variable and you are ready to go:
Remember the host and port you set when you run the emulator when setting the environment variable, in case that you used the default configuration, HOST will be
localhost and PORT will be
cbt will automatically know that you want it to connect to the emulator, instead of the real Cloud Bigtable. It is important to remember that while this variable is set,
cbt will connect to the emulator, that is why you need to unset it when you are done, you can do this with the following command:
Now that the emulator is running and
cbt will send the commands to it, we can execute some sample commands. To create a new table:
cbt -project foo -instance bar createtable foobar
You can indicate the project (in the emulator can be anything, I recommend using a random name, so you know you are connected to the emulator) with the
-project flag, in this case, our project is named
foo. To indicate the instance, use the
-instance flag, in this case, our instance’s name is
bar. Finally, we are indicating that we want to create a new table in the instance inside our project using the
createtable keyword, the name of our new table, in this case, will be
When you run the command above, you will get something like this:
2017/06/14 16:24:15 -creds flag unset, will use gcloud credential
Another command we can run is:
cbt -project foo -instance bar ls
This command will show us the tables we have in the project and instance indicated, in this case, we will get:
2017/06/14 16:24:17 -creds flag unset, will use gcloud credential foobar
foobar, the table previously created. To consult more commands you can run using the
cbt tool, visit this site.
Incorporating the Bigtable emulator to your project
In this section of the tutorial, we will talk about a way we can incorporate the Bigtable emulator to our project.
First of all, let’s talk about some considerations. It would be easy to assume that if we have the emulator running locally and the environment variable set, this would be enough for our application to connect to the emulator during testing, and in some cases, that assumption is fair and right.
If you are using a client library to connect to Cloud Bigtable provided by Google (or by the community with a fair amount of support) and you are running the application locally, then it is more than fair to assume that your application will, in fact, connect to the emulator with just this local configurations. But this is not a very portable model, it requires some configuration and manual steps before you can run the tests that will connect to the emulator.
If you wrote (or are still writing) a client library, then you have to make sure that your library has a way to recognize the environment variable is set and connect to the emulator.
The following steps are to configure the Bigtable emulator in your project using docker-compose:
The first step is to create a container that runs the emulator, for this, we can use the google/cloud-sdk docker container from Dockerhub. Add this to your
services: ... bigtable: command: gcloud beta emulators bigtable start --host-port=0.0.0.0:8086 image: google/cloud-sdk:latest volumes: - .:/opt/folder_name/my_application:delegated working_dir: /opt/folder_name/my_application environment: - BIGTABLE_EMULATOR_HOST=bigtable:8086 ports: - 8086:8086
As we can see, a command to start the emulator is executed when the container is started, and the ports we defined are the default mentioned earlier (8086). Now when we run:
docker-compose up bigtable
We will have a container with the emulator running, great!
Then, we can add a service that runs our application when we want to run tests, and add our
bigtable container as a service our application depends on:
services: my_application_tests: depends_on: - bigtable volumes: - .:/opt/folder_name/my_application:delegated - ~/.config/gcloud:/root/.config/gcloud working_dir: /opt/folder_name/my_application environment: - BIGTABLE_EMULATOR_HOST=bigtable:8086 ...[other configurations needed]
In the example above, we are not adding all the other configurations needed to run our application in a container, because that depends on each application, so, when creating a container that runs your application for testing, do not forget to add all the configurations needed.
Now, when we launch our application container for testing, the bigtable container will also be started:
docker-compose up my_application_tests
The command above will start both the application’s container and the bigtable container, we will not nned to run
docker-compose up bigtable to have the emulator running.
Now we have two containers running, one with our application and one with the Bigtable emulator, the next thing we want to do is run the tests, now that the environment variable is set in both
my_application_tests containers, we do not need to worry to set it anywhere else. We will use a bash file to start the containers and run the tests.
I also recommend having a bash file that specifies the specific commands needed to run the tests in our application (we will not include an example of this file, because it is highly reliant of the language your application is written in).
Following is an example of a bash file that starts the containers and run the tests (by calling another bash file that contains the exact commands needed to start the tests execution):
# run_test_bash echo "=> Starting containers" docker-compose up -d my_application_tests echo "=> run tests for our application" docker-compose exec my_application_tests ./test_bash
In this bash, we are assuming a bash file that specifies commands to run tests already exists, we are also assuming that it is in the same folder that this other bash file (
run_test_bash) is and we called it
Let’s run our tests by running the bash above (we are calling it
run_test_bash in here) and wait for the results:
Assuming the bash
run_test_bash is in the folder we are currently in when running these commands.
And we are done.