|
|
Locust is an open source load testing tool that uses Python to write the tests.
Because it uses Python to define the tests, it is more friendly for developers and inherited all potentiality of being driven by a language such as Python.
Even though it does not provide out of the box the ability to define and use KPIs, it has hooks and extension mechanisms that will allow an implementation to support this need.
For this example, we will use Locust to define a series of Performance tests and using the extension capacities of Locust, we will define two extensions to:
We will need:
|
We will start to define a simple load test in Locust that will target a demo site (travel agency) supplied by BlazeMeter that you can find here.
The test will exercise 3 different endpoints:
To start using Locust please follow the documentation.
The test will be simple as we have defined above, so let's look in more detail on what are each part is exactly doing
class FlightSearchTest(TaskSet): @task def open_login_page(self): self.client.get("/login") @task def find_flight_between_Paris_and_Buenos_Aires(self):self.client.post("/reserve.php", { 'fromPort': 'Paris', 'toPort': 'Buenos+Aires' }) @task def purchase_flight_between_Paris_and_Buenos_Aires(self):self.client.post("/purchase.php", { 'fromPort': 'Paris', 'toPort': 'Buenos+Aires', 'airline': 'Virgin+America','flight': 43, 'price': 472.56 }) class MyLocust(HttpUser): tasks = [FlightSearchTest] host = "http://blazedemo.com" |
The file will have two classes, one to define the requests we want to do and another to specify the endpoint and the tasks we want to execute.
The first Class we see is FlightSearchSimulator that is of the type TaskSet this allows you to mimic your site structure and organize the tests the same way your site is organized (in this case we are using to demonstrate the capacity because as the example is simple we will not need it).
Within the class we can see @tasks decorators that will indicate to Locust that these are the tasks to be executed in the test.
The second class, MyLocust, is used to define what are the tasks to be executed, and this is done setting the tasks with a list of tasks to be executed:
tasks = [FlightSearchTest] |
In our case only one: FlightSearchTest, but we could define a list of tasks and even define weights of executions.
Finally the target host that will be used as the target of the tests.
In order to execute the tests you can use two ways, using the UI or the command line, in our case we will use the command line.
In the command line we can define a series of parameters that will define the way Locust will handle the test, such as:
The final command will be f the form:
locust -f LocustScript.py --headless -u 50 -r 1 -t 1m --csv=example |
This command will execute the LocustScript.py in headless mode, will start with one user and grow 1 user each second until reaching the maximum of 50. It will also generate the CSV files and will be executing for 1 minute.
This will be enough to execute performance tests, however a manual validation of results must always be done in the end to assess if the performance is enough of not, and looking at CSV files is not always easy.
We need the ability to:
In order to do that we will extend Locus and add listeners that will interpret the results and evaluate their output and also redirect the logs to Graphana to be viewed.
In order to use performance tests in a pipeline we need those to be able to fail the build if the result is not the expected, for that we need to have the ability to automatically assess if the performance tests were successful (within the parameters we have defined) or not.
For demonstration purposes we will use the following implementation present in GitHub, this will allow us to define KPIs per request, the KPIs being used are:
This enable us to define KPis for each request and fail the build in case they are not followed, to do that in our class we will add the following lines:
def __init__(self, environment): super().__init__(environment) @events.init.add_listener def my_locust_init(environment, **_kwargs): KPI_SETTINGS = [{'/login': [('percentile_90', 5), ('rps', 500), ('error_rate', 0)]}, {'/reserve.php': [('percentile_90', 5), ('rps', 500), ('error_rate', 0)]}, {'/purchase.php': [('percentile_90', 5), ('rps', 500), ('error_rate', 0)]}] KpiPlugin(env=environment, kpis=KPI_SETTINGS) |
For each one of the endpoints we have defined 3 KPIs (but we could have defined only one or two), in each KPI we define a value that if exceeded will cause an error.
In this example particularly if:
Once an error is generated we can react and fail the build based on these results.
In the output these results will look like this:
Now we are executing Tests to validate the performance of our application and we are capable of defining KPIs to validate each performance indicator in a build (enable us to add these Tests to CI/CD tools given that the execution time is not long), so what we need is to be able to ship these results to Xray to bring visibility over these types of Tests also.
We have seen that Locust produces CSV files of the requests and errors (valuable to perform a post analysis but hard to convert into proper pass or fail result) but we need to produce a result that will hold all the information produced and will bring value to our project, to do so, we are going to create an Xray Json report to hold these results.
First let us explain the approach we are taking towards these performance Tests, in Xray we have defined 3 Tests:
Each one of them will gather the results of those indicators in our application and failure or success will depend on all the endpoints for each indicator we are considering for our application.
If any of those fail we can check the details and understand what particular endpoint have failed, but overall what is important is that the overall result will dictate if the it was successful or not.
The last change will done in the "kpi_listener.py" where we are going to add logic to create the Xray Json file, append the CSV to the file and write the output to a file, for that we have created new methods that will handle that logic:
... @staticmethod def injectCSVFile(fileName): with open(fileName, 'rb') as open_file: byte_content = open_file.read() return b64encode(byte_content).decode('utf-8') def appendToXrayResult(self, testkey, metric, name, value, comment, status): done = False if self.data['tests']: for tests in self.data['tests']: for key, value in tests.items(): if key == 'testKey' and value == testkey: tests['results'].append({ 'name': metric + ' for ' + name, 'log': comment, 'status': status }) done = True if not done: info = { 'info': { 'summary': ' Perf test', 'description': 'Perf test', 'project': 'COM', }, } self.data['tests'].append({ 'testKey': testkey, 'comment': metric, 'status': status, 'results': [ { 'name': metric + ' for ' + name, 'log': comment, 'duration': 5, 'status': status } ], 'evidences': [ { 'data': self.injectCSVFile('example_exceptions.csv'), 'filename': 'performanceexceptions.csv', 'contentType': 'text/csv' }, { 'data': self.injectCSVFile('example_failures.csv'), 'filename': 'performancefailures.csv', 'contentType': 'text/csv' }, { 'data': self.injectCSVFile('example_stats_history.csv'), 'filename': 'performancstatshistory.csv', 'contentType': 'text/csv' }, { 'data': self.injectCSVFile('example_stats.csv'), 'filename': 'performancstats.csv', 'contentType': 'text/csv' } ] }) info.update(self.data) self.data = info def writeToXrayResultFile(self): with open('xrayResults.json', 'w') as outfile: json.dump(self.data, outfile) |
Let's look to each addition in more detail:
Now when we execute the tests we will have the CSV files produced by Locust and one extra file named "xrayResults.json" that will have all relevant information in a way that Xray will ingest this file and create the corresponding Test Execution with details.
On example of the Xray Json file is:
{"info": {"summary": " Perf test", "description": "Perf test", "project": "COM"}, "tests": [{"testKey": "COM-160", "comment": "percentile_90", "status": "FAILED", "results": [{"name": "percentile_90 for /login", "log": "percentile_90 for '/login' is 720.0, but expected it to be better than 5", "duration": 5, "status": "FAILED"}, {"name": "percentile_90 for /reserve.php", "log": "percentile_90 for '/reserve.php' is 600.0, but expected it to be better than 5", "status": "FAILED"}, {"name": "percentile_90 for /purchase.php", "log": "percentile_90 for '/purchase.php' is 630.0, but expected it to be better than 5", "status": "FAILED"}], "evidences": [{"data": "Q291bnQsTWVzc2FnZSx...UcmFjZWJhY2ssTm9kZXMNCg==", "filename": "performanceexceptions.csv", "contentType": "text/csv"}, {"data": "TWV0aG9kLE5hbWUsRX...Jyb3IsT2NjdXJyZW5jZXMNCg==", "filename": "performancefailures.csv", "contentType": "text/csv"}, {"data": "VGltZXN0YW1wLFVzZ...yMzExDQo=", "filename": "performancstatshistory.csv", "contentType": "text/csv"}, {"data": "VHlwZSxOYW1lLF...JlcXVlc3QgQ291b==", "filename": "performancstats.csv", "contentType": "text/csv"}]}, {"testKey": "COM-159", "comment": "rps", "status": "FAILED", "results": [{"name": "rps for /login", "log": "rps for '/login' is 18.366666666666667, but expected it to be better than 500", "duration": 5, "status": "FAILED"}, {"name": "rps for /reserve.php", "log": "rps for '/reserve.php' is 19.559322033898304, but expected it to be better than 500", "status": "FAILED"}, {"name": "rps for /purchase.php", "log": "rps for '/purchase.php' is 19.559322033898304, but expected it to be better than 500", "status": "FAILED"}], "evidences": [{"data": "Q291bnQsTWVzc2FnZSx...UcmFjZWJhY2ssTm9kZXMNCg==", "filename": "performanceexceptions.csv", "contentType": "text/csv"}, {"data": "TWV0aG9kLE5hbWUsRXJyb3IsT...2NjdXJyZW5jZXMNCg==", "filename": "performancefailures.csv", "contentType": "text/csv"}, {"data": "VGltZXN0YW1wLFVzZXIgQ291...zExDQo=", "filename": "performancstatshistory.csv", "contentType": "text/csv"}, {"data": "VHlw...IwMDANCg==", "filename": "performancstats.csv", "contentType": "text/csv"}]}]} |
This is just an example of one possible integration, you can reuse it or come up with one that better suites your needs.
Most of the time the reports and logs generated by performance tests are not human friendly, either by the number of results or because they are hard to read, even during the test execution it is hard to understand if it is going as expected or not.
In order to provide a way to visualize those results in Graphana (known as the open observability platform) we have extended Locust and registered a listener that will ship the logs to Graphite (time series database).
We have followed the BlazeMeter tutorial and, as described there, we use a Docker image to run an instance of Graphite and Graphana where we can validate our local performance tests, for that use the image present here:
git clone https://github.com/kamon-io/docker-grafana-graphite.git |
Once you have extracted the contents, enter the directory and run:
make up |
This command will start and Docker instance of Graphana in localhost:80 and of Graphite in localhost:81
We have extracted the code in a class in order to be easily used in your tests, the python script that have resulted from that extraction is below
class GraphanaPlugin(): sock = None request_success_stats = [list()] request_fail_stats = [list()] def __init__( self, env: locust.env.Environment, ): self.sock = socket.socket() self.sock.connect( ("localhost", 2003) ) self.env = env self.errors = [] events = self.env.events events.request_success.add_listener(self.hook_request_success) events.request_failure.add_listener(self.hook_request_fail) atexit.register(self.exit_handler) events.quitting.add_listener(self.exit_handler) def hook_request_success(self, name, response_time, **_kwargs): message = "%s %d %d\n" % ("performance." + name.replace('.', '-'), response_time, time.time()) self.sock.send(message.encode()) def hook_request_fail(self, request_type, name, response_time, exception, **_kwargs): self.request_fail_stats.append([name, request_type, response_time, exception]) def exit_handler(self, environment): self.sock.shutdown(socket.SHUT_RDWR) self.sock.close() |
In more detail we can see that we have created a socket connected to the default port of the Graphite tool and two lists with the successful requests and the failed ones, the way we manage to obtain those is by plugin two listeners in Locust:
In order to use this new class in our Locust tests we must add these lines to our test script
@events.init.add_listener def graphana_init(environment, **_kwargs): GraphanaPlugin(env=environment) |
Finally we must add the following template in Graphana to define the dashboard that will enable us to see the execution details, BlazeMeter blog also refers to this template and made it available here. Once it is imported in Graphana it will create a new dashboard.
With all of this in place we can execute the tests again and check in Graphana (and in Graphite) all the requests being made:
Notes:
As we saw in the above example, where we are producing Xray Json reports with the result of the tests, it is now a matter of importing those results to your Jira instance, this can be done by simply submitting automation results to Xray through the REST API, by using one of the available CI/CD plugins (e.g. for Jenkins) or using the Jira interface to do so.
In this case we will show how to import via the API.
APIOnce you have the report file available you can upload it to Xray through a request to the REST API endpoint for Xray Json. Xray Json resultsThe API request we will use, as we have generated an Xray Json report, will be the one to import Xray Json results.
With this command we are creating a new Test Execution that will have the results of the Tests that were executed. Once uploaded the Test Execution will look like the example below With Title and Description we have defined in the code and linked to the Tests we have created beforehand to hold Performance results. In order to check the details we click on the details icon next to each Test (below the red arrow in the screenshot), this will take us to the Test Execution Details Screen In the details we have the following relevant information:
Bringing the information of performance tests to your project will allow a complete view over the Testing process and bring that visibility up front for the team to have all the elements necessary to deliver a quality product. |
.toc-btf { position: fixed; } |