Master Apache JMeter From load testing to DevOps.

Copyright

Presentation of the authors

1. Antonio Gomes Rodrigues
2. Bruno Demion (Milamber)
3. Philippe Mouawad (Philippe M.)

About the reviewers

1. Felix Schumacher

Preface by Alexander Podelko

1. Alexander Podelko bio

How to read this book

1. I just want to discover Apache JMeter
2. There is nothing and I have to do everything
3. I am joining a performance team
4. I need to do a load test from scratch
5. I need script load test
6. I need to help developers to make load test
7. I need to analyse and presents results from a load test
8. I need to make shift left strategy

Quick start with JMeter

1. Our process
2. Setup JMeter to record our browsing session
3. Configure your browser
4. Save our script
5. Validate our script with only one user
6. Setup our load test and launch it
7. Run our load test and analyze it

JMeter overview

1. JMeter overview
2. Types of load tests supported
3. Supported test protocols
4. Reporting
5. Conclusion

Why is it still important to make load test today?

1. Why performance matters?
2. Better user experience
3. Better SEO (Search Engine Optimization)
4. Better conversion rate
5. Reduce cost
6. What has changed in the last years and why it is not enough?
7. Provisioning a server is easiest with Cloud
8. More powerful server
9. Autoscaling is easy with Cloud/Kubernetes
10. Canary release deployment
11. Blue/Green deployment
12. Feature Flipping allows to quickly disable a faulty feature
13. Design for failure
14. Other reasons why it is still important to make load test today
15. Performance culture
16. Don’t live with Broken Windows
17. Put into production the healthiest application
18. Fine tune as much as possible the architecture and application
19. Understand the weak points of the application
20. Reproduce a performance or multithreading issue detected in production to fix it
21. Give business an idea of future response times when a new application goes live
22. Anticipate Sales period
23. Compare two versions of the same application
24. Confirm/tune the alerting system
25. Validate/improve monitoring (metrics, tools, dashboards, etc.)
26. Find the soft limits of the AWS infrastructure
27. Conclusion

Load testing team organization

1. Introduction
2. Goals of the team
3. Organization/team topology
4. Load testing team in its ivory tower
5. DevOps and load testing team collaboration
6. One performance tester integrated on DevOps team
7. Hybrid Model
8. No performance tester
9. Team of evangelist’s performance testers (Center of Excellence)
10. Skills needed in a load testing team
11. Technical skills
12. Analytical skills
13. Soft skills
14. Business knowledge
15. Key Roles
16. Performance testing manager
17. Senior performance engineers
18. Performance test engineers
19. Database performance specialists
20. Infrastructure specialists
21. Network performance specialists
22. Trainer
23. Monitoring specialists
24. Data/AI specialists
25. Team interactions
26. Load testing team & FinOps teams
27. Load testing team & Development teams
28. Load testing team & Platform Engineering/SRE/ops teams
29. Load testing team & DBA teams
30. Load testing team & Network teams
31. Load testing team & Business/Product teams
32. Load testing team & Security teams
33. Load testing team & Purchasing teams
34. Conclusion

Designing a test case

1. Introduction
2. A load test is a simulation
3. Netflix Technology Blog
4. Define your goal
5. Type of load test
6. Smoke Test
7. Performance/Load Test
8. Stress Test
9. Mark Tomlinson’s quote
10. Breakpoint Test
11. Soak/Endurance Test
12. Failover/Resilience Test
13. Spike Test
14. Other
15. Summary
16. Load model
17. Closed model vs Open model
18. Closed model
19. Open model
20. Why is it important to choose the right model?
21. Implementation in JMeter
22. Concurrency user (Closed model)
23. Throughput (Open model)
24. Arrivals rate (Open model)
25. Arrivals rate model
26. Poisson process with Precise Throughput Timer
27. Constant arrival rate with Thread Group
28. Constant arrival rate with Constant Throughput Timer
29. Arrival rate with Open Model Thread Group
30. Think time
31. Constant think time with Constant Timer
32. Uniform distribution think time with Uniform Random Timer
33. Gaussian/Normal distribution think time with Gaussian Random Timer
34. Poisson distribution think time with Poisson Random Timer
35. All think time we want with JSR223 Timer
36. Structure of a test case
37. Ramp Up
38. Plateau/Step
39. Ramp Down
40. Example: step load test
41. Identify critical business scenarios to script
42. Frequent
43. Vital/Business critical
44. Technically risky
45. Legally risky
46. Brand image risk
47. Summary
48. How many virtual users do I need?
49. Identify Service Level Agreements (SLAs)
50. Why SLA?
51. Identify SLAs, use case 1: SLAs already exists
52. Identify SLAs, use case 2: SLAs don’t exist, but business has a clear idea of them
53. Identify SLAs, use case 3: SLAs don’t exist and business does not have a clear idea of them
54. Best practices
55. Implement them in Apache JMeter
56. Parameters to take into account when creating a scenario
57. Vary user input data
58. Reuse the same values
59. Pre-calculate input data vs random input data
60. Vary the type of users
61. Vary and adjust think times
62. Adjust the ramp up time
63. Adjust the test duration
64. Error handling
65. User connection type
66. Size of the scenarios
67. Meeting point/rendez-vous
68. Cookie Management
69. Cache Management
70. Conclusion

Important concepts in JMeter

1. Scoping
2. Elements execution order
3. How Timers scope works
4. Rule 1: Timers are executed before each sampler in their scope.
5. Rule 2: If there is more than one timer in the scope, all the timers will be processed before the sampler.
6. Control the execution order of Timers
7. How Assertions scope works
8. Rule 1: Assertions are executed after each sampler in their scope or to parent samplers
9. Rule 2: Failed assertions cause the failure of the entire Transaction Controller
10. Rule 3: Assertions can validate main sample and/or sub-samples
11. Rule 4: Be careful with low performing assertions
12. Our advice
13. How properties differ from variables
14. How properties are created
15. How variables are created
16. How ${} differs from ${_P}

Preparing the test environment (injectors and tested system)

1. Introduction
2. Prepare the injectors
3. Never host injectors on the same servers as the tested system
4. Calibrating your test
5. Monitor injectors
6. Prepare Apache JMeter
7. Use the latest version
8. JMeter memory configuration
9. Prepare the network
10. IP Spoofing
11. Setting up IP Spoofing:
12. Avoid load testing behind a proxy
13. Prepare the tested system
14. A tested environment
15. An environment consistent with the target
16. Datasets
17. Take into account calls to third party services
18. Disable protection systems
19. Isolated environment
20. License Management
21. Summary
22. Monitoring the tested system
23. Monitoring/profiling tools and their features
24. Overhead
25. Metrics
26. Logs
27. Instrumentation
28. Sampling
29. Distributed tracing
30. Memory monitoring
31. Thread monitoring
32. Exception monitoring
33. Database monitoring
34. Continuous Profiling/APM (Application Performance Monitoring)
35. Network monitoring
36. Events monitoring
37. Monitoring methodology
38. USE method (Utilisation, Saturation & Errors)
39. RED method (Rate, Errors & Duration)
40. Four Golden Signals (Latency, Traffic, Errors & Saturation) method
41. Our advice
42. Conclusion

Being productive with JMeter

1. Introduction
2. Before starting the recording of the script
3. Choose if static resources are important or not
4. Solution 1: Do not take them into account
5. Solution 2: Let JMeter manage them
6. Solution 3: Manage it ourselves
7. Solution 4: Let JMeter manage only first party resources (1P)
8. Apply a naming convention for transactions/requests
9. Modify or learn keyboard shortcuts
10. Have a script base quickly
11. Pre-populate HTTP Request Defaults before recording the script
12. Use the HTTP(S) Test Script Recorder element
13. BlazeMeter Chrome Extension
14. Use Tools / Import from cURL
15. Import requests from HAR file
16. Recording HTTPS website
17. Save time when working on the script
18. Global Assertion
19. Re-Use script parts
20. Share an object between different Thread Groups
21. Get the most out of CSV files
22. The different modes/options of CSV reader
23. CSV file and JSON content
24. Data coming from several CSV files
25. CSV and Groovy
26. Using Regular Expression Extractor on content with spaces and multiple lines
27. Add headers to our HTTP requests
28. Waiting with While Controller
29. Wait for the end of execution: example 1
30. Wait for the end of execution: example 2
31. Wait for three products to be added to the cart.
32. Exit from While if the last action returns an error.
33. The right Extractor at the right time
34. Handle cookies
35. Set HTTP status code 404 as success in JMeter
36. Debugging a script
37. Using View Results Tree
38. Capture errors
39. Debug Sampler
40. jp@gc - Dummy Sampler
41. Log Viewer
42. BlazeMeter Step-by-step Debugger
43. The old-fashioned way in the console
44. Use a third-party tool
45. Finalise a script
46. Change the name of a transaction according to a parameter
47. Mark a response as an error
48. Execute an action based on the status of the previous one
49. Use timeouts
50. Conclusion

Load Testing a website

1. A bit of theory
2. Set up with JMeter
3. Methodology
4. Put into practice with JMeter
5. Example 1: Find a owner in Spring PetClinic application
6. Our scenario to script
7. The recording
8. Step 1: Work on “Go to the home page”
9. Step 2: Work on “Do a search on the owner of a pet”
10. Vladimir Sitnikov’s quote (JMeter team member)
11. Step 3: Work on “View the details of the owner”
12. Example 2: Add an owner/Display last owner in Spring PetClinic application
13. Step 1: Add an owner
14. Step 2: Display last owner
15. Example 3: Technical tests with Byteman
16. Example 4: Technical tests with JProfiler
17. Example 5: Tricking captchas with a Java Request
18. Conclusion

Load Testing web services

1. What we will learn
2. A few concepts
3. REST (REpresentational State Transfer) web services
4. GraphQL (Graph Query Language)
5. REST vs GraphQL
6. Set up with JMeter
7. Tips
8. Methodology
9. Practice with JMeter
10. Load test a REST web service: Spring PetClinic REST
11. Load test a GraphQL web service: Spring PetClinic GraphQL
12. Conclusion

Load Testing a database server

1. A bit of theory
2. Methodology
3. Setup with JMeter
4. Putting theory into practice with JMeter
5. Example 1: Load test of a database
6. Avoiding duplication using Include Controller
7. Avoiding duplication using Module Controller
8. Example 2: Studying indexes impact on performance
9. Example 3: ETL
10. Conclusion

Load Testing Message Oriented Middleware (MOM) via JMS

1. What we will learn
2. A bit of theory
3. Asynchronous messages
4. Decoupling
5. Technology decoupling
6. Geographical Decoupling (Location transparency)
7. Time decoupling
8. Back pressure
9. Communication models
10. Types of delivery semantics
11. At Most Once
12. At Least Once
13. Exactly Once
14. Summary
15. What is a message composed of?
16. Set up with JMeter
17. Installation of MOM libraries
18. JMS Point-to-Point
19. JMS Resources and JNDI Properties
20. Communication style
21. Priority
22. Use non-persistent delivery mode?
23. JMS Selector
24. Example with Apache ActiveMQ Classic
25. Publish/Subscribe
26. JMS Publisher
27. JMS Subscriber
28. Methodology
29. Put into practice with JMeter
30. Example 1: Test the configuration of a MOM server with point-to-point
31. Example 2: Test the performance of our consumer with JMS Publish/Subscribe
32. Example 3: Testing the configuration of a MOM server with Publish/Subscribe
33. Example 4: Test any MOM
34. Conclusion

Performing a load test

1. Introduction
2. Methodology
3. Before the execution of the load test
4. Testing your script/dataset with the graphical interface
5. Check the injectors
6. Be careful with the use Extractors to avoid injection problems
7. Regular Expression Extractor
8. XPath Extractor
9. Extractor performances
10. Be careful with the use Assertions to avoid injection problems
11. Be careful with the use of custom scripting to avoid injection problems
12. Generate reports at the end of the test
13. Execute load test
14. Running your test from the command line
15. Running your test on the command line with Taurus
16. Running your test from Apache Maven
17. Running your test from Jenkins
18. Running your test from Apache Ant
19. Conclusion

Visualising, analysing and reporting the load testing results

1. Introduction
2. What we will learn
3. Prepare our analyse
4. Have data to analyse
5. Have RAW data
6. Having the right granularity on the data
7. Have enough data to analyse
8. Have the right definition of metrics
9. Visualising
10. Visualising the results with listeners
11. View Results Tree
12. Sampler result tab
13. Request tab
14. Response data tab
15. Summary Report
16. Aggregate Report
17. Backend Listener
18. Report Dashboard
19. Particularities of the visualization of results for a distributed load test before JMeter 5.0
20. Visualizing the results with PaaS load testing tools
21. Redline13
22. Blazemeter
23. Visualizing the results with third party visualizing tools
24. Analyse the results
25. Cognitive biases
26. Confirmation bias
27. Authority bias
28. Overconfidence bias
29. Observer effect
30. Availability heuristic
31. Other bias
32. Anti methodologies
33. Tuning by Folklore Anti-Method
34. Blame-Someone-Else Anti-Method
35. Random Change Anti-Method
36. Methodology
37. Example 1: High response time and Low CPU/memory/network/io disk usage
38. Example 2: High response time and High CPU usage
39. Some tips to analyse the results
40. Correlation does not imply causation
41. Performance is about trade offs
42. Check response time distribution
43. Check for spurious events (errors and bad response time)
44. Worrying About Your Tail
45. Be careful with response time at beginning of the load test
46. Have separate statistics for failed and successful response times
47. Have separate statistics for requests
48. Do not extrapolate on ad hoc hypothèses
49. Some examples
50. Is the application support increasing number of clients?
51. Is errors having an impact in response time?
52. Is Garbage Collector (GC) is optimal?
53. Is my application is CPU bound?
54. Is my application is memory bound?
55. Reporting
56. Prepare the reporting
57. Who is the target of the report?
58. What is the purpose of the report
59. Don’t expose sensitive information
60. Methodology
61. Structure our load test report
62. Choose the right chart types
63. Don’t use pie charts
64. Prefer bar graphs when we have sparse metrics
65. Present errors with toplist
66. Timeseries graphs and single-value summaries
67. Consider box plot for response time distributions
68. Consider heatmaps for identifying patterns in large datasets
69. Choose the right key metrics
70. Define the number of metrics to display in a time series line graphs
71. Same key metrics in report and in SLAs/load model
72. Choose the right data aggregation
73. Prefer percentiles over average
74. View aggregated data by phase
75. Don’t average percentiles
76. Define the resolution/granularity of the measures
77. Choose the correct display for the chart
78. Don’t forget to add clear labels, legends and units in graphs
79. Axis Forced 0
80. Make sure data can be easily seen
81. Add annotations for important events
82. Add Thresholds in graph
83. Conclusion

Integration of JMeter in the DevOps tool chain

1. Introduction
2. Why
3. Questions to ask before automation and Shift-Left strategy
4. Question 0: Which tool?
5. Question 1: How many times do we need to create and maintain scripts?
6. Question 2: How can we stabilize the test environment (injectors and tested system) to allow us to compare results?
7. Question 3: How long should a test take?
8. Question 4: When do we need to execute automated load test?
9. Question 5: Do we have someone to analyze the load test results?
10. Question 6: When is it acceptable to slow down one part of the application to speed up another?
11. Question 7: How to detect slow performance degradation?
12. Question 8: How to evaluate the effectiveness of load tests?
13. Question 9: How to choose threshold alerting and avoid false positive?
14. Question 10: Can we have a production-like environment for exclusive use?
15. Question 11: Do we need to inject huge amounts of test data before the test?
16. Question 12: Do we need to wait for someone before the test?
17. Question 13: How to load test new features?
18. Question 14: How to maintain a realistic workload?
19. Question 15: How to handle dependencies on external systems?
20. Question 16: How to reduce organisational resistance?
21. Question 17: How to justify the cost?
22. Setting up Shift-Left strategy
23. Modification in JMeter script to implement Shift-Left strategy
24. Integrating JMeter in our software factory
25. Integrating with Jenkins using Maven
26. Ensuring the load tests are run on the correct nodes
27. Configuring the load test using jmeter-maven-plugin
28. Fail the build if error rate of load test is over a threshold
29. Configure HTML report
30. Configure memory used by JMeter
31. Enhancing integration with Jenkins using Performance Plugin
32. Integrate JMeter with APM monitoring tools
33. Automation
34. Example 1: Non-regression testing of memory consumption with EJ JProfiler
35. Example 2: Detecting a Memory leak during an endurance test with EJ JProfiler
36. Example 3: Retrieving SQL queries executed during a load test with YourKit Java Profiler
37. Example 4: Analysis of the Garbage Collector (GC) log file with GCViewer
38. Conclusion

Old Stuff

1. Introduction
2. Load testing web application Java EE 6 Pet Catalog
3. Vladimir Sitnikov’s quote (JMeter team member)
4. Load test a SOAP/XML-RPC web-service
5. WS-* web services
6. In JMeter
7. REST Web-Service: Discussion Forum
8. Load test a REST web service: Customer Database
9. Example 7: Non-regression testing of Web APIs with Dynatrace APM AppMon
10. Registering to Dynatrace using setup Thread Group
11. Using the testId in other Thread Group

What’s new

1. 2025/02/28 release
2. Load Testing Organization
3. Integration of JMeter in the DevOps tool chain
4. Visualising, analysing and reporting the load testing results
5. Designing a test case
6. Being productive with JMeter
7. Preparing the test environment (injectors and tested system)
8. 2024/11/29 release
9. Chapter “Designing a test case”
10. Chapter “Load Testing web services”
11. Chapter “Preparing the test environment (injectors and tested system)”
12. Chapter “Quick start with JMeter”
13. Chapter “Performing a load test”
14. Chapter “JMeter overview”
15. Chapter “Why is it still important to make load test today?”
16. Chapter “Load Testing Message Oriented Middleware (MOM) via JMS”
17. Chapter “Visualising, analysing and reporting the load testing results”
18. Load Testing a website
19. Chapter “Old Stuff”
20. 2020/04/28 release
21. Chapter “Designing a test case”
22. Chapter “Preparing the test environment (injectors and tested system)”
23. Chapter “Why is it still important to make load test today?”
24. New chapter “What’s new” at the end of the book