Synthetic testing and Real User Monitoring (RUM) are two website performance monitoring tools with a few overlaps, and several complimentary differences. When used together, they complement one another to cover all what an expected user and a real user will experience when accessing the website or application.
While real user monitoring tools provide the insights into how the real visitor interacts with the website, the synthetic monitoring provides a similar assessment of what an expected user will experience. The synthetic testing goes further to include third-party impact, availability monitoring, and consistent base lining monitoring.
Website performance monitoring tools
The web monitoring tools can effectively identify what is slowing down the performance, see whether it is the applications, users, or protocols that are using most of the bandwidth, or whether link failures, network congestion, hardware or software issues are the cause of packet losses.
Knowing where the problem is, and what is causing it, enables one to take the remedial action to rectify and manage the network issues in a timely and effective way.
Using both the Synthetic and RUM monitoring helps mitigate business risks by enabling the IT teams to;
- Identify performance bottlenecks in the infrastructure or web app.
- Compare the performance of different versions of an application as well as different system configurations
- Evaluate the services provided by third parties such as ISPs, CDNs, Cloud servers etc
- Collect performance related data both for a real user or an expected user.
- Determine the capacity of the website or application and measure if they can handle anticipated load
Figure 1: website performance monitoring image: newrelic
To better understand these monitoring methods, it is important to look at how each of them works, their benefits and pitfalls, and how the two complement one another.
Real User Monitoring (RUM)
The RUM is a passive monitoring process that listens to all the website traffic as users navigate through the site. The RUM gathers data from all the users accessing the site from anywhere in the world regardless of the device, browser, or network. The real user monitoring tools collects data directly from a real user’s browser or website application, in real-time.
A typical monitoring uses a dynamic lightweight snippet of JavaScript to measure real page loads as the user navigates through the website. This enables the collection of data on performance metrics such as page load times and bandwidth, as well as engagement metrics such as conversion and bounce rates. Additionally, the tools gather user metrics such as device type, location, application usage and carrier speeds to get the full picture of user experience.
What the real user monitoring shows
- The performance of the pages and the slowest components
- Whether you are fulfilling the SLA
- Whether changed codes or latest updates effect user experience
- Impact of user experience on other areas such as support
- Type of users enjoying fast responses and those experiencing slow response times
- Performance and response times across different tiers such as users accessing from different locations, different devices, browsers, etc
Advantages of Real User Monitoring
- Wider geographical coverage
- Real user, browser and network conditions
- Tests a large sample size
- Whether the customer is experiencing problems
- Get an insight into browser, location, device or network problems
- An insight into user behavior based on web or application performance, uncover critical targets for optimization
Pitfalls of Real User Monitoring
- Does not provide detailed performance metrics
- Limited analysis of assets
- JavaScript must be installed
- No performance waterfall charts
RUM allows one to reveal each website visitor’s experience regardless of what the devices they use and their location. Analyzing the user’s data helps to uncover hidden performance issues and to correlate conversion, engagement, abandonment or other user behaviors with how the pages or particular application responds.
RUM monitors the availability and performance of the services for all users from all locations and in real-time. It provides data on users’ experience across tiers, measure response times, capture live user sessions, know what users clicked and identify pages that could be causing performance problems. It also allows one to replay the user sessions which can help create test scripts based on real user behavior.
Synthetic monitoring
Synthetic monitoring is a form of network monitoring method usually conducted on the servers in a data center. The simulated testing uses a throttled connection and is designed to mimic conditions a web user might experience. The synthetic monitoring uses desktop and mobile browsers such as the Firefox, Chrome and Internet Explorer, to accurately simulate user’s journeys.
The testing measures the user experience from different locations around the world. The data from such a test is used to ensure that the websites and applications, including those optimized for mobile devices, are available and performing as expected.
A typical test involves one defining what is to be tested and this could be to execute a specific transaction within a web application, or a simple ping to a page. The provider sends the test to the agents using the specified geographical location and browser. The agent makes requests to the website the same way a normal browser would do, and the site responds with the transaction of page requested. As the agent receives the page requested, it measures and captures other performance data such as the load and response times.
The test data is sent back to the provider where it is sorted out for analysis. The analyzed data can then be displayed using waterfall a chart, giving a visual representation of every request the page makes over the total execution time, and provide an easy way of detecting and identifying the performances bottlenecks.
The synthetic testing relies on controlled and predictable environment. And even though it does not represent the experience of a real user, it provides a regular, reliable and consistent insight of the website and application performance. This happens even when there are no users visiting the website.
What the synthetic testing shows
Synthetic monitoring data indicates the website performance in terms of
- Website speed
- Availability
- Whether there are slow downs and at what point in the infrastructure
- Whether all transactions are working
- Whether third-party components are working as required
- Whether there is a correct balance between cost and performance
- The overall performance
Among the important things that synthetic monitoring leaves out are
- Problems with specific devices and browsers.
- Does not cover some individual actions such as adding an item to cart, and users might be experiencing problems when they perform such specific actions.
- Problems may be associated with some paths which have not been tested
- Synthetic testing may not measure the actual geographical locations between the user and the server. The test is usually between servers under controlled conditions which might not be there in the real situation.
- Intermittent problems may exist in between intervals when the synthetic is not running
Advantages synthetic testing method
- No installation or code injection is required
- Ability to measure and compare website performance
- Consistent and stable baseline
- Understands the performance of the third-party services
- Provides performance waterfall charts
- Detailed and consistent testing helps diagnosing and repair problems at both the network and application level
Pitfalls of synthetic testing method
- Synthetic data does not reflect the variables in real life
- The test is data center based and only average network conditions can be mimicked
- Limited geography and sample
Benefits of using the Synthetic and RUM together for website performance monitoring
- RUM data is not detailed as the Synthetic data. However, the RUM tools collect huge amounts of performance data from the actual users on real devices from real locations and connections from all over the world. When used well, the RUM can dramatically improve the end-user performance visibility, and especially for those companies and organizations with broad international reach.
- The real user monitoring is great as long as users are accessing the website. However, one needs synthetic testing to ensure that the website works even when there are no visitors or customers. The synthetic testing allows you to pick any availability issues when users are not visiting the site and help you fix the problems before the real users experience them. In addition, Synthetic testing can check for problems in the infrastructure along paths where there are no users.
- Synthetic testing assumes the user will take certain steps when making a transaction. However the user may hit the “Back” button when adding items into a cart, or go to another page before completing an online purchase. These actions and others that deviate from the intended click path, have impact on the user experience as well as the web performance, as such, one can use RUM to monitor such actions.
- The RUM captures all the interactions of every visit and therefore determines if the application or website is working for everyone and from all countries. The tool identifies what is not working and what is the likely cause of the problem if any. In addition, it will help provide data on users along different click-paths, browsers and from different locations from those used for synthetic testing.
Figure 1: Illustration of the last mile network for a real user
- One can perform synthetic testing from hundreds of locations. However, the testing cannot simulate the true diversity of the real user and does not test the last mile network that is essential in determining the response time for a real user. The RUM on the other hand utilizes tens of thousands of ISPs, collects data from millions of end-points in almost every country.
When to use Synthetic testing and Real User Monitoring in Website performance monitoring
| Activity | Synthetic testing | Real User Monitoring |
| Website, application, or new feature pre-launch testing | X | |
| Monitoring day to-day performance | X (known paths) | X |
| Test new market / location | X | |
| Impact of a failure or slowdown on all users | X | |
| Third-party assets | X | |
| Identify cause of issues on pages and applications | X | |
| Monitoring all traffic from real users | X | |
| SLA | X | X |
| Transaction performance | X | X |
| API performance | X | |
| Monitoring Website and mobile app availability | X | |
| Scaling and sizing planning | X |
Conclusion
Website performance monitoring helps to find out why there are slow website pages and other performance issues that impact on business operations. However, this will only be possible if the correct data is available.
Combining the complementary synthetic testing and Real User Monitoring methods gives a complete overview of what the real users experience along with clues of what is affecting performance.
Using both the RUM and synthetic testing eliminates the blind spots as well as providing a complete view of the real and potential user’s experience, the system, and infrastructure. Both the synthetic and RUM data is useful in determining if there are problems, eliminating false positives and speeding the time to resolve issues and hence enhance user experience.