EPM for Mac performance reports

EPM-M 24.7 performance report

Introduction

The aim of this document is to provide data on agreed performance metrics of the Privilege Management for Mac desktop client compared to the previous release.

The content of this document should be used to provide general guidance only. The are many different factors in a live environment which could show different results such as hardware configuration, macOS configuration and background activities, 3rd party products, and the nature of the EPM policy being used.

Performance benchmarking

Test Scenario

The tests were ran on a virtual machine with the following configuration:

• macOS: 13.4.1
• Apple Silicon
• 8GB RAM

Tests were completed with the GA releases.

• 24.5
• 24.7

Test Names

• Quick Start policy in High flex, admin user
• Running the Automation Tests Suite with a Monitoring tool
• Installing and building Qt5 from source

Test Method

We have included three methods to test our product, all make use of the caching feature. To find out how to use our caching feature, see the information in our administrator guides in the Configure Caching on Policies section.

There were three types of tests conducted:

• The first one uses the default quick start policy where we expect to match on an allowed binary with no dialog presented to the user, ensuring consistency and reducing user interaction.

The quick start policy is the policy that's most commonly used as a base for all of our customers. It can be applied by the MMC and WPE with Import Template. It was chosen as it's our most common use case. The binary we're launching is bash with the -help argument, and we run it through the hyperfine tool during the test process to produce the minimum, maximum and mean time of our rule matching engine.

• The next one involves running our Automation Tests Suite and measuring the components resource consumption. We conduct tests on all of our core functionality with our automation suite and keeps our software under load for approximately an hour. We’ve used the Automation Tests Suite to portray as close as possible a real life environment usage and to monitor PMfM through all types of intensive and non intensive scenarios.
• The last one is about installing the Qt5 package on the system and building it from source to simulate a developer environment trying to set it up and compile code. We use the hyperfine tool to run the 'HOMEBREW_NO_AUTO=1 brew reinstall qt@5' command which executes it 10 times in a row and times the minimum and maximum in seconds for each run, also producing the mean values.

Summary

Comparing 24.7 results against 24.5, we can observe a small increase in performance for the 24.7 candidate with or without caching enabled for the 1 and 3 synthetic tests, starting from 2% and going up to 12%. This is because we’ve either removed some of our logging messages or we’ve not included them into our release builds. The tests were done at a micro level focusing on a single binary and at a macro level focusing on multiple binaries being run while installing and building from source.

On the other hand, looking at our automation performance data for the 24.7 release candidate, we can see a small increase in resource consumption for some components when compared to the previous release of PMfM, and this can be explained by the new addition of the JIT Admin and Application features.

Overall, we can observe a small variable increase in performance on focused time-boxed tests and a slight performance degradation in resource consumption when running the full automation suite.

Results

Rule matching latency

Note: Hyperfine v1.19.0 used both 24.5 and 24.7 for the results of tests 1 and 3.

µs is microsecond. ms is millisecond.

Automation Tests resource consumption per component

EPM-M 24.7 caching disabled

EPM-M 24.5 caching disabled

Installing and building Qt5 from source with a QuickStart High Flexibility policy applied

Hyperfine --runs 10 'HOMEBREW_NO_AUTO=1 brew reinstall qt@5'

EPM-M 24.5 performance report

Introduction

The aim of this document is to provide data on agreed performance metrics of the EPM-M desktop client compared to the previous release.

The content of this document should be used to provide general guidance only. There are many different factors in a live environment which could show different results such as hardware configuration, macOS configuration and background activities, 3rd party products, and the nature of the EPM policy being used.

Performance benchmarking

Test scenario

The tests were ran on a physical machine with the following configuration:

• macOS: 14.5
• Apple Silicon
• 16GB RAM

Tests were completed with the GA releases.

• 24.3
• 24.5

Test names

• Quick Start policy in High flex, admin user
• Running the Automation Tests Suite with a Monitoring tool
• Installing and building Qt5 from source

Test method

We have included three methods to test our product, all make use of the caching feature.

There were three types of tests conducted:

• The first one uses the default quick start policy where we expect to match on an allowed binary with no dialog presented to the user, ensuring consistency and reducing user interaction.

The Quick Start policy is the policy that's most commonly used as a base for all of our customers. It can be applied by the MMC and WPE with Import Template. It was chosen as it's our most common use case. The binary we're launching is bash with the -help argument, and we run it through the hyperfine tool during the test process to produce the minimum, maximum and mean time of our rule matching engine.

• The next one involves running our Automation Tests Suite and measuring the components resource consumption. We conduct tests on all of our core functionality with our automation suite and keep our software under load for approximately an hour. We’ve used the Automation Tests Suite to portray as close as possible a real life environment usage and to monitor EPM-M through all types of intensive and non intensive scenarios.
• The last one is about installing the Qt5 package on the system and building it from source to simulate a developer environment trying to set it up and compile code. We use the hyperfine tool to run the 'HOMEBREW_NO_AUTO=1 brew reinstall qt@5' command which executes it 10 times in a row and times the minimum and maximum in seconds for each run, also producing the mean values.

Summary

Comparing 24.3 results against 24.5, we can observe no major differences between these two builds when caching is disabled but there is a 10-15% improvement when caching is enabled. This is due to a new performance improvement feature that introduces caching for file reads on the system.

The tests were done at a micro level focusing on a single binary and at a macro level focusing on multiple binaries being run while installing and building from source.

You may see a negative impact on performance when compared to the previous release of EPM-M but this can be attributed to a increase of automation especially around URM and the addition of URM which is a Just In Time procedure so modifications are made instantly.

Overall there is no performance degradation and the resource consumption differences between the automation runs are negligible, with a substantial performance improvement when caching is enabled.

Results

Rule matching latency

ℹ️

Note

Hyperfine v1.18 used both 24.3 and 24.5 for the results of tests 1 and 3.

µs is microsecond. ms is millisecond.

Automation tests resource consumption per component

EPM-M 24.5 caching disabled

EPM-M 24.3 caching disabled

Installing and building Qt5 from source with a QuickStart High Flexibility policy applied

Hyperfine --runs 10 'HOMEBREW_NO_AUTO=1 brew reinstall qt@5'

EPM-M 24.3 performance report

Introduction

The aim of this document is to provide data on agreed performance metrics of the EPM-M desktop client compared to the previous release.

The content of this document should be used to provide general guidance only. The are many different factors in a live environment which could show different results such as hardware configuration, macOS configuration and background activities, 3rd party products, and the nature of the EPM policy being used.

Performance benchmarking

Test scenario

The tests were ran on a Parallels virtual machine with the following configuration:

• macOS: 13.4.1
• Apple Silicon
• 8GB RAM

Tests were completed with the GA releases.

• 24.1
• 24.3

Test names

• Quick Start policy in High flex, admin user
• Running the Automation Tests Suite with a Monitoring tool
• Installing and building Qt5 from source

Test method

We have included three methods to test our product, all make use of the caching feature.

There were three types of tests conducted:

• The first one uses the default quick start policy where we expect to match on an allowed binary with no dialog presented to the user, ensuring consistency and reducing user interaction.

The Quick Start policy is the policy that's most commonly used as a base for all of our customers. It can be applied by the MMC and WPE with Import Template. It was chosen as it's our most common use case. The binary we're launching is bash with the -help argument, and we run it through the hyperfine tool during the test process to produce the minimum, maximum and mean time of our rule matching engine.

• The next one involves running our Automation Tests Suite and measuring the components resource consumption. We conduct tests on all of our core functionality with our automation suite and keep our software under load for approximately an hour. We’ve used the Automation Tests Suite to portray as close as possible a real life environment usage and to monitor EPM-M through all types of intensive and non intensive scenarios.
• The last one is about installing the Qt5 package on the system and building it from source to simulate a developer environment trying to set it up and compile code. We use the hyperfine tool to run the 'HOMEBREW_NO_AUTO=1 brew reinstall qt@5' command which executes it 10 times in a row and times the minimum and maximum in seconds for each run, also producing the mean values.

Summary

Comparing 24.1 results against 24.3, we can observe no major differences between these two builds when caching is disabled but there is a 10-15% improvement when caching is enabled. This is due to a new performance improvement feature that introduces caching for file reads on the system.

The tests were done at a micro level focusing on a single binary and at a macro level focusing on multiple binaries being run while installing and building from source.

Overall there is no performance degradation and the resource consumption differences between the automation runs are negligible, with a substantial performance improvement when caching is enabled.

Results

Rule matching latency

ℹ️

Note

Hyperfine v1.18 used both 24.1 and 24.3 for the results of tests 1 and 3.

µs is microsecond. ms is millisecond.

We’ve included a visualization of the resources our components are using when running the following command on the 24.1 release candidate with caching disabled:

Hyperfine --runs 1000 'bash -help'

This shows how our components under load using the command and for a short time after, which is due to our components needing additional time to fully compute the 1000 actions in this test case.

All resources drop to near zero after that.

Automation tests resource consumption per component

Defendpoint component

Process load ( % )

Memory (MB)

Energy Impact

PrivilegeManagement component

Process load ( % )

Memory (MB)

Energy impact

EndpointSecurity component

Process load ( % )

Memory (MB)

Energy impact

Interrogator Component

Process load ( % )

Memory (MB)

Energy impact

Install and build Qt5 from source with a QuickStart high flexibility policy applied

Hyperfine --runs 10 'HOMEBREW_NO_AUTO=1 brew reinstall qt@5'

EPM-M 24.1 performance report

Introduction

The aim of this document is to provide data on agreed performance metrics of the EPM-M desktop client compared to the previous release.

The content of this document should be used to provide general guidance only. The are many different factors in a live environment which could show different results such as hardware configuration, macOS configuration and background activities, 3rd party products, and the nature of the EPM policy being used.

Performance benchmarking

Test scenario

The tests were ran on an Parallels virtual machine with the following configuration:

• macOS: 13.4.1
• Apple Silicon
• 8GB RAM

Tests were completed with the GA releases.

• 23.9.1.1
• 24.1

Test names

• Quick Start policy in High flex, admin user
• Running the Automation Tests Suite with a Monitoring tool
• Installing and building Qt5 from source

Test method

We have included three methods to test our product, all make use of the caching feature.

There were three types of tests conducted:

• The first one uses the default quick start policy where we expect to match on an allowed binary with no dialog presented to the user, ensuring consistency and reducing user interaction.

The quick start policy is the policy that's most commonly used as a base for all of our customers.  It can be applied by the MMC and WPE with Import Template.  It was chosen as it's our most common use case.  The binary we're launching is bash with the -help argument, and we run it through the hyperfine tool during the test process to produce the minimum, maximum and mean time of our rule matching engine.

• The next one involves running our Automation Tests Suite and measuring the components resource consumption. We conduct tests on all of our core functionality with our automation suite and keep our software under load for approximately an hour. We’ve used the Automation Tests Suite to portray as close as possible a real life environment usage and to monitor PMfM through all types of intensive and non intensive scenarios.
• The last one is about installing the Qt5 package on the system and building it from source to simulate a developer environment trying to set it up and compile code. We use the hyperfine tool to run the 'HOMEBREW_NO_AUTO=1 brew reinstall qt@5' command which executes it 10 times in a row and times the minimum and maximum in seconds for each run, also producing the mean values.

Summary

Comparing 23.9.1 results against 24.1, we can observe no major differences between these two builds when caching is disabled but there is a 10-15% improvement when caching is enabled. This is due to a new performance improvement feature that introduces caching for file reads on the system.

The tests were done at a micro level focusing on a single binary and at a macro level focusing on multiple binaries being run while installing and building from source.

Overall there is no performance degradation and the resource consumption differences between the automation runs are negligible, with a substantial performance improvement when caching is enabled.

Results

Rule matching latency

Note: Hyperfine v1.18. used both 23.9.1 and 24.1 for the results of tests 1 and 3.

µs is microsecond. ms is millisecond.

We’ve included a visualization of the resources our components are using when running the following command on the 24.1 release candidate with caching disabled:

Hyperfine --runs 1000 'bash -help'

This shows how our components under load using the command and for a short time after, which is due to our components needing additional time to fully compute the 1000 actions in this test case.

All resources drop to near zero after that.

Automation tests resource consumption per component

Defendpoint component

Process load ( % )

Memory (MB)

Energy impact

PrivilegeManagement component

Process load ( % )

Memory (MB)

Energy impact

EndpointSecurity component

Process load ( % )

Memory (MB)

Energy impact

Interrogator component

Process load ( % )

Memory (MB)

Energy impact

Install and build Qt5 from source with a QuickStart high flexibility policy applied

Hyperfine --runs 10 'HOMEBREW_NO_AUTO=1 brew reinstall qt@5'