Octopus supports delta compression for package transfer using our delta compression library. Delta compression will speed up the package acquisition phase of your deployments, especially when the limiting factor is transfer bandwidth, or if the delta achieves significant size reduction.
Delta compression is not available when a package is downloaded directly on your machine(s). Delta compression is used by default when a package is uploaded from the Built-in repository to the remote machine(s). Delta compression is used by default when a package is downloaded from an external feed to the Octopus Server’s package cache and then uploaded to the remote machine(s).
A typical scenario in Octopus Deploy is frequent deployments of small changes to packages. For example, you might push some code updates but all of your libraries are unchanged. In the past Octopus Deploy would upload the entire package to each machine, regardless of how little had changed. With the introduction of delta compression, only the changes to your package will be uploaded to each machine.
A package deployment in Octopus Deploy now looks something like this
- Identify all of the versions of the package available on the target machine by calling Calamari.
- Calamari then attempts to match these packages with packages available on the Octopus Server. If the PackageId, Version and file hash are identical then create a signature file for the package.
- Build the delta file between the previous package and the package being transferred from the Octopus Server.
- If the delta file meets the size criteria (see note below), the server will upload the delta file to the Tentacle and call Calamari to apply the delta file to the transferred package. N.B. If any of these issues are experienced during the creation or application of the delta, then the entire package will be uploaded to the tentacle.
- Once the delta is applied, the signature file from step 2 will be compared with the final applied package on the target to determine if the change was successful. If the signatures don’t match, calamari will request the server to re-upload the entire package.
Delta file size If the final size of the delta file is within 80% of the new package, we upload the full package instead of uploading and applying the delta file as this will save on server resources as applying the delta file can be quite resource heavy with big delta files.
What if something goes wrong?
If any of the below occurs the full package will be uploaded:
- The signature file fails to create.
- The delta file fails to create.
- Applying the delta fails.
- The package details (size and file hash) don’t match after applying the delta.
Running a deployment that generates a delta file
When running a deployment that creates and applies a delta file, you will see the following in the logs under the Acquire packages section
Delta progress logging As can be seen in the screenshot above, the logging of the progress of applying the delta doesn’t look like it’s successfully completed as it’s reporting only 20% and 0%(!!). Don’t worry though, this is due to a problem with how our delta compression library (Octodiff) reports progress (we will be fixing this logging issue) and it’s actually applying the full delta.
Optimizing delta compression
The best way to guarantee the best size reduction is to use the tools provided by Octopus Deploy when creating your packages. All of our packaging tools are automatically tested to ensure the package contents are bundled in a delta-compression-friendly format.
If you want to use your own tools, that is fine! Just be sure to test the performance of delta compression to make sure you have configured everything correctly.
The most common mistake causing delta compression to yield minimal size reduction is when artificial differences are injected into the package file. One example is when timestamps are changed each time the package is built. The tools provided by Octopus Deploy are designed to yield high size reductions based on the actual content of your packaged files.
Turning delta compression off
To turn this feature off, create a project variable named Octopus.Acquire.DeltaCompressionEnabled with a value of False.
Delta calculations can be CPU intensive You should consider disabling delta compression if package transfer bandwidth is not a limiting factor (all the machines are in the same network segment), or if the CPU on the Octopus Server is pegged at 100% during your deployments.
Are you really benefiting from delta compression? The deployment logs will tell you the % saving delta compression is achieving. If you are constantly transferring 50% or more of the original package, perhaps delta compression is actually becoming a bottleneck and should be disabled.
Help us continuously improve
Please let us know if you have any feedback about this page.
Page updated on Sunday, January 1, 2023