Apache Subversion 1.9 Release Notes

Python (optional dependency) must be ≥2.7 ¶

The Python programming language is an optional dependency. It is required for running the test suite and for building nightly versions, but not for building from packaged releases.

Subversion 1.8 supported Python 2.5 and newer. Subversion 1.9 requires Python 2.7. Older versions of Python are no longer guaranteed to work.

We break compatibility with Python 2.6 and older since it has been end-of-life for nearly two years, and in order to introduce compatibility with Python 3.x in patch releases of the 1.9.x line.

Next release will be numbered 1.10 ¶

The next minor release after 1.9.0 will be numbered 1.10.0.

Since "1.10.0" is smaller than "1.9.0" when considered as ASCII strings, scripts that compare Subversion versions as strings may fail to correctly determine which of "1.10.0" and "1.9.0" is the more recent one. Such scripts should be changed to compare Subversion version numbers correctly: as tuples of integers, with an optional suffix for development or pre-release versions. See the Subversion release numbering documentation for details. (Programs written against the C API or the various bindings should refer to the svn_version_* interfaces.)

The next patch releases on the 1.9.x line will be numbered 1.9.1, 1.9.2, …, as usual.

Format bump ¶

The default filesystem format is now a new format, numbered 7. (The new command svnadmin info displays the filesystem format number of a repository.) In contrast to previous releases, format 7 is a major overhaul with the general goal of I/O reduction. Major changes include:

• Revision content is no longer addressed by physical location (offset) within the rev file but by logical item numbers. Two indexes provide the necessary mapping information. You can no longer manipulate the rev / pack file contents manually.
• svnadmin pack reorders the revision data such that related information is put next to each other and can often be read with a single I/O.
• Block reads are an option now and will fetch data in larger blocks of configurable size (64kB by default) and cache all of their contents. This eliminates much of the OS overhead.
• Explicit flags for svn:mergeinfo changes speed up mergeinfo-related operations like svn log -g.
• Full checksum coverage of all revision data, including meta data and structural information.
• Allow commits while packing is in progress.

As with earlier releases, you may simply run svnadmin upgrade on your repository. However, the new addressing, checksumming and packing scheme will not be used in that case but only in repositories that got created as format 7. For best performance and to enable all features, it is recommended to perform a full dump / load cycle. You can tell whether all format 7 features are enabled by looking for FSFS Logical Addressing: yes in the output of svnadmin info: if that line is printed, then the repository has all format 7 features enabled.

The next subsection compares performance characteristics of repositories created directly as format 7 with those upgraded to format 7 from older formats.

Service quality ¶

The FSFS changes in 1.9 are aimed at better service quality and do not translate into client-side features being available or not. Depending on your setup, some of the improvements may be relevant while others are not. This table shall help you decide what features are relevant to your environment and whether you need to upgrade or dump and load your repositories to reap the benefits.

Most users will want to not only svnadmin upgrade to migrate their repositories but to eventually migrate them to format 7. For some, it will be the fast verification feature, others will need the disk I/O improvements. Note that the key I/O characteristics here is the Bandwidth Delay Product of your storage, which is usually between 100kB and 1MB. Even with SSDs you will see a speed-up, unless your storage bandwidth is severely limited.

There is no appreciable difference in CPU usage between the new format and the older ones. Hence, the few setups that work almost entirely from RAM due to very large caches will see little extra performance with format 7. These environments will still benefit from the improved checksum coverage and the support for quick verification.

Format 7 tuning options ¶

New configuration options are available for tuning read block sizes (format 7 only). They may be changed at any time without causing consistency issues with existing revisions. Changing them is rarely necessary, but may result in slightly improved performance with your specific storage backend or when dealing with multi-TB repositories. See the commentary in fsfs.conf for more information. (You may have to create a new repository to see the documentation about these settings in fsfs.conf.)

  [io]
  block-size = 256
  l2p-page-size = 32768
  p2l-page-size = 16384

Zlib compresssion is now optional ¶

FSFS uses a combination of two methods to reduce on-disk data size. First, we determine the changes (delta) against some previous version of the same file. This process can be controlled by fsfs.conf settings since 1.8. The result would then be compressed using ZIP/deflate. That relatively costly operation limits commit speeds of large incompressible files. You may now specify the compression level in fsfs.conf or disable compression entirely:

  [deltification]
  compression-level = 0

Changed deltification defaults ¶

Directory and property deltification are now enabled by default for FSFS repositories in 1.6 format or newer. Various optimizations were added to the delta creation code to minimize the read amplification effect and to provide a net speedup when using deltification vs. not using it. Keep txdelta caches enabled when using directories deltification. Otherwise, you may suffer a significant performance hit.

To prevent Subversion 1.9 from deltifying properties and directories in older format repositories, set the following options in the repository’s fsfs.conf file:

    [deltification]
    enable-dir-deltification = false
    enable-props-deltification = false

Changes to the caching ¶

The caching logic has been enhanced to cope with the extra load caused by the access indirection introduced in FSFS format 7. While most of this simply means more net speed, some changes may affect your configuration settings.

• Fulltext and txdelta caching are now enabled by default for all servers as well as for local repository access.
• Revision property caching is disabled and the respective server option will be ignored.
• Directories 3x as large as in 1.8 can now be cached.
• Caching is quasi-perfect on short time scales (requesting data equal or less than approx. 10% of the cache size). This greatly improves performance when e.g. many clients check out the same project.
• Long-term caching is now priority-based with text deltas having the lowest priority. This results in slower but more reliable heating up, i.e. it takes multiple similar requests until all frequently used data is kept in cache, but it will reach this point eventually while older releases might not.

Overview ¶

Where the svnadmin tool covers typical administrative tasks in a mostly back-end agnostic way, svnfsfs is a specialist tool for analyzing and manipulating of FSFS repositories. It is not intended for everyday use.

stats sub-command replacing the fsfs-stats tool ¶

The fsfs-stats tool first released with Subversion 1.8 has been replaced by the svnfsfs stats sub-command and is no longer part of Subversion 1.9. Both produce similar output.

Reading and writing format 7 indexes ¶

During forensics or data recovery, it is necessary for experts to directly "look into" the raw database files. While the FSFS on-disk format is fully documented, the indirect addressing and reordering added in format 7 makes it hard for humans to trace internal references. This is where the svnfsfs dump-index sub-command is used. It produces a table describing all items in revision and pack files:

    $ svnfsfs dump-index /path/to/repo -r0
       Start       Length Type   Revision     Item Checksum
           0           11 drep          0        3 60232b75
          11           59 node          0        2 403dbe48
          6a            1 chgs          0        1 f28a4f1d

Because the index information must always match the actual file contents, updating the index data after every revision / pack file manipulation is mandatory in format 7. svnfsfs load-index allows you to do that. It consumes the same table format as produced above, except that the checksum field is optional and will be ignored if given. See the known issues list for problems in released versions of that tool.

Known issues ¶

In 1.9.0, the svnfsfs load-index does not work as described in its documentation. The following restrictions and workarounds apply:

• The lines must be sorted by offset (first column).
• The item number (5th column) must be given as hexadecimal. However, svnfsfs dump-index produces decimal numbers in that column.
• The first entry must refer to the first revision in the pack file. This is a non-issue for unpacked revisions. A simple solution is inserting a line for empty section of length 0:

         Start       Length Type   Revision     Item 
             0            0 none       3000        0
             0          25a chgs       3999        1
           25a          3b9 chgs       3998        1
         ...
       

These problems have been fixed in 1.9.1.

Overview ¶

Since its inception 10 years ago, FSFS has been improved greatly and the improvements provided with format 7 are another major step forward. However, there are limits to what can be done in a meaningful and backward compatible way. FSX is being designed to overcome these limitations. Here some of the intended improvements:

• 90% reduction in metadata overhead.
• Efficient handling of very large files.
• Higher overall compression rates, in particular for office-style documents.
• Information required for log and merge operations is more readily available.
• O(1) handling of large directories.
• Versioned revision properties.
• Partitionable storage.
• Arbitrary meta data storage and indexing facilities.

Development of FSX was started as a fork of FSFS to guarantee functional completeness and testability. The current code still contains remnants of FSFS legacy code and not all of the above improvements have been implemented yet. Later releases will close the gap and remove transitional code.

Usage scenarios ¶

FSX is nowhere near as stable and reliable as FSFS. Furthermore, there will be no support for 1.9-style FSX repositories in 1.10 (see Incompatibility). Think, therefore, of all data in a FSX repository as being potentially corrupt. Its improved performance and storage characteristics, though, might make FSX a viable option for the following use-cases:

• Running analysis or reporting tools whose output does not need to be 100% reliable. Use FSX as a high-speed data source in a read-only mirror.
• Investigate how FSX may fit into your future infrastructure. How well does it perform with your data set concerning disk, I/O, RAM and CPU usage?

Incompatibility ¶

The FSX code and storage representation is in an intermediate state with respect to the feature set that its developers have in mind. For as long as it keeps its experimental status, there will be neither forward nor backward compatibility between FSX repositories of different Subversion feature releases. Subversion 1.10 and later will recognize 1.9 FSX repositories and error out on them. You may use dump and load to upgrade an FSX repository from one release to another.

Dump and load is also the only upgrade path between FSFS and FSX.

Finally, there is no guarantee that FSX will eventually be released at all; as an experimental backend, we make no promise that future releases will support reading or writing FSX repositories.

New sub-command svn auth ¶

The new svn auth sub-command can be used to view or remove authentication credentials saved in any of the supported password caches. Authentication credentials include usernames, passwords, SSL certificates, and SSL client-certificate passphrases.

Specific credentials can be selected with an arbitrary number of pattern arguments which are matched against the attributes of each credential. For example, view cached SSL certificates for the apache.org domain, matched via credential kind (svn.ssl.server) and authentication realm and certificate subject (apache.org):

$ svn auth svn.ssl.server apache.org
------------------------------------------------------------------------
Credential kind: svn.ssl.server
Authentication realm: https://svn.us.apache.org:443
Subject: C=US, ST=Maryland, L=Forest Hill, O=The Apache Software Foundation, OU=Infrastructure, CN=*.apache.org
Valid from: 2015-04-29 02:00:00 +0200 (Wed, 29 Apr 2015)
Valid until: 2017-04-30 01:59:59 +0200 (Sun, 30 Apr 2017)
Issuer: C=US, O=Symantec Corporation, OU=Symantec Trust Network, CN=Symantec Class 3 Secure Server CA - G4
Fingerprint: 4ad722dd0442043657d176f9c81aab66094d4223
Hostnames: *.apache.org
Automatic certificate validity check failed because:
  The certificate's Common Name (hostname) does not match the remote hostname.

  Credentials cache in '~/.subversion' contains 1 matching credentials

For more information, see svn help auth.

svn info --show-item=arg [--no-newline] ¶

The svn info command can now display the value of one of the fields and nothing else, for easier consumption by scripts.

Subversion 1.8 and earlier had two output modes: the default, human-oriented output, mode and the XML mode for scripted use. Subversion 1.9 adds a third output mode, whereby exactly one attribute will be displayed:

## Display the youngest revision of a repository:
% svn info --show-item=revision https://svn.apache.org/repos/asf/subversion/trunk
1693514

## Find the root directory of a working copy:
% svn info --show-item=wc-root
/home/jrandom/src/svn/trunk

Incidentally, Subversion will also attempt to offer the correct selector name if the argument was misspelled:

% svn info --show-item=wcroot
svn: E205000: Try 'svn help info' for more information
svn: E205000: 'wcroot' is not a valid value for --show-item; did you mean 'wc-root'?

The list of valid arguments to --show-item may be found in its help message, svn help info. As of 1.9, the valid values are kind, url, relative-url, repos-root-url, repos-uuid, revision, last-changed-revision, last-changed-date, last-changed-author, and wc-root.

The --no-newline argument instructs svn not to emit a cosmetic newline (\n) after the value.

svn propget --no-newline ¶

The svn propget sub-command has a new --no-newline option. It is equivalent to the old --strict option which is now deprecated.

New HTTPS-related argument --trust-server-cert-failures ¶

The new --trust-server-cert-failures option is intended to be used by scripts which for some reason must accept SSL certificates which fail validation for various reasons (e.g. expired certificates).

If at all possible, fixing a certificate problem is preferable to using this option.

The --trust-server-cert-failures option only works in conjunction with the --non-interactive option.

Previous versions of Subversion in non-interactive mode could only ignore certificates with an unknown certificate authority, but expired or otherwise invalid SSL certificates could not be accepted.

New --pin-externals option for svn copy ¶

The svn copy sub-command now supports a new --pin-externals option. Use of this option is recommended when creating tags.

If this option is used, svn copy pins the URLs in svn:externals properties in the copied tree to their current revision, such that externals keep their current contents when the copied tree is checked out at a later time.

Note that external references within externals cannot be pinned as this would require svn copy to make commits to any foreign repositories referenced by externals.

New options for svn cleanup ¶

The svn cleanup command has new --remove-unversioned and --remove-ignored options which can be used to remove unversioned and ignored files from the working copy.

svn cleanup can now recurse into externals with the new --include-externals option.

Interactive conflict resolver changes ¶

In the interactive conflict resolver, the (m) merge command now tries an external file merge tool if one is defined. Else the internal file merge tool is used.

In Subversion 1.8, the (m) merge command always triggered the internal file merge tool. The (i) internal merge tool command was added to Subversion 1.9 and has the old meaning of the (m) merge command.

svn status now takes --revision (-r) ¶

The status command, when given the --show-updates (-u) flag, can now compare the working copy to specific remote revision, given by the -r (--revision) flag. (Previously, the remote revision would default to HEAD.)

Committing the result of large merges ¶

Our best practices suggest that projects should branch and merge at the project root level. Each merge will then usually change the svn:mergeinfo property at the project base folder. To commit that change, pre-1.9 servers require the client to have the latest version of that base folder. Whenever there is a commit in that tree, its base folder will get a new revision as well, though. Thus, in large, busy projects, it is likely that by the time a large merge commit would actually have reached its finalization phase, some other commit got through and the merge commit is rejected for being out of date.

Subversion 1.9 now allows to modify directory properties based on old revisions as long as the directory contents itself did no change. Directory contents changes would either be property changes, added or removed entries. Sub-tree or file contents changes no longer prevent the property change from going through. Of course, true file and tree change conflicts will still result in out-of-date errors.

This new behavior is not restricted to svn:mergeinfo changes but applies to any directory properties. Changes to svn:ignore are another common scenario previously prone to update / commit / out-of-date races.

Rate limiting svnserve ¶

In threaded mode, svnserve starts one thread per network connection - potentially consuming a large amount of resources. Subversion 1.9 now uses a thread pool with a configurable minimum and maximum number of threads. Use the command line options --min-threads and --max-threads if the defaults don't suite your needs.

Whether or not the new options are available depends on the platform and APR capabilities. Also, the default for --max-threads is lower on 32 bit systems than on 64 bit ones. Run svnserve --help to see your system's defaults and whether the options are available at all.

New sub-command svnadmin info ¶

This prints detailed format information for the given repository.

  $ svnadmin info /repos/apache/
  Path: /repos/apache
  UUID: ac336b0e-000b-11e0-b354-23d019ddd9ed
  Repository Format: 5
  Compatible With Version: 1.8.0
  Repository Capability: mergeinfo
  Filesystem Type: fsfs
  Filesystem Format: 6
  FSFS Sharded: yes
  FSFS Shard Size: 1000
  FSFS Shards Packed: 1631/1631
  FSFS Logical Addressing: no
  Configuration File: /repos/apache/db/fsfs.conf

svnadmin pack ¶

Packing now takes a cache size parameter (-M) which is used for efficient operation on FSFS format 7 repositories.

  $ svnadmin pack -M 1000 /path/to/repository

svnadmin verify ¶

Verification will terminate by default once a problem has been found. The new --keep-going option instructs svnadmin to continue with the next revision such that multiple issues may be found in a single run. If a revision has multiple issues and depending on the verification logic, still only the first problem may be reported.

The new --check-normalization option reports any path names within the same directory or svn:mergeinfo property which differ only in character representation (e.g. Unicode precomposed and decomposed character representations), but are otherwise identical.

Thorough verification is expensive and becomes slower as the repository history grows. The new --metadata-only flag will skip the expensive internal consistency checks and reconstruction of all user data. In format 7 repositories, this will still perform a quick on-disk checksum test of all data in rev / pack files but will only detect cases where external forces (storage crash, rogue scripts etc.) modified committed data. Also, the checksums are weaker (about a 1 in a billion chance that a random change remains undetected) than the ones used to protect the user data. If you suspect internal repository corruption caused by a bug within SVN itself, you still need to run a full verification.

svnadmin setrevprop --transaction and svnadmin delrevprop ¶

The setrevprop command now takes the --transaction flag. It can be used from pre-commit hooks to set revision properties on the transaction (commit-in-progress) before it becomes a commit.

The new delrevprop command deletes a revision property from either a revision or a transaction. It complements setrevprop which adds or modifies revprops.

svnadmin load --ignore-dates ¶

The svnadmin load command has a new flag, --ignore-dates, which causes it to ignore revision datestamps (svn:date) in the input dumpfile.

Tool renamed from svn-bench to svnbench ¶

With Subversion 1.9 this tool is now part of the standard set of SVN binaries. In keeping with the existing programs, it has been renamed to svnbench.

New sub-command svnbench null-blame ¶

To investigate bottlenecks in svn blame, we added a protocol driver variant of it to svnbench. The new null-blame sub-command requests the same data from the server as the normal command line client and shows a summary of the amount of data received.

  $ svnbench null-blame -g http://example.org/repo/path/to/file
               34 revisions
               31 deltas
        1,161,077 bytes in deltas