AVM Testing: It Is Not as Daunting as You Might Think

Increasingly lenders are using automated valuation models (AVMs) at multiple points in the lending process. In order to ensure compliance, it is important that AVMs periodically undergo rigorous performance testing. Rob Walker, executive vice president, collateral solutions for First American Real Estate Solutions answers some of the common questions lenders have about AVM testing.

Q: What is AVM testing?
A: AVM testing is the process of asking an AVM to value properties where the actual sale prices are known to you but not to the AVM. The AVM valuations are then compared to the actual sale prices, and performance metrics are calculated to determine the AVM’s overall level of accuracy. This ability to measure AVM accuracy compares favorably to the traditional appraiser environment where the preparer of the report can be subject to transactional pressures, and users are not aware of the statistical accuracy of the product.

Q: What are the benefits of AVM testing?
A: One of the biggest benefits of AVM testing is that it enables lenders to measure valuation error, confidence score validity, hit rate and other parameters, and thus make comparisons among multiple AVM brands. AVM testing also provides the information needed for constructing AVM cascades and for determining when AVMs can be used in lieu of more costly or time-intensive valuation methods such as field appraisals.

Q: I’ve heard that blind testing is important – what exactly does “blind” testing mean?
A: Blind testing requires the AVM to calculate current values without knowledge of the most recent sale price. It is accomplished by accessing recent sales information, either from a provider of real estate information or by accessing closed loan information from your pipeline and comparing it to AVM values. Of course, it is critical that you access the data and test before the data is available to AVM models, because a test that includes the most recent sale price of a subject property is considered a non-blind test.

On a side note, it is important to state that Uniform Standards of Professional Appraisal Practice (USPAP) guidelines prohibit appraisers from engaging in appraisals of pending sales without knowledge of the proposed sales price. Consequently, appraisers cannot perform a blind test the way that an AVM can, and this has led some to contend that transaction pressure unduly influences traditional appraisals.

Q: What is a reasonable range for values to fall within compared to the sale price?
A: The true value of a property is what a buyer and seller agree upon in the absence of fraud or duress. It is widely accepted that each purchase transaction is unique, resulting in natural variances in actual purchase prices for properties that are identical. While every lender will have to establish a reasonable variance tolerance according to its own lending guidelines, industry literature suggests that a range of plus or minus 6 to 12 percent is acceptable.

Q: What is the best way to go about testing?
A: The first step is to determine what level of risk (variance) you are willing to accept. If you are an equity lender whose average consumer has great credit, you may be willing to accept a greater variance from AVM values to the owners’ estimated value. You may determine that your minimum AVM criterion is a median absolute error of plus or minus 10%. If this is your assumption, then you can evaluate all AVMs against your pre-defined minimum standards. Once tolerance guidelines are established, it is possible to embark on large and exhaustive tests. Historically, AVM testing has been provided at little or no cost to lenders. All that is usually required is assembling a sample data set and testing a variety of properties across any given geographic footprint.

Q: How do I evaluate all these AVM values vs. recent sales prices?
A: Lenders must individually design their test methodologies to include a selection of reference values, sourcing of sufficient samples for each test cell and AVM performance measurements. There are many approaches and there is no one correct method. Some firms use a combination of a statistical measure like median or average error and/or the probability of generating an “accurate value.” Either way, you should devise a ranking or scoring mechanism that can help you see the best AVMs in a given market based upon your firm’s specific needs.

Q: As a small bank, that does not have in-house analysts, how can I accomplish AVM testing and cascade development cost effectively?
A: Almost all AVM developers provide prospective and current customers with due diligence reports upon request. These reports reflect the valuation accuracy that the AVM provider believes they can achieve based upon their testing of their product.

The next step is to corroborate the provider’s information based upon blind, localized testing data. To achieve this, go to a reputable source of information on pending sales and prices. In many instances, this could be a real estate information company or your own loan pipeline. In terms of sample size, you should secure enough data to equal between 10 and 20 percent of the residential purchase transactions funded in that county each month.

Based upon this data, determine whether the representation of accuracy provided by the AVM in your market is consistent with your results from the localized blind test. If the results are consistent, then your due diligence mission is accomplished for that AVM. Repeat this process for multiple AVMs. If the results are inconsistent among one particular brand, reject that AVM in favor of brands that have results that you can support via your own testing efforts.

For further information, please contact Robert L. Walker, CMB, CMT, executive vice president collateral solutions for First American Real Estate Solutions at 714-250-6684; or by email at robwalker@firstam.com.

About First American Real Estate Solutions
First American Real Estate Solutions is a member of The First American Family of Companies and America’s largest provider of advanced property and ownership information, analytics and services. First American RES’ databases cover more than 2,900 counties representing 99 percent of the United States population. With more than 600,000 users nationwide, First American RES products are used by companies to improve customer acquisition and retention, detect and prevent fraud, improve mortgage transaction cycle time and cost efficiency, measure the value of residential and commercial properties, identify real estate trends and neighborhood characteristics, track market performance and increase market share. First American RES is a joint-venture company 80-percent-owned by The First American Corporation and 20-percent-owned by Experian. More information about First American RES can be found on the Internet at www.firstamres.com.