Enforcement beyond ALPR

When automatic license plate recognition fails, an image retrieval system from Brazil is helping with vehicle search and identification.

Automatic license plate recognition (ALPR) is a widely deployed technology for vehicle identification, used in traffic surveillance, automatic access control, and toll collection systems worldwide. However, in some situations ALPR can fail, or is not even an option.

A complementary method, called content-based image retrieval, can be used for those situations where ALPR fails or cannot be used for some reason. ALPR has plenty of drawbacks. First of all, it requires a visible and legible license plate for successful recognition to take place. If the plate is dirty, bent, or obstructed by road debris such as paper or plastic bags, recognition might fail. For these and other reasons, recognition rates for current systems are around 98%. Second, ALPR systems are easily deceived, for example by painting over some of the characters on a license plate. Third, reading a certain license plate does not guarantee that the vehicle or driver is the one registered for that license plate – the plate could be entirely fake.

98%: The standard recognition rate for ALPR cameras. Content-based image retrieval can help identify the missing 2%.

All of these failures can be intentionally caused by criminals. For example, in Brazil it is not uncommon to see trucks going through automatic toll lanes with their license plates deliberately obstructed, in order to avoid paying the toll, while also avoiding a ticket due to the violation.

What we see, in fact, is that with the proliferation of ALPR cameras, traffic law violators are adapting and always finding new ways of deceiving identification hardware. In the context of public security, it is desirable to have other means of automatically identifying the vehicle in such situations. Content-based image retrieval methods extract features from the vehicle’s image, such as its color, make, model and other visual characteristics, which can then be used to find the same vehicle in other images, possibly taken from different cameras. In some of those other images the license plate might be visible, which would enable easy identification. Even if none of the images allow recognition of the vehicle’s license plate, a real-time system that offers matching vehicle images based on visual features could, for example, alert a police officer when a vehicle known to have committed a traffic violation has been found nearby.

How does it work?

Image retrieval methods work as follows. When an image of a vehicle is captured, an algorithm encodes its visual features into a compact representation, often called a signature. The encoding process is designed to ensure that similar vehicles have similar signatures and that different vehicles have different signatures. Signatures can be compared very efficiently, which enables a quick search for a particular signature among millions of images, even with a regular desktop computer.

Whenever an unidentified vehicle needs to be identified, its signature is searched for in a database of similar signatures from other images. Those signatures are ranked according to similarity to the query signature, and the top-ranking ones are very likely to be the same vehicle, or at least very similar vehicles. At this point, a human operator can triage the top few results for true matches. Whether such a system can be fully automated depends on the level of precision required. For public security applications, a human-in-the-loop will probably be required to remove false positive matches.

Working together

Image retrieval does not make ALPR systems obsolete. On the contrary, they complement each other very well. About 98% of vehicle images will have the license plate read correctly by a properly designed ALPR system. In the remaining 2% of cases, image retrieval can then be used to attempt matching and identification of the vehicle in question. Here, ALPR data becomes essential to reduce the search space by filtering out vehicles that have already been recognized elsewhere and that, due to timing and spatial constraints, cannot possibly be the vehicle that is being searched for. The smaller the geographical search area, the higher the probability that true matches will be ranked at the top of the search results given by the system. On the other hand, due to the large number of practically identical vehicles of the same make, model and color, the recognition rate of ALPR will always be higher than that of any image retrieval system.

A human operator can pick out true matches from the top results given by the systems.

Another feature of image retrieval is that it can be useful even when a query image is not available.

For instance, suppose we only have a verbal description of an unrecognized vehicle used in a crime scene – a blue sedan, a green van, etc. By automatically clustering signatures according to broad characteristics such as color and shape, all images of vehicles that match the description can be quickly found.

In the end, ALPR will no doubt continue to be an essential technology for traffic surveillance and enforcement of all types, but complementary methods will have an important role in handling more difficult and complex requirements – such as finding a vehicle without a visible and legible license plate.