Live text and visual search: Use your iPhone camera for more than just photos

Apple’s Photos app offers incredibly powerful technology for what, at a glance, looks like an indescribable photo library. The app has well-known AI capabilities that come with a host of editing features. Not only can Photos crop, enhance, sharpen, and otherwise edit your photos, but AI recognizes text in images with Live Text, letting you highlight, copy, and even translate words. With iOS 16, this feature is even more advanced, allowing you to highlight text in videos as well.

That same technology powers Apple’s Visual Look Up, a new iOS 16 feature that lets you identify and isolate the subject of a photo from its background. Moreover, it allows you to discover more information about the said subject by extracting data from the Internet and Siri. On a superficial level, Visual Look Up can highlight the subject of a photo and copy and paste it into another app, such as Mail or Messenger. However, Visual Look Up has enormous potential; although still in its infancy, its practical applications are impressive.

Live text and visual search are by far the most notable features of iOS 16, and are the ones we had the most fun exploring in Apple’s mobile operating system. These are our first thoughts.

(Credit: PCMag)

Live text gets new tricks

Live Text lets you copy and share text located in photographs. The AI ​​identifies key terms in the photo and suggests relevant functions. To test this, I photographed a page from Julia Child’s Master the art of French cuisine. Looking at the images in the Photos app, I tapped the Live Text icon in the lower right corner of the screen, which highlighted the text. Measurement conversion options have appeared at the bottom of the photo for some of the key instructions. They displayed and converted the default cooking temperature from Fahrenheit to Celsius, and changed cup measurements to imperial units, such as grams and liters. If you’re passionate about meal planning, this is a great tool to have at your disposal. This is just one example, but Live Text can also identify addresses, contact details and foreign languages ​​in photographs to offer relevant information. It is an extremely practical tool.

That said, Live Text isn’t new to the iPhone, but it gets improvements in iOS 16. In particular, you can now use Live Text with recorded video. To test this, I recorded a video presentation of my home office. The AI ​​had trouble identifying text if the camera was moving too fast. Live Text couldn’t glean much from slightly blurry little words. In such cases, Live Text simply does not offer suggestions and may not even allow you to highlight text.

This works best when the iPhone is used to focus on a particular subject or area. For example, a slow pan over my desk drew in a treasure trove of information. My iPhone 13 Pro Max offered map links for addresses, as well as call and message options for phone numbers it identified. Even if the recognized text wasn’t associated with a specific app or tool, I could still highlight it and share it with other apps, such as Mail, Messages, and Notes.

Visual Look Up recognizes monuments and landmarks

(Credit: PCMag)

The magic of Visual Look Up

Visual Look Up is a new Photos feature introduced in iOS 16 that offers similar functionality to Live Text, but for the subject of a photo instead of text. You can highlight a person or object of interest in a photo by pressing and holding your finger on it. You can then copy, share or save the highlighted topic. It’s an amazing addition to the iOS ecosystem, giving you a selection of Photoshop-like subjects without needing to open a Photoshop program. You can create a collage or make crazy stickers of your cat.

In addition to copying images, Visual Look Up lets you run an online search based on the subject of the photo (provided the AI ​​can correctly identify it). It works with common objects in photographs, such as animals, art, landmarks, pets, plants, and vehicles. If the AI ​​identifies objects in a photo, you’ll see a star graphic on the information icon at the bottom of the screen. Simply swipe up on the photo or tap the information icon to reveal basic photo details, plus any additional Visual Look Up content available.

It works surprisingly well, although it has some limitations. When used with photos of a Japanese skyline, Visual Look Up correctly identified the Tokyo Sky Tree landmark, offering basic Wikipedia information via Siri Knowledge. It also provided Maps data which included distance, opening hours, contact number and ticket information. When it works, it’s almost magical.

Recommended by our editors

In another test, I took a picture of a wasp parasitized tomato worm in my garden. Admittedly, this test subject was a curveball, and something I had never seen firsthand. Visual Look Up misidentified the dead caterpillar as a gourd or bitter melon due to its coloring and overall shape. No bad guesses, but still off the mark.

Visual Look Up can also be used for illustrations, but with much less precision than photos. It easily identifies famous paintings, such as that of Vincent van Gogh Starry Night. However, when it comes to more modern works of art, the functionality becomes noticeably less precise. For example, Visual Look Up recognized an artwork from Final Fantasy IX, but incorrectly identified the artist who created it. In another example, I’ve saved art from artist Jun Suemi, who created some awesome paints for the game Brandish. Visual Look Up recognized the image as a work of art, but could not give me any results for the artist or source. In some cases, Visual Look Up was completely unavailable. Many of my family photos have fallen into this category.

PC Mag Logo Handy with the Apple iPhone 14 range

These technologies are great, but Apple has limited these features to certain handsets. First, Live Text and Visual Look Up are only available on iPhone 11, iPhone 12, iPhone 13, iPhone 14, iPhone XR, and iPhone XS. Additionally, Live Text is limited to specific languages, including English dialects, French, German, Japanese, Korean, and Spanish. You can see the specific languages ​​supported by Live Text here(Opens in a new window). Finally, Visual Look Up is also limited by region and language, which you can learn more about here(Opens in a new window).

If you want to know more about iOS 16, read our impressions. If you’re an iPad enthusiast, many of the features of iOS 16 carry over to the tablet. which you can read about in our iPadOS 16 preview.

Apple fan?

Subscribe to our Weekly Apple Brief for the latest news, reviews, tips and more straight to your inbox.

This newsletter may contain advertisements, offers or affiliate links. Signing up for a newsletter indicates your consent to our Terms of Service and Privacy Policy. You can unsubscribe from newsletters at any time.

About Geraldine Higgins

Check Also

Florida begins to prepare for Tropical Depression Nine, governor declares state of emergency

Tropical Depression Nine is currently forecast to make landfall off the southwest coast of Florida …