Facebook is giving away the software it uses to understand objects in photos

Facebook is open sourcing a set of computer vision software tools that can identify both the variety and the shape of objects within photos. The tools, developed by the Facebook AI Research (FAIR) team, are called DeepMask, SharpMask, and MultiPathNet, and all three work in tandem to help break down and contextualize the contents of images. These technologies, though not in active use in consumer Facebook products right now, are similar to the software the company uses to describe photos to blind users, a feature it calls “automatic alternative text” that launched back in April. DeepMask and SharpMask are more experimental research projects and focus on what the FAIR team calls segmentation. While human beings can discern the various elements of a photograph in mere seconds, the process is much harder for computers, which perceive pixels as a series of number values corresponding to changes in color. It’s not easy then to help software make sense of where an image’s background becomes its subject, or which parts of the foreground can be broken down into distinct objects. It’s also difficult to then have the computer identify the objects correctly.”