We are happy to announce a new release of our Web SDK. It takes a more modern approach to web development and aims to ease the integration with your web apps. This blog post will list all the exciting updates ^v4.0.0 has to offer.

‍

‍

About DeepAR Web

DeepAR makes it easy to add augmented reality features to any website.  From virtual try-on for makeup, sneakers, or glasses to removing background from a video call, augmented reality is fast becoming a staple ingredient to modern web interactions.  

Our demo projects can be used to get AR up and running on iOS, Android, MacOS or the web in under 30 minutes.  We have hundreds of readymade AR effects you can use to create a variety of experiences to sell, entertain and engage.  Using DeepAR Creator Studio, you can create custom AR effects with the features of those on Instagram or SnapChat and more.

TLDR

• DeepAR Web is now available on NPM.
• DeepAR Web is now written in Typescript.
• API reference and documentation inside your IDE.
• DeepAR JS library is now available as an ES module.
• Easy integration with bundlers like Webpack and Rollup.
• DeepAR Web now expects paths for all asset files, like ^deepar.wasm.
• 40% smaller javascript file and 20% smaller WebAssembly file.
• Performance and stability improvements.
• DeepAR preview can work even if the browser tab is not focused.
• We have some breaking changes. See migrating to ^v4.0.0 for more info.

‍

Public NPM Package

NPM is a public repository of many Web frameworks and libraries. Publishing code to NPM will massively increase the availability of the DeepAR Web library and improve the ease of the integration process for new developers.

👉 Yeah, you heard that right. DeepAR Web is now available on NPM.

‍

You are just one command away from installing the most powerful AR framework for the Web.

   
       npm install deepar
    
  

‍

And 13 lines of code away from running your first demo. It's that easy!

   
        import { DeepAR } from 'deepar';
        const deepAR = new DeepAR({
          licenseKey: 'your_license_key_here',  
          canvas: document.getElementById('deepar-canvas'),  
          deeparWasmPath: 'path/to/deepar.wasm',
          callbacks: {     
              onInitialize: () => {         
                  deepAR.startVideo(true);
                  deepAR.switchEffect(0, 'mask', 'path/to/super_cool_mask_filter');
              } 
          }
        });
        deepAR.downloadFaceTrackingModel('path/to/models-68-extreme.bin');
    
  

‍

Javascript Modules

For a long time, Javascript did not natively support having code separated into multiple files (modules). Having the ability to organize code into files helps to divide functionality into smaller, more manageable,  parts. This comes bluntly apparent for any project larger than 400 lines of code. If a project is developed by multiple people, this is a must.

👉 The new DeepAR Web is now an ES module. From a practical perspective, this means that it works nicely with Javascript bundlers like Webpack and Rollup.

‍

Bundlers are Javascript’s non-official answer to separating code into modules. Let’s look at the DeepAR example.

DeepAR Web comes with ^{deepar.esm.js} file. It contains all the functionality of the DeepAR SDK in one file. Most web apps compose of many libraries and files. To import DeepAR functionality into your code:

   
        import { DeepAR } from 'deepar.esm';
    
  

‍

Here is an example of how to use ES modules directly in HTML:

   
        <script type="module">    
          import { DeepAR } from 'path/to/deepar.esm.js';    
          let deepAR = new DeepAR({ ... });
        </script>
    
  

‍

‍

Types, Definitions, Autocomplete, and Inline Docs

If there is one thing that all developers love, it is when IDE helps them to be more productive. This is the main reason we rewrote our entire Web SDK.

👉 DeepAR Web is now written in Typescript.

‍

Every Typescript app will now have a straightforward integration with DeepAR Web. For all you Javascript folks, fear not. Because you also will reap most of the benefits that come with a Typescript library.

Typescript code is transpiled into Javascript code. Therefore both Typescript and Javascript projects can consume it. But alongside emitted Javascript code, ^tsc generates Typescript declaration files.

📓 transpile - fancy combination of words transform and compile.

^tsc - a Typescript compiler.

‍

What is a Typescript Definition File?

DeepAR Web comes with a pair of these files:

• ^deepar.js
• ^deepar.d.ts

Extension ^.d.ts denotes a Typescript definition file. It describes all the classes, methods, and properties of the ^deepar.js file. How can this help a developer?

• You can inspect a ^.d.ts file and see all the available API, what classes and types are exported, how methods are called, and what their parameters are. All this without looking at the documentation.
• An IDE will pick up and parse the definition file resulting in code auto-completion.
• The definition file also contains the documentation of all the methods and properties. IDE can nicely preview the docs for you.

💡 In conclusion: with Typescript definition ^.dt.s files you have API reference and autocomplete inside your IDE.

‍

Bundler-Friendly API

Webpack and Rollup merge code into a single bundle. By default, they rename and place assets in different directory structures. Previous versions of SDK had problems when integrating with bundlers. The main issue was with fetching the WebAssembly and machine learning models.

In previous versions of SDK, DeepAR assumed the paths for those assets and developers could not change them. This changes with DeepAR ^v4.0.0.

👉 DeepAR Web now expects paths for all asset files.

‍

Webpack ↔ DeepAR Example

A cool example here is using Webpack’s asset modules in tandem with deepar npm package. Add this to your ^{webpack.config.js}:

   
      module.exports = {
        // ...
        module: {
          rules: [
            {
              test: /\.(wasm)|(bin)|(obj)$/i,        
              include: [
                path.resolve(__dirname, 'node_modules/deepar/'),        
              ],
              type: 'asset/resource',      
            },      
            {
              include: [
                path.resolve(__dirname, 'effects/'),        
              ],        
              type: 'asset/resource',      
            },    
          ],  
        },
        // ...
    
  

‍

Then you can include DeepAR SDK and its assets:

• ^deepar.wasm - WebAssembly file containing most DeepAR functionality that enables it to run fast in the browsers.
• ^{models-68-extreme.bin} - Machine learning model for face tracking.
• Effect files - All 3D masks, filters, background removal feature, and so on are represented by effect files in DeepAR. You can load them into DeepAR to preview the effect. Download a free filter pack here.

  	
      import { DeepAR } from 'deepar';
      import deeparWasmPath from 'deepar/wasm/deepar.wasm';

      import faceTrackingModelPath from 'deepar/models/face/models-68-extreme.bin';
      import someEffect from './path/to/effect_file';
  	
  

‍

Then initialize DeepAR:

   
      const deepAR = new DeepAR({
        licenseKey: 'your_license_key_here',
        canvas: document.getElementById('deepar-canvas'),
        deeparWasmPath,
        callbacks: {
          onInitialize: function() {
            deepAR.startVideo(true);
            deepAR.switchEffect(0, 'mask', someEffect);
          },
        },
      });

      // Download the face tracking model. This is required in order to track face.
      deepAR.downloadFaceTrackingModel(faceTrackingModelPath);
    
  

‍

Web Engine Upgrades

DeepAR SDK is a multi-platform framework working across all modern platforms:

• iOS
• Android
• Web
• MacOS

With support for some major mobile frameworks:

• Flutter
• Unofficial react-native support

DeepAR core SDK developers have an unthankful job of developing and maintaining all the fun features that clients expect across a range of very different platforms. To make the development and maintenance of DeepAR more manageable and productive, core devs had to do some major refactoring and introduce some upgrades on the Web side of SDK.

🚘 The DeepAR car did not get just a shiny new body and bumpers. It also got a massive engine upgrade!

‍

‍

Emscripten Upgrade

The core of the DeepAR SDK is written in C++ and we compile it down to WebAssembly with Emscripten. After a few years, we finally managed to upgrade our Emscripten toolchain to the latest version. What held us back from upgrading earlier is the inability to compile some C++ dependencies with the new Emscripten toolchain because we did not have an access to the code of those dependencies.

In addition to some of our internal code optimizations and code size analysis, these are the results:

🔎 40% smaller javascript file and 20% smaller WebAssembly file → Faster website loading!

🔥 Performance and stability improvements → More FPS!

‍

The camera preview freezes when the browser tab does not have a focus.

Some web apps need the camera and effects preview from DeepAR even when the browser tab is not focused. A typical example is a video calling app that utilizes background blurring. When you are in a video call, you will at one point need to switch browser tabs to check something on the web. The last thing you want to happen is for your camera preview to freeze.

❄️ The new DeepAR Web SDK can render in the background even if the browser tab is not focused.

‍

A typical app that does some processing and rendering has something called a game loop. In web browsers, the game loop cannot be infinite since the app needs to yield the execution back to the browser for a preview of the web page.

DeepAR renders its preview to an HTML canvas element. Before rendering it needs to do face tracking and calculate the engine state. All this happens on callbacks from ^{requestAnimationFrame} by the browser. Unfortunately, ^{requestAnimationFrame} stops calling when the tab is not focused to save up on processor consumption.

The natural choice seems to use the ^setTimeout function and call the game loop update function 30 times a second (if you want 30 FPS). But this also doesn’t yield good results since browsers don’t call ^setTimeout with given intervals when the tab is not focused. It is called with much longer intervals and you end up with 5 FPS instead of 30 FPS.

So how do we solve this issue? There is one funny hack. What can play in the background even when the tab is not focused? Don’t know? Sound!

We used Web Audio API to connect the Gain node and Oscillator node. We play the oscillator node 30 times a second and attach a callback every time the sound playing ends. The callback is attached to the game loops update function. Now you are asking yourself, will I be hearing a high-frequency beeping when I use DeepAR in the unfocused tab? Of course not, we set the gain of the gain node to ⁰, you silly!

We used 30 FPS here as an example. DeepAR by default previews in 30 FPS but that can be configured via API.

An example of this in action is our Vonage video calling API integration example. Check it out here. The DeepAR API to enable this is:

   
       deepAR.setOffscreenRenderingEnabled(true);
    
  

‍

How to use Javascript library inside C++ with Emscripten

Some time ago we transitioned from the native C++ TensorFlow runtime to the TensorFlow.js runtime for performance boost in browsers. This made our code scattered between cumbersome C++ and JS interop. We no longer had calls from C++ code to do inference on the camera image. Rather, all the logic was written in JS and passed results down to C++ which consumed them by awkwardly reading and setting some global flags. This makes the code completely incomprehensible unless you knew exactly what was happening.

We don’t have a problem with writing the logic in JS but communicating with global flags is a big no-no! What we wanted is to call the JS code directly from C++ with standard C functions.

We had the TensorFlow.js library wrapped in our internal tracking JS library that was created with a bundler. Here is how we used that JS library in our Emscripten toolchain and called the JS logic directly from C++.

DeepAR Web is finally created with Webpack. Inside our client-facing JS code, we import the internal library and pass it to Emscripten via module. To enable this you need to enable ^MODULARIZE and ^EXPORT_ES6 features of Emscripten (see settings.js).

   
      import { A, B, C } from 'internal_js_library.js';
      import Module from 'emitted_emscripten_js_code.js';

      // When initializing Emscripten Module
      Module({
        // Pass the library exports inside the Module
        internal_lib: { A, B, C }
      }).then((module) => {
        // Use emitted functions inside module
      });
    
  

‍

Then inside your Emscripten ^pre.js add the next line to make the library exports accessible across all your Emscripten js code.

   
       var internal_lib = Module['internal_lib'];
    
  

‍

Now you can access the library anywhere in your JS code like this:

   
      internal_lib.A.foo(args...);
      internal_lib.B.bar(args...);
      internal_lib.C.baz(args...);
    
  

‍

Considering you are generating production-level code, you will try to optimize and reduce emitted JS code size with ^emcc flags ^-Os or ^-Oz. Emscripten will use Google’s closure compiler for this job. The closure will do all sorts of heavy optimizations that you need to keep in mind otherwise your code will not work (see Google Devs). In this instance, closure will rename ^foo, ^bar, and ^baz function calls and in runtime, it would throw an exception saying that no such functions exist. To fix this you can do the following:

‍

1. Call the functions with strings because Closure never renames strings.

     
        internal_lib.A.[’foo’](42, true);
     
  

‍

2. Create a closure externs file.

   
      /**
      * @param {number} a
      * @param {boolean} b
      * @return {string}
      */
      internal_lib.A.foo = function(a, b) {};
    
  

‍

Then we wrapped the JS logic that called our internal library inside Emscripten’s js-library code. That enabled us to have Javascript functions that we could call directly from C++ code.

‍

Migrating to ^v4.0.0 from Previous Versions

Building new technology sometimes means that you have to break the older tech. We are always sad when we have to break backward compatibility with previous versions of SDK. It was a tough decision but we finally made it because it is a step in the right direction.

📌 All changes are made to help developers with easier and more standard/modern integration.

‍

Don't be intimidated! Changes are not big.

We prepared a handy document that lists all the things you need to know when migrating to DeepAR Web ^v4.0.0 from previous versions.

⏱ Migrating to the new version takes 5-10 minutes.

‍

That's all folks,

‍

DeepAR Team 💙

‍