OpenCV 3.4.9 for Android with a little bits of native C++ code

Sun, 12 Jan 2020 in howto, opencv, android, cpp, jni

Another task on the horizon: integrate OpenCV 3.4.9 into an Android application written in Kotlin. Additional requirement complicated this task a bit. Let’s also try to have some C++ code around, that is shared with other platforms and that interacts and configures OpenCV video processing. Of course nobody expects any rough edges nor problems, right…

One might think it’s a typical scenario and lots of mobile apps use OpenCV this way. So it should be well documented, confirmed by official tutorials or sample codes, explanations or wiki pages and an obvious step would be just to open them and job done. That, I must admin it, was a terribly missed idea, since I haven’t found there anything valid. Among tutorials there isn’t a single one dedicated for Android. Also based on the doxygen CSS-style I immediately felt like in ‘90 of previous century.

The single information I took from those sites is that I need a dedicated SDK, called OpenCV4Android and binaries can be found in Releases section. If you do the same mistake twice and open the Android tutorials from mentioned OpenCV4Android page, that remember Eclipse IDE days of 2013, it means you have learned nothing so far! There are however plenty of links to Android-platform documentation, or some general JNI best-practices (better here), 2 archaic samples (Red-eye detection or Google Glass ya-da-ya-da SDK, which both are really old), but not a single word, how to configure the Android Studio, nor Gradle project! No single word, how to pack it into AAB or generate multiple APKs, each for specific ABI to reduce their size (because all the opencv*.so shared libraries for all supported CPU architectures consume 120MB! and could be more, if you start using other extensions). Instead they convince to install OpenCV Manager and use its shared-libraries to reduce occupied space, but as of Jan-2020, this app is no longer accessible in Google PlayStore. I am really sorry, if you had to dig it all though…

Well, turned out I even had a book Android Application Programming with OpenCV, received one day via PacktPub Free-learning program, but it turned out to be from 2013, using OpenCV 2.4.5 and again based on Eclipse-type of projects. So again, no go!

OK, man, stop whining and do something useful. This project itself is helpful so there must be a way!

Last try - let’s use Google.com and GitHub.com to check, if we can find something newer created by the community. And thankfully I found those 3 gem-like projects that one-by-one step-by-step showed me, how to configure the stuff in detail. I am really grateful authors for their publications, as I was really loosing any hope up till now. And my mix of all their techniques I present later in this post.

Camera Calibration Project - basics, what should be copied and where to have OpenCV and Camera1 API working
Native OpenCV with Android Studio - advanced, using OpenCV 3.4, Camera2 API, uses C++ code in application module just for spicing
Native OpenCV for Android with Android NDK - advanced, using OpenCV 4.1, Camera2 API, passes video-frames via JNI call and hosts OpenCV outside of the repository; it’s actually improved version of previous example.

Then, to complete the big-picture I highly recommend reading following articles, that I discovered meanwhile:

LFS

All mentioned above projects show some valid aspects of the task and combining them together could bring a really nice result I could be happy with. However after thinking a bit, there is still one subject that I haven’t said anything - GiT Large File Storage (tutorial). Storing OpenCV4Android outside of the repository looks very promising, but in turns complicates Continuous Integration builds. Having the huge opencv*.so shared-library binaries (and updating them in the future) could very quickly bloat the whole repository and make version browsing very cumbersome or impossible. LFS solves both of this issues - by keeping the contents of big files outside of the regular history of GiT repository. In practice LFS mounts some hooks into git system and versions lightweight “pointers” (along with hashes) to selected files and replaces those pointers with expected content loaded directly from remote server each time a branch is checked out and “pointer” stored on that branch indicates to different version of the file’s content.

Configure Android project with OpenCV and C++

Download OpenCV4Android 3.9.4
Extract it to any OPENCV_SDK_PATH (e.g. T:\codetitans\OpenCV-android-sdk)

Create empty GiT repository (outside of the OpenCV4Android) for the Android project in Kotlin

> mkdir opencv_app && cd opencv_app
> git init .
Initialized empty Git repository in T:/codetitans/opencv_app/.git/

Install LFS support inside the repository

> git lfs install
Updated git hooks.
Git LFS initialized.

Create new regular Android application with Kotlin as default language
Tips: I used here APIv21 to start with the introduction of Android Camera2 API.
Click FINISH to complete the wizard.

Add project to GiT

> git add src/
> git commit -m "Created empty project"
[master (root-commit) 9188a11] Created empty project
40 files changed, 892 insertions(+)
.
.
.
create mode 100644 src/settings.gradle

Import Java/JNI library from extracted OPENCV_SDK_PATH > sdk > java and name it libOpenCV module
Navigate to File > Import Module....
And follow the wizard.
Now, it’s a good time to update module’s build.gradle file and set Android SDK properties identically to the app module.
```
android {
    compileSdkVersion 29
    buildToolsVersion "29.0.2"

    defaultConfig {
        minSdkVersion 21
        targetSdkVersion 29
    }
    .
    .
    .
}
```
Then remove <uses-sdk /> tag from libOpenCV > src > main > AndroidManifest.xml. As this is some Eclipse-type project leftover, not needed, when building with Gradle.

Add new module to GiT

First ask GiT to ignore folder with binaries:

 > echo "/libOpenCV/build/" >> src/.gitignore

Commit module

> git add src/
> git commit -m "Imported libOpenCV module"
[master 5ce6960] Imported libOpenCV module
144 files changed, 62007 insertions(+)
create mode 100644 src/import-summary.txt
create mode 100644 src/libOpenCV/build.gradle
create mode 100644 src/libOpenCV/lint.xml
create mode 100644 src/libOpenCV/src/main/AndroidManifest.xml
create mode 100644 src/libOpenCV/src/main/aidl/org/opencv/engine/OpenCVEngineInterface.aidl
create mode 100644 src/libOpenCV/src/main/java/org/opencv/android/AsyncServiceHelper.java
create mode 100644 src/libOpenCV/src/main/java/org/opencv/android/BaseLoaderCallback.java
.
.
.
create mode 100644 src/libOpenCV/src/main/java/org/opencv/videoio/VideoWriter.java
create mode 100644 src/libOpenCV/src/main/java/org/opencv/videoio/Videoio.java
create mode 100644 src/libOpenCV/src/main/res/values/attrs.xml

Copy OpenCV selected shared-libraries (see note at the end) from OPENCV_SDK_PATH > sdk > native > libs into the openCVLibrary project > sdk > libs folder

Add copied shared-libraries .so to LFS

> git lfs track src/libOpenCV/sdk/**/*.so
Tracking "src/libOpenCV/sdk/**/*.so"

> git add src/libOpenCV/sdk
> git add .gitattributes
> git commit -m "Added OpenCV native libraries"
[master fe090ef] Added OpenCV native libraries
5 files changed, 13 insertions(+)
create mode 100644 .gitattributes
create mode 100644 src/libOpenCV/sdk/libs/arm64-v8a/libopencv_java3.so
create mode 100644 src/libOpenCV/sdk/libs/armeabi-v7a/libopencv_java3.so
create mode 100644 src/libOpenCV/sdk/libs/x86/libopencv_java3.so
create mode 100644 src/libOpenCV/sdk/libs/x86_64/libopencv_java3.so

As a confirmation, that instead of big binary content - a small “pointer” was added into repository, run gitk and notice the details consisting of a spec version, SHA256 and original size in bytes.

LFS Commit

Depend app on libOpenCV module
Simply navigate to File > Project Structure...
There select Dependencies > app > Module Dependency

Configure app module to include OpenCV’s shared-libraries via build.gradle, and also to build separate APKs for selected processor architectures (multiple ABIs can be comma-separated) for testing:

android {
    .
    .
    .
    splits {
        // multiple APKs based on ABI
        abi {
            enable true
            reset()
            include "x86" // just build these!
            universalApk false // disable the one with all ABIs
        }
    }

    sourceSets {
        main {
            jniLibs.srcDirs = ['../libOpenCV/sdk/libs']
        }
    }
}

Setup correctness on this stage could be checked, if APK is build via Build > Build Bundle(s) / APK(s) > Build APK(s) and produces a single app-x86-debug.apk file under project’s location T:\codetitans\opencv_app\src\app\build\outputs\apk\debug.

Commit all changes

> git add src/
> git commit -m "Added libOpenCV dependency"

Make sure NDK is installed along with cmake and lldb.
Navigate to Tools > SDK Manager and verify checkboxes next to mentioned items. If not, select those items and hit Apply button.
Copy OpenCV’s cmake files from OPENCV_SDK_PATH > sdk > native > jni into the libOpenCV project > sdk > jni folder

Add those files regularly to GiT (as they are small enough)

> git add src/libOpenCV/sdk/jni/
> git commit -m "Added JNI cmake files"
[master f31fdc9] Added JNI cmake files
296 files changed, 128308 insertions(+)
create mode 100644 src/libOpenCV/sdk/jni/OpenCV-arm64-v8a.mk
create mode 100644 src/libOpenCV/sdk/jni/OpenCV-armeabi-v7a.mk
create mode 100644 src/libOpenCV/sdk/jni/OpenCV-x86.mk
create mode 100644 src/libOpenCV/sdk/jni/OpenCV-x86_64.mk
create mode 100644 src/libOpenCV/sdk/jni/OpenCV.mk
.
.
.
create mode 100644 src/libOpenCV/sdk/jni/include/opencv2/videostab/ring_buffer.hpp
create mode 100644 src/libOpenCV/sdk/jni/include/opencv2/videostab/stabilizer.hpp
create mode 100644 src/libOpenCV/sdk/jni/include/opencv2/videostab/wobble_suppression.hpp

Add another module into the project (libOpenCvLogic - to keep whole future C++ code).

Configure libOpenCvLogic to contain C++ code and operate with OpenCV natively

To do so, create a file libOpenCvLogic > src > main > cpp > CMakeLists.txt with following content:

# For more information about using CMake with Android Studio, read the
# documentation: https://d.android.com/studio/projects/add-native-code.html

# Sets the minimum version of CMake required to build the native library.

cmake_minimum_required(VERSION 3.4.1)
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/../../../../libOpenCV/sdk/jni/include)
add_library( lib_opencv SHARED IMPORTED )
set_target_properties(lib_opencv PROPERTIES IMPORTED_LOCATION ${CMAKE_CURRENT_SOURCE_DIR}/../../../../libOpenCV/sdk/libs/${ANDROID_ABI}/libopencv_java3.so)

# Creates and names a library, sets it as either STATIC
# or SHARED, and provides the relative paths to its source code.
# You can define multiple libraries, and CMake builds them for you.
# Gradle automatically packages shared libraries with your APK.

add_library( # Sets the name of the library.
        my-opencvlogic-extensions

        # Sets the library as a shared library.
        SHARED

        # Provides a relative path to your source file(s).
        test.cpp
        ****list here all C/C++ files to compile ****
        )

# Searches for a specified prebuilt library and stores the path as a
# variable. Because CMake includes system libraries in the search path by
# default, you only need to specify the name of the public NDK library
# you want to add. CMake verifies that the library exists before
# completing its build.

find_library( # Sets the name of the path variable.
        log-lib

        # Specifies the name of the NDK library that
        # you want CMake to locate.
        log)

# Specifies libraries CMake should link to your target library. You
# can link multiple libraries, such as libraries you define in this
# build script, prebuilt third-party libraries, or system libraries.

target_link_libraries( # Specifies the target library.
        my-opencvlogic-extensions

        # Links the target library to the log library
        # included in the NDK.
        ${log-lib}
        lib_opencv)

And then extend the module’s build.gradle file with configuration of required native build:

android {
    .
    .
    .
    defaultConfig {
        .
        .
        .
        externalNativeBuild {
            cmake {
                cppFlags "-frtti -fexceptions"
            }
        }
    }
    .
    .
    .
    externalNativeBuild {
        cmake {
            path "src/main/cpp/CMakeLists.txt"
            version "3.10.2"
        }
    }
}

At this stage it’s very important to have at least a single C/C++ source file created and included inside CMakeLists.txt. This file can be even empty. Otherwise the Android Studio will behave strangely and will hide whole cpp branch in project source-tree.

It behaves similarly wrong, when there were typos in the CMakeLists.txt.

Tip: cmake configuration already defines folders for OpenCV headers from libOpenCV/sdk/jni/include, shared libraries and dependant cmake files to allow further compilation.

Depend app module on libOpenCvLogic
Tip: again, creating x86 APK (and changing it’s extension to .zip), could reveal, that there are now two native libraries (in lib folder) possibly loaded by the application: libmy-opencvlogic-extensions.so and libopencv_java3.so.

Add new module to GiT

> git add src/
> git commit -m "Added libOpenCvLogic module"

Build the project and now you are ready to add you code into the project. If you need something for startup go to Native OpenCV for Android with Android NDK.
Done! But that was far too long!

Now, if you need some solid JNI knowledge - refer to JNI Cookbook and its samples.

Note

Finally one note about native libraries. Since Android 4.4 (API level 19), NDK r17 and Android Plugin for Gradle 3.0.1 the following ABIs are not supported anymore: armeabi (aka ARMv5/ARMv6), mips and mips64 (docs and here). All the effort should go to support ARMv7/ARM64 and x86/x64 architectures (docs).