Coding with Titans

so breaking things happens constantly, but never on purpose

OpenCV 3.4.9 for Android with a little bits of native C++ code

Another task on the horizon: integrate OpenCV 3.4.9 into an Android application written in Kotlin. Additional requirement complicated this task a bit. Let's also try to have some C++ code around, that is shared with other platforms and that interacts and configures OpenCV video processing. Of course nobody expects any rough edges nor problems, right…

One might think it's a typical scenario and lots of mobile apps use OpenCV this way. So it should be well documented, confirmed by official tutorials or sample codes, explanations or wiki pages and an obvious step would be just to open them and job done. That, I must admin it, was a terribly missed idea, since I haven't found there anything valid. Among tutorials there isn't a single one dedicated for Android. Also based on the doxygen CSS-style I immediately felt like in ‘90 of previous century.

The single information I took from those sites is that I need a dedicated SDK, called OpenCV4Android and binaries can be found in Releases section. If you do the same mistake twice and open the Android tutorials from mentioned OpenCV4Android page, that remember Eclipse IDE days of 2013, it means you have learned nothing so far! There are however plenty of links to Android-platform documentation, or some general JNI best-practices (better here), 2 archaic samples (Red-eye detection or Google Glass ya-da-ya-da SDK, which both are really old), but not a single word, how to configure the Android Studio, nor Gradle project! No single word, how to pack it into AAB or generate multiple APKs, each for specific ABI to reduce their size (because all the opencv*.so shared libraries for all supported CPU architectures consume 120MB! and could be more, if you start using other extensions). Instead they convince to install OpenCV Manager and use its shared-libraries to reduce occupied space, but as of Jan-2020, this app is no longer accessible in Google PlayStore. I am really sorry, if you had to dig it all though…

Well, turned out I even had a book Android Application Programming with OpenCV, received one day via PacktPub Free-learning program, but it turned out to be from 2013, using OpenCV 2.4.5 and again based on Eclipse-type of projects. So again, no go!

OK, man, stop whining and do something useful. This project itself is helpful so there must be a way!

Last try - let's use Google.com and GitHub.com to check, if we can find something newer created by the community. And thankfully I found those 3 gem-like projects that one-by-one step-by-step showed me, how to configure the stuff in detail. I am really grateful authors for their publications, as I was really loosing any hope up till now. And my mix of all their techniques I present later in this post.

Then, to complete the big-picture I highly recommend reading following articles, that I discovered meanwhile:

LFS

All mentioned above projects show some valid aspects of the task and combining them together could bring a really nice result I could be happy with. However after thinking a bit, there is still one subject that I haven't said anything - GiT Large File Storage (tutorial). Storing OpenCV4Android outside of the repository looks very promising, but in turns complicates Continuous Integration builds. Having the huge opencv*.so shared-library binaries (and updating them in the future) could very quickly bloat the whole repository and make version browsing very cumbersome or impossible. LFS solves both of this issues - by keeping the contents of big files outside of the regular history of GiT repository. In practice LFS mounts some hooks into git system and versions lightweight “pointers” (along with hashes) to selected files and replaces those pointers with expected content loaded directly from remote server each time a branch is checked out and “pointer” stored on that branch indicates to different version of the file's content.

Configure Android project with OpenCV and C++

  1. Download OpenCV4Android 3.9.4

  2. Extract it to any OPENCV_SDK_PATH (e.g. T:\codetitans\OpenCV-android-sdk)

  3. Create empty GiT repository (outside of the OpenCV4Android) for the Android project in Kotlin

    > mkdir opencv_app && cd opencv_app
    > git init .
    Initialized empty Git repository in T:/codetitans/opencv_app/.git/
    
  4. Install LFS support inside the repository

    > git lfs install
    Updated git hooks.
    Git LFS initialized.
    
  5. Create new regular Android application with Kotlin as default language

    New Android Project

    New Project Configuration

    Tips: I used here APIv21 to start with the introduction of Android Camera2 API.

    Click FINISH to complete the wizard.

  6. Add project to GiT

    > git add src/
    > git commit -m "Created empty project"
    [master (root-commit) 9188a11] Created empty project
    40 files changed, 892 insertions(+)
    .
    .
    .
    create mode 100644 src/settings.gradle
    
  7. Import Java/JNI library from extracted OPENCV_SDK_PATH > sdk > java and name it libOpenCV module

    Navigate to File > Import Module....

    Import Menu

    And follow the wizard.

    Import OpenCV

    Import Finalization

    Now, it's a good time to update module's build.gradle file and set Android SDK properties identically to the app module.

    android {
        compileSdkVersion 29
        buildToolsVersion "29.0.2"
    
        defaultConfig {
            minSdkVersion 21
            targetSdkVersion 29
        }
        .
        .
        .
    }
    

    Then remove <uses-sdk /> tag from libOpenCV > src > main > AndroidManifest.xml. As this is some Eclipse-type project leftover, not needed, when building with Gradle.

  8. Add new module to GiT

    • First ask GiT to ignore folder with binaries:
     > echo "/libOpenCV/build/" >> src/.gitignore
    
    • Commit module
    > git add src/
    > git commit -m "Imported libOpenCV module"
    [master 5ce6960] Imported libOpenCV module
    144 files changed, 62007 insertions(+)
    create mode 100644 src/import-summary.txt
    create mode 100644 src/libOpenCV/build.gradle
    create mode 100644 src/libOpenCV/lint.xml
    create mode 100644 src/libOpenCV/src/main/AndroidManifest.xml
    create mode 100644 src/libOpenCV/src/main/aidl/org/opencv/engine/OpenCVEngineInterface.aidl
    create mode 100644 src/libOpenCV/src/main/java/org/opencv/android/AsyncServiceHelper.java
    create mode 100644 src/libOpenCV/src/main/java/org/opencv/android/BaseLoaderCallback.java
    .
    .
    .
    create mode 100644 src/libOpenCV/src/main/java/org/opencv/videoio/VideoWriter.java
    create mode 100644 src/libOpenCV/src/main/java/org/opencv/videoio/Videoio.java
    create mode 100644 src/libOpenCV/src/main/res/values/attrs.xml
    
  9. Copy OpenCV selected shared-libraries (see note at the end) from OPENCV_SDK_PATH > sdk > native > libs into the openCVLibrary project > sdk > libs folder

    Copied SDK JNI libs

  10. Add copied shared-libraries .so to LFS

    > git lfs track src/libOpenCV/sdk/**/*.so
    Tracking "src/libOpenCV/sdk/**/*.so"
    
    > git add src/libOpenCV/sdk
    > git add .gitattributes
    > git commit -m "Added OpenCV native libraries"
    [master fe090ef] Added OpenCV native libraries
    5 files changed, 13 insertions(+)
    create mode 100644 .gitattributes
    create mode 100644 src/libOpenCV/sdk/libs/arm64-v8a/libopencv_java3.so
    create mode 100644 src/libOpenCV/sdk/libs/armeabi-v7a/libopencv_java3.so
    create mode 100644 src/libOpenCV/sdk/libs/x86/libopencv_java3.so
    create mode 100644 src/libOpenCV/sdk/libs/x86_64/libopencv_java3.so
    

    As a confirmation, that instead of big binary content - a small “pointer” was added into repository, run gitk and notice the details consisting of a spec version, SHA256 and original size in bytes.

    LFS Commit

  11. Depend app on libOpenCV module

    Simply navigate to File > Project Structure...

    Project Structure Menu

    There select Dependencies > app > Module Dependency

    Module Dependency

    libOpenCV Module Dependency

  12. Configure app module to include OpenCV's shared-libraries via build.gradle, and also to build separate APKs for selected processor architectures (multiple ABIs can be comma-separated) for testing:

    android {
        .
        .
        .
        splits {
            // multiple APKs based on ABI
            abi {
                enable true
                reset()
                include "x86" // just build these!
                universalApk false // disable the one with all ABIs
            }
        }
    
        sourceSets {
            main {
                jniLibs.srcDirs = ['../libOpenCV/sdk/libs']
            }
        }
    }
    

    Setup correctness on this stage could be checked, if APK is build via Build > Build Bundle(s) / APK(s) > Build APK(s) and produces a single app-x86-debug.apk file under project's location T:\codetitans\opencv_app\src\app\build\outputs\apk\debug.

  13. Commit all changes

    > git add src/
    > git commit -m "Added libOpenCV dependency"
    
  14. Make sure NDK is installed along with cmake and lldb.

    Navigate to Tools > SDK Manager and verify checkboxes next to mentioned items. If not, select those items and hit Apply button.

    Module Dependency

  15. Copy OpenCV's cmake files from OPENCV_SDK_PATH > sdk > native > jni into the libOpenCV project > sdk > jni folder

    Copied SDK JNI cmake

  16. Add those files regularly to GiT (as they are small enough)

    > git add src/libOpenCV/sdk/jni/
    > git commit -m "Added JNI cmake files"
    [master f31fdc9] Added JNI cmake files
    296 files changed, 128308 insertions(+)
    create mode 100644 src/libOpenCV/sdk/jni/OpenCV-arm64-v8a.mk
    create mode 100644 src/libOpenCV/sdk/jni/OpenCV-armeabi-v7a.mk
    create mode 100644 src/libOpenCV/sdk/jni/OpenCV-x86.mk
    create mode 100644 src/libOpenCV/sdk/jni/OpenCV-x86_64.mk
    create mode 100644 src/libOpenCV/sdk/jni/OpenCV.mk
    .
    .
    .
    create mode 100644 src/libOpenCV/sdk/jni/include/opencv2/videostab/ring_buffer.hpp
    create mode 100644 src/libOpenCV/sdk/jni/include/opencv2/videostab/stabilizer.hpp
    create mode 100644 src/libOpenCV/sdk/jni/include/opencv2/videostab/wobble_suppression.hpp
    
  17. Add another module into the project (libOpenCvLogic - to keep whole future C++ code).

    Add new module library

    Finalize new module library

  18. Configure libOpenCvLogic to contain C++ code and operate with OpenCV natively

    To do so, create a file libOpenCvLogic > src > main > cpp > CMakeLists.txt with following content:

    # For more information about using CMake with Android Studio, read the
    # documentation: https://d.android.com/studio/projects/add-native-code.html
    
    # Sets the minimum version of CMake required to build the native library.
    
    cmake_minimum_required(VERSION 3.4.1)
    include_directories(${CMAKE_CURRENT_SOURCE_DIR}/../../../../libOpenCV/sdk/jni/include)
    add_library( lib_opencv SHARED IMPORTED )
    set_target_properties(lib_opencv PROPERTIES IMPORTED_LOCATION ${CMAKE_CURRENT_SOURCE_DIR}/../../../../libOpenCV/sdk/libs/${ANDROID_ABI}/libopencv_java3.so)
    
    # Creates and names a library, sets it as either STATIC
    # or SHARED, and provides the relative paths to its source code.
    # You can define multiple libraries, and CMake builds them for you.
    # Gradle automatically packages shared libraries with your APK.
    
    add_library( # Sets the name of the library.
            my-opencvlogic-extensions
    
            # Sets the library as a shared library.
            SHARED
    
            # Provides a relative path to your source file(s).
            test.cpp
            ****list here all C/C++ files to compile ****
            )
    
    # Searches for a specified prebuilt library and stores the path as a
    # variable. Because CMake includes system libraries in the search path by
    # default, you only need to specify the name of the public NDK library
    # you want to add. CMake verifies that the library exists before
    # completing its build.
    
    find_library( # Sets the name of the path variable.
            log-lib
    
            # Specifies the name of the NDK library that
            # you want CMake to locate.
            log)
    
    # Specifies libraries CMake should link to your target library. You
    # can link multiple libraries, such as libraries you define in this
    # build script, prebuilt third-party libraries, or system libraries.
    
    target_link_libraries( # Specifies the target library.
            my-opencvlogic-extensions
    
            # Links the target library to the log library
            # included in the NDK.
            ${log-lib}
            lib_opencv)
    

    And then extend the module's build.gradle file with configuration of required native build:

    android {
        .
        .
        .
        defaultConfig {
            .
            .
            .
            externalNativeBuild {
                cmake {
                    cppFlags "-frtti -fexceptions"
                }
            }
        }
        .
        .
        .
        externalNativeBuild {
            cmake {
                path "src/main/cpp/CMakeLists.txt"
                version "3.10.2"
            }
        }
    }
    

    At this stage it's very important to have at least a single C/C++ source file created and included inside CMakeLists.txt. This file can be even empty. Otherwise the Android Studio will behave strangely and will hide whole cpp branch in project source-tree.

    It behaves similarly wrong, when there were typos in the CMakeLists.txt.

    Tip: cmake configuration already defines folders for OpenCV headers from libOpenCV/sdk/jni/include, shared libraries and dependant cmake files to allow further compilation.

  19. Depend app module on libOpenCvLogic

    libOpenCV Module Dependency

    Tip: again, creating x86 APK (and changing it's extension to .zip), could reveal, that there are now two native libraries (in lib folder) possibly loaded by the application: libmy-opencvlogic-extensions.so and libopencv_java3.so.

  20. Add new module to GiT

    > git add src/
    > git commit -m "Added libOpenCvLogic module"
    
  21. Build the project and now you are ready to add you code into the project. If you need something for startup go to Native OpenCV for Android with Android NDK.

  22. Done! But that was far too long!

Now, if you need some solid JNI knowledge - refer to JNI Cookbook and its samples.

Note

Finally one note about native libraries. Since Android 4.4 (API level 19), NDK r17 and Android Plugin for Gradle 3.0.1 the following ABIs are not supported anymore: armeabi (aka ARMv5/ARMv6), mips and mips64 (docs and here). All the effort should go to support ARMv7/ARM64 and x86/x64 architectures (docs).