Using a 3rd party crash reporter is one of the best things you can do to ensure a high quality app. If you don't know something's wrong how would you know it needs to be fixed? Non production builds can create a lot of noise making it difficult to separate real issues from normal development related crashes. You should draw a line between development, alpha, beta, and productions builds to focus your bug fixing efforts.

Conditionally turn it off

This works great if you have a simple set up consisting of development and production builds. You simply pass the BuildConfig.DEBUG flag, disabling the reporting at run time.

Crashlytics:

Fabric.with(this, Crashlytics.Builder()
                .core(CrashlyticsCore.Builder().disabled(BuildConfig.DEBUG).build())
                .build())

Instabug:

Instabug.Builder(this, "APP_TOKEN")
                .setCrashReportingState(if (BuildConfig.DEBUG)
                    InstabugFeature.State.DISABLED else InstabugFeature.State.ENABLED)
                .build()

Firebase:

FirebaseCrash.enableCrash(BuildConfig.DEBUG)

Segment builds with version numbers

This one is simple and keeps all of your crash data in one place. Simply change versionName in your build file to be something more descript. Maybe something like 1.2.3-alpha, 1.2.3-beta, etc. You now rely on the crash reporting dashboard to be able to cleanly segment and filter this data. One problem I see here is that it messes with aggregated statistics and it's harder for other people to jump in and extract data.

It's by far the simplest, covering the most cases with a single solution. It covers other things you might not think of either. Say for example you add BuildConfig.VERSION_NAME to your WebView or API user agent. You've now signaled to your backend you may be testing things out and those weird requests probably aren't anything to worry about.

// app/build.gradle
productFlavors {
    alpha {
        versionNameSuffix "-alpha"
    }
    beta {
        versionNameSuffix "-beta"
    }
}

Completely different buckets

Another options is to create entirely new apps or accounts for each variant of your app. I find this interesting because you get the best of all worlds, but it can be tricky to set up. This isn't my first choice, but for the right situation it's certainly a reasonable option.

Crashlytics:

// app/build.gradle
productFlavors {
    alpha {
        applicationIdSuffix ".alpha"
    }
    beta {
        applicationIdSuffix ".beta"
    }
}

Instabug:

// app/build.gradle
productFlavors {
    alpha {
        buildConfigField("String", "INSTABUG_TOKEN", "\"ALPHA_APP_TOKEN\"")
    }
    beta {
        buildConfigField("String", "INSTABUG_TOKEN", "\"BETA_APP_TOKEN\"")
    }
}

// Application#onCreate
Instabug.Builder(this, BuildConfig.INSTABUG_TOKEN).build()

Firebase:

You'll need to create multiple Firebase projects each with its own package name. From there you can generate a google-services.json for each.

// app/build.gradle
productFlavors {
    alpha {
        applicationIdSuffix ".alpha"
    }
    beta {
        applicationIdSuffix ".beta"
    }
}

// Add files in these directories
app/
    google-services.json
    src/alpha/google-services.json
    src/beta/google-services.json
    ...

So there you have it, multiple ways to group crash data with various crash reporting libraries. You'll no longer have to explain why there are 823 crashes from one user, every single day.