Enable package compatibility checks with older Elastic stacks

[mtojek]

What does this PR do?

This PR enables package compatibility checks with older Elastic stacks. A package will be installed (not tested) in Kibana.

Checklist

- [ ] I have reviewed [tips for building integrations](https://github.com/elastic/integrations/blob/master/docs/tips_for_building_integrations.md) and this pull request is aligned with them.
- [ ] I have verified that all data streams collect metrics or logs.
- [ ] I have added an entry to my package's changelog.yml file.

How to test this PR locally

CI will verify it.

Related issues

• Fixes Enable backward compatibility tests #537

[mtojek]

Blocked by: elastic/beats#24163

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #740 updated

• Start Time: 2021-03-01T14:01:24.218+0000

• Duration: 38 min 43 sec

• Commit: ed26c02

Test stats 🧪

Test	Results
Failed	0
Passed	1723
Skipped	3
Total	1726

Trends 🧪

[mtojek]

I found a weird behavior when used -SNAPSHOT for constraint checks:

../../build/elastic-package install -c kibana.version=7.11.0-SNAPSHOT -v
— Check compatibility with 7.11.0-SNAPSHOT
<1s
[2021-02-25T12:40:26.436Z] + ../../build/elastic-package install -c kibana.version=7.11.0-SNAPSHOT -v
[2021-02-25T12:40:26.436Z] 2021/02/25 12:40:26 DEBUG Enable verbose logging
[2021-02-25T12:40:26.436Z] Check conditions for package
[2021-02-25T12:40:26.436Z] 2021/02/25 12:40:26 DEBUG Verify Kibana version (constraint: ^7.9.0, requirement: 7.11.0-SNAPSHOT)
[2021-02-25T12:40:26.436Z] Error: checking conditions failed: Kibana constraint unsatisfied: [0] 7.11.0-SNAPSHOT is a prerelease version and the constraint is only looking for release versions

I will try with releases, e.g. 7.11.0, 7.10.0

[mtojek]

Ok, without SNAPSHOT tags, the elastic-package properly considers the version as valid/compatible. It fails because of #745 (need to merge this one first).

[mtojek]

I think I have to improve the elastic-package, as it supports only released version:

docker pull docker.elastic.co/elasticsearch/elasticsearch:7.11.0 - works

docker pull docker.elastic.co/elasticsearch/elasticsearch:7.12.0 - doesn't work, as 7.12.0 is not released

[mtojek]

@ycombinator it's not ready for merge yet (-SNAPSHOT case), but feel free to comment if you want to.

EDIT:

I think I will fix the condition in a follow up PR. This one already polishes a couple packages in terms of compatibility.

[ycombinator]

So it seems we are calling a package as compatible if we are able to install it. Is this sufficient or do we also want to run tests for the package against the previous stack versions?

[mtojek]

No, I prefer to stick to installation as testing will elongate test execution much (+20minutes).

[ycombinator]

Understood. Does installing a package result in any testing? In other words, what are the reasons this step would fail in CI?

[mtojek]

e.g. installation can check if all Kibana objects can be installed (correct migrationVersion field).

[ycombinator]

I didn't understand why we need these kibana.version bumps here and in other packages in this PR?

[mtojek]

It's because of the fact that these packages are not compatible with older Kibana stacks.

[ycombinator]

I see. Is there a reason these changes are part of this PR? In other words, do we need these changes anyway even without this PR? Did the compatibility checks introduced in this PR catch that these packages needed kibana version bumps?

[mtojek]

Did the compatibility checks introduced in this PR catch that these packages needed kibana version bumps?

Yes, exactly this. Otherwise the CI for this PR will be red.

[ycombinator]

LGTM. This is a nice addition!