Update kubernetes namespace field. breaking change

[MichaelKatsoulis]

What does this PR do?

In agent 8.0 a new breaking changed is introduced.

As part of elastic/beats#16483 and after merging of elastic/beats#27917
following changes where introduced:

• Updates kubernetes.namespace from keyword to group field.
• Moves kubernetes.namespace which was the name of the namespace to kubernetes.namespace.name field.
• Renames namespace_labels, namespace_annotations namespace_uid metadata to namespace.labels, namespace.annotations, namespace.uid.

kubernetes integration must also be updated accordingly to support the updated fields.
An ingest pipeline is also added in each updated data_stream to support also agents before 8.0 which may also send data to 8.0 stack. This can happen due to the update process of each user.

In this PR Kubernetes namespace field gets updated from keyword to group type and name,labels,annotations, uid are added as fields under it.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• If I'm introducing a new feature, I have modified the Kibana version constraint in my package's manifest.yml file to point to the latest Elastic stack release (e.g. ^7.13.0).

##Screenshots

1. Agent 8.0 and stack 8.0

2. Agent 7.16 and stack 8.0

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[MichaelKatsoulis]

This needs to be merged after elastic/beats#27917 is merged

[mtojek]

If this is a breaking change, you should go with 2.0.0

[mtojek]

type: breaking-change ?

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-10-19T13:26:53.309+0000

• Duration: 55 min 19 sec

• Commit: 990188b

Test stats 🧪

Test	Results
Failed	0
Passed	114
Skipped	0
Total	114

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[ruflin]

Can you share some details on how a user is upgrading from 7.x to 8.0 and the order a user would upgrade? I see now changes to the ingest pipeline in this PR but would have expected the ingest pipeline to provide some backward compatbility.

[MichaelKatsoulis]

@mtojek I added the ingest pipelines to support old agent versions as well! I think that now it is not considered a breaking change for the package anymore. So we don't need to set the version to 2.0 and we can remove the breaking change label.
What do you think?

[MichaelKatsoulis]

Can you share some details on how a user is upgrading from 7.x to 8.0 and the order a user would upgrade? I see now changes to the ingest pipeline in this PR but would have expected the ingest pipeline to provide some backward compatbility.

@ruflin I expect that the upgrade process goes like this:

1. User updates the stack(Kibana, Elasticsearch, fleet server) to 8.0
2. User then updates elastic-agents

Between steps 1 and 2 there will be 7.x agents sending data to 8.0 stack at the same time with 8.0 agents.
In this PR I added ingest pipelines which take care of the backward compatibility.
In case they detect old fields (like kubernetes.namespace in 7.x) they will rename them to the new fields.
If the fields are new (kubernetes.namespace does not exist anymore like in 8.0) then nothing will happen.
These pipelines will only run in package version with ^8.0 Kibana constraint which means that the stack can only be 8.0 and up. So in older versions of the stack they will not run.

[ruflin]

I assume step 1.5 is that the user also needs to ensure to have the most recent k8s package version.

[mtojek]

@mtojek I added the ingest pipelines to support old agent versions as well! I think that now it is not considered a breaking change for the package anymore. So we don't need to set the version to 2.0 and we can remove the breaking change label.
What do you think?

I have exactly the same understanding. Please proceed with these changes (lower version, no breaking changes).

[mtojek]

There are some Git conflicts here.

[MichaelKatsoulis]

There are some Git conflicts here.

@mtojek I resolved them