[Discuss] Proposal: support integration sets

[mtojek]

Hi everyone,

The goal of this issue is to discuss challenges, benefits and drawbacks of introducing a new package structure (potentially v2?), which can satisfy the requirement of service granularity (reported in #122). I've prepared a sample package to visualize the proposal.

MySQL pseudo-integration: https://github.com/mtojek/integrations/tree/v2-integration-sets/packages/mysql-2

Requirements:

• fit in the new service granularity model proposed by @sorantis
• have single manifest file (for package and all data streams), so it would be easier for Kibana and EPR to load and process package description
• keep large contents (pipeline definitions, icons, screenshots, agent's configs) in separate resources
• support extending resources, e.g. manifests contain vars on every level, fields are on multiple levels in the directory tree

Notes/observations:

• Integration policies defines groups of inputs: logs, metrics, packets (packetbeat), synthetics (uptime). I suppose we'll need also traces
• With this approach, we'll drop the original idea of a package - a group of dashboards, pipelines, etc. The new concept changes the meaning of not just integrations.
• data_stream names are aligned with the manifest structure, e.g.:

  integration_sets:
  - name: basic
    integration_policies:
      logs:
        inputs:
          - name: error
            ilm_policy: custom-hot-warm-logs
            vars: ~ # ...

  .. gives - logs-mysql-basic.error-shared. I suppose there should be an option for the end-user to correct/change it.

Please do not consider this is as future approach or a kick-off for adjustments, it's rather an exercise to understand and estimate the amount of work in multiple places. It would be nice to hear some feedback from all parties.

EDIT:

Do not look at the manifest as it's the final/target format. Please rather focus on concepts/requirements.

[mtojek]

Let me ask few people to join the discussion.

Kibana: @jen-huang @skh
AWS Integration/Integrations: @kaiyan-sheng @masci @andrewkroh
EPR: @ruflin @ycombinator
Fleet: @ph @nchaulet

It would be great to justify first if we're able to introduce such changes and when would be the right time.

[ph]

Let's keep @mostlyjason @andresrc in the loop

[sorantis]

Proposed integration architecture

[sorantis]

Thank you for starting this discussion @mtojek.

One use case I was thinking about is the usage of vars in the new architecture. I see two cases for common configuration:

1. vars that are applicable to all policies. I think you have it covered in your example and it should work like a charm for AWS account details for example.
2. vars that are applicable to one policy only. For example, for NGINX metrics the user might want to use HTTP and SSL configuration to access password protected NGINX server, while for logs it’s not really necessary.

UPDATE
I see that vars can also be specified for each policy as well as input. Which should cover the 2 cases!

integration_policies:
      logs:
        inputs:
          - name: error
            ilm_policy: custom-hot-warm-logs
            vars: ~ # ...
          - name: slowlog
            ilm_policy: hot-warm
            vars: ~ # ...
        vars: ~ # ...

[kaiyan-sheng]

Thanks @mtojek and @sorantis! I like the design of having vars at the integration_policies so we can share these vars across metrics and logs policy. For AWS, this is where the common credentials will be. One question I have is: why we need the inputs layer/folder under logs and metrics? Can we put the individual inputs directly under logs or metrics folder? For example:

.
├── README.md
├── fields
│   └── README.md
├── integration_sets
│   ├── basic
│   │   ├── fields
│   │   │   └── README.md
│   │   └── integration_policies
│   │       ├── logs
│   │       │       ├── error
│   │       │       │   ├── agent
│   │       │       │   │   └── stream.yml.hbs
│   │       │       │   └── fields
│   │       │       │       └── README.md
│   │       │       └── slowlog
│   │       └── metrics
│   │               └── perf
│   ├── enterprise
│   ├── galera
│   ├── traffic
│   └── uptime
└── manifest.yml

[mtojek]

Hey @kaiyan-sheng , thanks for taking a look.

I assume you're referring to this level: https://github.com/mtojek/integrations/tree/v2-integration-sets/packages/mysql-2/integration_sets/basic/integration_policies . The goal of the additional directory with inputs (input dir) is to leave an open door for potential extensions in the future, if we decide to put sth aside to inputs. Maybe dedicated vars for all "logs" inputs?

[ycombinator]

One of the stated requirements is:

have single manifest file (for package and all data streams), so it would be easier for Kibana and EPR to load and process package description

I would not optimize for machine reading (Kibana and EPR) when thinking about the format/structure of the package manifests. We can always use elastic-package build to stitch individual manifests into a single one to optimize for machine consumption if needed.

When thinking of the package structure I would optimize for human consumption instead, as this will impact package developers more frequently. If, by that measure, a single manifest file is more convenient then let's go with that. But if multiple files, grouped by logical folders (like we have today) is more easily understood and maintainable by humans, I would lean towards that.

[ycombinator]

With this approach, we'll drop the original idea of a package - a group of dashboards, pipelines, etc. The new concept changes the meaning of not just integrations.

Where would stack assets (index templates, dashboards, pipelines, etc.) be defined? Don't they need to come from somewhere associated with a package version so Kibana can install them?

Can you flesh out one more integration set in your sample package? I'm curious to know if there's any repetition of contents between two integration sets (e.g. agent policies or fields).

Speaking of fields, I see fields folders in the sample spec but no field definitions under them (just READMEs so far). I also don't see field definitions in the top-level manifest.yml file. Where would field definitions come from? This may be related to the first question above about stack assets.

[mtojek]

Thank you, Shaunak, for looking into this.

I would not optimize for machine reading (Kibana and EPR) when thinking about the format/structure of the package manifests. We can always use elastic-package build to stitch individual manifests into a single one to optimize for machine consumption if needed.

Agree, but I'm not sure if the current structure is sufficient to model the "integration-sets" concept. If so, then all names will be wrong (policy templates -> integration policies?). In this case building procedure will mean mapping entities from one domain to another. I'm not sure if this is the expected way to go, definitely it will hard to debug/RCA anything if integrations stay as is and the Kibana side changes (new domain).

Side note - if we keep the human model aligned with machine model (or even have a single one), the build procedure will be less error prone.

But if multiple files, grouped by logical folders (like we have today) is more easily understood and maintainable by humans, I would lean towards that.

Sure, we can split the single manifest into multiple entries. It just depends what is the current user experience, I mean, is it easier to design new integration while looking at all integration policies or it's fine to navigate between directories.

Where would stack assets (index templates, dashboards, pipelines, etc.) be defined? Don't they need to come from somewhere associated with a package version so Kibana can install them?

Frankly speaking, I skipped them intentionally in the initial draft to agree first on the core model (integration sets, integration policies, etc.). Things like vars, fields, kibana saved objects can be added later on whenever we want (even on every level).

Speaking of fields, I see fields folders in the sample spec but no field definitions under them (just READMEs so far). I also don't see field definitions in the top-level manifest.yml file. Where would field definitions come from? This may be related to the first question above about stack assets.

As I stated above I wanted to focus on the core structure first. Regarding the repetitions, I don't think we'll be able to get rid of them. Imagine the case in which some fields are valid for 2/3 metrics integration policies. The presented model doesn't cover this.

[ycombinator]

Agree, but I'm not sure if the current structure is sufficient to model the "integration-sets" concept.

I don't think I was suggesting reusing the current structure (as in, what live packages use today). All my comments here are referring to the new proposed structure.

To be clear: I'm suggesting that the new proposed structure that you are demonstrating in https://github.com/mtojek/integrations/tree/v2-integration-sets/packages/mysql-2 should not optimize for machine consumption but rather human consumption to make package authors' lives easier. Via elastic-package build we could transform the human-friendly structure into anything that's more machine-friendly so we get the best of both worlds, each optimized for consumers at different points of the package lifecycle - creation/maintenance (done by humans) vs. serving/parsing (done by machines). This is similar to the idea of a high-level programming language coupled with a compiler.

Side note - if we keep the human model aligned with machine model (or even have a single one), the build procedure will be less error prone.

I'm not sure about this. Certainly if the two are the same there's no build process as such so it's completely error-prone. But I'm not sure this is worth making the developer experience poorer either. We can have tests to make sure the build procedure produces expect output for a given input.

Now, it may turn out that humans actually would prefer a single large manifest file. In that case, I'm all for it! Maybe this is something we need more feedback from current package authors on? I do think that to provide such feedback it would be helpful to have the sample package more fleshed out. This would give a more accurate picture of what a package developer would need to create and maintain.

Frankly speaking, I skipped them intentionally in the initial draft to agree first on the core model (integration sets, integration policies, etc.). Things like vars, fields, kibana saved objects can be added later on whenever we want (even on every level).

As I stated above I wanted to focus on the core structure first.

It is to inform decisions about the core structure that I'd like to see it more fleshed out 🙂.

Regarding the repetitions, I don't think we'll be able to get rid of them. Imagine the case in which some fields are valid for 2/3 metrics integration policies. The presented model doesn't cover this.

I think repetitions could be avoided by referencing, just as an example. But before we jump to a solution, it would be good know, at least for me, where repetitions could show up in the structure.

I guess I'm uncomfortable having a strong opinion on the proposed structure without seeing more use of it — both within a single integration (more fleshing out of the sample package) but also across different types of integrations (maybe pick a few diverse examples of integrations from @sorantis's doc and create sample packages for those as well?) You don't need to flesh out every field or every asset in general, but seeing 2-3 instances of every type of asset would be enough to see patterns, I think.

I'm on PTO at the moment but, if you prefer, I can do all the work I'm asking for here once I'm back via a PR to your branch.

[ph]

@jen-huang This is the other issue related to the doc, I've shared with you.

[sorantis]

Can you flesh out one more integration set in your sample package? I'm curious to know if there's any repetition of contents between two integration sets (e.g. agent policies or fields).

@kaiyan-sheng I think this should be a good exercise to see how the proposed changes look like in the AWS case.