A flexible synthetic data generator in Go, designed to produce realistic logs and metrics for testing and development.

---
config:
  look: handDrawn
  theme: default
---
flowchart LR
 subgraph Config["Config Source"]
        YAML["config.yaml"]
        ENV["Env Vars"]
  end
 subgraph Tool["Synthetic Data Generator"]
    direction TB
        G["<b>Generator</b><br>Logs &amp; Metrics"]
        P["<b>Processor</b><br>Throttling &amp; Batching"]
        E["<b>Exporter Engine</b>"]
  end
 subgraph AWS["AWS Services"]
        S3["S3 Bucket"]
        CW["CloudWatch Logs"]
        FH["Kinesis Firehose"]
  end
 subgraph Azure["Azure"]
        EH["Event Hub"]
  end
 subgraph Local["Local Storage"]
        LF["Local File"]
  end
    G --> P
    P --> E
    Config --> Tool
    E --> AWS & Azure & Local

    style E fill:#d1e7ff,stroke:#004a99,stroke-width:2px
    style Tool fill:#f9f9f9,stroke:#D50000,stroke-width:2px,color:#000000

• Clone the repository (alternatively download a release binary from releases)

• Copy following to config.yaml

  input:
    type: LOGS
    delay: 500ms
    batching: 2s
    max_batch_size: 500_000
    max_runtime: 5s
  output:
    type: DEBUG
    config:
      verbosity: detailed

• Run data generator with one of the following commands,

  From source,

  go run cmd/main.go --config ./config.yaml

  Using a release binary,

  ./dataGenerator_darwin_arm64 --config ./config.yaml

Observe the terminal for generated logs.

Following program commands are supported,

• --config <file_path> : Path to configuration file. Default is ./config.yaml.
• --metrics <file_path> : Path to write internal metrics. If ignored, metrics will be logged to console. (see supported metrics below)
• --debug : Enable debug mode for verbose logging.

For example, to run with a custom config file at ./myconfig.yaml, writing internal metrics to metrics.json at end and enabling debug mode,

./dataGenerator_darwin_arm64 --config ./myconfig.yaml --metrics ./metrics.json --debug

List below explains supported internal metrics.

• startTime : Timestamp when the generator started.
• endTime : Timestamp when the generator ended.
• totalBatches : Total number of batches exported.
• totalElements : Total number of data elements generated.
• totalBytes : Total number of bytes generated.

Given below are the supported configuration options. Check config.sample.yaml for reference.

Given below are supported input types and their related environment variable overrides,

Given below are supported type values for input,

Example:

input:
  type: LOGS             # Input type LOGS
  delay: 500ms           # 500 milliseconds between each data point
  batching: 10s          # Emit generated data batched within 10 seconds
  max_batch_size: 10000  # Limit maximum batch size to 10,000 bytes. The output is capped at 1000 bytes/second max
  max_data_points: 10000 # Exit input after generating 10,000 data points

Given below are supported output configurations and their related environment variable overrides,

Given below are supported output types,

Sections below provide output specific configurations

Example:

output:
  type: S3
  config:
    s3_bucket: "testing-bucket"
    compression: gzip
    path_prefix: "datagen"

Example:

output:
  type: FIREHOSE
  config:
    stream_name: "my-firehose-stream"

Example:

output:
  type: CLOUDWATCH_LOG
  config:
    logGroup: "MyGroup"
    logStream: "data"

Example:

Using with connection string & event hub name:

output:
  type: EVENTHUB
  config:
    connection_string: "Endpoint=sb:xxxxxx"
    event_hub_name: "<event_hub_name>"

Using with connection string & event hub name (requires IAM role assigned to the role running this application):

output:
  type: EVENTHUB
  config:
    event_hub_name: "<event_hub_name>"
    namespace: "<namespace>"

Example:

output:
  type: FILE
  config:
    location: "./data"

Example:

aws:
  region: "us-east-1"
  profile: "default"

Generate ECS-formatted log every 2s, batch them in 10 seconds and forward to S3 bucket. Default delay is 1s between data points.

input:
  type: LOGS
  delay: 2s
  batching: 10s
output:
  type: s3
  config:
    s3_bucket: "testing-bucket"

Generate ALB logs. No delay between data points (continuous data generating). Limit batching to 10 seconds and max batch size is set to 10MB. This translates to ~1 MB/second data load. S3 files will be in gzip format.

input:
  type: ALB
  delay: 0s
  batching: 10s
  max_batch_size: 10_000_000  # 10 MB max bytes per batch
output:
  type: s3
  config:
    s3_bucket: "testing-bucket"
    compression: "gzip"

Generate WAF logs. Delay of 100ms between data points. Limit batching to max 1000 elements per batch. Send to S3 in gzip format.

input:
  type: WAF
  delay: 100ms
  max_batch_elements: 1000  # Max 1000 elements per batch
output:
  type: s3
  config:
    s3_bucket: "testing-bucket"
    compression: "gzip"

Generate VPC logs and limit to 200 data points. Then upload it to S3 in gzip format.

input:
  type: VPC
  delay: 1s
  max_data_points: 200  # Limit to 200 data points total of the runtime
output:
  type: s3
  config:
    s3_bucket: "testing-bucket"
    compression: "gzip"

Generate CLOUDTRAIL logs and limit to generator runtime of 5 minutes.

input:
  type: CLOUDTRAIL
  delay: 10us       # 10 microseconds between data points
  batching: 10s
  max_runtime: 5m   # 5 minutes
output:
  type: s3
  config:
    s3_bucket: "testing-bucket"
    compression: "gzip"