Improved documentation

[pbatk]

Thank you for your contribution orion is a very cool system!

I have managed to get a working orion cluster up after a couple of days. I will have to write some more documentation on it so that others can work on the cluster in my org. Would you like me to offer it as a PR?

There isn't a major rush on it for us so it may take a couple of weeks as I'll be doing it in my spare time.

Some of the areas I had problems figuring out:

• Configuration files that were not clear
  • .serverset
  • brokerset.json
  • topicassignment.json
• How to deploy an agent with a kafka node
  • It wasn't clear if orion was expecting to install the kafka node itself or if I was responsible to install
• OpenTSDB integration
  • I didn't even know this was part of the system until the warning logs started

P.S. I still haven't solved the brokerset.json or topicassignment.json problem so if you guys have some examples that could be shared that would be awesome

[ambud]

Yes, it would be great to get documentation improvements as a PR and thank you for offering to help improve it. Sorry for the delayed response.

We will update our brokerset and topicassignment.json examples.

We have updated the intake process to make it more flexible, you can now have a assignment file per topic e.g. __consumer_offset.json for the __consumer_offset topic

{
    "topicName": "__consumer_offsets",
    "replicationFactor": 3,
    "brokerset": "internal",
    "stride": 0,
    "project": "kafka",
    "config": {
        "compression.type": "producer",
        "segment.bytes": 104857600,
        "min.insync.replicas": 2,
        "cleanup.policy": "compact",
        "unclean.leader.election.enable": "true"
    }
}

brokerset.json

[
   {
      "brokersetAlias": "Capacity_B30_P10_0",
      "entries": [
            {
                "startBrokerIdx": 3101,
                "endBrokerIdx": 3130
            }
      ],
  "partitions": 10
  },
  {
    "brokersetAlias": "internal",
    "entries": [
        {
          "startBrokerIdx": 3101,
          "endBrokerIdx": 3130
        }
    ],
    "partitions": 50
  }
]