Skip to content

[FEA] Create script(s) to validate on-prem environment for readiness for Spark RAPIDS #239

Description

@mattahrens

Is your feature request related to a problem? Please describe.
I would like to have a tool that would validate a Spark environment for its readiness for acceleration with GPU for the Spark RAPIDS plugin.

Describe the solution you'd like
The solution should include the following:

  • Environment analyzer to collect information about the nodes and setup -- including number of nodes, number of cores per node, amount of memory on each node, amount of disk storage on each node, disk bandwidth, network bandwidth, and GPU type and memory (if applicable)
  • Wrapper script for the NDS power run to generate data, run the benchmark and generate an event log and driver log
  • Report results from environment analyzer and NDS power run along with event log and driver log
  • The solution can allow for a configurable data size to be used along with data type (e.g. SF3K, parquet).

Describe alternatives you've considered
N/A

Additional context
N/A

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions