Feature Request Thread

[zachary62]

😊 Please submit and upvote features you want

[brsolo]

Would be nice to have easier support for all OpenAI models (or mainly the latest gpt-4-turbo and gpt-3.5-turbo). Right now to use a different OpenAI model, I manually change the model parameter in init (below). I had tried to just make another elif statement, but openai.api_type is used in util.py within the openai library, so this raises an error.

One quick fix is to add the elif statement below, but it seems like bad form redefining an environ variable.

Maybe a better solution is to have two separate environ variables. One for openai.api_type and one for model.

Current code:

    elif openai.api_type == 'open_ai':
        
        response = openai.ChatCompletion.create(
            model="gpt-4-1106-preview",
            temperature=temperature,
            top_p=top_p,
            messages=messages
        )

Potential quick fix:

  elif openai.api_type == 'gpt-3.5-turbo-0125':

       os.environ['OPENAI_API_TYPE'] = 'open_ai'
       
       response = openai.ChatCompletion.create(
           model="gpt-4-1106-preview",
           temperature=temperature,
           top_p=top_p,
           messages=messages
       )

[jptoor]

Feature: Exclude/null out all PII columns by default

Details:
Security & privacy wise PII should not be sent to a 3rd party in nearly any case. Hashing may be enough for some data points, but unless they're a PK, I'd prefer remove them from all table samples. In order to test Cocoon, I've been creating sample sets of tables with non-sensitive data, and nulling out anything close to PII.

Snowflake has some built in functionality that may be helpful as well.

[zachary62]

Feature: Exclude/null out all PII columns by default

Details: Security & privacy wise PII should not be sent to a 3rd party in nearly any case. Hashing may be enough for some data points, but unless they're a PK, I'd prefer remove them from all table samples. In order to test Cocoon, I've been creating sample sets of tables with non-sensitive data, and nulling out anything close to PII.

Snowflake has some built in functionality that may be helpful as well.

Would a step that detects PII based on only column name (without any rows), verifies with human the recommended PII, and then removes the column be sufficient?

[Palkers76]

Create non-interactive mode with parameters, in many cases you will use same options but for different tables. I can imaging that we could run it recursivly for many tables.

[Palkers76]

generate pyspark code in addition to dbt yaml or sql, and instead of saving it to file, add as a cell or save as notebook if contain more than 1 cell / step

[Palkers76]

Feature: Exclude/null out all PII columns by default

Details: Security & privacy wise PII should not be sent to a 3rd party in nearly any case. Hashing may be enough for some data points, but unless they're a PK, I'd prefer remove them from all table samples. In order to test Cocoon, I've been creating sample sets of tables with non-sensitive data, and nulling out anything close to PII.

Snowflake has some built in functionality that may be helpful as well.

generating syntetic data based on schema and sample of data maight be the solution. there are dozen of solutions maybe re-use of one of open-source https://github.com/statice/awesome-synthetic-data

[zachary62]

Create non-interactive mode with parameters, in many cases you will use same options but for different tables. I can imaging that we could run it recursivly for many tables.

I will implement this first this week.

[zachary62]

Create non-interactive mode with parameters, in many cases you will use same options but for different tables. I can imaging that we could run it recursivly for many tables.

Implemented a step to specify the options for "format" express mode:
! pip install cocoon_data==0.1.129

You shall see the following UI. Let me if these options are enough or not.

[zachary62]

Feature: Exclude/null out all PII columns by default

Details: Security & privacy wise PII should not be sent to a 3rd party in nearly any case. Hashing may be enough for some data points, but unless they're a PK, I'd prefer remove them from all table samples. In order to test Cocoon, I've been creating sample sets of tables with non-sensitive data, and nulling out anything close to PII.

Snowflake has some built in functionality that may be helpful as well.

Hey sorry for the very late reply, but PII detection has already been supported.
You can use LLMs to detect PII (e.g., at schema level), and optionally remove it

[Palkers76]

Create non-interactive mode with parameters, in many cases you will use same options but for different tables. I can imaging that we could run it recursivly for many tables.

Implemented a step to specify the options for "format" express mode: ! pip install cocoon_data==0.1.129

You shall see the following UI. Let me if these options are enough or not.

do you have guide how to us it? can I export config now and pass it into cocoon method ? it's not clear sorry :)

[Palkers76]

I want to migrate processing from legacy SQL base system like oracle or other to databricks + dbt,
is there a way to provide some additional metadata like SQL's or contects to generate data pipeline (dbt) code bases on i.e source systems schemas and sql code which was used to process data in legacy system?

[zachary62]

I want to migrate processing from legacy SQL base system like oracle or other to databricks + dbt, is there a way to provide some additional metadata like SQL's or contects to generate data pipeline (dbt) code bases on i.e source systems schemas and sql code which was used to process data in legacy system?

Hi @Palkers76 could you try to go to your dbt repo and enter

pip install cocoon_data -U
cocoon_data

It can help you RAG existing dbt projects (https://cocoon-data-transformation.github.io/page/pipeline)
Oracle databases need more time to support, but if you know the tables to use, you can enter these into the contexts for the chatbots

[Palkers76]

I want to migrate processing from legacy SQL base system like oracle or other to databricks + dbt, is there a way to provide some additional metadata like SQL's or contects to generate data pipeline (dbt) code bases on i.e source systems schemas and sql code which was used to process data in legacy system?

Hi @Palkers76 could you try to go to your dbt repo and enter

pip install cocoon_data -U
cocoon_data

It can help you RAG existing dbt projects (https://cocoon-data-transformation.github.io/page/pipeline) Oracle databases need more time to support, but if you know the tables to use, you can enter these into the contexts for the chatbots

is there a way to use cocoon in non-interactive mode, call some cli/methods with parameters (i,e. passed in config file) to do specific job?

[zachary62]

I want to migrate processing from legacy SQL base system like oracle or other to databricks + dbt, is there a way to provide some additional metadata like SQL's or contects to generate data pipeline (dbt) code bases on i.e source systems schemas and sql code which was used to process data in legacy system?

Hi @Palkers76 could you try to go to your dbt repo and enter

pip install cocoon_data -U
cocoon_data

It can help you RAG existing dbt projects (https://cocoon-data-transformation.github.io/page/pipeline) Oracle databases need more time to support, but if you know the tables to use, you can enter these into the contexts for the chatbots

is there a way to use cocoon in non-interactive mode, call some cli/methods with parameters (i,e. passed in config file) to do specific job?

Yeah! Which methods do you want to call?
E.g., the cleaning config can be called via:

para = {"cocoon_stage_options": {'table_desc': True, 'explain_missing_values': True, 'detect_regex_patterns': True, 'parse_variant_types': True, 'unique_test': True, 'category_test': True, 'expression_tests': False, 'projection': True, 'rename': True, 'deduplication': True, 'standardize_values': True, 'detect_disguised_missing_values': True, 'transform_data_type': True, 'trim': True, 'generate_html_report': False, 'description_access_control': 'Read Data Access', 'cardinality_threshold': 300, 'pii': False, 'pii_schema_only': True, 'pii_model': None}}
create_cocoon_workflow(con=con, output=widgets.Output(), para=para)