Researching our idea

[paekva]

#45

• Idea description @sunny-26
• Context: an overview of the subject area, target audience @sunny-26
• Problem: it is hard to manually clean data, the context of how they are cleaned @Nikvasil
• Solution: description + high-level requirements @paekva
• Value: competitors, what value we bring @Ovaday
• Prototype (Figma) @paekva @sunny-26
• Architectural overview diagram (draw.io) @sunny-26 @Ovaday https://drive.google.com/file/d/1P_uYlbbcIHsUI1V_RvutlCsuewhEI3me/view?usp=share_link
• Django @Ovaday

[Ovaday]

https://drive.google.com/file/d/1P_uYlbbcIHsUI1V_RvutlCsuewhEI3me/view?usp=share_link

[sunny-26]

Idea Description:
A survey is a method to collect data from a predefined group of respondents to gain information and insights into various topics of interest.
Unfortunately, it is impossible to control every responder and as a result, the raw data will include not only meaningful research responses but also responses that don’t contain any valuable information. To get only meaningful information, firstly the collected data should be cleaned.
Survey data cleaning involves identifying and removing responses from individuals who either don’t match your target audience criteria or didn’t answer your questions thoughtfully.
Our Idea:
To create a tool where it will be possible:

• To upload data in .csv format.
• Data cleaning based on the user-predefined requests (Filter, where the user should initialize from what he or she wants to clean the data as well as define the dependable questions(the answers to which should be the same) etc).
• Download the cleaned data in .csv format.
• The basic visualization of the data in the tool.

Cleaning includes:

• Remove partially completed responses.
• Remove answers from the untargeted group.
• Remove the responses which were done too fast.
• Remove straighteners (the same answers in all questions (I am not sure, zB)).
• Remove unrealistic responses.
• Remove outliers.
• Remove inconsistent answers (the user should predefine the dependencies which the tool later can analyze and detect inconsistency).

[sunny-26]

Context: Research Institutes, Universities, and Trading companies. They all do surveys to get useful information.
Why it is important for business:

• Providing customer opinion on the products and services.
• Providing structured data based on which the company can make decisions.
• Revealing the strengths and weaknesses of your products or services.

[Nikvasil]

Problem:

Many researchers are faced with bad and dirty data in surveys. And working with this data will cause problems, affecting the data quality. Also, having poor quality data can be harmful to further data analysis and lead to improper or inaccurate results and lead you to make ill-informed business decisions.

Data cleaning is one of the main parts of any survey data collection performed before data analysis. And sometimes it's overlooked by the researcher as its time consuming. Undertaking data cleaning will help identify and correct any errors or minimize their impact on the data quality and study results. Data cleaning will identify any hidden issues and don’t influence the data analysis and results.

[Ovaday]

What value do we bring / competitors resume.
We provide research institutes, small companies and individuals with a solution for analyzing and cleaning up data after surveying. Our tooling allows our customers to automatically convert the data into a single format, to clean the data from basic problems, such as missing values, inconsistent and too fast answers in the survey, in addition to carry out correlation analysis and draw conclusions on the data, allowing on this basis to further resurvey already with information on insides from the data.
At the moment, the proposed analogues are either offline solutions, or provide expensive or too much redundant functionality. Our proposal will solve the problems students, scientists and small companies have in processing the answers of the interviewees, for a notional fee.

Analysis here:
https://docs.google.com/document/d/14y2Eb2sgFB3ktsalimttr_Bj20qbG3Mz7JKma2rhO9w/edit?usp=share_link

[paekva]

Solution: description + requirements:

Survey data cleansing tool:

• It is a web application that provides users the possibility to clean provided data by one or more predefined cleansing rules and downloads or visualizes resulting data
• The rules are described with a YAML config or selected via the interface
• Rules may include removing unanswered questions from the statistics, checking controlling questions on answers, removing irrelevant answers, etc

Main workflow:

1. The user uploads the file with data to be cleaned
   1.1 Validate the uploaded file
2. Pre-cleaning statistics & visualization (DEFINE what is useful to show: list of questions, with guessed type)
3. The user uploads the cleansing config (as a file) or user defines it on the interface
   3.1 (if the config is uploaded) The config is validated
4. The cleaning is performed
5. The result is ready
   5.1 The cleaned data returned in the same format and could be downloaded
   5.2 Post-cleaning statistics & visualization
6. Another iteration of cleaning if needed (return to 3)

Requirements:

• The user should be able to define cleaning configuration via interface or via file upload

• The user should be able to define the main types of rules for cleansing:

  • empty response check (overall)
  • incomplete answer check (rule for how to check response)
  • not relevant answer check (mark questions as dependent on each other)
  • control questions check (mark questions that should have the same or opposing answers)

• The user should be able to define the main types of handling issues in config:

  • Removing
  • Replacing
  • Notifying

• The user should be able to see if the provided config is valid

• The user should be able to load data files for cleansing

• The user should be able to download the resulting file with cleaned data

• The user should be able to get some basic visualization charts (e.g. pie charts)

[paekva]

Figma protopype: https://www.figma.com/file/e9ZctsqYKcSntPBnMvTLMW/Cleansing-tool?node-id=1%3A21&t=adR62mG6d5rixHhV-0

[sunny-26]

Our value: the tool can decrease the time that analysts spend on the data cleaning which allows to spend less time on data processing to get the survey results faster.

[paekva]

Discussion 03.12

• Write a Value statement @sunny-26
• Add more specific target groups to the context @Ovaday
• Decide on which data we store (how much data? how sensitive?) @mraxemax
• Think about the Security of our application - what approaches do we need? (what parts do we need to secure?) @mraxemax
• User management: registration & login flow (should be a part of our high-level req) @Ovaday
• .
• Create app use example in document: Data + cleaning rules + data analysis report (before and after cleaning) @Nikvasil
• Draw closely how the rules could be defined on the UI & config @Ovaday
• Come up with a tool name @Nikvasil
• Figma demo update @paekva
• Usecase diagram @sunny-26
• Arch. overview diagram @Ovaday
• .
• Create a presentation structure @sunny-26
• .
• Define scenarios for cleaning: action, how it could be described by the user, how we on BE will be able to apply it) @Ovaday @paekva
  empty response check (overall)
  filter (provide rule)
  control questions check (mark questions that should have the same or opposing answers)

App usage scenarios:
https://docs.google.com/document/d/1SlZ2LIDs6FX3mTBAU6XQbbh7Q_Dx_-W1zHsFDb5QaC0/edit?usp=share_link

[sunny-26]

Cleaning includes:

Remove partially completed responses.
Remove answers from the untargeted group.
Remove the responses which were done too fast.
Remove straighteners (the same answers in all questions (I am not sure, zB)).
Remove unrealistic responses.
Remove outliers.
Remove inconsistent answers (the user should predefine the dependencies which the tool later can analyze and detect inconsistency).

[mraxemax]

Security:

Application Security:
We can use encryption for the database mean hash method and HTTPS for data flow to the database.
User should use mixed password mean special characters, numbers, and alphabets.

DNS Security:

We should implement DNS security so the attacker cant trace our hosting.
for this purpose we can use cloudflare which is using by big companies. and it is free of cost. so it will help us for making our website live without denial services. also we need to add a captcha google provides a captcha service that allows the real user only not robots.

Hosting Security:

Where we can store our data?
We will use AWS Ec2 and RDS Database System.
Why these services because EC2 provides top security with rules and scaling. So you can only allow specific ports such as HTTP and HTTPS. for SSH we can add specific ips so it will help us only we can connect.

For more security, we can use AWS loadbalancer as well which will create a new layer between the user and hosting.

Why RDS?

RDS is one of the best database systems which allow us scaling and top security so the attacker can't access our data because it is privately available. And only communicate with the ec2 server.

[Nikvasil]

Product name variants:

• Cleansyd (like "Cleansed", but "y" instead of "e" )
• Cleanify (just a cool name, but it's easy to confuse with Spotify)
• Sudacle (SUrvey DAta CLEaning)
• Easy Survey (simple, but still...)

I also tried different name generators and it works in the next way: Adjective + "Survey/Data" or "Clean/Survey" + some ending. So, if you do not like any of these options, I can simply throw more ideas from the generator. Or, we can associate the product with some animal, for example, an ant or a snail (if you know better cleaning animal examples in nature, pls let me know =) ).

[Ovaday]

Sample config:
https://docs.google.com/document/d/17ETKTROwesbzZZ6r0AGvhaMDNzz1mEDC1YXaNwX7okw/edit?usp=share_link
UI config (my version) in figma and discussed with Kate

[Ovaday]

User management in Django:
Django provides us with built-in tools to store user data with a ORM layer to protect from developer and architectural miatakes.

First of all, there is provided simple interface by default, which allows us to handle basic registration, user login and management panel.

User structure includes two types of records: admin and regular user, they are defined by giving the permission from a root and further represented by a flag on a user account .isAdmin.
Only admins can access the management panel and add/edit/see other users.

Passwords are not stored, only the hash of them, with a special algorythims that are used by Django.
The database for user records and technical information from Django is SqlLite

[Ovaday]

there are 2 possible ways of how we organise our architecture. They depend on the question, do we store temporary files or not.

in case if we want to store:

in case if not: