Skip to content

Hail via Livy on PIC-SURE #79

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
DeniseKersjes wants to merge 27 commits into
base: master
Choose a base branch
from
Open

Hail via Livy on PIC-SURE #79

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
6d441b9
Initial commit
DeniseKersjes Aug 9, 2018
8ef3f2c
Add Hail resource
DeniseKersjes Aug 10, 2018
03cb499
Suppress the warning of duplication code (from HAIL.java)
DeniseKersjes Aug 22, 2018
8905157
ignore
jgeofil Aug 22, 2018
d65f0c9
Create a Livy session via IRCT
DeniseKersjes Aug 23, 2018
81c21f4
Create a livy session via IRCT
DeniseKersjes Aug 23, 2018
c52fc83
Create a POST request with a PySpark template containing Hail variables
DeniseKersjes Aug 24, 2018
ceeafc9
Map the Livy state to the ResourceState
DeniseKersjes Aug 27, 2018
25ab838
Reformat the code
DeniseKersjes Aug 27, 2018
760759e
Set the session kind to PySpark
DeniseKersjes Aug 27, 2018
f067cbd
Get the Hail output result from the output JSON file
DeniseKersjes Aug 28, 2018
8165307
Export the data output
DeniseKersjes Aug 28, 2018
17728a0
Reformat the Hail template to handle multiple variables in the fields…
DeniseKersjes Aug 30, 2018
3593c4f
Parse the output data in correct table format
DeniseKersjes Aug 31, 2018
0ad0f88
Soft-coded the datafile selection
DeniseKersjes Sep 4, 2018
21bf1b1
Hail and datafiles are imported during the IRCT set up instead of dur…
DeniseKersjes Sep 5, 2018
32dc34d
Removed unnecessary code
DeniseKersjes Sep 5, 2018
0069e50
Handle .tsv files and remove unused import statements
DeniseKersjes Sep 6, 2018
6040764
Add json dependency
DeniseKersjes Sep 7, 2018
a2fc02b
Solve issue of parsing the output
DeniseKersjes Sep 7, 2018
c1de650
Resolve the issue of skipping one data row while parsing the json
DeniseKersjes Sep 10, 2018
ab31193
Remove 'index' column from parsed header
DeniseKersjes Sep 10, 2018
44fceda
Add documentation to the (existing) functions
DeniseKersjes Sep 12, 2018
a5c976c
Add assumptions
DeniseKersjes Sep 12, 2018
40a2162
Revert "Add Hail resource"
DeniseKersjes Sep 17, 2018
08ed4ef
Revert "ignore"
DeniseKersjes Sep 17, 2018
f2eec8b
Add note about the docker image usage
DeniseKersjes Sep 17, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions IRCT-RI/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,11 @@
<artifactId>jaxp-ri</artifactId>
<version>1.4.5</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.json</groupId>
<artifactId>json</artifactId>
<version>20180813</version>
</dependency>
</dependencies>
<build>
Expand Down
Loading