streaming functions for doobie and slick with tests#108
streaming functions for doobie and slick with tests#108salamonpavel wants to merge 93 commits intomasterfrom
Conversation
…ts and mixed into db functions with status
slick/src/it/scala/za/co/absa/fadb/slick/SlickStreamingResultFunctionTest.scala
Show resolved
Hide resolved
lsulak
left a comment
There was a problem hiding this comment.
Found a few tiny things and Jacoco coverage action is still failing (not on insufficient coverage though), so please take a look. Otherwise LGTM.
I pulled it, reviewed, compiled and ran tests.
# Conflicts: # README.md # build.sbt # core/src/main/scala/za/co/absa/fadb/DBFunction.scala # core/src/main/scala/za/co/absa/fadb/DBSchema.scala # doobie/src/main/scala/za/co/absa/fadb/doobie/DoobieFunction.scala # project/Dependencies.scala # slick/src/main/scala/za/co/absa/fadb/slick/SlickFunction.scala
JaCoCo code coverage report - scala 2.12.17
Files
|
JaCoCo code coverage report - scala 2.13.12
Files
|
Fixed this. The problem was in node version of Github's updated runner. Solved by an update of the jacoco github action version. |
|
@lsulak @jakipatryk Thanks for the initial review here and on the calls we had yesterday. Based on your comments I have made the streaming part for Slick as a separate package since additional dependencies are needed and we don't want to be importing transitive dependencies into our projects when streaming is not used. At the moment there is core module for non-streaming, streaming module for streaming, slick module for slick non-streaming, slick-streaming module for slick streaming and doobie module. Doobie module provides both non-streaming and streaming functions since Doobie already bring all the dependencis needed (works with fs2 streams under the hood) so there would be no benefit in splitting this into separate modules. |
| ${{ github.workspace }}/streaming/target/scala-${{ matrix.scalaShort }}/jacoco/report/jacoco.xml, | ||
| ${{ github.workspace }}/slick/target/scala-${{ matrix.scalaShort }}/jacoco/report/jacoco.xml | ||
| ${{ github.workspace }}/slick-streaming/target/scala-${{ matrix.scalaShort }}/jacoco/report/jacoco.xml | ||
| ${{ github.workspace }}/doobie/target/scala-${{ matrix.scalaShort }}/jacoco/report/jacoco.xml |
There was a problem hiding this comment.
I do not agree with removing the code coverage fail-check logic - see the comment on the end of the file.
As I know there is no hard merge block enabled.
There was a problem hiding this comment.
@miroslavpojer I am not satisfied with this situation either. Unfortunately I couldn't perform merge of previous pull request with this check in place. Let's work together to find a good solution so we can perform tests against a database both locally and within a Github pipeline; for instance with a dockerized Postgres instance.
| paths: > | ||
| ${{ github.workspace }}/core/target/scala-${{ matrix.scalaShort }}/jacoco/report/jacoco.xml, | ||
| ${{ github.workspace }}/streaming/target/scala-${{ matrix.scalaShort }}/jacoco/report/jacoco.xml, | ||
| ${{ github.workspace }}/slick/target/scala-${{ matrix.scalaShort }}/jacoco/report/jacoco.xml |
There was a problem hiding this comment.
| ${{ github.workspace }}/slick/target/scala-${{ matrix.scalaShort }}/jacoco/report/jacoco.xml | |
| ${{ github.workspace }}/slick/target/scala-${{ matrix.scalaShort }}/jacoco/report/jacoco.xml, |
| ${{ github.workspace }}/core/target/scala-${{ matrix.scalaShort }}/jacoco/report/jacoco.xml, | ||
| ${{ github.workspace }}/streaming/target/scala-${{ matrix.scalaShort }}/jacoco/report/jacoco.xml, | ||
| ${{ github.workspace }}/slick/target/scala-${{ matrix.scalaShort }}/jacoco/report/jacoco.xml | ||
| ${{ github.workspace }}/slick-streaming/target/scala-${{ matrix.scalaShort }}/jacoco/report/jacoco.xml |
There was a problem hiding this comment.
| ${{ github.workspace }}/slick-streaming/target/scala-${{ matrix.scalaShort }}/jacoco/report/jacoco.xml | |
| ${{ github.workspace }}/slick-streaming/target/scala-${{ matrix.scalaShort }}/jacoco/report/jacoco.xml, |
lsulak
left a comment
There was a problem hiding this comment.
LGTM, just read the code
Added support for streaming for both Slick and Doobie. Since Slick works with Futures but for streaming it relies on IOs or other effect types there was a need to abstract the streaming operation in a new DBStreamingEngine.
There are following modules:
Closes #107