[WIP] DL4J toSameDiff conversion method

[rnett]

Adding a toSameDiff method + overloads to MultiLayerNetwork and ComputationGraph, and adding the used define* methods for layers, vertices, activations, and losses.

Need a SameDiff Mish function.
Currently no support for Truncate convolution mode in SameDiff.

rectifiedTanh and rationalTanh were in SameDiff.math but not SameDiff.nn..

Needed new SameDiff functions for optimization:

• a RELU that supports leaky and a custom threshold at the same time.
• A non-weighted cross entropy loss. The weightedCrossEntropyWithLogits javadoc says it supports null weights, but I get an exception. Same for BinaryCrossentropy
• For LossSparseMCXENT, a version of OneHot that takes depth as a SDVariable. The current version takes a double and should be passed input.shape[-1], which I can't get w/o offline shape inference.
• A 1d ops for subsampling and upsampling op.
• a way to set depthMultiplier for depthwise convolutions
• weight format on all convolution configurations

Not being implemented in the first pass:

• Losses
  • LossMultiLabel
  • OCNNLossFunction
  • LossMixtureDensity
• Activations
  • ActivationMish
• Layers
  • Mask Layer (no mask support yet)
  • GlobalPoolingLayer (no SameDiff op?)
  • MaskZeroLayer
  • TFOpLayer
  • Yolo2OutputLayer
  • CenterLossOutputLayer (should be done as a loss function)
  • OCNNOutputLayer maybe?
  • EmbeddingLayer (JVM crash issues)
  • EmbeddingSequenceLayer
  • Autoencoder and VAE
• Vertices (for the most part, because there's no way to get int rank from a SDVariable. A SDIndex.ellipses() would solve this too)
  • SubsetVertex
  • DuplicateToTimeSeriesVertex
• Every dropout but Dropout (needs SameDiff support).

Need to add support for custom ILossLayer.computeScore methods, e.g. CnnLossLayer.

What's the difference between FrozenLayerWithBackprop and FrozenLayer?

• toSameDiff
  • MultiLayerNetwork
  • ComputationGraph
• define*
  • Layers
  • Vertices
  • Activations
  • Losses
  • Dropout (and support).
• Updater state
• MNIST test
• Add to layer tests
• Add to loss gradient tests

[AlexDBlack]

Overall looks good so far. See comments though.

[AlexDBlack]

Hm... it's only global ATM, true. Might be worth adding (just not in this PR). Another issue to be opened perhaps.

[AlexDBlack]

loss.shape().get(SDIndex.point(0)) -> sd.sizeAt(loss, 0)

[rnett]

Can we add that to SDVariable (not this PR)? squeeze and expandDims too.

[AlexDBlack]

Only half way through the second review. I'll submit another review for the rest soon.

Main issues here are:
(a) We'll need to work out a way to do reshapes, permutes, etc of weights at the INDArray level (not SDVariable level) for performance and memory reasons.
(b) We'll need some sort of coverage checking. There's 2 models we can use for that:
https://github.com/eclipse/deeplearning4j/blob/master/deeplearning4j/deeplearning4j-core/src/test/java/org/deeplearning4j/nn/dtypes/DTypeTests.java
or
https://github.com/eclipse/deeplearning4j/blob/master/nd4j/nd4j-backends/nd4j-tests/src/test/java/org/nd4j/OpValidationSuite.java

Having the tests in the gradient checks is definitely good, and we don't want to needlessly write redundant tests...
That said, DTypeTests approach allows us to assert that all layers/preprocessors/etc are checked though (and fail a test if not), which is definitely nice. i.e., it stops us introducing a bug if we add a new layer and forget to write/test the SameDiff conversion.

[AlexDBlack]

It would also be great to test fitting - i.e., parameters are the same after each fit step, for say 3 steps.

[AlexDBlack]

Let's try to isolate this crash.
If it's the squeeze (bad shape) we can maybe wokr around it via .reshape(-1).castTo(DataType.INT64)

Also int32 input should be fine for this too
https://github.com/eclipse/deeplearning4j/blob/master/libnd4j/include/ops/declarable/generic/transforms/gather.cpp#L88