colmetadata does not read custom metadata with multiple writes

[JakeZw]

This problem is described in more detail in discourse.

Specifically the following MWE works when the for loop executes once, but the colmetadata gives nothing when the for loop executes more than once.

`using Arrow
using Tables

go to runtest.jl in Arrow package for hints on how to use metadata and colmetadata

cd(@DIR)
filename = "test10.arrow"
meta = ["file" => "test.arrow", "when" => "now", "why" => "gain experience"]
metacol1 = ["id" => "chan1", "comment" => "Chan 1 comment"]
metacol2 = ["id" => "chan2", "comment" => "Chan 2 comment"]
writer = open(Arrow.Writer, filename, metadata=meta, colmetadata = Dict(:Column1 => metacol1, :Column2 => metacol2))

for i in 1:1
result = rand(2, 2) .+ i
Arrow.write(writer, Tables.table(result)) #, header=[:a,:b])
end
close(writer)

tbl = Arrow.Table(filename)

tblmeta = Arrow.getmetadata(tbl)
tblmeta
tblmeta["why"]
tblmeta["when"]

tblmetachan1 = Arrow.getmetadata(tbl.Column1)
tblmetachan1
tblmetachan1 == nothing
`

For once through the for loop we get
julia> tblmetachan1 = Arrow.getmetadata(tbl.Column1) Base.ImmutableDict{String, String} with 2 entries: "comment" => "Chan 1 comment" "id" => "chan1"

When we go through the for loop twice (or more) we get
`julia> tblmetachan1 = Arrow.getmetadata(tbl.Column1)

julia> tblmetachan1

julia> tblmetachan1 == nothing
true`

[ericphanson]

Setup code:

using Arrow
using Tables

# Write an Arrow file with table metadata and column metadata.
# The same schema is written in multiple record batches when nbatches > 1.
function write_issue485_file(path; nbatches::Int)
    rm(path; force=true)

    filemeta = ["file" => path, "note" => "issue #485 repro"]
    colmeta1 = ["id" => "chan1", "comment" => "Column1 metadata"]
    colmeta2 = ["id" => "chan2", "comment" => "Column2 metadata"]

    writer = open(
        Arrow.Writer,
        path;
        metadata=filemeta,
        colmetadata=Dict(:Column1 => colmeta1, :Column2 => colmeta2),
    )

    try
        for i in 1:nbatches
            # Use the same write path as the original issue:
            # a matrix converted with Tables.table(...)
            data = rand(2, 2) .+ i
            Arrow.write(writer, Tables.table(data))
        end
    finally
        close(writer)
    end
end

# Read the file back and print metadata at both levels:
#   1) the top-level combined Arrow.Table column
#   2) each partition / record batch separately
function inspect_file(path)
    tbl = Arrow.Table(path)

    println("file: ", path)
    println("typeof(tbl.Column1): ", typeof(tbl.Column1))
    println("table metadata: ", Arrow.getmetadata(tbl))
    println("top-level Column1 metadata: ", Arrow.getmetadata(tbl.Column1))
    println("top-level Column2 metadata: ", Arrow.getmetadata(tbl.Column2))

    # Partition-local columns keep their metadata even in the failing case.
    println("partition-local Column1 metadata:")
    for (i, part) in enumerate(Tables.partitions(tbl))
        println("  partition ", i, ": ", Arrow.getmetadata(part.Column1))
    end
    println()
end

gives

julia> write_issue485_file("issue485_one.arrow"; nbatches=1)

julia> inspect_file("issue485_one.arrow")
file: issue485_one.arrow
typeof(tbl.Column1): Arrow.Primitive{Float64, Vector{Float64}}
table metadata: Base.ImmutableDict("note" => "issue #485 repro", "file" => "issue485_one.arrow")
top-level Column1 metadata: Base.ImmutableDict("comment" => "Column1 metadata", "id" => "chan1")
top-level Column2 metadata: Base.ImmutableDict("comment" => "Column2 metadata", "id" => "chan2")
partition-local Column1 metadata:
  partition 1: Base.ImmutableDict("comment" => "Column1 metadata", "id" => "chan1")

so things work as expected with top-level column metadata when we have 1 batch, but with 2:

julia> write_issue485_file("issue485_two.arrow"; nbatches=2)

julia> inspect_file("issue485_two.arrow")
file: issue485_two.arrow
typeof(tbl.Column1): SentinelArrays.ChainedVector{Float64, Arrow.Primitive{Float64, Vector{Float64}}}
table metadata: Base.ImmutableDict("note" => "issue #485 repro", "file" => "issue485_two.arrow")
top-level Column1 metadata: nothing
top-level Column2 metadata: nothing
partition-local Column1 metadata:
  partition 1: Base.ImmutableDict("comment" => "Column1 metadata", "id" => "chan1")
  partition 2: Base.ImmutableDict("comment" => "Column1 metadata", "id" => "chan1")

now the top-level column metadata is nothing, but the individual partitions still have the metadata.

I think there is a missing getmetadata method, e.g. this works:

julia> using Arrow, SentinelArrays

julia> function Arrow.getmetadata(x::SentinelArrays.ChainedVector{T,A}) where {T,A<:Arrow.ArrowVector}
           isempty(x.arrays) && return nothing

           m0 = Arrow.getmetadata(first(x.arrays))
           for a in Iterators.drop(x.arrays, 1)
               m = Arrow.getmetadata(a)
               if m != m0
                   throw(ArgumentError("inconsistent column metadata across ChainedVector chunks"))
               end
           end
           return m0
       end
getmetadata (generic function with 5 methods)

julia> inspect_file("issue485_two.arrow")
file: issue485_two.arrow
typeof(tbl.Column1): ChainedVector{Float64, Arrow.Primitive{Float64, Vector{Float64}}}
table metadata: Base.ImmutableDict("note" => "issue #485 repro", "file" => "issue485_two.arrow")
top-level Column1 metadata: Base.ImmutableDict("comment" => "Column1 metadata", "id" => "chan1")
top-level Column2 metadata: Base.ImmutableDict("comment" => "Column2 metadata", "id" => "chan2")
partition-local Column1 metadata:
  partition 1: Base.ImmutableDict("comment" => "Column1 metadata", "id" => "chan1")
  partition 2: Base.ImmutableDict("comment" => "Column1 metadata", "id" => "chan1")

[JakeZw]

What is the next step. It would be super to have this merged in Arrow.jl. Would it go into \src\table.jl?