Manual C header

[Cryoris]

Summary

This PR implements the headers for the C API manually and removes the dependency on cbindgen.
This needs rebasing on top of #14340.

Details

The header structure here is proposed as

cext/crates/
   include/
       qiskit.h  // main header that we include in C programs. Can also hold metadata, like the version.
       qiskit/  // dependencies
           circuit.h
           observable.h
           <... and all other headers...>

Why remove cbindgen?

cbindgen is a tool we currently use to automatically generate the header for the C FFI for Rust, off the .rs files that expose Rust structs, enums and functions. It's been super useful in starting the C API, but we're already hitting some limitations and expecting some more serious limitations in the future. The two main points IMO are:

• cbindgen treats all objects as part of a global namespace: this means that trying to expose an object some_namespace::Thing to C only works if there is no other object called Thing in the parsed Rust API. This is a fundamental limitation in cbindgen, which they also mention in their docs.
• the project is maintained on a rather infrequent basis. This doesn't generally need to be an issue, but there are a series of small problems/missing features we have that I think could be solved rather easily in cbindgen. But that requires active PR and issue reviews.

Smaller problems we can work around (but that get a bit annoying and sum up):

• cbindgen pulls the Rust docs of objects we expose as opaque pointers. For example, we expose SparseObservable as typedef struct SparseObservable SparseObservable to C, which causes the header to include the Rust docstring. We currently solve this by manually writing the Sphinx docs for these objects. This also affects enums we directly expose, such as BitTerm.
• We need to manually define object renames, due to missing flexibility in cbindgens export control.
• We require some custom headers anyways (e.g. complex number definitions) and end up with a mix of generated headers and tracked headers, which we need to orchestrate.

Why keep cbindgen?

• If we write the header manually, we need to include the docs there. This either leads to doc duplication or less extensive docs in the Rust code that exposes the C FFI.
• So far we've been able to work around all issues. The smaller issues listed above could be resolved with PRs to cbindgen (as we're trying with Allow excluding prefixes for objects mozilla/cbindgen#1064). The global namespace could be a dealbreaker in the longer run, though.

Todos

• Rebase on top of Directly expose QkComplex64 to the C API #14340, once merged
• Verify the docs are still rendered correctly
• Decide whether the Rust docstrings should be trimmed down, to avoid doc duplication

[qiskit-bot]

One or more of the following people are relevant to this code:

• @Eric-Arellano
• @abbycross
• @beckykd
• @Qiskit/terra-core

[jakelishman]

Shall we close this PR? For the time being at least, we don't seem to be heading in this direction, and the PR's just going to get super stale if left open.

[jakelishman]

Since Julien didn't reply, I'll just close it (since I assume that's the correct state) and see if he re-opens it to indicate I was wrong.

[Cryoris]

Whoops sorry, this fell off my radar. I would expect that sooner or later we'll want to do this for the reasons listed in the description, but I'm also fine keeping it for now since we were able to circumnavigate problems so far.