Why does do glove embeddings get dropped differently

[zkokaja]

Why is there a special case for glove? If operating on base dfs, it should generalize to any type of model.

247-encoding/scripts/tfsenc_read_datum.py

Lines 184 to 188 in 03e73d2

	if args.emb_type == "glove":
	df = df.dropna(subset=["embeddings"])
	else:
	df = drop_nan_embeddings(df)
	df = remove_punctuation(df)

[zkokaja]

Related to #57. We should test whether the else case works for glove then we can remove the special case.

[zkokaja]

Glove puts None for words with no embeddings, so it should work https://github.com/hassonlab/247-pickling/blob/4588392872b7491a8a7a52cee553968ac025722e/scripts/tfsemb_main.py#L603-L607

[zkokaja]

next step: we just need to test whether glove will work with drop_nan_embeddings and we can remove this if condition. once ken figures out the token thing

[VeritasJoker]

I tried drop_nan_embeddings with glove. Got this error:

*** TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Will look into it

[zkokaja]

If x is None this could cause the error

247-encoding/scripts/tfsenc_read_datum.py

Line 22 in 03e73d2

df["is_nan"] = df["embeddings"].apply(lambda x: np.isnan(x).all())