sml-punycode

Punycode (RFC 3492, "A Bootstring encoding of Unicode for Internationalized Domain Names") in pure Standard ML, plus a thin, mechanical IDNA-style ToASCII/ToUnicode label wrapper.

No dependencies, no FFI, no threads, no clock, no randomness: the same inputs always produce the same outputs under MLton and Poly/ML.

Representation

Unicode text is represented as int list, one element per Unicode code point (not UTF-16 code unit, not UTF-8 byte). SML's string/char are byte sequences, not code-point sequences, so an SML string cannot directly hold arbitrary Unicode text. This library requires the caller to have already decoded their input (e.g. from UTF-8) into a list of code points before calling encode, and to re-encode the list decode/toUnicode return back into whatever external representation they need. UTF-8 decoding itself is out of scope. toAscii/toUnicode still take/return plain SML string, since the Bootstring/base36 payload itself is guaranteed pure ASCII by construction (that's the whole point of Punycode).

toAscii/toUnicode are a deliberately thin, mechanical wrapper, not a full IDNA (RFC 3490/5891) implementation: there is no Nameprep/IDNA2008 mapping, no bidi rule checking, and no label-length (63 octet) enforcement.

API

signature PUNYCODE =
sig
  (* Unicode code points -> Bootstring/base36 ASCII payload, no "xn--"
     prefix. NONE on an invalid code point (negative, > U+10FFFF, or a
     UTF-16 surrogate U+D800..U+DFFF) or RFC 3492 sec 6.4 overflow. *)
  val encode : int list -> string option

  (* Bootstring/base36 ASCII payload (no "xn--" prefix) -> Unicode code
     points. NONE if malformed, truncated, or overflowing. Accepts
     mixed-case base36 digits per the RFC's decoder requirement. *)
  val decode : string -> int list option

  (* Full (mechanical) IDNA-label ToASCII: pure-ASCII input passes through
     unchanged; otherwise Punycode-encoded and prefixed with "xn--". *)
  val toAscii : int list -> string

  (* Full (mechanical) IDNA-label ToUnicode: strips a literal "xn--" prefix
     and decodes the remainder, or passes non-prefixed input through as
     code points unchanged. *)
  val toUnicode : string -> int list option
end

Example

val SOME puny = Punycode.encode
  [0x0644,0x064A,0x0647,0x0645,0x0627,0x0628,0x062A,0x0643,0x0644,
   0x0645,0x0648,0x0634,0x0639,0x0631,0x0628,0x064A,0x061F]
(* puny = "egbpdaj6bu4bxfgehfvwxn"  -- RFC 3492 sec 7.1 sample (A), Arabic *)

val label = Punycode.toAscii
  [0x0644,0x064A,0x0647,0x0645,0x0627,0x0628,0x062A,0x0643,0x0644,
   0x0645,0x0648,0x0634,0x0639,0x0631,0x0628,0x064A,0x061F]
(* label = "xn--egbpdaj6bu4bxfgehfvwxn" *)

Running examples/demo.sml with make example prints:

Punycode encode/decode (RFC 3492 sec 7.1 sample strings):
  (A) Arabic  encode -> "egbpdaj6bu4bxfgehfvwxn"
             decode <- [1604,1610,1607,1605,1575,1576,1578,1603,1604,1605,1608,1588,1593,1585,1576,1610,1567]
  (D) Czech   encode -> "Proprostnemluvesky-uyb24dma41a"
             decode <- [80,114,111,269,112,114,111,115,116,283,110,101,109,108,117,118,237,269,101,115,107,121]
  (M) amuro   encode -> "-with-SUPER-MONKEYS-pc58ag80a8qai00g7n9n"
             decode <- [23433,23460,22856,32654,24693,45,119,105,116,104,45,83,85,80,69,82,45,77,79,78,75,69,89,83]

IDNA-style label wrapper (toAscii / toUnicode):
  toAscii pure-ASCII "cafe"    = "cafe" (unchanged, no xn--)
  toAscii Arabic label         = "xn--egbpdaj6bu4bxfgehfvwxn"
  toUnicode of that label       = [1604,1610,1607,1605,1575,1576,1578,1603,1604,1605,1608,1588,1593,1585,1576,1610,1567]
  toAscii Czech label           = "xn--Proprostnemluvesky-uyb24dma41a"
  toUnicode of that label       = [80,114,111,269,112,114,111,115,116,283,110,101,109,108,117,118,237,269,101,115,107,121]

Build & test

Requires MLton and/or Poly/ML.

make test        # build + run the suite under MLton
make test-poly   # run the suite under Poly/ML
make all-tests   # both, plus the byte-identical gate
make example     # build + run the demo
make clean

Installing with smlpkg

smlpkg add github.com/sjqtentacles/sml-punycode
smlpkg sync

Reference lib/github.com/sjqtentacles/sml-punycode/punycode.mlb from your own .mlb (MLton / MLKit), or feed sources.mlb to tools/polybuild (Poly/ML).

Layout

sml.pkg                                       smlpkg manifest
Makefile                                      MLton + Poly/ML targets
.github/workflows/ci.yml                      CI: MLton + Poly/ML
lib/github.com/sjqtentacles/sml-punycode/
  punycode.sig   PUNYCODE signature
  punycode.sml   Bootstring encode/decode + toAscii/toUnicode
  sources.mlb    ordered source list
  punycode.mlb   public basis
examples/
  demo.sml       RFC sample-string encode/decode + IDNA wrapper walkthrough
test/
  harness.sml    shared assertion harness
  test.sml       RFC 3492 sec 7.1 sample-string vectors + round trips (47 checks)
  entry.sml / main.sml
tools/polybuild  Poly/ML build wrapper

Tests

47 deterministic checks, primarily transcribed from RFC 3492 section 7.1 "Sample strings": Arabic, Chinese (simplified), Czech, Russian, and a mixed Japanese/ASCII sample (<amuro><namie>-with-SUPER-MONKEYS), each checked both directions (encode and decode) plus full round trips (encode(decode(x)) = x, decode(encode(x)) = x).

The Russian vector uses the corrected Punycode string per IETF errata #3026 against RFC 3492: the published RFC text has a typo'd uppercase D in b1abfaaepdrnnbgefbaDotcwatmq2g4l that should be lowercase d (b1abfaaepdrnnbgefbadotcwatmq2g4l), confirmed by the RFC's own decoding/encoding pseudocode.

Also covered: mixed-case base36-digit decoding (RFC 3492 sec 5's decoder requirement), invalid-code-point rejection (negative, beyond U+10FFFF, UTF-16 surrogates), malformed/truncated Punycode payload rejection, the Bootstring algorithm's pure-ASCII trivial case (encode appends a bare - delimiter, per sample (S) -> $1.00 <- -> -> $1.00 <--), and the toAscii/toUnicode IDNA-label wrapper (which, unlike raw encode, passes pure-ASCII labels through completely unchanged with no trailing delimiter).

Run make all-tests to verify identical output under both compilers.

License

MIT. See LICENSE.