Feature proposal: MQTT v5.0 Enhanced Authentication (AUTH packet) support

[guclumhg]

Is your feature request related to a problem?

coreMQTT 5.0 supports single-step authentication (Authentication Method + Authentication Data sent once in CONNECT, accepted or rejected by the broker), but it does not support the MQTT v5.0 Enhanced Authentication flow that uses the AUTH control packet for multi-step, challenge-response exchanges.

As a result, SASL-style mechanisms such as SCRAM-SHA-256, Kerberos/GSSAPI, and OAuth token exchange / refresh cannot be used, and re-authentication on an established connection is not possible. These are commonly required by enterprise brokers (HiveMQ, EMQX) and managed services.

Current state on main (5be5f95)

The AUTH packet is recognized by the serializer/property layer but not driven end to end:

• During the connect handshake, any non-CONNACK packet (i.e. an incoming AUTH) is rejected — with an explicit placeholder:
  

  coreMQTT/source/core_mqtt.c

  Lines 3517 to 3529 in 5be5f95

  	if( pIncomingPacket->type == MQTT_PACKET_TYPE_CONNACK )
  	{
  	status = receiveConnackPacket( pContext,
  	*pIncomingPacket );
  	}
  	/* TODO: Handle AUTH packets here as well. */
  	else
  	{
  	LogError( ( "Incorrect packet type %X received while expecting"
  	" CONNACK(%X).",
  	( unsigned int ) pIncomingPacket->type,
  	MQTT_PACKET_TYPE_CONNACK ) );
  	status = MQTTBadResponse;

  (/* TODO: Handle AUTH packets here as well. */, then MQTTBadResponse)
• In the main receive loop, MQTT_PACKET_TYPE_AUTH (0xF0) is not a case in the packet switch, so an incoming AUTH falls through to default and returns MQTTBadResponse.
• There is no public API to send an AUTH packet (MQTT_Auth) and no MQTT_REASON_CONTINUE_AUTHENTICATION (0x18) / MQTT_REASON_REAUTHENTICATE (0x19) reason codes.

Notably, the building blocks already exist: the Authentication Method (0x15) and Authentication Data (0x16) properties are defined with MQTTPropAdd_AuthMethod/MQTTPropAdd_AuthData and MQTTPropGet_*, MQTT_Connect already accepts a MQTTPropBuilder_t, and MQTTEventCallback_t already carries both an incoming-property buffer and an outgoing-property buffer.

Describe the solution you'd like

A backward-compatible addition (opt-in; no change for users who don't authenticate this way), which would be a natural minor release (5.1):

1. Send path — a new MQTT_Auth() API that serializes an AUTH packet with a reason code (0x18 Continue, 0x19 Re-authenticate) and Authentication Method/Data properties, usable both mid-handshake and on an established connection.
2. Receive path — handle incoming AUTH in receiveConnack (the TODO above) and in the MQTT_ProcessLoop packet switch, delivering it to the application through the existing MQTTEventCallback_t. The callback signature already fits: pGetPropsBuffer exposes the broker's Authentication Data, and pSendPropsBuffer lets the application supply the next response.
3. Reason codes — add MQTT_REASON_CONTINUE_AUTHENTICATION and MQTT_REASON_REAUTHENTICATE to MQTTSuccessFailReasonCode_t.

The mechanism-specific computation (SCRAM, Kerberos, ...) stays in application code; the library only moves AUTH packets and hands their properties to/from the callback, consistent with how coreMQTT already treats properties elsewhere.

Questions for maintainers

Before proposing an implementation I'd like to check scope and preferred shape:

• Is Enhanced Authentication in scope for a future release, or intentionally left to the application layer?
• For the receive path, do you prefer reusing MQTTEventCallback_t as above, or a dedicated auth callback?
• For the send path during the handshake, should MQTT_Connect drive the AUTH exchange internally until CONNACK, or should it return control to the application between AUTH round-trips?

I'm happy to work on this (with unit tests and CBMC proofs matching the existing style) if there's interest and agreement on the shape.