You can register custom categories using the RegisterAdvancedAssetCategory function from the AssetTools module. You’d then need to use the registered category flag in the GetCategories function of your custom type asset actions.
The AI module provides an example of doing this. See FAIModule::StartupModule for the registration, and FAssetTypeActions_Blackboard::GetCategories for an example of using the registered flag in the asset type actions.
I notice that tutorial doesn’t actually cover creating asset type actions, however they’re pretty simple. FAssetTypeActions_Enum provides a pretty basic example, just remember to register them with the AssetTools module (via RegisterAssetTypeActions).