How we wrote a GPU-based Gaussian Splats viewer in Unreal with Niagara

Written By Alessio Regalbuto

In this article, I want to share our journey of writing a fully functional Gaussian Splat viewer for Unreal Engine 5, starting right from the ground up.

Getting the ball rolling

First of all, let鈥檚 quickly recap what Gaussian Splatting is. In short, it鈥檚 a process that produces something similar to a point cloud, where instead of each point, a colored elliptic shape is used. This changes and stretches, depending on the camera position and perspective to blend into a continuous space representation. This helps to keep visual information such as reflections and light shades intact in the captured digital twin, retaining details as realistically as possible.

For more info, feel free to check out my previous articles:

The first challenge was to understand specifically what format a Gaussian Splat file uses, and which one is the most commonly accepted by the industry. After in-depth research, we identified two main formats that are currently popular: .ply and .splat.

After some consideration, we chose the .ply format as it covered a wider range of applications. This decision was also driven by looking at other tools such as , which allows importing Gaussian Splats in the form of .ply files only, even if it also offers to export them as .splat files.

What does a .PLY file look like?

There are two different types of .ply files to start with:

  • ASCII based ply files, which store data in textual form.

  • Binary based ply files, which are less readable.

We can think of a ply file as a very flexible format for specifying a set of points and their attributes, which has a bunch of properties defined in its header. With those, it instructs the parser on how the data contained in its body should be interpreted. For reference, is a very informative guide on the generic structure of .ply files.

Here is an example of what a typical Gaussian Splat .ply file looks like:

ply
format binary_little_endian 1.0
element vertex 1534456
property float x
property float y
property float z
property float nx
property float ny
property float nz
property float f_dc_0
property float f_dc_1
property float f_dc_2
property float f_rest_0
(... f_rest from 1 to  43...)
property float f_rest_44
property float opacity
property float scale_0
property float scale_1
property float scale_2
property float rot_0
property float rot_1
property float rot_2
property float rot_3
end_header

  • The first line ensures this is a ply file. 

  • The second line establishes if the format of the data stored after the header is ASCII based or binary based (the latter in this example).

  • The third line tells the parser how many elements the file contains. In our example, we have 1534456 elements, i.e. splats.

  • From the fourth line until the 鈥渆nd_header鈥 line, the entire structure of each element is described as a set of properties, each with its own data type and name. The order of these properties is commonly followed by most of the Gaussian splat .ply files. It is worth noting that regardless of the order, the important rule is that all the non-optional ones are defined in the file, and the data follows the declared structure.

Once the header section ends, the data to be parsed for each element is provided by the ply body. Each element after the header needs to respect imperatively the order that was declared in the header to be parsed correctly.

This can give you an idea of what to expect specifically when we want to describe a single Gaussian Splat element loaded from a ply file:

  • A position in space in the form XYZ (x, y, z);

  • [Optional] Normal vectors (nx, ny, nz);

  • Zero order Spherical Harmonics (f_dc_0, f_dc_1, f_dc_2), which dictate what color the single splat should have by using a specific mathematical formula to extract the output RGB value for the rendering;

  • [Optional] Higher order Spherical Harmonics (from f_rest_0 to f_rest_44), which dictate how the color of the splat should change depending on the camera position. This is basically to improve realism for the reflections or lighting information embedded into the Gaussian splat. It is worth noting that this information is optional, and that files that embed it will be a lot larger than zero-order-only based ones;

  • An opacity (opacity), which establishes the transparency of the splat;

  • A scale in the form XYZ (scale_0, scale_1, scale_2);

  • An orientation in space in the quaternion format WXYZ (rot_0, rot_1, rot_2, rot_3).

All this information has its own coordinate system, which needs to be converted into Unreal Engine once loaded. This will be covered in more detail later in this article.

Now that you are familiar with the data we need to deal with, you are ready for the next step.

Parsing a .PLY file into Unreal

For our implementation, we wanted to support both ASCII and binary ply files, so we needed a way to quickly parse their data and store them accordingly. Luckily, ply files are not new. They have been used for 3D models for a long time, even before Gaussian Splats became popular. Therefore, several .ply parsers exist on GitHub and can be used for this purpose. We decided to adapt the implementation of , a general purpose open source header-only ply parser written in C++ (big kudos to the author ).

Starting from the implementation of Happly, we adapted its parsing capabilities to the coding standard of Unreal and ported it into the game engine, being mindful of the custom garbage collection and data types expected by Unreal. We then adapted our parsing code to align with the previous Gaussian Splat structure.

The next logical step, once we knew how the data looked and how to read it from a file, was to store it somewhere. This meant we needed a class or a struct that could hold all this data for a specific lifetime within the Engine. Time to dig into some C++ code!

How could we define a single Gaussian Splat in Unreal?

The easiest way to store each Gaussian Splat data was to define a custom USTRUCT in Unreal, optionally accessible by Blueprints, implemented along the following lines:

/**
 * Represents parsed data for a single splat, loaded from a regular PLY file.
 */
USTRUCT(BlueprintType)
struct FGaussianSplatData
{

GENERATED_BODY()

// Splat position (x, y, z)
UPROPERTY(EditAnywhere, BlueprintReadWrite)
FVector Position;

// Normal vectors [optional] (nx, ny, nz)
UPROPERTY(EditAnywhere, BlueprintReadWrite)
FVector Normal;

// Splat orientation coming as wxyz from PLY (rot_0, rot_1, rot_2, rot_3)
UPROPERTY(EditAnywhere, BlueprintReadWrite)
FQuat Orientation;

// Splat scale (scale_0, scale_1, scale_2)
UPROPERTY(EditAnywhere, BlueprintReadWrite)
FVector Scale;

// Splat opacity (opacity)
UPROPERTY(EditAnywhere, BlueprintReadWrite)
float Opacity;

// Spherical Harmonics coefficients - Zero order (f_dc_0, f_dc_1, f_dc_2)
UPROPERTY(EditAnywhere, BlueprintReadWrite)
FVector ZeroOrderHarmonicsCoefficients;

// Spherical Harmonics coefficients - High order (f_rest_0, ..., f_rest_44)
UPROPERTY(EditAnywhere, BlueprintReadWrite)
TArray<FVector> HighOrderHarmonicsCoefficients;

FGaussianSplatData()
: Position(FVector::ZeroVector)
	, Normal(FVector::ZeroVector)
	, Orientation(FQuat::Identity)
	, Scale(FVector::OneVector)
	, Opacity(0)
	{
	}
};

One instance per splat of this struct was generated during the parsing phase and added to a TArray of splats to use this data for our visualization in the following steps.

Now that we have the core data, let鈥檚 dive into the most challenging and fun part: transferring the data to the GPU so a Niagara system can read it!

Why Niagara for Gaussian Splats?

Niagara is the perfect candidate to represent particles inside Unreal. Specifically, a Niagara system is made up of one or multiple Niagara emitters, which are responsible for spawning particles and updating their states every frame.

In our specific case, we will use a single Niagara emitter to make a basic implementation. As an example, we will call it 鈥GaussianSplatViewer鈥

Now that we have our new shiny emitter, we need a way to 鈥減ass鈥 the splats鈥 data into it, so that for each splat we can spawn a relative point in space, representing it. You might wonder, is there anything in Unreal we could use out of the box to do that for us? The answer is yes, and it is called the 鈥淣iagara Data Interface (NDI)鈥.

What is a Niagara Data Interface (NDI) and how to write one

Imagine you want to tell the Niagara emitter, 鈥淗ey, I have a bunch of points I read from a file that I want to show as particles. How can I make you understand what position each point should be in?鈥 Niagara would reply, 鈥淢ake me a beautiful NDI that I can use to understand your data and then retrieve the position for each particle from it鈥.

You might wonder, how do I write this NDI and what documentation can I find? The answer is simple: most of the Engine source code uses an NDI for custom particle systems, and they鈥檙e an excellent source of inspiration for building your own! The one we took the most inspiration from was the 鈥UNiagaraDataInterfaceAudioOscilloscope鈥.

Here鈥檚 how we decided to structure a custom NDI to make each splat 鈥渦nderstandable鈥 by Niagara when passing it through. Keep in mind that this class will hold the list of Gaussian Splats we loaded from the PLY file so that we can access their data from it and convert it into Niagara-compatible data types for use within the particles.

Firstly, we want our NDI class to inherit from UNiagaraDataInterface, which is the interface a Niagara system expects to treat custom data types via NDI. To fully implement this interface, we needed to override several functions, which I present below.

GetFunctions override

When overriding this function, we are telling Niagara 鈥淚 want you to see a list of functions I am defining, so that I can use them inside your Niagara modules鈥. This instructs the system to know what input and output each of these functions should expect, the name of the function, and if it鈥檚 static or non-static.

// Define the functions we want to expose to the Niagara system from
// our NDI. For example, we define one to get the position from a
// Gaussian Splat data.
virtual void GetFunctions(TArray<FNiagaraFunctionSignature>& OutFunctions) override;

Here is a sample implementation of GetFunctions, which defines a function GetSplatPosition to the Niagara system using this NDI. We want GetSplatPosition to have exactly 2 inputs and 1 output:

  • An input that references the NDI that holds the Gaussian splats array (required to access the splats data through that NDI from a Niagara system scratch pad module);

  • An input of type integer to understand which of the splats we request the position of (this will match a particle ID from the Niagara emitter, so that each particle maps the position of a specific Gaussian splat);

  • An output of type Vector3 that gives back the position XYZ of the desired Gaussian splat, identified by the provided input Index.

void UGaussianSplatNiagaraDataInterface::GetFunctions(
    TArray<FNiagaraFunctionSignature>& OutFunctions)
{   
   // Retrieve particle position reading it from our splats by index
   FNiagaraFunctionSignature Sig;
   Sig.Name = TEXT("GetSplatPosition");
   Sig.Inputs.Add(FNiagaraVariable(FNiagaraTypeDefinition(GetClass()),
       TEXT("GaussianSplatNDI")));
   Sig.Inputs.Add(FNiagaraVariable(FNiagaraTypeDefinition::GetIntDef(),
       TEXT("Index")));
   Sig.Outputs.Add(FNiagaraVariable(FNiagaraTypeDefinition::GetVec3Def(),
       TEXT("Position")));
   Sig.bMemberFunction = true;
   Sig.bRequiresContext = false;
   OutFunctions.Add(Sig);
}

Similarly, we will also define other functions inside GetFunctions to retrieve the scale, orientation, opacity, spherical harmonics, and particle count of our Gaussian splats. Each particle will use this information to change shape, color, and aspect in space accordingly.

GetVMExternalFunction override

This override is necessary to allow Niagara to use the functions we declared in GetFunctions by using Niagara nodes so that they become available within Niagara graphs and scratch pad modules. This combines with the DEFINE_NDI_DIRECT_FUNC_BINDER macro in Unreal designed for this purpose. Following is an example of the GetSplatPosition function definition.

// We bind the following function for use within the Niagara system graph
DEFINE_NDI_DIRECT_FUNC_BINDER(UGaussianSplatNiagaraDataInterface, GetSplatPosition);


void UGaussianSplatNiagaraDataInterface::GetVMExternalFunction(const FVMExternalFunctionBindingInfo& BindingInfo, void* InstanceData, FVMExternalFunction& OutFunc)
{
   if(BindingInfo.Name == *GetPositionFunctionName)
   {
       NDI_FUNC_BINDER(UGaussianSplatNiagaraDataInterface,
         GetSplatPosition)::Bind(this, OutFunc);
   }
}


// Function defined for CPU use, understandable by Niagara
void UGaussianSplatNiagaraDataInterface::GetSplatPosition(
  FVectorVMExternalFunctionContext& Context) const
{
   // Input is the NDI and Index of the particle
   VectorVM::FUserPtrHandler<UGaussianSplatNiagaraDataInterface> 
     InstData(Context);


   FNDIInputParam<int32> IndexParam(Context);
  
   // Output Position
   FNDIOutputParam<float> OutPosX(Context);
   FNDIOutputParam<float> OutPosY(Context);
   FNDIOutputParam<float> OutPosZ(Context);


   const auto InstancesCount = Context.GetNumInstances();


   for(int32 i = 0; i < InstancesCount; ++i)
   {
       const int32 Index = IndexParam.GetAndAdvance();


       if(Splats.IsValidIndex(Index))
       {
           const auto& Splat = Splats[Index];
           OutPosX.SetAndAdvance(Splat.Position.X);
           OutPosY.SetAndAdvance(Splat.Position.Y);
           OutPosZ.SetAndAdvance(Splat.Position.Z);
       }
       else
       {
           OutPosX.SetAndAdvance(0.0f);
           OutPosY.SetAndAdvance(0.0f);
           OutPosZ.SetAndAdvance(0.0f);
       }
   }
}

Note that the definition of GetSplatPosition is implemented to make this NDI CPU compatible.

Copy and Equals override

We also need to override these functions, so that when we copy or compare an NDI that uses our class, Niagara will understand how to perform these operations. Specifically, we instruct the engine to copy the list of Gaussian Splats when one NDI is copied into a new one, and to establish if two NDIs are the same if they have the same exact Gaussian Splats data.

virtual bool CopyToInternal(UNiagaraDataInterface* Destination) const override;
virtual bool Equals(const UNiagaraDataInterface* Other) const override;

This function is required to let the Niagara system understand if our NDI functions need to be executed on the CPU or on the GPU. In our case, initially we wanted it to work on the CPU for debugging, but for the final version we changed it to target the GPU instead. I will explain this choice further later.

virtual bool CanExecuteOnTarget(ENiagaraSimTarget Target) const override { return Target == ENiagaraSimTarget::GPUComputeSim; }

Additional overrides required for our NDI to work on the GPU too

We also need to override the following functions, so that we can instruct Niagara on how our data will be stored on the GPU (for a GPU compatible implementation) and how the functions we declared will be mapped onto the GPU via HLSL shader code. More on this later.

// HLSL definitions for GPU
virtual void GetParameterDefinitionHLSL(const FNiagaraDataInterfaceGPUParamInfo& ParamInfo, FString& OutHLSL) override;


virtual bool GetFunctionHLSL(const FNiagaraDataInterfaceGPUParamInfo& ParamInfo, const FNiagaraDataInterfaceGeneratedFunction& FunctionInfo, int FunctionInstanceIndex, FString& OutHLSL) override;


virtual bool UseLegacyShaderBindings() const override { return false; }


virtual void BuildShaderParameters(FNiagaraShaderParametersBuilder& ShaderParametersBuilder) const override;


virtual void SetShaderParameters(const FNiagaraDataInterfaceSetShaderParametersContext& Context) const override;

CPU vs GPU based Niagara system

Each emitter of a Niagara particle system can work on the CPU or on the GPU. It鈥檚 very important to establish which of the two to choose, because each of them has side effects.

Initially, for a simple implementation, we went for the CPU based Niagara emitter. This was to make sure that the splat data and coordinates were correctly reproduced in terms of position, orientation, and scale inside the Niagara system.

However, there are some important limitations for CPU based emitters: 

  • They cannot spawn more than 100K particles;

  • They rely only on the CPU, which means they might consume additional time taking it away from other scripts鈥 execution every frame, resulting in lower frame rates especially when dealing with the maximum amount of supported particles;

  • GPUs can handle much better than CPUs. This makes GPUs better suited than CPUs to large volumes of particles.

While it makes sense for debugging to accept the CPU 100K particle limits, it鈥檚 definitely not the right setup to scale up, especially when you want to support bigger Gaussian Splats files that may contain millions of particles.

In a second iteration, we decided to switch to a GPU based emitter. This not only relies on the GPU completely without affecting the CPU but can support up to 2 million particles spawned, which is 20x more than what is supported on the CPU.

The side effect of executing on the GPU is that we also needed to take care of GPU resource allocations and management, requiring us to get dirty with HLSL shader code and data conversion between CPU and GPU.

How? You guessed it, by extending our beautiful custom NDI.

From PLY file to the GPU via the NDI

Thanks to our custom NDI, we have full control over how our data is stored in memory and how it is converted into a Niagara compatible form. The challenge now is to implement this via code. For simplicity, let鈥檚 break our goal down into two parts:

  1. Allocate memory on the GPU to hold Gaussian Splat data coming from the CPU.

  2. Transfer Gaussian Splat data from the CPU to the prepared GPU memory.

Prepare the GPU memory to hold Gaussian Splat data

The first thing to be aware of is that we cannot use Unreal data types like TArray (which holds the list of Gaussian Splats in our NDI) when we define data on the GPU. This is because TArray is designed for CPU use and is stored in CPU-side RAM, which is only accessible by the CPU. Instead, the GPU has its own separate memory (VRAM) and requires specific types of data structures to optimize access, speed, and efficiency.

To store collections of data on the GPU, we needed to use GPU buffers. There are different types available:

  • Vertex Buffers: store vertex such as positions, normals, and texture coordinates;

  • Index Buffers: used to tell the GPU the order in which vertices should be processed to form primitives;

  • Constant Buffers: store values such as transformation matrices and material properties that remain constant for many operations across the rendering of a frame;

  • Structured Buffers and Shader Storage Buffers: more flexible as they can store a wide array of data types, suitable for complex operations.

In our case, I decided to follow a simple implementation, where each Gaussian Splat information is stored in a specific buffer (i.e. a positions buffer, a scales buffer, an orientations buffer, and a buffer for spherical harmonics and opacity).

Note that both buffers and textures are equally valid data structures to consider for splat data on the GPU. We elected for buffers as we felt the implementation was more readable, while also avoiding an issue with the texture-based approach where the last row of pixels was often not entirely full.

To declare these buffers in Unreal, we needed to add the definition for a 鈥Shader parameter struct鈥, which uses an Unreal Engine Macro to tell the engine this is a data structure supported by HLSL shaders (hence supported by GPU operations). Here is an example:

BEGIN_SHADER_PARAMETER_STRUCT(FGaussianSplatShaderParameters, )
   SHADER_PARAMETER(int, SplatsCount)
   SHADER_PARAMETER(FVector3f, GlobalTint)
   SHADER_PARAMETER_SRV(Buffer<float4>, Positions)
   SHADER_PARAMETER_SRV(Buffer<float4>, Scales)
   SHADER_PARAMETER_SRV(Buffer<float4>, Orientations)
   SHADER_PARAMETER_SRV(Buffer<float4>, SHZeroCoeffsAndOpacity)
END_SHADER_PARAMETER_STRUCT()

It is worth noting that these buffers can be further optimized since the W coordinate remains unused by position and scales (they only need XYZ). To improve their memory footprint it would be ideal to adopt channel packing techniques, which are out of the scope of this article. It is also possible to use half precision instead of full floats for further optimization.

Before the buffers we also define an integer to keep track of the splats we need to process (SplatsCount), and a GlobalTint vector, which is an RGB value that we can use to change the tint of the Gaussian Splats. This definition goes into the header file of our NDI class.

We also need to inject custom shader code for the GPU to declare our buffers so that they can be referenced later on and used by our custom shader functions. To do it, we inform Niagara through the override of GetParameterDefinitionHLSL:

void UGaussianSplatNiagaraDataInterface::GetParameterDefinitionHLSL(
  const FNiagaraDataInterfaceGPUParamInfo& ParamInfo, FString& OutHLSL)
{
  Super::GetParameterDefinitionHLSL(ParamInfo, OutHLSL);


  OutHLSL.Appendf(TEXT("int %s%s;\n"), 
    *ParamInfo.DataInterfaceHLSLSymbol, *SplatsCountParamName);
  OutHLSL.Appendf(TEXT("float3 %s%s;\n"),
    *ParamInfo.DataInterfaceHLSLSymbol, *GlobalTintParamName);
  OutHLSL.Appendf(TEXT("Buffer<float4> %s%s;\n"),
    *ParamInfo.DataInterfaceHLSLSymbol, *PositionsBufferName);
  OutHLSL.Appendf(TEXT("Buffer<float4> %s%s;\n"),
    *ParamInfo.DataInterfaceHLSLSymbol, *ScalesBufferName);
  OutHLSL.Appendf(TEXT("Buffer<float4> %s%s;\n"),
    *ParamInfo.DataInterfaceHLSLSymbol, *OrientationsBufferName);
  OutHLSL.Appendf(TEXT("Buffer<float4> %s%s;\n"),
    *ParamInfo.DataInterfaceHLSLSymbol, *SHZeroCoeffsBufferName);

Effectively, this means that a Niagara system using our custom NDI will have this shader code generated under the hood. This allows us to reference these GPU buffers within our HLSL shader code for next steps. For convenience we defined the names of the parameters as FString, and they are used to make the code more maintainable.

Transfer Gaussian Splat data from CPU to GPU

Now the tricky part: we need to 鈥減opulate鈥 the GPU buffers using C++ code as a bridge between the CPU memory and the GPU memory, specifying how the data is transferred.

To do it, we decided to introduce a custom 鈥Niagara data interface proxy鈥 鈥 a data structure used as a 鈥bridge鈥 between the CPU and the GPU. This proxy helped us push our buffer data from the CPU side to the buffers declared as shader parameters for the GPU. To do it, we defined in the proxy the buffers, and the functions to initialize and update them respectively.

I know this seems to be getting very complicated, but from a logical point of view it is quite simple, and I can help you understand the system by visualizing the full concept in this diagram:

Now that we have a complete overview of our system, there are some final little details we need to refine in order for it to be fully operational.

We already have the buffers鈥 definitions for the GPU as HLSL code via the GetParameterDefinitionHLSL function. Now, we need to do the same for the functions we previously defined in GetFunctions, so the GPU understands how to translate them into HLSL shader code.

Let鈥檚 take the GetSplatPosition function for example, we previously saw how it was defined for use with the CPU. Now we need to extend its definition to be also declared for the GPU. We can do this by overriding the GetFunctionHLSL in our custom NDI:

bool UGaussianSplatNiagaraDataInterface::GetFunctionHLSL(
  const FNiagaraDataInterfaceGPUParamInfo& ParamInfo, const
  FNiagaraDataInterfaceGeneratedFunction& FunctionInfo, 
  int FunctionInstanceIndex, FString& OutHLSL)
{
   if(Super::GetFunctionHLSL(ParamInfo, FunctionInfo,
     FunctionInstanceIndex, OutHLSL))
  {
    // If the function is already defined on the Super class, do not
    // duplicate its definition.
    return true;
  }
  
  if(FunctionInfo.DefinitionName == *GetPositionFunctionName)
  {
    static const TCHAR *FormatBounds = TEXT(R"(
      void {FunctionName}(int Index, out float3 OutPosition)
      {
        OutPosition = {PositionsBuffer}[Index].xyz;
      }
    )");
    const TMap<FString, FStringFormatArg> ArgsBounds =
    {
     {TEXT("FunctionName"), FStringFormatArg(FunctionInfo.InstanceName)},
     {TEXT("PositionsBuffer"),
       FStringFormatArg(ParamInfo.DataInterfaceHLSLSymbol + 
         PositionsBufferName)},
    };
    OutHLSL += FString::Format(FormatBounds, ArgsBounds);
  }
  else
  {
    // Return false if the function name does not match any expected.
    return false;
  }
  return true;
}

As you can see, this part of the code simply adds to the OutHLSL string the HLSL shader code that implements our GetSplatPosition for the GPU. Whenever Niagara is GPU based and the GetSplatPosition function is called by the Niagara graph, this shader code on the GPU will be executed.

For brevity I did not include the other HLSL shader code for the scale, orientation, spherical harmonics, and opacity getter functions. However, the idea is the same, we would just add them inside GetFunctionHLSL.

Finally, the actual code to transfer data from the CPU to the GPU via the DIProxy is handled by the override of SetShaderParameters:

void UGaussianSplatNiagaraDataInterface::SetShaderParameters(
  const FNiagaraDataInterfaceSetShaderParametersContext& Context) const
{
  // Initializing the shader parameters to be the same reference of 
  //our buffers in the proxy
  FGaussianSplatShaderParameters* ShaderParameters =
    Context.GetParameterNestedStruct<FGaussianSplatShaderParameters>();
  if(ShaderParameters)
  {
    FNDIGaussianSplatProxy& DIProxy = 
      Context.GetProxy<FNDIGaussianSplatProxy>();


      if(!DIProxy.PositionsBuffer.Buffer.IsValid())
      {
        // Trigger buffers initialization
        DIProxy.InitializeBuffers(Splats.Num());
      }


      // Constants
      ShaderParameters->GlobalTint = DIProxy.GlobalTint;
      ShaderParameters->SplatsCount = DIProxy.SplatsCount;
      // Assign initialized buffers to shader parameters
      ShaderParameters->Positions = DIProxy.PositionsBuffer.SRV;
      ShaderParameters->Scales = DIProxy.ScalesBuffer.SRV;
      ShaderParameters->Orientations = DIProxy.OrientationsBuffer.SRV;
      ShaderParameters->SHZeroCoeffsAndOpacity =
        DIProxy.SHZeroCoeffsAndOpacityBuffer.SRV;
  }
}

Specifically, this transfers the buffer data from the NDI proxy (DIProxy) into the relative HLSL shader parameters, ruled by the FGaussianSplatShaderParameters struct.

That was a lot of code! If you managed to follow the full process, congratulations! You are now pretty much done with the low-level implementation. Let鈥檚 back up one level and finish some of the leftovers to complete our Gaussian Splat viewer!

Register our custom NDI and NDI proxy with Niagara

One last thing required to access our custom NDI inside the Niagara property types is registering it with the FNiagaraTypeRegistry. For convenience, we decided to do it inside the PostInitProperties of our NDI, where we also create the NDI proxy that will transmit data from the CPU to the GPU.

void UGaussianSplatNiagaraDataInterface::PostInitProperties()
{


  Super::PostInitProperties();


  // Create a proxy, which we will use to pass data between CPU and GPU
  // (required to support the GPU based Niagara system).
  Proxy = MakeUnique<FNDIGaussianSplatProxy>();
 
  if(HasAnyFlags(RF_ClassDefaultObject))
  {
    ENiagaraTypeRegistryFlags DIFlags =
      ENiagaraTypeRegistryFlags::AllowAnyVariable |
      ENiagaraTypeRegistryFlags::AllowParameter;


    FNiagaraTypeRegistry::Register(FNiagaraTypeDefinition(GetClass()), DIFlags);
  }


  MarkRenderDataDirty();
}

Here is a screenshot of our updated shiny Niagara system making use of our custom NDI and getter functions exposed in its graph!

The big challenge of converting from PLY to Unreal coordinates

There is hardly any documentation currently available online to explicitly specify the conversions required to transform data coming from a PLY file into Unreal Engine. 

Here are some funny painful failures we had to go through before finding the right conversions.

After many trials and mathematical calculations, we were finally able to establish the proper conversion. For your convenience, here is the list of operations to do it:

Position (x, y, z) from PLY 
Position in UE = (x, -z, -y) * 100.0f

Scale (x, y, z) from PLY
Scale in UE = (1/1+exp(-x), 1/1+exp(-y), 1/1+exp(-z)) * 100.0f

Orientation (w, x, y, z) from PLY
Orientation in UE = normalized(x, y, z, w)

Opacity (x) from PLY
Opacity in UE = 1 / 1 + exp(-x)

In order to keep performance optimal, these conversions are performed on load rather than at runtime, so that once the splats are in the scene, no update is required per frame.

Here is how the resulting Gaussian Splats viewer will show by following the right calculations at the end of the process I described in this article.

There are some more bits and bobs of code to deal with further geometric transformations and clipping, but those remain outside of the scope of this article.

The final result with some more feedback

This has been a very long journey, resulting in a very long article I admit. But I hope it has inspired you to better understand how Niagara in Unreal can be customized to interpret your custom data; how it is possible to optimize its performance via GPU-based HLSL shader code injected from your custom Niagara Data Interface and Niagara Data Interface Proxy; and finally how Gaussian Splat can be viewed in the viewport after all this hard work!

Thank you for following this journey and feel free to and on LinkedIn for more tech-based posts in the future!

Happy coding! 馃檪

Alessio Regalbuto
Previous

Meet the Magnopians: Daksh Sahni
Next

Meet the Magnopians: Chris Kinch