Virtual fields allow you to derive new values from your data in real time, eliminating the need for up-front data structuring, enhancing flexibility and efficiency.
Virtual fields allow you to derive new values from your data in real time.One of the most powerful features of Axiom are virtual fields. With virtual fields, there is no need to do any up-front planning of how to structure or transform your data. Instead, send your data as-is and then use virtual fields to manipulate your data in real-time during queries.The feature is also known as derived fields, but Axiom’s virtual fields have some unique properties that make them much more powerful.
Virtual fields are query-time representations only. They don’t create new data in storage.
This page introduces you to virtual fields, their features, how to manage them, and how to get the best out of them.
Select the dataset where you want to create the virtual field.
Click Virtual fields in the top right. You see a list of all the virtual fields for the dataset.
Click Add virtual field.
Fill in the following fields:
Name and Description help your team understand what the virtual field is about.
Expression is the formula applied to every event to calculate the virtual field. The expression produces a result such as a boolean, string, number, or object.
The Preview section displays the result of applying the expression to some of your data. Use this section to verify the expression and the resulting values of the virtual field.
The power of virtual fields is in letting you manipulate data on read instead of on write, allowing you to adjust and update virtual fields over time as well as easily add new ones without worrying that the data has already been indexed.
Virtual fields are available as parameters to visualizations but, as the type of a virtual field can be any of the supported types, it’s important to make sure that you use a virtual field that produces the correct type of argument.
Virtual fields are available in the filter menu and all filter options are presented. It’s important to ensure that you are using a supported filter operation for the type of result your virtual field produces.
For high-performance use cases, parsing fields at ingest time is often the better choice:
Query performance: Virtual fields re-parse data on every query execution. For frequently queried fields, especially during incident response when teams run many ad-hoc queries, this adds latency. Parsed fields at ingest time reduce your storage and query hours usage. For more information, see Virtual fields for simple transformations.
Compression benefits: When you parse fields from a JSON body to top-level fields, Axiom uses specialized data-aware compression techniques based on the detected data types. This results in more efficient storage compared to keeping data as unparsed strings. For more information, see Overusing runtime JSON parsing.
Parsing fields from a body to top-level attributes doesn’t double your storage usage. Axiom’s compression is optimized for typed, structured data at the top level, and stores parsed fields efficiently. However, there is some data redundancy if you keep the original body field alongside the parsed fields. To minimize redundancy, remove the parsed fields from the body after extraction during your ingest pipeline.
For a balance between flexibility and performance, parse your most frequently queried fields at ingest time while using virtual fields for less common or experimental transformations.
Virtual fields are APL expressions and share all the same functions and syntax as APL expressions. For more information, see Introduction to APL.The list of APL scalar functions:
Virtual fields may reference other virtual fields. The order of the fields is important. Ensure that the referenced field is specified before the field that references it.
