How the dedup Command Works Dedup has a pair of modes. Typical examples of a dedup produce a single event for each host or a pair of events for each sourcetype. Remove only consecutive duplicate events. The Splunk dedup command, short for deduplication, is an SPL command that eliminates duplicate values in fields, thereby reducing the number of events returned from a search. Keep results that have the same combination of values in multiple fieldsįor search results that have the same combination of source AND host values, keep the first 2 that occur and remove all subsequent results. Remove duplicate search results with the same host value and sort the events by the _size field in descending order. Sort events after removing duplicate values | from main order by ASC _time | dedup source 4. Remove duplicate results with the same source value. Sorting the events ensures that the oldest events are listed first. For search results that have the same value, keep the first 3 that occur and remove all subsequent results. Use the order by clause in the from command to sort the events by time in ascending order, the default order. Remove duplicate search results with the same value and sort the results by the field in descending order. Sort events in ascending order before removing duplicate values Keep the first 3 duplicate resultsįor search results that have the same source value, keep the first 3 that occur and remove all subsequent results. Remove duplicate search results with the same host value. Remove duplicate results based on one field To learn more about the dedup command, see How the dedup command works.ġ. This strategy would give me a 'most recent date' that is inaccurate by up to x-1 days, but would cut a 9-hour extract to something more manageable.The following are examples for using the SPL2 dedup command.

Initially, I can randomly backdate the most recent report to spread them out. The Splunk dedup command, short for deduplication, is an SPL command that eliminates duplicate values in fields, thereby reducing the number of events returned from a search. That would be much more versatile for these kinds of optimization problems.Īlternatively, I guess I'll need to pursue some kind of custom command.Īt the moment I'm leaning toward creating a lookup table (local=f) and killing all events where the last copy is less than x days old.

Ideally, I'd love an option to force streamstats to run at each indexer (perhaps with a different name to avoid confusion). Obviously, I'll have some non-streamable commands after I pull them all together, but it seems like a pre-sort strategy for dedup would save a lot of transmission bandwidth, if nothing else, and could push the work out to the indexers instead of dropping it on the search head. Is there any streamable command or strategy that can kill the lion's share of those duplicates? The results distribution is likely to be neither sparse nor dense, but long-tail - a few combinations will predominate, with hundreds of thousands of dups each over the time period, and lots of combinations will be rare, with only a few dozen dups each. There will be 10s of thousands of distinct value combinations and 10's of millions of dups, across a few dozen indexers. Depending on your data (number of results, size of raw events, time of each search execution) there could be different ways to do that. I'm running a search against a huge amount of data (minimum dozens of terabytes) and need only one example of each combination of a couple of fields (let's say 3 at most). Is there a way to have the equivalent of dedup running against each index's results before they get sent to the search head? Background in a moment, but here's the question:

Splunk search dedup