At Hurricane Labs, we love logs and we love video games. Any chance to combine the two is exciting, and recently I stumbled on just such an opportunity.
My friends and I have been hooked on 7 Days to Die, a survival horror game in which players scavenge an apocalypse-ridden world and build a defense to survive a massive zombie horde that attacks every seven days. One of my friends had found an itema book, specificallythat wouldve been a significant upgrade for another player, but when they met up to exchange the item it had disappeared. We dismissed it as a glitch, but I began to wonderdid the game have logs, and could they help us discover what had happened? Naturally, my weapon of choice was Splunk.
Server-side logs are actually pretty rare in the gaming world these days. The concept of hosting a local server is antiquated, replaced by online servers hosted by game companies themselves. 7 Days to Die, however, allows players to operate their own server and configure it however theyd like. We didnt realize this at firstmy brother was the first one to start the game and we all joined him, unaware that we were actually joining a server running locally on his computer, which also happened to be the one with the least amount of resources. Eventually he sent me the server files and I started hosting from mine.
I was pleasantly surprised by the logs that had been generated. Every log starts with a consistent and readable timestamp and a generally uniform message format. For example:
Copy to Clipboard
2021-01-20T23:31:49 14350.539 INF GMSG: Player xAweSamx left the game
Scrolling through the unexpectedly large file, I noticed that in addition to operational logs, there were logs about player behavior and AI behavior: kills, deaths and even how many zombies were spawned! It was time to get these log files into Splunk and really start digging in.
I chose to use 7daystodie:output as the custom sourcetype name to reflect both the software and the log file name (output_log) in case there is ever another log filetype we want to ingest.
The first thing I noticed is that every log file begins with a header that has interesting information but isnt something we want to index every time:
Copy to Clipboard
Mono path[0] = D:/SteamLibrary/steamapps/common/7 Days To Die/7daystodie_Data/ManagedMono config path = D:/SteamLibrary/steamapps/common/7 Days To Die/MonoBleedingEdge/etcInitialize engine version: 2019.2.17f1 (8e603399ca02)[XR] Discovering subsystems at path D:/SteamLibrary/steamapps/common/7 Days To Die/7daystodie_Data/UnitySubsystemsForcing GfxDevice: Direct3D 11GfxDevice: creating device client; threaded=1Direct3D:Version: Direct3D 11.0 [level 11.1] Renderer: NVIDIA GeForce RTX 2070 SUPER (ID=0x1e84)Vendor:VRAM: 8011 MBDriver: 27.21.14.6079Begin MonoManager ReloadAssembly Completed reload, in 2.962 seconds Initializing input.
Input initialized.
Initialized touch support.
UnloadTime: 6.034400 ms
To omit this, I took advantage of the FIELD_HEADER_REGEX props setting. This uses Regular Expression to identify the last line of the header and in this case tells Splunk not to index anything above it:
Copy to Clipboard
[7daystodie:output]FIELD_HEADER_REGEX = UnloadTime
Line-breaking and timestamp parsing was the next objective. Fortunately, it uses a pretty standard and consistent timestamp, which is already more than I can say for some other more common data sources. When I first loaded this into the data preview through the web interface, Splunk actually did a good job identifying and separating events without any further configuration. The best practice though is to be as specific as possible here to avoid any potential issues now or in the future.
At Hurricane Labs we like to cover at least TIME_PREFIX, TIME_FORMAT, MAX_TIMESTAMP_LOOKAHEAD, and TIMEZONE. Between these settings, weve specified where in the log the timestamp occurs, how far from that point to consider part of the timestamp, what the timestamp looks like, and what timezone the log was written in. We also prefer to have control over the event line-breaking, in other words specifying what constitutes a single event. In most cases, such as ours, the event in the log begins with a timestamp so weve told Splunk not to line merge automatically and we specify our own condition in the LINE_BREAKER setting.
As youll recall from the brief example above, our timestamp looks like this:
Copy to Clipboard
2021-01-20T23:31:49
Our props configuration so far is:
Copy to Clipboard
[7daystodie:output]FIELD_HEADER_REGEX = UnloadTimeLINE_BREAKER = ([rn]+)d{4}-d{2}-d{2}Td{2}:d{2}:d{2}MAX_TIMESTAMP_LOOKAHEAD = 19SHOULD_LINEMERGE = falseTIMEZONE = US/EasternTIME_FORMAT = %Y-%m-%dT%H:%M:%STIME_PREFIX = ^
I ingested a single log file to a test index to make sure everything behaved as expected and started playing with the data to see what was available. Splunk automatically extracted some fields that were already key=value pairs, but some more specific field extractions were necessary. I had data about when players connected and disconnected, how long their session was, how many zombies (and what types) they killed, and if they died. I also had some internal system messages such as audio files failing to load, no space to load in a resource, andmost relevant to our original goal of discovering the fate of our missing itemwhen a resource fell off the world.
Some examples of the field extractions:
Copy to Clipboard
EXTRACT-playerinfo = INFs+Players+(?.+?)s+disconnecteds+afters+(?d+.d+)s+minutesEXTRACT-spawnerinfo = INFs+(?S+)s+spawneds+(?S+)EXTRACT-vendorprio = d{4}-d{2}-d{2}Td{2}:d{2}:d{2}s+d+.d+s+(?S+)s+EXTRACT-killerinfo = Entitys+(?S+)s+d+s+killeds+bys+(?S+)s+EXTRACT-component = d{4}-d{2}-d{2}Td{2}:d{2}:d{2}s+d+.d+s+WRNs+(?NosS+|D+?)s+
If youre not familiar with reading Regular Expression one of the above extractions pulls out the priority: INF (informational) or WRN (warning). Narrowing down on the warning messages I began to see events such as:
Copy to Clipboard
2021-01-24T22:36:46 8357.174 WRN Entity [type=EntityItem, name=resourceScrapIron, cnt=150] fell off the world, id=51777 pos=(1327.2, -0.5, 108.7)
Fell off the world is an interesting fate for an item, but the game is technically in early access (meaning it is still being worked on). Honestly, open world survival games usually have bugs similar to these simply due to the scope of the environment and the resources, so you kind of know what youre getting into. But is it what happened to our item, a book that wouldve taught a much-needed skill for one of our players?
The log above involves resourceScrapIron, which as you can probably guess is not the book in question. But with that name extracted, we can easily perform an all-time search for the fell off warning message and display all the different items that have been logged:
Continued here:
Splunking the Zombie Apocalypse - Security Boulevard
- History will not be kind to Tokyo's 'zombie Olympics' - Nikkei Asia - April 5th, 2021
- Researchers discover "Zombie Genes" in the brain that come to life after death - CMU The Tartan Online - April 5th, 2021
- Geoengineering the climate: The zombie idea that just won't die - Resilience - April 5th, 2021
- Hilarie Burton Worried She'd Make a 'Bad Zombie' On The Walking Dead - TooFab - April 5th, 2021
- The Guitar Zombie Apocalypse Is Real - Premier Guitar - April 5th, 2021
- Call of Duty: Warzone's Zombies Move to a New Location - ComicBook.com - April 5th, 2021
- Corey Taylors Zombie Versus Ninja movie will commence shooting this year - Louder - April 5th, 2021
- Is Zombie Army 4 multiplayer? Heres all we know about the Rebellion Developments release - Republic TV - April 5th, 2021
- What are 'zombie mines'? What do they have to do with Malcolm Turnbull? - ABC News - April 5th, 2021
- The Best Zombie Movie According To Reddit Isn't What You'd Think - Looper - April 5th, 2021
- Zombie (The Cranberries song) - Wikipedia - March 9th, 2021
- How to Survive a Zombie Apocalypse, According to the CDC - Inside Edition - March 9th, 2021
- The CDC wants you to prepare for a zombie apocalypse. (Yes, you read that right.) - USA TODAY - March 9th, 2021
- Call of Duty's Zombie Easter Eggs Were Too Hard, So Cold War Season 2 Is Simplifying Things - Kotaku - March 9th, 2021
- Resident Evil 3 Did Army of the Dead's Zombie Vegas 14 Years Ago - Screen Rant - March 9th, 2021
- The perfect Call of Duty zombies mode is here, and it's not Outbreak - PC Gamer - March 9th, 2021
- CDC Offers Guide to Survive Zombie Apocalypse and Other Natural Disasters - Complex - March 9th, 2021
- Zombie Soil: The Harmful Result of Industrial Farming | Earth 911 - Earth911.com - March 9th, 2021
- Call of Duty Warzone Crossover With Zombies Teased on Twitter - Screen Rant - March 9th, 2021
- 'Zombie' cicadas could be on your doorstep this summer - Times-West Virginian - March 9th, 2021
- How to Find Flawless Aetherium Crystals in COD: Zombies Outbreak - Screen Rant - March 9th, 2021
- Resurrected The Walking Dead star Lauren Cohan tells Conan her best zombie kill nicknames - The A.V. Club - March 9th, 2021
- Warzone yellow access cards - how to survive the new shipwreck zombies event - Gamesradar - March 9th, 2021
- WTOPs Beer of the Week: 3 Floyds Zombie Dust Pale Ale - WTOP - March 9th, 2021
- Indias zombie home fund set to deliver thousands of apartments - The Real Deal - February 20th, 2021
- Zombie infection threat as country unlocks 50,000-year-old viruses - Yahoo News Australia - February 20th, 2021
- Stubbs the Zombie rises from his grave this March for PC via Steam - PC Invasion - February 20th, 2021
- Zombie Hit Train To Busan Is Getting A US Remake - GameSpot - February 20th, 2021
- Rob Zombie says the meat industrys future is unsustainable - NME.com - February 20th, 2021
- Stubbs The Zombie In Rebel Without A Pulse - Yahoo Entertainment - February 20th, 2021
- Batista Wanted To Play A Zombie On The Walking Dead, Rejected For Being "Too Big" - Fightful - February 20th, 2021
- Dave Bautista told he was too big to play zombie on The Walking Dead - NME - February 17th, 2021
- Dave Bautista Offered To Play A Zombie For Free On The Walking Dead - Screen Rant - February 17th, 2021
- Is plurilateralism making the WTO an institutional zombie? - East Asia Forum - February 17th, 2021
- REVIEW: Zom 100: Bucket List of the Dead, Vol. 1 is a Comedic Zombie Apocalypse - CBR - Comic Book Resources - February 17th, 2021
- Don't Eat Pete is a zombie survival game that's available now for Android in Early Access - Pocket Gamer - February 17th, 2021
- Call of Duty Warzone seems to be teasing Zombies - Gamesradar - February 17th, 2021
- James Gunn Weighs In On Whether Zombies Or Vampires Are More Dangerous - Looper - February 17th, 2021
- The Walking Dead back on the march - Stuff.co.nz - February 17th, 2021
- The Walking Dead Turned Dave Bautista Down for a Zombie Role - Superherohype.com - February 16th, 2021
- Army of the Dead Image Teases Zombie Heist Movie - /FILM - February 16th, 2021
- The Real Life Diet of Rob Zombie, Who Thinks Eating Vegan Is Metal - British GQ - February 16th, 2021
- Rob Zombie says veganism is about 'fighting the establishment' - Vegan Food and Living - February 16th, 2021
- The Next Generation of Marvel Zombies Is Already in the Marvel Universe - CBR - Comic Book Resources - February 16th, 2021
- It Looks Like Zombies Are Coming To Warzone Season 2 | TheGamer - TheGamer - February 16th, 2021
- ROB ZOMBIE Explains Why He Went Vegan "It Is Anti-Establishment" - Metal Injection - February 14th, 2021
- Zombie Baseballs and MLBs Home Run Problem - The Wall Street Journal - February 14th, 2021
- Rob Zombie Thinks Eating Vegan Is Metal - GQ - February 14th, 2021
- This is what the DoD has planned for a zombie apocalypse - We Are The Mighty - February 14th, 2021
- 3 Cookbooks and guides to teach you how to survive during the eventual zombie apocalypse - Undead Walking - February 14th, 2021
- Call Of Duty Zombies: All Zombies Crews, Ranked - TheGamer - February 14th, 2021
- Tesla : Which is the best EV to own to survive a zombie apocalypse? - marketscreener.com - February 14th, 2021
- Here's How You Can Watch Every Season Of iZombie - Looper - February 14th, 2021
- What's The 1st Thing You'd Grab If Zombies Invaded Twin Falls? - kezj.com - February 14th, 2021
- Kennedy: Banks, communities need to collaborate on zombie property upkeep - The Daily Gazette - February 10th, 2021
- Zombies Attack Bend! [With Video] - The Source Weekly - February 10th, 2021
- Ten Zombie Comedies That Won't Rot Your Braaaaaaaains - tor.com - February 10th, 2021
- These Bulletproof Vehicles Are Perfect For Surviving The Zombie Apocalypse - HotCars - February 10th, 2021
- How a Left 4 Dead Remaster Could Revive the Zombie Shooter Genre - CBR - Comic Book Resources - February 10th, 2021
- Rise of Barnegat Light's Zombie Whale; It Starts With Dead Bigfoots, the Devils - The SandPaper - February 10th, 2021
- The Senate should do everything it can to avoid a zombie impeachment trial - CNN - February 9th, 2021
- 8 Video Game Zombies That Break the Traditional Mold - Bloody Disgusting - February 9th, 2021
- Zombie Reddy Four Days Box Office Collection, Teja Sajja's Film Is Ruling The Box Office. - Sakshi Post - February 9th, 2021
- Colm O'Regan: The perfect Valentine's Day gift - a zombie plan for the one you love - Irish Examiner - February 9th, 2021
- Zombie Reddy Third Day Box Office Collection, Teja Sajja's Film Passes The Weekend On A Phenomenal Note - Sakshi Post - February 9th, 2021
- How to Play the Witch Role Mod in Among Us | Screen Rant - Screen Rant - February 9th, 2021
- Reanimation and Zombies: Discussion with author Frank Swain - WLNS - February 9th, 2021
- Stubbs the Zombie Is Returning In a Unique Re-release From Aspyr Media - Happy Gamer - February 9th, 2021
- Author discussion to explore reanimation, zombies - The Hillsdale Daily News - February 9th, 2021
- Not Ready to Die: These 5 'zombie' cars were sold alongside their replacements - Driving - February 5th, 2021
- Zombie Reddy movie review: When the zombies attack Seema - The Hindu - February 5th, 2021
- Stubbs the Zombie comes back from the dead in March, achievements revealed - TrueAchievements - February 5th, 2021
- Prasanth Varma: Zombie Reddy doesnt make a joke out of zombies - The Indian Express - February 5th, 2021
- Stubbs the Zombie Re-Release Launching in March | Game Rant - GameRant - February 5th, 2021
- Shang-Chi Finally Found the Cure for the Newest Kind of Marvel Zombie - CBR - Comic Book Resources - February 5th, 2021
- Free-Roam VR Unleashed - The Source Weekly - February 5th, 2021
- Releasing Today: Zombie Attack On Box Office.. - Greatandhra.com - February 5th, 2021
- ROB ZOMBIE's New Album Was Completed A Couple Years Ago - Metal Injection - February 5th, 2021
- Zombie nation: Third of adults walking around in concussion-like daze due to stress, lack of sleep - Study Finds - February 5th, 2021
- Stubbs the Zombie Achievements Listing Hints a Re-release in March - The Nerd Stash - February 5th, 2021
Reviewed and Recommended by Erik Baquero